Inverse Factorized Soft Q-Learning for Cooperative Multi-agent Imitation Learning
Authors: The Viet Bui
Tien Anh Mai
Thanh Hong Nguyen
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-agent Imitation Learning
Inverse Q Learning
Centralized Learning
Primary area: reinforcement_learning
TL;DR: An Inverse Q-Learning Algorithm for Multi-Agent Imitation Learning

Abstract: This paper concerns imitation learning (IL) in cooperative multi-agent systems.
The learning problem under consideration poses several challenges, characterized by high-dimensional state and action spaces and intricate inter-agent dependencies. In a single-agent setting, IL was shown to be done efficiently via an inverse soft-Q learning process. However, extending this framework to a multi-agent context introduces the need to simultaneously learn both local value functions to capture local observations and individual actions, and a joint value function for exploiting centralized learning.
In this work, we introduce a new multi-agent IL algorithm designed to address these challenges. Our approach enables the
centralized learning by leveraging mixing networks to aggregate  decentralized Q functions.
We further establish conditions for the mixing networks under which the multi-agent IL objective function exhibits convexity within the Q function space.
We present  extensive experiments conducted on some challenging multi-agent game environments, including an advanced version of the Star-Craft multi-agent challenge (SMACv2), which demonstrates the effectiveness of our algorithm.

Inverse Factorized Soft Q-Learning for Cooperative Multi-agent Imitation Learning
The Viet Bui
Tien Mai
Singapore Management University, Singapore theviet. bui 2023@phdcs. smu. edu.sg
Singapore Management University, Singapore atmai @smu. edu.sg
Thanh Hong Nguyen
University of Oregon Eugene, Oregon, United States thanhhngScs. uoregon. edu
Abstract
This paper concerns imitation learning (IL) in cooperative multi-agent systems. The learning problem under consideration poses several challenges, characterized by nigh-dimensional state and action spaces and intricate inter-agent dependencies. In a single-agent setting, IL was shown to be done efficiently via an inverse soft-Q learning process. However, extending this framework context introduces the need to simultaneously learn both local value functions to capture local observations and individual actions, and a joint value function for exploiting centralized learning. In this work, introduce new multi-agent IL algorithm designed to address these challenges. Our approach enables the centralized learning by leveraging mixing networks to aggregate decentralized Q functions. We further establish conditions for the mixing networks under which the multi-agent IL objective function exhibits convexity within the Q function space. We present extensive experiments conducted on some challenging multi-at multi-agent game environments, including an advanced version of the Star- StarCrrfft multi-agent challenge (SMACv2), which demonstrates the effectiveness of our algorithm.
multi-agent
Introduction
Imitation learning (II.) is powerful approach for sequential decision making in complex environ- ments in which agents can learn desirable behavior by imitating expert. There applications of IL in several real-world domains, including healthcare [40] (4) and autonomous driving [43]
17 23
A rich body of IL literature focuses simple single-agent settings [8]
B
44
32
while
many recent works develop new multi-agent IL methods that are tailored to either cooperative or non-
19 37]. Leading methods in [6 42 explore solution concepts for Markov games such as Nash equilibria. Findings on underlying properties of these solution concepts integrated extend the single-agent IL models to multi agentt settings. While these methods show promising results, they face difficulties in training, as the underlying adversarial optimization process involves biased and high variance gradient estimators, leading a highly unstable learning process.
cooperative settings 36
42
22
4
Our work studies in cooperative multi-agent settings. We leverage recent advanced findings in both single-agent IL and multi-agent reinforcement learning (MARL) to build unified multi-agent IL algorithm, named Multi-agent Inverse Factorized Q-learning (MIFQ). MIFQ is built upon inverse soft Q-learning ((Q-Learn) 14), leading single-agent IL method of which advantage is leam single Q-function that implicitly defines both reward and policy functions, thus avoiding adversarial training. To adapt this idea for multi- agent settings, following the well-known paradigm of centralised training with decentralised execution (CTDE) in MARL 26 21,, we develop centralized learning
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

approach based on the concepts of mixing and hyper- netwrrs [16]. This approach facilitates the integration of individual state-value and reward functions into unified learning objective, enabling the training of decentralized imitation policies for agents in centralized manner.
In this work, we formulate the IL. problem multi-agent inverse soft-Q learning task, where the objective is to match the occupancy distribution of the joint learning policy with that of the expert. To make the learning process practical and facilitate the CTDE paradigm, we propose factorizing both the global Q and V functions using mixing networks with non-negative weights and convex activation functions (such as ReLU and ELU). We further examine the
Soodddal-llobll-aaat (IGM) principle 129 (35, a core concept in factorization learning, which suggests that the optimal joint actions across agents are equivalent to the collection of individual optimal actions for each agent. However, we argue that this principle does not apply for our problem setting since our objective is to recover soft policies based on max-entropy reinforcement learning. Therefore, we introduce a generalized Sewriiial--Gobal--ooooo
version of IGM, called joint actions produced by the optimal joint policy matches the collection of local action distributions produced by the individual optimal policies. We then demonstrate that IGC holds in approach.
(IGC), which requires that the distribution of
A key advantage of single-agent inverse soft Q-learning (IQ-Learn) that the original training problem, which is max-min optimization, can be conveniently converted into a (non-adversarial) concave maximization problem [4] This conversion ensures that the optimization objective is well-behaved, contributing to the effectiveness of the original IQ-Learn. Interestingly, we provide theoretical results showing that under our factorization approach, the multi-agent training objective can also be converted into a adverraaraaiil) concave maximization problem. This advantage holds with any multi-layer werward mixing networks with non-negative weights and convex activations.
Finally, we conduct extensive experiments in three domains: SMACv2 [9], Gold Miner [[2], and MPE (Multi Particle Environments) (25). We show that our MIFQ outperforms other baselines in all these environments. To the best of our knowledge, our experiments with SMACv2 mark the first time IL algorithms are employed and evaluated on such a challenging multi-agent environment.
Concretely, we make the following main contribution:
(i) We introduce a new multi-agent IL method based inverse soft-Q learning and factorization.
(ii) We show that our approach satisfies the IGC, generation of the IGM principle.
(iii) We show that the max-min learning objective can be converted into concave maximization problem, under any mixing networks of non- non-negative weights and convex activations, which helps avoid adversarial training and ensure well-behaved learning within the Q-space.
(iv) We empirically show state-of-art results on several challenging multi-agent game tasks, including an advanced version of the Star-Craft multi-agent challenge (SMACv2).
2 Related Work
Single-Agent Imitation Learning. There is a rich body of existing works focusing on generating policies that mimic an expert's behavior given data of expert demonstrations in single-agent settings. A classic approach is behavioral cloning which casts IL as supervised learning problem, attempting to maximize the likelihood of the expert's trajectories [32 28). This approach, while simple, requires large amount of data to work well due to its compounding error issue. An alternative approach is to recover the reward function (either implicitly or explicitly) for which the expert's policy is
optimal [13 18 30 20 10]. Leading methods [18] 13 follow an adversarial optimization process (which is similar to GAN (15)) to train the imitation policy and reward function alternatively. These adversarial training- based methods, however, suffer instability, A more recent work by [14] overcomes this instability issue by introducing an inverse soft-Q learning process. For a comprehensive review of literature on this topic, we refer readers to the survey article in [2].
Multi-Agent Imitation Learning. Most of single-agent IL works, however, do not apply directly to multi-agent settings. Literature on multi-agen! IL is rather limited. A few works study multi-agent IL
either cooperative environments 3 22 37|5 3 or competitive environments 224 36 42
Recent
leading methods employ equilibrium solution concepts in Markov games to extend some existing single-agent II. methods to the multi-agent settings [36 42). However, these methods still suffer the instability challenge during training as they still rely on adversarial training. Our work focuses

---

on multi-agent IL in a cooperative Dec-POMDP environment. We utilize the idea of inverse soft-Q Q learning in single-agent IL [4] to avoid adversarial training. We extend the idea the multi-agent settings under the paradigm of centralized training decentralized execution (CTDE) from MARI, allowing an efficient and stable learning process.
MARL. The literature of MARL encompasses a number of advanced methods; many follow the well-known CTDE paradigm. In [1]. they employ actor-critic architectures and train a centralized critic that utilizes global information. Value-decomposition (VD) methods represent the joint Q-
function as a function of agents' local Q-functions [3] 29 QMIX [2]] offers a more advanced VD
method for consolidating agents' individual local Q-functions through the utilization of mixing and hyper-network [16] concepts. Later value function factorization approaches, such as QTRAN [35] and QPLEX (39] introduce new factorization methods for MARL based on the IGM principle, which necessitates consistency between joint and local action selections. There are also policy gradient based MARL algorithms, Independent PPO (IPPO). a decentralized MARL, can achieve high success rates in several challenging SMAC maps 77. MAPPO, a PPO version for MARI, achieves SOTA results on expert and generate expert demonstrations. We also employ a hyper-network architecture [6] with mixing networks of non-negative weights and convex activations to facilitate our CTDE. While our mixing network techniques are similar to those used in QMIX [29]. we show that this configuration offers several unique advantages in the context of inverse Q-learning. Specifically, the training objective can be transformed from max min problem into a (non-adversarial) concave muximization problem, and it upholds the IGC principle. Other methods, such as QTRAN and QPLEX 335 39). primarily based on the IGM principle and are designed for different objective structures, making them unsuitable for our context.
several tasks [4]]. Our work utilizes MAPPO to train
3 Preliminaries
multi-agent cooperative system can be described as a decentralized partially observable Markov decision process (Dec-POMDP), defined by a tuple {S, O,N, A. P. R) (27]. where S is the set of global states, O is the set of local observations of agents, and N is all the agents. In addition. A = I Ar is the set of joint actions of all the agents, A, is the set of actions of an agent E N. and is the transition dynamics of the multi-agent environment. Finally, in Cooperative MARL (Coop-MARL.) all agents share the same reward function, R. that can take inputs as global states and actions of all the agents and return the corresponding rewards.
At each step, given a global state S, each ally agent i takes an action a; E A, based on his policy (a; | oz), where o; his local observation. The joint action is defined as A {a; and the joint policy is defined accordingly: II(A S) = lle (a; o;). After all the agent actions are executed, the global state is updated to a new state S' E S with the transition probability P(S' A, S). The objective of Coop-MARI. is find joint policy IT(- (S = II. mz(- o;) that maximizes the expected long-term joint reward, formulated as follows:
maxm En
En0 ((AA,,
7"
Multi-agent Inverse Q-Learning
In IL, the objective is to recover an expert reward or expert policy from some expert demonstration data. Our new multi-agent IL algorithm is built upon an integration of recent advanced findings in single-agent IL (i.e., the new leading IQ-Learn [14) and cooperative MARL. In the following, we first present the main idea of IQ-Learn, with some direct centralized and decentralized adaptations to multi-agent settings. We then discuss key shortcomings of such simple adaptations in the context of a Dec-POMDP. Finally, present our new algorithm which tackles all those shortcomings.
4.1
Inverse Soft Q-Learning
4.1.1
Centralized Inverse Q-Learning
In general, IQ-Learn can be directly adapted for II. a fullly-obeervable cooperative multi-agent set- ting. Given expert samples DE = =(T {(A,, St), t t=0, 1.. IQ-Learn aims to solve the following
3

---

maximin problem, which is also the objective of adversarial learning-basee IL approaches 18 13
maxę minn LRR, II) = (S.A))= [RS, A)] - E [R(S, A)] - Epn In [[)
(I)
where / (S, A) is the occupancy measure of visiting state 5 and joint action A, under the policy II:
x AS) = (1 (1- ()
II PS, - |
6 is the occupancy measure of the expert policy = and E,n [In [I,S A)] is the entropy regularizer. typical method solve TT to run an adversarial optimization process over rewards and policies, of which idea is similar to generative adversarial networks [18 [3]. However, such approach suffers instability, a well-known challenge of adversarial training. Thus, IQ-Learn avoids adversarial training by learning single soft Q-function, as defined in the following.
Definition 4.1 (Soft Q-function
[4]). Given policy IL, the soft Bellman operator B is defined as:
(B"Q)( A) = R(S, + P/s where VT (S) (E [Q(S, A) log (T(A S)]
(2)
1 is contractive and defines a unique soft Q-function for the reward function R. i.e. Q = BTQ.
Essentially, in [14), they show that ① is equivalent to the following single minimization problem which only requires optimizing over the soft Q-function: |
maxq ( {J(Q) = (S ((SS))) - (ESS--( [VR( - -e (S}
(3)
where Q x A - R is the soft Q-function, 8 is the initial state, and (Q is computed as (Q(S) = AEA exp(Q(S, A))). A shorteoming of centralized IQ-Learn described above is that the computation of V (S) or VQ(S) is not tractable as it requires to sample joint actions, an the joint action space grows exponentially in the number of agents. Furthermore, it requires full observations for agents, which is not applicable in a Dec-POMDP with local partial observations.
4.1.2 Independent Inverse Q-Learning
An alternative approach to overcome the shortcomings of centralized IQ-Learn is consider separate II. problem for each individual agent, considering its local observations. That is, one can set the objective recover local Q function Q: a;), as function of local observation o; and local action a; for each agent i E N. The local IQ-Learn loss function can be formulated as follows:
maxmin { Ji(Qs, Ti) = Eps Q ax) - -Eos-(P()SsA)) [V..T [o9]] - -(1 - E [v. (o") Q
Here, of P(- S, means of is the local observation of agent i corresponding to the new state S' ~ P(- S,A. The value function can be computed as [4]:
V (o.) (a.()))):) [Qr a;) log (a:lo:)
(5)
This approach is more tractable than the centralized method, making it suitable for Dec-POMDP environment by enabling decentralized policies for agents. However, it has limitations in addressing the interdependence between agents and the global information available during the training process.
4.2 Inverse Factorized Soft Q-Learning
We now present our new multi-agent IL algorithm designed to recover Q functions in the multi multi-agent setting, following the CTDE paradigm. A possible straightforward approach is to directly factorize the centralized objective function 3 This factorization can be achieved by decomposing the global Q-function Q(S, A) into local Q-functions Qi az), and computing the global V-functions using the formula 54 (}( log(E exp(Q(S,
Exp(((,) A))). However, this means that the global policy
We convert the maximization formulation in 14 into minimization to facilitate our later theoretical analyses.

---

has to be computed based on softmax of the global Q-function as (I) = E. (S.AI
exp(Q(S.4)
This requirement of global policy computation violates the decentralized execution under CTDE. According to this analysis, we will instead start from the original max-min formulation in (I. keeping in mind the necessity of having R; support CTDE.
Our key ideas involve creating (i) agent local Q-value networks that output local V-values V.O.T (o;, ai) and (ii) mixing the networks that utilize global state information to combine local values of agents into joint values that comprise the objective of the inverse soft Q-learning: and (iii) hyper-networks that provide rich representation for the weights of the mixing networks, allowing us to govern their value ranges. There are two important aspects driving our mixing architecture: the consistency between local and global policies, and whether we can leverage the advantages of single-agent IQ-learn [4), specifically non- aonaaversarial training and convexity within the Q-space. We will first describe our factorization approach, followed by an analysis of these two aspects.
4.2.1 Multi-agent IL Network Architecture
Overall, our network architecture comprises of three different types of networks, described below.
Agent local Q networks. Let Q(S, A) = {Qi(o; ) of local soft Q-values of agents where m = [W] is the number of agents and (o,, a;) E (S, A), E N. For an abuse of notations, we denote by Qi(a;, a;; 0:) as the local Q-value network of the agent of which learnable parameter is 0, Given (Q, II), we then use (S,AA) to represent the corresponding state-value vector vo. () (v.0TT (o=) where v,Q (o,) is computed in (5). Q and vQ.11 will be passed to the corresponding mixing networks to induce the joint values Qear and Q.11. These joint values will be then incorporated into computing the objective of the inverse soft Q-learning. Here, we do not assume any specific model for the policies т; As we show later, there exists a closed-form solution to compute optimal policies, which allows us to eliminate the variable m, from the training problem.
Value mixing networks. We create two mixing networks to combine local Q and V values into the joint values Qtot and / respectively. Let's denote these networks as Maaa (-) and Mav (-). Here th and Φy are corresponding weights of these two networks. In particular, we have:
tot (S = -M ( A))
V (S) -Mar (VQ.I(S))
Note that, instead of directly mixing Q(S), we mix the negative of this vector to achieve the IGC principle and the convexity. We can now formulate the objective function of our multi-agent inverse factorized Q-learning w.r.t the local Q-values and polices, and these mixing networks, follows:
maxq minn {J(Q, th, vv) - E(S.AAeDS QLoa
-BS ( [Voot (S)(
(S,A)EDE
- -11 (1 go [Vich
We further assume that the two mixing networks are feed-foorrr networks, constructed with non-negative weights and convex activation functions (e.g., ReLU, ELU, and Maxout). This configuration is widely-used in the literature and, as shown in the next section, is sufficient to ensure consistency between the global and local policies (a form of the IGM property, as shown later) and, importantly, the convexity of the training objective function within the Q-space.
Hyper-networks. Finally, we create two hyper-networks corresponding the two mixing networks. These hyper-networks take the global state S as an input and generate the weights Bhy and the of the mixing networks accordingly. The creation of such hyper-networks allows to have rich representation of the weights that can be governed to ensure the convexity of the objective J(Q, 11, vi2 vy) as well as the IGC property (as described below). We can write v/v = tvv (S; wv) and wie vo wR) where wv and wi denote trainable parameters of the state-value and reward hyper-networks. We can alternatively write the objective J( II, the vvv) as J(,, wR: wy) when the local soft Q-values Q are parameterized by f = (01. .2.... 8m and the weights of the mixing networks (gv, vhq) are parameterized by wy and WR respectively. The optimal policy, as shown later. can be computed as soxt-max of Q. We note that we will use either J(Q 1I, uQ. vvv) or [ wn, wv) depending on the context of our analysis.
In practical implementation, we consider shared local Qavalue network for all agents.
5

---

4.2.2
(IGC)
One of the key aspects of CTDE is ensuring consistency between global learning and local policy execution. In previous MARL approaches based on mixing Q-functions, this often expressed through the IGM principle [35 29 39), implying that the optimal joint actions across agents (obtained by maximizing the joint Q- Q-function) are equivalent to the collection of individual optimal actions of each agent (obtained by maximizing their local Q-functions). In our context. there are significant differences that prevent direct application of this principle. First, our recovered policy is not derived from directly maximizing the Q-function. Instead, it is obtained by solving a diittibbttionmmacciiin objective, which results in soft policies computed as the soft-max of Q-values. To adapt this principle in the context of inverse Q-learning, (IGC), which is a generalization of the IGM, defined in the following.
introduce the concept of Individual-Glabatt Siniiiuud-Gllaaa-CCnstttttenn Consistency
Definition 4.2 Madividual-Globall Conossal-y (IGC)), A factorized learning approach is said to adhere the IGC principle if and only if the optimal joint policy (obtained from solving the global training problem) is equivalent to the collection of individual optimal policies for each agent (obtained by solving their respective local training objectives).
The following result states that our factorized inverse Q-learning approach satisfies the IGC principle.
Theorem 4.3. Let II = argmin (Q. II, Uh, vy)}, then there are a set of optimal local policies { such that के = argming, Js, m.), Vi N, and II" {N. i E N..
The theorem implies that the distribution of joint actions, produced by the optimal joint policy, is the same as the collection of local action distributions produced by the local optimal policies. Moreover, as additional note, the IGM principle with respect to the joint Q-function also holds under our mixing network architecture, i... for all S E S:
argmax, A ( A)] = fargmax,,, (Qr(0, NEN
4.2.3 Non-adversarial Training and Convexity
One of the main advantages of the single-agent inverse Q-learning algorithm is that the training problem can be equivalently transformed into a concave maximization problem over the Q-space, making the training process highly stable and well-behaved. We demonstrate below that these features still hold under our mixing network architecture.
Proposition 4.4. The max-min problem in (6) is equivalent to the following maximization problem:
maxe {Jl VQ. vvv) vv,+vv
= (S.A)EDR
(S, - -.s.))).ssss [5')]]
[Qtat (S,
[S]]]
- -11-29 - Eso vrdo((So)]
(7)
where Vio = Mar ( and v (2 (0)) def log(Ea, exp(Qi(o;, a))),, i ieN)
Moreover, let Q' be optimal 77. then the global optimal policy can be recovered as follows:
exp(Qi(os, a+)))
II" n. (os. 0s) = Ea; exp(Qu(or,a)) at))
iEN
(8)
Furthermore, it can be shown that the objective in 7 is concave in Q which is essential property that make the Q-learning procedure well-behaved and stable [14]
Theorem 4.5. J(, vi. v/v) is (strictly) concave in Q. As a result, alwways yield a unique solution within the Q-space.
Theorem 4.5 indicates that the global training objective JQQ, VQ, vv) is concave in Q when using any layer feed-forward mixing networks with non-negative weights and convex activation functions. This result is highly general and notably non-trivial due to the nonlinearity and complexity of vi (as functions of Q)) and the mixing networks. Prior work often relies on a two-layer mixing structure (29, noting that such a two-layer structure is sufficient for the mixing network to approximate any monotonic function arbitrarily closely in the limit of infinite width (8]

---

4.3 MIFQ Algorithm
We now present our practical algorithm, MIFQ. and details of our implemented network architecture. Mixing and Hyper Networks. We employ the following two-layer feed-forvee forward network structure:
Maa (X) = ELU(X x W x 1 + bg
(9)
Mør (X) ELU(X x | b W + by
(10)
for mixing networks, where dQ = (WP, W bi 89 and v/v = (WY, W. bY, bb are the weight and bias vectors. The absolute operations ensure that all the weights non-negative. Moreover, aiQ and g'y are an output of hyper-network taking the global state as input. Each hyper- network consists of two fully-connected layers with a ReLU activation. Finally, ELU is employed to mitigate the issue of gradient vanishing and to ensure that negative inputs remain negative. Indeed, ELU is convex and the two mixing networks Ma and May have non-negative weights, implying that the loss function J (Q, QQ.. viv) is concave in Q and the IGC holds.
Practical Implementation. Similar to 14,, we use a X² -regularize Φ(x) + 1=2² for the first terms of the loss function in (6). This convex regularizer bounded even when Q; for some i E N. to -cc, which is crucial to keep the learning process stable. In addition, instead of directly estimating EEs. (Viot (So)), we utilize the following equation to approximate Es, [V (So)] which can stabilize training:
useful to ensure that this is lower-
(1 E) (S)) (( (V - 1Is (V
for any value function Vg (-) and occupancy measure [4], we can estimate Es. (V))
(So)] by (V (S) 2 (5') instead. In
sampling (S, A) from replay buffer and estimate
summary, we will employ the following practical loss function:
min {J(0, W() = E(S,A)eDE ((Q'0 7V2 Ownnay
E.s....s) [V (S) VV2l [
(11)
Fig. T shows our network architecture and the details of our MIFQ can be found in the appendix.
yrt (55
(S)
(5,A)
Q. a.)
Hyper Network
Mixing Natwork
Mixing Network
LogSumEssp
Linear
Qu(oi )
gather
bY
BLU
(o)) ((
(Qa Qm (Ora en)
Linear
Linear
Af
H
GRU
Linear
W
Agent Network
Agent Network
Linear
(Rw)
(ois 41)
(Ome dme)
Og
Figure 1: An overview of our network architecture.
Experiments
Environments. We three environments in which enemy agents' policies are fixed and controlled by a simulator. We collect expert trajectories from well-trained ally agents and build IL to mimic them. The resulting imitating agents are evaluated by letting them play against the simulator's enemies.
SMACv2 19. SMAC a well-known multi-agent environment built based on StarCraft II. We employ S [9, an enhanced version of SMACv1 [3], that introduces a more formidable environment
7

---

Expert
(Det)
(Soft)
0.6-
0.8
0.4
0.4
0.3M
0.50
Protoss: 5ys5
Terran: 5vs5
Zerg: 5vs5
Protoss: 10vs10
Terran: 10vs/0
Zerg: 10vs10
0.5
-50
0.4
20
-100-
0.5M
0.500
0:3M
Miner: easy
Miner: medium
Miner: hard
MPE: reference
MPE: spread
MPE: speaker
Figure 2: Learning curves
Table
Winrate and reward comparisons
IQ- VDN
MA- SQIL
MA- AIRL
MIFQ (ours)
Scenarios
Expert
BC
IIQ
GAIL
Det
Soft
5ys5
86.7%
19.5%
33.6%
39.8%
47.7%
42.6%
42.6%
64.8% 61.7%
72.7
Protoss
10vs10
90.3%
5.5%
16.6%
38.3%
36.8%
19.8%
28.3%
77.3% %
5vs5
81.7%
18.0%
19.7%
32.0%
24.6%
10.9%
10.9%
55.9%
71.9%
Terran
10vs10
81.7%
6.2%
14.1%
35.9%
0.5%
2.3%
1.0%
53.8%
72.7 %
5vs55
73.5%
10.9%
18.6%
33.6%
14.8%
5.3%
18.8%
46.7%
60.9%
Zerg
10vs10
76.3%
9.4%
16.4%
17.2%
27.1
1 1.2%
31.2%
51.0%
59.4%
easy 82.4%
31.2%
21.9%
18,8%
36.7%
35.2%
34.6%
52.7%
60.2%
Miner medium
74.9%
28.1%
9.8%
14.8%
28.3%
21.1%
26.0%
43.8%
49.2%
hard
69.8%
12.5%
6.6%
11.7%
19.9%
17.2%
20.9%
39. 19.1%
39,8%
reference
-17.3
-18.3
-18.6
-19.0
-36.6
-40.7
-40.0
-23.4
-20.5
MPE spread
-11.1
-20.5
-23.6
-21.6
-23.0
-26.5
-24.3
-23.2
-24.2
speaker
-19.4
-27.5
-29.1
-29.5
-104.3
-125.5
-78.7
-30.8
-26.3
Gold Miner SMACx2
O21
215
212
1911
552
214
647
65
2039
19C
275
1955
211
IQVDN
MASQIL
MAAIRL
MAGAIL
MIRQ (Det)
MIFQ (Soft)
Figure 3: Comparison with different numbers of demonstrations. X-axis: winning rate. Y-axis: number demonstrations.
for evaluating MARL algorithms. In SMACv2, scenarios are procedurally generated, compelling agents to adapt to previously un-encountered situations. This benchmark has 6 sub-tasks, including Protoss, Terran, and Zerg which feature 5 to 10 agents. These agents have the flexibility to engage with opponents of differing difficulty levels.
Gold Miner [2]. This game originates from a MARL competition, which two teams, ally and enemy. navigate through 2D terrain to find gold. A team win if they mined larger amount of gold than the other. Winning this game is challenging since the allied agents must compete against exceptionally well-developed heuristic-based enemies. We consider three sub-tasks, each involves two ally and two enemy agents: (i) Easy: The enemies employ simple shortest-path strategy to reach the gold deposits; (ii) Medium: One enemy agent follows greedy approach, while the other

---

follows the algorithm developed by the second ranking team in the competition; and (iii) Hard: the enemies consists of the first- and becond-ranking teams.
Mulii Particle Environments (MPE) (25]. MPE contains multiple communication-orienee determin- istic multi-agent environments. We use three cooperative scenarios available in MPE for evaluating, including: (i) Simple spread: three agents learn to avoid collisions while covering all of the landmarks; (ii) Simple reference: two agents learn to get closer to the target landmarks. Each target landmark is known only by the other agents, so all agents have to communicate to each others; and (iii) Simple speaker listener: similar to simple reference, but one agent (speaker) can speak but cannot move, and one agent (listener) can move but cannot speak.
Expert Demonstrations. For each task, trained expert policy by MAPPO with large-scale hyper-parameters (e.g., multi layers, higher dimensions, longer training steps, and using recurrent neural network, etc). In term of expert buffer collection, test each method with different numbers of expert trajectories: up to 128 trajectories for MPEs and up to 4096 trajectories for Miner and SMAC-v2. Note that MPEs are not dynamic environments, so we do not need a large number of expert demonstrations for evaluation. For each collected trajectory, we used a different random seed. For fair comparison, each method uses the same saved expert demonstrations for the training.
Baselines. We compare our MIFQ against other multi-agent IL algorithms, which either originate from the multi-agent IL literature, be adapted from SOTA single-agent IL algorithms. These baselines include: (i) Behavior Cloning (BC); (ii) Independent IQ- (IIQ) this is a simple adaption of IQ-Leamn for multi-agent setting, described in Section 4. 4.1.2. (iii) IQ-Learm with Value Decomposition Network (IQVINN) - this is another adaptation of IQ-Leemn, but instead of using mixing and hyper networks aggregate agent Q functions, employ Value Decomposition (VDN) [38]; (iv) MASQIL- - we adapt SQIL for multi-agent settings [30]. MASQIL shares some similar advantages with our MIFQ, such as being non-adversarial and enabling decentralized learning through centralized learning; (v) MAGAIL- - a multi-agent IL algorithm introduced by [36]; and finally (vi) MAAIRL- this algorithm proposed by 42 Moreover, since our MIFQ algorithm relies
on soft policies while most previous factorized Q-learning algorithms [29 35 39 use deterministic policies, we include a deterministic-polie ppoicy version of MIFQ for comparison purposes. In this version. the deterministic optimal local actions are determined as at = argmax,, (Q, a,) and the local
functions are 1.9 (o:) = maxa, Qi(o,, oz). We denote our main MIFQ algorithm as MIFQ (Soft), and the deterministic-polen poliic version as MIFQ (Det).
Comparison Results. We train each task with our algorithms and the different baselines, using 128 trajectories for MPE, and 4096 trajectories for SMACv2 and Miner. As a standard practice, we report winrates for SMACv2 & Gold Miner, and reward scores for MPEs. Figure 2 shows the learning curves and Table I reports the final winrates and average scores (the full table with std values can be found in appendix). Our methods MIFQ (Soft) significantly outperforms all other baselines on SMACv2 & Gold Miner tasks, and has competitive performance on MPEs, with note that MPEs a deterministic environment and its tasks are considerably easier than SMACv2 and Gold Miner. We also observe that MIFQ (Der) significantly outperforms other baselines, but remarkably worse than MIFQ (Soft), indicating the advantage of using soft-policy learning in our context. Moreover, to evaluate the efficiency of our method with different numbers of expert demonstrations, we compare our MIFQ with the baselines on two dynamic tasks: SMACv2 & Miner. Figure 3 shows box and whisker plots of the average winning rate of each method each task for data summarising analysis. shows that MIFQ (Det and Sofi) offer higher winrates, and converges better with smaller standard errors. Moreover, to evaluate how the convexity help enhance our algorithm, we compares MIFQ with those that rely on non-convex mixing networks, which are obtained by replacing the convex activation ELU with non-convex ones such as sigmoid or tanh. More details can be found in the appendix.
6 Conclusion and Limitations
Conclusion. We developed a multi-agent IL algorithm based on the inversion of soft-Q functions. By employing mixing and hyper-network architectures, our algorithm, MIFQ, is non-adversarial and enables the CTDE approach. We demonstrated that, with some commonly used two-layer mixing network structures, our IL loss function is convex within the Q-function space, making

---

learning convenient. Extensive experiments conducted across several challenging multi-agent tasks demonstrate the superiority of our algorithm compared to existing approaches.
Limitations. Some limitations of the work includes: (i) while MIFQ achieves impressive perfor- mance across various tasks, it falls short of reaching the expertise levels (increasing the number of expert demonstrations could solve the issue, but it would introduce additional computational challenges, as the replay buffer will become too large and the training process will be very costly); (ii) MIFQ, along with other baseline methods, struggles when confronted with very large-scale tasks, such as some of the largest games in SMACv2; and (iii) our multi multi-agent IL (and other baselines) still requires large amount of (clean) expert demonstrations, which would be not always available in practice. These limitations could pave the way for future work.
Broader Impacts. Our research focuses on imitation learning in multi-agent systems, it may have potential applications similar areas where imitation learning has been impactful, such as autonomous driving, healthcare, and game theory. There are also potential negative impacts. For instance, imitation learning could be used for surveillance purposes, following and monitoring individuals in public spaces, or for developing autonomous weapons.
Acknowledgments
This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the Al Singapore Programme (AISG Award No: AISG2-RP-2020-017),
References
[1] Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty- first international conference on Machine learning, pp- 1, 2004.
[2] Arora, S. and Doshi, P. survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
[3] Barrett, S., Rosenfeld, A., Kraus, S., and Stone, P. Making friends on the fly: Cooperating with new teammates. Artificial Intelligence, 242:132-171, 2017.
[4] Bhattacharyya, R. P., Phillips, D. J., Wulfe, B., Morton, J., Kuefler, A., and Kochenderfer, M. J. Multi-agent imitation learning for driving simulation. In 2018 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 1534-1539. IEEE, 2018.
[5] Bogert, K. and Doshi, P. Multi-robot inverse reinforcement learning under occlusion with interactions. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pp. 173-180, 2014.
[6] Boyd, S. and Vandenberghe, L. Convex optimization. Cambridge university press, 2004.
[7] de Witt, C. Gupta, T., Makoviichuk, D., Makoviychuk, V., Torr, P. H., Sun, M., and Whiteson, S. Is independent learning all you need in the stareraft multi-agent challenge? arXiv preprint 2..i::2011.0553:: 2020.
[8] Dugas, C., Bengio, Y., Bélisle, F., Nadeau, C., and Garcia, R. Incorporating functional knowledge neural networks. Journal of Machine Learning Research, 10(6), 2009.
[9] Ellis, B., Moalla, S., Samvelyan, M., Sun, M., Mahajan, A., Foerster, J. N., and Whiteson, Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2212.07488, 2022.
[10] Finn, C., Christiano, P., Abbeel, P., and Levine, S. A connection between generative adver- sarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint 161iv:1611 116611111222.. 2016.
[11] Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
10

---

[12] FPT. Fpt reinforcement learning competition.
https://github.. http://tuuub.oooopppogg.ggg
cocop2000 2020.
[13] Fu, J., Luo, K., and Levine, S. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint urXiv: 1248, 2017.
[14] Garg, D.. Chakraborty, S., Cundy, C., Song, J., and Ermon, S. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028-4005 2021.
[15] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B.. Warde-Farley. D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[16] Ha, D., Dai, A. M., and Le, Q. V. Hypernetworks. In International Conference on Learning Representations, 2017. URL http:://wppppreeeeeeeeeeettreata
[17] Hawke, J., Shen, R., Gurau, C., Sharma, S., Reda, D., Nikolov. N., Mazur, P., Micklethwaite, S.. Griffiths, N., Shah, A., al. Urban driving with conditional imitation learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 25 257- IEEE, 2020.
[18] Ho, and Ermon, Generative adversarial imitation learning. Advances neural information processing systems, 29, 2016.
[19] Jeon, W., Barde, P., Nowrouzezahrai, D., and Pineau, J. Scalable and sample-efficient multi- agent imitation learning. In Proceedings of the Workshop on Artificial Intelligence Safety, co-located with 34th AAAI Conference on Artificial Intelligence, SafeAI@ AAAI, 2020.
[20] Kostrikov, I. Nachum, O., and Tompson, J. Imitation learning via off-policy distribution matching. arXiv preprint 11XV::19/2.050322200
[21] Kraemer, L. and Banerjee, B. Multi-agent reinforcement learning rehearsal for decentralized planning. Neurocomputing, 190:82-94 2016.
[22] Le, H. M., Yue, Y., Carr, and Lucey, P. Coordinated multi-agent imitation learning. In International Conference on Machine Learning, pp. 1995-2003, PMLR, 2017.
[23] Le Mero, L., Yi, D., Dianati, M., and Mouzakitis, survey on imitation learning techniques for end-to-end autonomous vehicles. IEEE Transactions on Intelligent Transportation Systems, 23(9):14128- 14147, 2022.
[24] Lin, X., Beling, P. A., and Cogill, R. Multi-agent inverse reinforcement learning for zero-sum games. arXiv preprint arXiv: 1403.5508, 2014.
[25] Lowe, R., Wu, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. Multi-agent actor-critic for mixed processing systems, 30, 2017.
environments. Advances in neural information
[26] Oliehock, F. A., Spaan, M. T., and Vlassis, N. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353. 2008.
[27] Oliehoek, F. A., Amato, C., et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
[28] Pomerleau, D. A. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97 1991.
[29] Rashid, T., Samvelyan, M., De Witt, C. Farquhar, G., Foerster, J., and Whiteson, S. Mono- tonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1):2334-2284 2020.
[30] Reddy, S., Dragan, D., and Levine, Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint 1rXiv: 2019.

---

[31] Reddy, T. S.. Gopikrishna, V., Zaruba, G., and Huber, M. Inverse reinforcement learning for decentralized non-cooperative multiagent systems. In 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1930-1935. IEER, 2012.
[32] Ross, S. and Bagnell, D. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-668. JMLR Workshop and Conference Proceedings, 2010.
[33] Samvelyan, M., Rashid, T., De Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G., Hung, C.-M., Torr, P. H., Foerster, J., and Whiteson, S. The starcraft multi-agent challenge. arXiv preprint arXiv; 1902.040433 2019.
[34] Shah, S. I. H., Coronato, A., Nacem, M., and De Pietro, G. Learning and assessing optimal dynamic treatment regimes through cooperative imitation learning. IEEE Access, 10:78148. 78158, 2022.
[35] Son, K., Kim, D., Kang, W. J., Hostallero, E., and Yi, Y. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pp. 5887-5896. PMLR, 2019.
[36] Song. J., Ren, H., Sadigh, D., and Ermon, S. Multi-agent generative adversarial imitation learning. Advances in neural information processing systems, 31, 2018.
[37] Sošić, A., KhudaBukhsh, W. R., Zoubir, A. M., and Koeppl, H. Inverse reinforcement learning in swarm systems. arXiv preprint arXiv: 1602. 05450, 2016.
[38] Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Somnerat, N., Leibo, J. Z., Tuyls, K., et al. Value-decomposition networks for cooperative multi agent learning. arXiv preprint arXiv: 05296, 2017.
[39] Wang. J., Ren, Z., Liu, T., Yu, Y., and Zhang, C. Qplex: Duplex dueling multi-agent q-learning. arXiv preprini arXiv:2008.01006 2020.
[40] Wang, L., Yu, W. He, X., Cheng, W., Ren, M. R., Wang, W., Zong, B., Chen, H., and Zha, H. Adversarial cooperative imitation learning for dynamic treatment regimes. Proceedings of The Web Conference 2020, pp. 1785-1795. 2020.
[41] Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., and Wu, Y. The surprising effec- tiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611 1-24644, 2022.
[42] Yu, L., Song. J., and Ermon, S. Multi-agent adversarial inverse reinforcement learning. In International Conference on Machine Learning, pp. 7194-7201. PMLR, 2019.
[43] Zhou, J., Wang, R., Liu, X., Jiang, Y., Jiang, S., Tao, J., Miao, J., and Song, S. Exploring imitation learning for autonomous driving with feedback synthesizer and differentiable rasteri- zation. In 202/ JEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1450-1457, IEEE, 2021.
[44] Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K., et al. Maximum entropy inverse reinforcement learning. In Aaai, volume pp. 1433-1438. Chicago, II, USA, 2008.
12

---

Appendices
A Missing Proofs
A.1 Proof of Theorem
Theorem. Let IIo = a (J(Q. II. th. vvv), then there are a set of optimal local policies {mt) such that पद argmin Ji(Qi, mi), vi E. and II {n], i N..
Proof. We first consider the local objection function:
JeQqs m4) EDs [Q:(0,, PC [V. (d)] 111-71 B) [1.0 (o.)
From this we can see that that Ji(Qi, mT)) is minimized (over T;) if each value function v.Q.T(o)) (of)
maximized (over T³). Morcover, from the definition V. (o.) = (a QQ((o, a,) log (e(a o,)], we see that the value of v.Q."(o, is only dependent of Qi a;) and o. (a, o;) for
all a; E A; follows that v.Q-=(..) achieves its maximization at:
(-lo,) = argmax,;;1lo) (Earm as) - log Te (as [;))]
[(((lo)))= = exp(Qi(o;;aa)) (:))
Eag exp(((o,, a()) Va, A
(12)
Now let II { E W be a joint policy (or a of local policies) defined in 12 It follows that V.Q.7 (o:) s V.R.=* (ot) for any E W.
We now look at the joint objective function JQ, 11, #2, v/),, defined
J(Q, II, tiQ, vvv) = E (S.A)EDE
=(( Qta A) - - - (vrot [o
- - Es0 [V
(Su)
Given Q. (5) = May (S)), our mixing structure ensures that M (VQ. is mono- tonically increasing with respect to each element of vQ. Additionally, for any joint policy
Il { i E W). we have (o.) < v. (o:) for any i E N. Combining this with the monotonieity of the mixing network, obtain:
Visa Viol Viot
which implies:
J(Q. II, 2Q, wv) 2 (QQ e0q vvv)
for any joint policy II. This also implies II" [ (J(Q. II, vie ) as desired.
A.2 Proof of Proposition 4.4
Proposition.
The max-min problem (6) is equivalent to the following maximization problem: (V))(
maxe { ((l))o E (S.A)EDE
E
[Q (S, - - V880t (5')]]
- B5 [vdrdt (So)]
where Vist (S) Mer (S),, and (o (S) = (v. (o.) def log(E., exp(Qi ad)),, i iEN).
Moreover, let Q* be optimal to 7 then the global optimal policy can be recovered as:
(o;,)((o;:a;)))
II
π. (os, as) = Ex, exp(Q:(o,,))
iEN
13

---

Proof. We can leverage the result stated in Theorem
4.3 to prove the proposition. That is, Theorem
4.3 tells us that (Q II, V2. vvv) ) is minimized IIo m.. E N/ such that
exp exp(Q((o,, m (os, 0:) Enj exp(Qi(or, a()
IT
iEN
Therefore, the max-min problem can be converted to a maximization one within the Q-space:
max Mexminn (J(Q, th dbv) max J[Q, "Q,",,,,"" thv)
Q
Moreover, is well-known the maximal value V.Q. of (o, 0z) [4] as:
. (o:) can be computed as a log-sum-exp function
v (0:)= log Σ Eexp(Qu(or,ai)) =1,9(0))
Rx
which confirm our claims.
A.3 Proof of Theorem
Theorem JQ, th vhv) is (strictly) concave in Q. As result, always yield a unique solution within the Q-space.
Proof. To prove the concavity, we will prove that each component of the objective function, i.c. Q A) and (S) are concave in Q. keeping in mind that the mixing networks have non- negative weights and convex activations. We will need the following lemmas to validate the main claim.
Lemma A.1. Any feed mixing networks M(X) constructed with non-negative weights and non-decreaiing convex activation functions (such as ReLU or ELU or Maxout) is convex and non-decreasing in X.
Proof. Any N-layer beed-forward network with input can be defined recursively as
f =
(13)
('(X) = o² (fm Wn + bn, 1...,, N
(14)
where o" is a set of activation functions applied to each element of vector f"- (X). and W, and b, the weights and biases, respectively, layer n. Therefore, we will prove the result by induction. i.e. (X) is convex and non-decreasing in X for n = 0, ..... Here note that f (X) is vector, so
when we say ("x (X) is convex and non-decreasing X", it means each element of (X) convex
and non-decreasing in X.
We first see that claim indeed holds for 0 Now let us assume that, (X) is convex and non-decreasing in X, we will prove that f (X) is also convex and non-decreasing in X. The non- decreasing property can be easily verified as we can see, given two vector X and X' such that X > X' (element-ve telement-wise wise comparison). then we have the following chain of inequalities
(0)
X
f
(X)
d"
(X)) 0"
(c)
n (((-1((X)) W,, bn 2 R" ((--1(Xx') Wn + b,
where (a) is due the induction assumption that f"-1(XX is nen-decreasing in X, (b) is because "º are also neceeecr,,,,ng, and (c) is because the weights W,, is non-negative.
To verify the convexity of ("(X),we we will show that for any X, X', and any scalar E (0, 1), the following holds
a. X (1-a))"
f" x)X)
(15)
14

---

To this end, we write
a (1 a0" (8)) (1-)))(('** x Wn + bn
4 nd (x) (1 a0) x Wn + bn
(1x (aX aaxxi) x Wn br
f + x)x
where (d) is due to the assumption that activation functions ṭ" are convex and W,, 2 0, and (e) is because af (X) +
a1laa)---(X 2 (aX + (1 - a X') (because f fx-1(X) is convex in
X. by the induction assumption) and the activation functions o" is non-decreasing and W, 2 0. So, we have
a f (X) + (1 a) (ax (1-a)
implying that fd ("( is convex in X. We then complete the induction proof and conclude that f convex and non non-decreasing in X for any n = 0, O.... N.
Lemma A.2. For any i E N.
V. (o:)) log Eexp(Q((o))a)))
are convex in Q.
Lemma A.2 can be easily verified, as V,P (o:) has the log-sum-exp form, so it is convex in Q 6
Lemma A.3. Let x(Q) (x,(2). (m(()) be vector of size where each element convex function function of Q2 i.e. (2(2) is corvex in Q for any i N, then M(X(2)) is convex in Q. where M(X(O)) mixing network of non-negative weights and convex activations.
Proof. We will make used general properties of convexity to verify the claim, i.e., we will prove that, for any two vector Q and Q' and any scalar a E (0, 1), the following inequality always holds
aM((X(Q) (1 a) 2 M(X(aQ + +(1 -1-aa)0 aa)a)))))
(16)
According to Lemma A.1 we know that M(X) is convex in X, thus:
aM(((Q)) (1 - ())))) M(ax(() + (1- (1-a)X(Q'))
(17)
(Q) + (1-a))x
=M aX2 + (1-a))x2(0
(18)
ax Q + (1-a) (m))(())))
Moreover, since (2 is convex in Q for all i E N. we have ax, ( + (1 (1- (x- >
X1(aQ + Q'). In addition, from the monotonicity of M(X) in each element X,, we have
ax,(Q) + (1 aX2(Q) + (1
X1(aQ
M
2 M
-M(X(a0+(1-0))
axm ( + (1
(19)
Combine 18) and
we get
a aM((()) + (1 a)M(x(()) (LX1--/MMX)0002XXX0000000000001- ((x(a@((1=)))
which validates [6
completing proof for Lemma A.3
15

---

We now go back to the main result. First, considering Muo (-Q(S,A)) we can see that each element of - (S, A) is linear in Q. Thus, Lemma A.3 tells us that that (- A)) is convex in Q. As result, Qto (S, A) is concave in Q. Similarly, (S) is also vector where each element (o:) convex in Q (Lemma A.2 thus May (V)) (and Vi (S)) also convex in Q Therefore, the objective function of our II, formulated as
(o:)
E (S.A)EDE (S. TIE (2 [5)]] - -1 (V2
J (Q, VQ, vvv
should be concave in Q. We further see that V. (o:), Vog, are strictly convex in Q. implying that [Q, t22, v is also strictly concave in Q. as desired.
It is important to note that the above result only holds if Qi are independent. It is not the case if Qi. for some share common network structure. This is also the case of the IQVDN considered in the main paper, i.e., the global reward and value function Ro and V are sums of the corresponding local functions, but Qi share the same neural network structure.
B
Additional Details
B.1 MIFQ Algorithm
The detailed steps of our MIFQ algorithm are shown in Algo.
below:
Algorithm 1: Multi-agent Inverse Factorized Q-Learning
Input: Parameters 0 = (O1,...,, .m.) and (wn, w expert's demonstrations DE policy replay buffer Dreplay 0, and learning rates Àg and Aw-
repeat
Compute local policies: (,(aloo)) = (xp(Q((o.a:00))
Vi E;
Eafea, exp(C.(o.,0
# Collect samples for replay buffer for a certain number of collecting steps do Execute policies my to collect new transitions ((S, A,S')) and add them to Dreplog;
Run gradient descent with mini-batches Dreplay for a certain number of training steps do
L 0 A Ag v [(, WR,Wy)) wy); (WR, (WR,WV) A V Wann J(B,up, wy);
until certain number of loops;
B.2 Global versus Local Value Equations
We discuss equations that describe the relationship between the and V functions, which we refer to as Value equations. Specifically, we consider Eq. f local value functions and Eq. 2 f global ones. So far, we have assumed that the Value equations hold for all local agents (i.e., Eq. S gholds), ensuring that distribution-mate matching is executed well at the individual level. An alternative approach to the factorized learning to assume that the Value equations hold at the global level, i.e.,
tot (S) E (a-I(()))))
[Q (S, A) log ([AAS))
(20)
Recall that the local value equations can be written
0 (o,) = (n,(a,)o.) [Qi(o, a;) log (a,lo))
We then can see that the local and global Value equations, generally, cannot hold simultaneously. in particular when Vot and Qiot have different mixing structures. Thus, we can only ensure the validity of either the local Value equations in 5 or the global one (2)p. It can seen that, while adhering to 5 is highly tractable due to several advantages shown above (i.c., IGC, aon-aaieraarial training,
16

---

and convexity), using 20 to compute V is highly impractical as due to several reasons. First, it cannot help avoid adversarial training as there is no closed form solution for the minimization over II, so one needs to learn both Q and II adversarially. Second, it requires sampling over joint actions approximate Viot, which shares the same computational issue as the centralized approach discussed in Section 4.1.1 namely, the exponentially large joint action space make the training process highly inefficient.
B.3
Experimental Settings
Each environment has different observation space, state dimension, and action space. Therefore, we use different syper-parameters each task to try to make all algorithms work stably. Moreover, due to limitations in computing resources, especially random access memory, we reduce the buffer size to 5000 on two hardest tasks, Miner and SMACv2, to be more efficient in running time with parallel workers. More details are available in Table 2 clusters for training and evaluating all tasks. Specifically, each HPC cluster has workload with an NVIDIA L40 GPU 48 GB GDDR6, 32 Intel-CPU cores, and 100GB RAM. In terms of model architecture, Figure T shows proposed model structure with mixing networks based QMIX algorithm 229
We use four
Computing (HPC)
Table 2: Hyper-parametens
Arguments
MPEs
Miner
SMACv2
Max training steps
100000
1000000
Evaluate times
32
Buffer size
100000
5000
Learning rate
2e-5
5c-4
Batch size
128
Hidden dim
256
Gamma
0.99
Target update frequency
Number of random seeds
B.4
Ablation Study Non-convex Mixing Networks
Recall that when convex activation functions are used in building mixing networks, our Theorem 4.5 shows that the objective function of the multi-agent inverse soft Q-learning is convex as long as the mixing networks are constructed with non-negative weights and convex activations. This convexity property guarantees the optimization objective is well-behaved and has an unique optimization solution with Q-space, leading the effectiveness of the IL process. Therefore, in the practical implementation of our algorithm, we chose the ELU activation function. In this section, we empirically demonstrate the impact of convex activation functions by examining the performance of our algorithm when non-conve con-conve activation functions such as sigmoid or tanh are used in mixing networks instead of convex once. Table JableBrepoos reports the winrates and rewards, and Figure 4 shows the learning curves of different approaches. These results clearly show that the ELU activation significantly outperform those using non-convex activations (i.e., sigmoid, tanh), for both versions MIFQ (Soft) and MIFQ (Det).
B.5 Ablation Study Comparison with SAC-based IQ-learn
In this section, we explore the performance of SAC-IQ, an adaptation of the Soft Actor-Criti (SAC) algorithm combined with IQ-Learn, tailored for aulti-agent settings. SAC-IQ employs a centralized
Q function alongside decentralized critics, maintaining adherence to the original objective of IQ- Learn. This approach is evaluated across 9 distinct tasks within two environments: SMACv2 and Gold Miner. For SMACv2, we consider scenarios with varying team sizes, specifically 5vs5 and 10vs10. In the Gold Miner environment, we assess performance across three difficulty levels: easy, medium, and hard.
17

---

Table 3: Winrate and reward comparisons between different activation functions (the values percentages for SMACv2 and Miner, and rewards for MPE.)
MIFQ-ELU
MIFQ-Sigmoid
MIFQ-Tanh
Expert
Det
Soft
Det
Soft
Det
Soft
5va5
86.7
64.8=4.1
72.7=10.00
49.2+:4.6
53.1+6.6
60,9=10.00
62.5+3311
Protoss
10va/0
90.3
61.7=:.88
77.34..6
59.2:::22
45.31:9.2
36.977722
64.4:1111
5ux5
81.7
$5.9100.4
71.9.13.8
48.4.12.9
35.2.1.0.55
45.1:12.7
6066.6..7
Terran
81.7
53.826.9
72.7:33.4
26.0:222
46.11:9.2
28.6:226
5ou5
73.5
46.71.5.2
60.91.14.7 14.7
39.8.15.1
21.111..1
27.9:1222
43.8.15.7
Zerg
10hs10
76.3
51.0=2.2
39.4:1.11
12.5:449
29.74:2.7
15.2433.9
41.3:222
easy
82.4
52.7-0.7
60.245.66
32.0+446
53.1+7.77
56.47.22
43.1
Miner
medium
74,9
43.8=2.2
49.24:5.1
24.8=+2.2
40.6:1:.88
45.7=10.1
38.7
hard
69.8
39.1
39.8:4:66
21.7=22.
25.8
34.823.9
10.8
reference
-17.2
-23.40..1
20..5::0.0
.0..
23.0:10.1
22.110.111
22177.0.11
spread
-10.7
-23.240.11
-24.240.1
166440.1
-24.74007
1.78
126.6
speaker
-19.7
-30.8+0.1
-26.3+0.0
-133,4=0.1
-28.9+0.0
-34.7+100
-28.2+0.0
Figure 4: Convergence curves of MIFQ with different activation functions
Expert
MIFQ (Det-ELU)
MIFQ (Soft-ELU)
MIFQ (Det-Sigmold)
MIFQ (Soft-Sigmoid)
MIFQ (Det-Tanh)
MIFQ (Soft-Tanh)
0.8
0.4-
0.5M
0.5M Terran: 5ys5
0.5M Zerg: 5ys5
1M
Protoss: 5py5
0.4-
0.5M
0.5M
0.5M
1M
Protoss: 10vs10
Terran: 10vs10
Zerg: 10vs10
0.4
0.5M
1M
0.5M
1M
0.5M Miner: hard
1M
Miner; easy
Miner: medium
The results, detailed in Table 4 illustrate that SAC-IQ demonstrates commendable performance when compared to other baselines such as Behavior Cloning (BC), Independent IQ-Learn (IQQ), and Multi-Agent Generative Adversarial Imitation Learning (MAGAIL). However, does not surpass the effectivvness of our proposed methods. Figure highlighting the training dynamics of SAC-IQ in comparison to other approaches. This visualization underscores the competitive nature of SAC-IQ while also affirming the superior performance of our methods in these challenging multi-agent environments.
a visual representation of the learning curves,
B.6 Experimental Details
In this section, we present the experimental details for the SMACv2, Miner, and MPE tasks, with varying numbers of expert demonstrations. These details complement the box plots shown in Figure ii the main paper. Tables 4-9 report win rates (as percentage values) for SMACv2 and Miner tasks with the number of trajectories ranging from 128 4096. Table 10 reports final rewards for MPE tasks with the number of trajectories ranging from to 128. We also plot the learning curves of these experiments in Figure]
18

---

Figure 5: Learning curves of SAC-IQ compared with other baselines and methods.
Expert
BC
11Q
MAGAIL
MIFQ (Det)
MIFQ (Soft)
SAC-1Q
0.8
0.4
0-
0.5M
1M
0.5M
IM
Protoss: 5vs5
Terran: 5vs5
Zerg: 5vs5
0
0.5M
1M
Protoss: 10vs 10
Terran: 10vs10
Zerg: 10vs10
0.8
0.4
0-
1M
0.5M
1M
0.5M
1M
Miner: easy
Miner: medium
Miner: hard
Table 4: Winning rate (percentage) of SAC-IQ compared with other baselines and our methods.
MIFQ
Task
Expert
MAGAHL
SAC-IQ
Det
Soft
5vs5 10vs10
86.7
42.62:4.6
64.844.11
72.7410.00
60.62332 64.4:66.2
Protoss
28.31:2.3
61.7445.8
77.34:5.6
Terran
5vs5
81.7
10.91:455
55.9110.4
71.9338
59.41:2.1
10vs10
81.7
1.0.0.77
53.81669
72.7:1344
53.7.6..
5us5
73.5
18.8:00.6
46.74:5.2
60.94:1477
55.616.2
Zorg
10vs10
76.3
31.2#3.6
51.0#:22
59.41:3.1
51.3=6.3
casy
82.4
34.62:3.3
52.710.77
60.24:5.6
35.0=5.33
Miner
medium
74.9
2..0.1..5
43.8.122
49.211..11
2..9
hard
20.9:2..4
39.14552
18.8
Table 5: Results in percentage on SMACv2 & Miner, the number of expert trajectories 128
IQ- VDN
MA- SQIL
MA- ATRL
MA- GATI.
MIFQ
Scenarios
Expert
Det
Soft
5ys5
86.7
17.2.147
13.3:1..6
8.612.66
0.8.1.44
0.8.0.66
0.0:0.00
18.0
41.4.146
Protoss
10vs10
6.2:44.4
2.341
3.133.8
0.0:00000
0.0:00.0
17225522
32.8:8..4
5vs5
81.7
944449
12.543.1
9.499.1
0.020.0
0.0:00.00
0.0:::.0
25.0=444
32.0::3.4
Teran
10vs/0
3.9.3.44
1.6.2.7
0.0:00000
0.0:0.00
0.0::0.0
0.0:0000
3.14:222
35.2:17.1
5vs5
73.5
7.0:14.6
10.2:1.56
4.2.14.2
3.9
2.0.1.22
3.1.3.88
5.512.6
24.2.14.1
Zery
10vs10
76.3
10.2:13.4
0.8:1144
0.0:10.0
0.0:0.00
2.14124
0.::0.00
10.9
18.9.19.2
easy
82.4
28.9.18.1
10:2::344
13.3::144
12.1:1200
14.114.77
10.5:13.7
14.1284
29.716..4
Miner
medium:
74.9
23.4::3.5
7.0+33.44
9,41388
11.54:355
13.3+14
10.0=244
113318
21.1+8.1
hard
69.8
15.6:1.5.8
5.5:1:14
7.0:1:14
7.41:666
4.742227
7.0:11.5
8.0:3322
14.8:14.1
19

---

Figure 6: Comparison with different numbers of demonstrations. X-axis: winning rate/final score. Y-axis: number demonstrations.
Gold Miner SMAC12
-EX
42
8
1
von
200
1999
2901
K2
200
900
922
522
212
590
1901
MPES
11Q
IQVDN
MASQIL
MAAIRL
MAGAIL
MIFQ (Det)
MIFQ (Soft)
Table 6: Results in percentage on SMACv2 & Miner, the number of expert trajectories is: 256
Scenarios
Expert
BC
11Q
1Q- VDN
MA-
MA- AIRL
MA- GAIL
MIFQ
Soft
5vu5
86.7
12.5.12.
16.5.244
1..1
1.611.6
72,32
14.114.7
40.611044
53.9.16.8
Protoss
10es10
90.3
6.2:2.9
3.9:2.66
29434
0.0:0.0
0.2:033
1500242
32.3=2.9
59.41.86
5vs5
81.7
10.944.7
12.542.2
1488-2.
08:114
0.0::000
0.0:0000
27.3-5.6
46.1475
Teran
10vs10
81.7
4.7=116
3.94-.11
0.0:0.00
0.0:0.00
0.0:::.00
0.0:0.00
0.0:0000
13.8
5vs5
73.5
8.6=2.6
11.72:66
7.86600
1.6=166
2.9:2222
6.222.2
12.5=388
35.945.2
Zorg
10us10
76.3
5.5.2:.6
2.3:1144
0.0:10.0
2.3.2..6
0.4:10.4
3.1
12.5=38
41.4134
easy
82.4
38.3:16.0
18.21:6.2
20.3:15.6
26.2.1288
14.81:466
19.9:12.5
26.622.44
34.41:444
Miner mediium
74.9
21.9-1108
109:164
10.2-26
15.0.14.
12553388
17.0
19.9-34
32.8.1.44
hard
69.8
20.3:11.6
6.24119
7.823.5
13.5:3.4
8.6:b4.1
10.4222
7.811.7
17.248.1
Table 7: Results in percentage SMACy2 & Miner, the number of expert trajectories is: 512
Scenarios
Expert
BC
I1Q
1Q- VDN
MA- SQIL
MA- ATRL
MA- GATI.
MIFQ
Det
Soft
5vs5
86.7
21.94666
17.24:166
19.5-8.1
0.0:0.00
4.411.4
1022-5.1
43.8-3,1
62,5-17777
Protoss
10vs10
90.3
5.5=2.6
7.0+22.66
9.1
0.0=0.00
0.0::0.0
18.0=7.1
35.91114
62.5:19.4
5vs5
81.7
9.444.9
18.0:3344
24.2=668
4.7.3..5
1.8:11.0
3.9:4.11
19.1
59.41186
Teman
10vs10
81.7
7.0:14.1
7.8:16.4
18.81 18.8
1.6:1
0.0.100.0
0.0:0..0
18.0
51.6.17.8
5ys5
73.5
6.2:15.8
10.9:::35
11.7=18.4
17.2::355
29:1:003
15.6:13.1
33.627.88
51.6:19.2
Zerg
10vs10
76.3
5.5.14.1
3.1.3..8
0.0:10..0
0.8 1.4
0.2.10.3
1.6.1..6
35.2.:.11
46.1
easy
82.4
30.5+5.11
13.9:22.9
1644660
25.8:4477
20.3::355
25.0=443
31.23311
53.944.1
Miner medium
74.9
18.8+3.11
14.3434
12.5-222
19.5455.6
117434
20,3-4.7
19.9=3.7
39.1
hard
69.8
15.6:15.8
6.11:0.9
5.54266
14.1422
7.0:2:66
12.5118
14.1=4.6
30.5:14.1
Table 8: Results in percentage on SMACv2 & Miner, the number of expert trajectories is: 1024
1Q- VDN
MIFQ
Scenarios
Expert
BC
11Q
SQIL
AIRL
GAIL
Det
Soft
5yu5
16.4:189
23.41:3.5
2..8::44
3.9.3.44
2444:2.5
43.8:16.6
5..2=5566
7..8:12.66
Protoss
10vx10
90.3
4.7.14.7
16411..1
29.2.1.33
0.0.10.0
3.9134
39166.8
53.1 1.8
71.11226
5ys5
81.7
12.5::544
27.34:344
33.6::1022
0.0:0.00
1.0::0666
0.8:114
52.3=266
60.9:16.88
10vs10
81.7
7.8.1.11
1..1....
13.31..4
2.3
0.0.10..
0.0:10.0
0.0:10.0
$8.6.12.66
Zerg
5us5
73.5
5.5224.6
9.4-12222
13.5
19.54114
4.4444
32.0- 10.5
38.3-5.11
58.6413.4
10vs10
76.3
9.4554
11.743
0.0:0.00
1.6=1.6
2.3:2.66
0.8:1.4
41.4-5.6
52.3-17.5
easy
82.4
34.4-18
17.84:2.3
15.6=5.4
35.9::42
22.74:344
31.1-30
36.9-3.1
$4.748..1
Miner mediuum
74.9
20.3:14.7
10.944.
22.1::24
42.2:11.6
hord
69.8
18.0-4:..6
9.0-12.3
8.6414.1
19.744..0
8.6.1.2.6
15.8
27.3-668
38.3-17.1
20

---

Table 9: Results in percentage SMACv2 & Miner, the number of expert trajectories is: 2048
Scenarios
Expert
BC
I1Q
1Q- VDN
MA- SQIL
MA- ATRL
MA- GAIL
MIFQ
Soft
5vu5
86.7
10.245.1
28.1+:4.9
28.1+10.4
39.1=1000
37.3444
56.2410066
65.6+1100
73.4+:522
Protoss
10vs10
90.3
2.3
16.41:34
25.0=599
43.8=10.6
9.4:11.5
39.8:6.44
45.3=166
78.145588
5vc5
81.7
10.2:16.0
28.1.:4.4
36.717.11
8.6.1.44
7.1.1.4
28.915.1
55.5.5.11
56.2.17.3
Teran
10vs10
81.7
5.5=2.6
24.24:2.6
43.8
0.0
0.0::0.0
0.0:0.00
46.9=888
09.5:77.1
5vs5
73.5
7.0=4.66
15.6+3388
26.3=5.3
28.1+33.8
39:11
33.6=7.5
48.1=800
$7.0=1022
Zong
10vs10
76.3
5,5=2..6
19.54334
10.4:3.6
25.8:88.1
1.
46.9=7.3
49.2=2.6
13.9
82.4
35.9+3..5
19.5+1.2
21.1=14
43.8422.2
26.61166
36.==7.1
50,0=4.9
64.8+6.0
Miner medium
74.9
19.5
9.8-44
17.2=3.5
30.5::4.6
211134
28.5=22
41.4=511
43.8+11.5
hard
69.8
10.2:43.
8.642.00
1022=2.
20.7+
14.1268
188==3.1
3988=8.1
45.344.77
Table 10: Results in percentage on SMACv2 & Miner, the number of expert trajectories is: 4096
IQ- VDN
MA- SQIL
MA- AIRL
MA- GAIL
MIFQ
Scenarios
Expert
BC
11Q
Det
Soft
5ys5
86.7
19.5+5.6
33.6#8.9
39,8=4,1
47.7422.
42.6+3399
42,6=4.6
64.8=4.1
10.7=10.0
Protoss
10vs10
90.3
5.5
16.6:00.3
38.328.9
3688228
19.812.9
28.3-233
61.7=588
77.3-15.6
Syu5
81.7
18.0.126
19.71:533
32.0.16.0
24.6.142
10.914.55
10.9:4.55
05.9.0.4
71.9.138
Teman
10vs10
81.7
6.2:222
35.924.7
0.5:0.5
2.31.44
1.0:0.77
53.8669
72.7134
5vs5
73.5
10.94:355
18.6 14
33.6=7.5
148813.1
5.3:122
188==066
46.7=5.2
60.9
Zorg
10vs10
76.3
94=388
16.4::0.6
17.2=3.5
27.1#
12:1.2
31.2=336
51.0=222
59.413.11
easy
82.4
31.2::5.8
21.91::44
188238
36.7::29
35.24::.66
34.6213.
52.7-0.7
60.2:15.66
Miner mediuu
74,9
28.1113.8
9.8.1.6
14.81.34
28.31119
21.1126
2..0.1..
43.8-22
49.2.15.1
hand
69.8
12.5:22
6.6:50.7
11.7-26
19.9220
172464
20.9-2.4
39.1
39.8::466
Table 11: Results MPEs, the number of expert trajectories is: [1, 4, 8, 16, 32, 64, 128)
Expert Traj.
Scenarios
Expert
BC
11Q
IQ- VDN
MA- SQIL
MA- AIRL
MA-
MIFQ
Det
Soft
reference
-17.2
-20.510.00
26.55.0.11
29.11:0.11
-3..5.0999
45.422
45.8003
428403
23.11:0.0
spread
-10.7
-21.710000
.0.31002
41.610.4
442.1011
28.91.06
311811.
30.41..1
3008.04
speaker
-28.540000 -19.7
-5..8.0.4
-37.70022
-138.94:04
1277420.1
-115.6211.6
-50550.3
-27.6:000
reference
-17.5
-24.1+002
-28440.11
.9.0+112
-9.7::0.1
-88620.11
43.420.1
44.6200
-22.4:0.11
spread
-11.4
-24.3::000
-3.331022
48.11:19
.588:0022
-3.66222
.252...1
27.5106
28.4404
speaker
.9.2
-27.6.0..0
43.7.0..4
56.4144
-119913.7
121.51199
107.6.0222
44.2.00
27.31000
reference
.17.5
.22.54:0.1
22622.044
.6.8:007
-2677000
48.2.0.3
-0.34402
45.240.2
-00000000
spread
-11.4
-23.8++0.0
-28.5+488
.4.221.11
46.9+0.22
-28.0=0.2
-3.5++0.2
25.90002
-3.11404
speaker
-1.2
-29.64:000
-67.3413
.10.0:1199
-11.1110666
13..00+.00
.2..5:12
125.41:22
29.910000
reference
-17.5
.19.0.1.11
25.610..1
15.99116
43.11006
46.51.
428111
13.1
20.20.11
spread
27.910.11
.9.33..22
.39.2-19
1588
53.11099
speaker
-19.2
-32.9+0.0
-3.9920.1
-3.6620.3
-111.10.2
133.0020.0
-9.4490.
59,9
-28.9-0.1
reference
-17.5
-18.2110.1
20.8
22,7710.2
.300.05
.7.822.1
45.410.
38,910.2
118810.11
16
spreud
-11.4
-21.3:10.0
-2421.02
38.74:1111
45.111022
-5.33.055
41.2110
31.0:133
-37.71066
speaker
.31.9.0.00
56.5.0.11
.3.22.0.5
.199110.
134.001000
134.0.000
27.910.00
reference
11..5
-18.3::000
-18.71022
21.740
46.010.2
3.041.5
30,7411
32,240.1
1864001
32
spread
11.44
20.6.10.0
17.1
34.710..
34910..
.2.110.7
3381004
248800
549104
speaker
-19.2
-29.0:10.0
-32.8002
-3..9.0.11
-1444:02
-132.5221
11488004
-6.88:1..0
28.
reference
-17.5
-18.1+00.0
-1,
18.9+0022
-36.4..77
-31.31.00
-2.6+002
-24.620.1
-0.4+0000
64
spread
-11.4
-20.3::0.0
-2.00:0.1
23.8::022
-2999:022
-2.4=000
-22740.1
23.5502
-3.74:222
speaker
.19.2
.28.4.10.0
30.3
.9.8..1.11
69.2.177
124.6.10.3
117.7.0.11
3344.00
26.7.000
reference
-17.2
.18.340..0
-18661022
19.04..11
36.6.155
40.713.2
40.040.2
23.410.11
20.540000
128
spread
-10.7
-20.5+00.0
-3,6+002
-2166202
-3.00:4
-265=033
-24.3+0.11
-3.220..1
-24240.1
speaker
-19.7
-27.5++0.0
-29.1.0.11
22.55:055
-10.33550
125.5500.7
-78.7449
-3.8-00.1
-66.34:000
21

---

Figure Learning curves with different numbers of demonstrations.
Expert
MASQIL
MAATRL
MAGAIL
(Det)
(Soft)
Protoss:
5xs3
0.3H
155
0.5H
0.SH
0.5H
O.SM
0.5H
Terran:
50s5
O.SH
0.SH
Zerg:
5v85
0.4
Protoss:
10vs10
0.SH
Terran:
10vs
0.5H
1M
0.5H
0.SH
0.5H
O.SM
3M
O.SH
Zerg:
10vs
0.5M
0.5M
0.3M
0.5M
Miner:
easy
Miner;
medium
LH
LH
Miner:
hand
0.5H
1M
0.5H
1M
0.5M
0.5M
0.5M
1/4
DLSM
128
256
512
1024
2048
4096
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Our abstract includes main claims reflecting main contributions and finding.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We created a subsection the Conclusion to discuss limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

All the proofs of the theorems and propositions stated in the main paper are provided in the appendix with clear references.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide details on the environments and hyper-parameter settings in the appendix. We also uploaded our source code for re-productivity purposes.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The data we used, along with our source code, has been uploaded with the main puper. We have also provided sufficient instructions for their use.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppscccppbllccguuudoooo Coaee//ddeeSSbbbmmma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the details are provided in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Following standard practices, we reported mean scores and standard devia- tions, computed by running several random seeds, for comparison.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provides all the details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper develops general imitation learning algorithm for multi-agent games, which we have tested only in simulated environments. As such, we do not foresee any direct societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We have provided clear citations the source code and data used in the paper.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our source code is submitted alongside the paper, accompanied by sufficient instructions. We will share the code publicly for re-producibility or benchmarking purposes.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "isfjEH7dis", "reviewer": "Reviewer_GGqd", "summary": "The paper addresses the imitation problem in cooperative Multi-Agent Reinforcement Learning (MARL). It extends inverse soft-Q learning to the multi-agent domain by leveraging value factorizations under the Centralized Training with Decentralized Execution (CTDE) paradigm. Experimental results demonstrate the effectiveness of the proposed approach across several environments.", "questions": "1.\tIs BC trained online, given that it shows learning curves with environment steps? If so, why not use DAGGER?\n2.\tCould the authors explain why MIFQ significantly outperforms IQVND? Is it solely due to the factorization structure?\n3.\tWhy does the paper state that QPLEX is unsuitable for the proposed method? QPLEX also has $\\partial Q/\\partial Q_i>0$.", "limitations": "Limitations are discussed in conclusion section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The study of imitation learning in MARL is a valuable and relevant research problem, and the paper provides promising solutions.\n- The experimental results are robust and convincingly support the proposed method's effectiveness.", "weaknesses": "- The paper's organization could be improved. The current structure alternates between theory and architecture without a clear flow.\n\n- The similarity between IGC and IGO[1] requires further clarification.\n\n- The objective function (6) introduces sub-optimality compared to the original objective (3) due to the restriction that $Q^{tot}$ and $V^{tot}$ must be monotonic. Additionally, since $Q^{tot}$ and $V^{tot}$ use different mixing networks, the relationship between them violates Equation (2). This indicates that Equation (6) does not represent the same objective as Equation (3), even without considering the sub-optimality introduced by factorization. These issues need further theoretical exploration and discussion.\n\n- Although the experimental results are promising, the superior performance seems to stem from the QMIX algorithm's advantage over other MARL algorithms. An important missing baseline is the soft actor-critic version of IQ-Learn, which uses a centralized Q function with decentralized critics and does not seem to violate the original objective.\n\n[1] Zhang, et al., FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning, ICML 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our paper and the insightful comments and questions. \n\n> The paper's organization could be improved.\n\nThank you for the feedback! We will revise our writing and improve our exposition.\n\n> The similarity between IGC and IGO[1] requires further clarification. \n\nThank you for mentioning this. IGO and our IGC are indeed similar. However, IGC is more specific in that it requires the local policies, obtained by solving the local objective functions, to be equivalent to the global policy obtained by the global objective function. \n\n> The objective function (6) introduces sub-optimality compared to the original objective (3) due to the restriction that  and  must be monotonic. Additionally, since  and  use different mixing networks, the relationship between them violates Equation (2). This indicates that Equation (6) does not represent the same objective as Equation (3), even without considering the sub-optimality introduced by factorization. These issues need further theoretical exploration and discussion. \n\nThank you for the insightful comments. We agree that the factorized Q-learning objective violates Eq. (3). The main reason we follow this approach is that the relationship between $V$ and $Q$ cannot simultaneously hold at both the global and local levels, i.e., (2) and (5) cannot hold simultaneously (we discussed this in Section B2 in appendix). On the other hand, maintaining the relationship as in (2) is impractical because it requires computing $V^{tot}$ via a global Q-function and global policy $\\Pi$. Therefore, we choose to keep (5) valid and build our factorization approach based on this (Computing local V functions via (5) , and compute $Q^{tot}$ and $V^{tot}$ by the mixing networks is indeed less challenging and more practical)  \n\nFurthermore, since (2) holds, each individual objective helps match the individual learning policy with the corresponding individual expert agent. Since IGC holds, our training can ensure global convergence and consistency across all agents. \n\n> Although the experimental results are promising, the superior performance seems to stem from the QMIX algorithm's advantage over other MARL algorithms. An important missing baseline is the soft actor-critic version of IQ-Learn, which uses a centralized Q function with decentralized critics and does not seem to violate the original objective. \n\nThank you for the comment and suggestion. There are two reasons we did not extend the soft actor-critic version of IQ-Learn to our multi-agent setting. First, as mentioned, it requires computing the global V function via the global Q function and global $\\Pi$, which is impractical for multi-agent settings. Second, soft-actor-critic (SAC) methods only work well for continuous-action-space environments. All the environments we considered (following prior SOTA MARL papers) have discrete action spaces, making direct Q-learning algorithms more suitable. To support this argument, we have conducted an additional experiment, detailed in the attached 1-page PDF, where we compare a SAC IQ-learn adapted to multi-agent tasks. The results generally show that SAC-IQ performs worse than our algorithm, MIFQ. We will include these results in the paper.\n\n>Is BC trained online, given that it shows learning curves with environment steps? If so, why not use DAGGER? \n\nOur BC was trained offline. The learning curves actually reflect our evaluations after certain training steps. We will clarify this in our paper. \n\n>Could the authors explain why MIFQ significantly outperforms IQVND? Is it solely due to the factorization structure? \n\nYes, IQVDN is simply a linear combination of local functions, while MIFQ leverages our two-layer mixing networks with learnable parameters. Previous work has also shown that QMIX generally outperforms VDN for this same reason. \n\n>Why does the paper state that QPLEX is unsuitable for the proposed method? QPLEX also has . \n\n The monotonicity of the Q function is simply a corollary of our mixing structure and is not a key target when constructing our learning objective. Additionally, while QPLEX utilizes the advantage function (A = Q - V), our objective is different and such an advantage function is unsuitable to use. We will elaborate more on this point in the updated paper. \n\n*We hope that the above responses address your concerns. If you have any other comments or concerns, we are more than happy to address them.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oDWjapxaMW", "reviewer": "Reviewer_vZKY", "summary": "This paper presents a novel algorithm, Multi-agent Inverse Factorized Q-learning (MIFQ), for cooperative multi-agent imitation learning (IL). It extends the inverse soft-Q learning framework to multi-agent settings by introducing a mixing network architecture for centralized training with decentralized execution. This enables learning local and joint value functions effectively. The authors conducted extensive experiments across multiple challenging environments, demonstrating that their approach outperforms existing methods.", "questions": "- In Figure 2, the semi-transparent curves are not standardly explained. If these do not represent standard deviations, what statistical measure do they depict?\n- Minor Error: On Line 62, the term \"generation\" is used where \"generalization\" might be intended. Could the authors clarify or correct this in the context?", "limitations": "Yes", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-  The introduction of a multi-agent extension of inverse soft-Q learning using factorized networks is a significant and novel contribution to the field of IL.  \n- This paper is well-written and organized, and provides a sound theoretical analysis.\n- The empirical results across three different environments, including a complex version of the StarCraft multi-agent challenge, are impressive. The proposed method outperforms existing baselines.", "weaknesses": "As someone who is not an expert in the field of imitation learning, I perceive no significant weaknesses in this paper from my perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our paper and for the positive feedback.\n> In Figure 2, the semi-transparent curves are not standardly explained. If these do not represent standard deviations, what statistical measure do they depict? \n\nThey present standard deviations. We will clarify this in the paper. \n> Minor Error: On Line 62, the term \"generation\" is used where \"generalization\" might be intended. Could the authors clarify or correct this in the context? \n\nThis is “generalization” and it is a typo. We will correct this. \n\n*We hope that the above responses address your concerns. If you have any other comments or concerns, we are more than happy to address them.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nZDCtmISHx", "reviewer": "Reviewer_v3qG", "summary": "This paper addresses the problem of extending a single-agent imitation learning algorithm, inverse soft-Q learning (IQ-learn, Garg et al. Neurips 21) to the multi-agent cooperative setting. The proposed algorithm, MIFQ, leverages the ideas of mixing networks and the individual-global-max (IGM) principle, to perform the extension. Experimental evaluations of MIFQ are conducted on SMAC-v2, MPE, and Gold Miner, and demonstrate that MIFQ improves over baselines across various domains and with varying numbers of demonstrations.", "questions": "1. Questions about experiments: \n    - What are some reasons why MIFQ does not achieve expert level performance? While the other methods also do not achieve expert level performance, the original IQ learn algorithm does have this ability. \n    - How does the method perform with demonstrations not sourced from MAPPO (an algorithm that learns gaussian policies)? For example, demonstrations sourced from QMIX, which learns 'hard max' policies? \n    - Why does the method need an order of magnitude more demonstrations than IQ Learn needs on complex single-agent tasks? \n   \n2. Method: \n    - Why is it necessary to maintain Q and V networks separately? Why not derive the global V function by computing the softmax of the Q functions as described in line 163-164? \n    - Why is it necessary to compute Q^tot via Q^tot = -M (-Q)? What is the purpose of the double negation? The stated justification is that this enable the method \"to achieve the IGC principle and the convexity\", but why exactly is this? Requiring the networks to be multi-layer feedforward w/nonnegative weights and convex activation functions (lines 194-195) is enough to ensure that Q^tot is monotonic w.r.t. the local Q functions, thus ensuring the IGC principle and convexity. \n    - Would major changes be necessary to enable this algorithm to operate on continuous action spaces? Did the authors consider continuous action space settings?", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper addresses the challenge of generalizing a key imitation learning (IL) algorithm from single-agent to multi-agent settings, offering a novel approach with MIFQ. The problem is clearly specified and represents an important contribution to the MARL literature. \n\nThe empirical results are robust:\n- MIFQ outperforms most baselines with various demonstrations.\n- Extensive experiments across multiple domains and tasks confirm MIFQ's superior performance.\n- Comprehensive comparisons with baselines (BC, independent IQ learning, alternative Q-factorization methods, etc.) highlight MIFQ's advantages.", "weaknesses": "1. Some aspects of the method do not seem fully justified to me: \n    - The authors claim in lines 143-148 that a shortcoming of the IQ learn method is that the objective depends on the centralized state and joint action. However, Section 5.4 of the IQ Learn paper presents a state-only objective (independent from the actions). I wonder if the authors could discuss whether a simple state-only extension of IQ Learn, where critic depends on the centralized state as usual, but the actor depends on the observations, would be sufficient to sidestep many of the concerns addressed by IQ Learn? \n    - The authors also claim in Section 4.1.2 that the straightforward Independent Inverse Q-learning is not a satisfactory solution because the method \"…has limitations in addressing the interdependence between agents and the global information available during the training process.\". Can the authors more explicitly discuss what the shortcomings of an independent version of IQ-learn is not satisfactory? Does it suffer from convergence problems? \n\t\t\t\t\t\n2. The current experimental analysis is somewhat shallow, and essentially amounts to a description of the plots. The authors could improve the analysis of MIFQ by considering the following additional questions: \n    - The original IQ learn paper plots the rewards to validate that their method recovers the ground truth reward. Can the same be done here? \n    - Why does MIFQ perform worse than BC on MPE, particularly the reference and spread tasks?\n3. There are some issues with how the experimental results have been reported. \n    - What is the number of trials for each of the results? Please include this in the main paper. \n    - The caption of Figure 2 is missing key information to understand the figure. What is the number of demonstrations used to train each of the methods? What does the shaded region mean? Based on the std devs reported in the Appendix, I assume it is the standard deviation; please see the note below and instead compute 95% confidence intervals. \n    - No measurements of uncertainty are provided in Table 2, and standard deviations are provided only in the Appendix. Standard deviations reflect the underlying variance in models learned by the algorithm, rather than providing a measure of statistical significance. Please also compute 95% confidence intervals to enable readers to judge the statistical significance of the gaps in mean test returns -- ideally, bootstrapped confidence intervals. See this paper for a reference on best practices:  https://arxiv.org/abs/2304.01315\n3. There are also some minor clarity issues: \n    - IGC is used in line 192, but is only explained in the following Section 4.2.2\n    - Definition 4.2 - this definition is not specific enough to be useful. It handwaves  by only requiring that the joint policy be 'equivalent' to the collection of individual optimal policies. Equivalent in what sense?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for carefully reading our paper and providing us with valuable questions and suggestions.\n\n>  I wonder if the authors could discuss whether a simple state-only extension of IQ Learn ...\n\nOur argument in lines 143-148 simply means that directly using the global Q, V, and global state would be impractical in multi-agent settings. This is not a limitation of IQ-Learn but a well-known challenge when extending single-agent models to multi-agent settings. This is also why the centralized training decentralized execution (CTDE) approach has become appealing for MARL. The state-only approach in Section 5.4 of the IQ-Learn paper is only useful when actions are not available. Applying this in our context is unsuitable because action observations are available. \n\n> Can the authors more explicitly discuss what the shortcomings of an independent version of IQ-learn ...\n\n If we learn the local Q independently by solving (4) for each agent, it implies that we neglect the interactions between agents. This approach ensures convergence to individual local policies but does not maintain consistency between local and global policies, as required by well-known principles such as IGO and IGM, which are necessary for a successful MARL algorithm. \n\n>The original IQ learn paper plots the rewards to validate that their method recovers the ground truth reward. Can the same be done here?  \n \nVisualizing rewards in multi-agent settings is much more challenging compared to the single-agent setting due to the vast joint state and action space. So far, we are unsure how to obtain meaningful visualizations for rewards in multi-agent tasks. Therefore, we will keep this for future investigation.\n\n>Why does MIFQ perform worse than BC on MPE, particularly the reference and spread tasks? \n\nAs mentioned in the paper, MPEs are deterministic environments (i.e., no dynamics), and BC typically performs well on such deterministic tasks.\n\n>What is the number of trials for each of the results?\n\nWe briefly mentioned these numbers in Table 2 of the appendix. Each number in Table 1 is computed based on 4 seeds and 32 evaluation runs per seed. We will add this information to the main paper.\n\n>What is the number of demonstrations used to train each of the methods? What does the shaded region mean?  ... \n\nThe number of trajectories is 128 for MPEs and 4096 for Miner and SMAC-v2. We will clarify this in the caption of Figure 2. Additionally, the reviewer's point regarding the 95% confidence interval is well taken. We will compute these and update the paper accordingly.\n\n>IGC is used in line 192, but is only explained in the following Section 4.2.2. Definition 4.2 - this definition is not specific enough to be useful. ...\n\nWe appreciate the reviewer for pointing these out. We will remove the mention of IGC in line 192. In Definition 4.2, equivalence means that the joint policy is equal to the product of local policies. We will clarify this.\n\n> What are some reasons why MIFQ does not achieve expert level performance?  ...\n\nThis was stated as a limitation of our approach (and other existing multi-agent IL algorithms as well). Multi-agent tasks are much more complex, making it difficult to recover the expert policy. Increasing the amount of expert demonstrations might help, but it also leads to an excessively large replay buffer, causing out-of-memory issues. Addressing this limitation will require further efforts, which we plan to pursue in future work. \n\n>How does the method perform with demonstrations not sourced from MAPPO ? ...   \n\n In our context, MAPPO achieves the best policy in MARL, so we use it as an expert. The main reason is that it is not reasonable to use a sub-optimal policy as an expert for imitation, as a sub-optimal solution to the imitation learning problem could yield better rewards than the expert, thus biasing the evaluation. \n\n>Why does the method need an order of magnitude more demonstrations than IQ Learn needs on complex single-agent tasks? \n\nThe main reason is that multi-agent tasks are much more complex than single-agent tasks, with much larger action and state spaces.  Therefore, much more data is needed to understand the environment, requiring significantly more demonstrations for the imitation learning.\n\n> Why is it necessary to maintain Q and V networks separately? Why not derive the global V function by computing the softmax of the Q functions as described in line 163-164?  \n\nWe have discussed this in Section B.4 of the appendix. The main reason for our approach is to make the algorithm practical. Directly computing $V$ through the global $Q$ is generally impractical because it requires sampling over a joint action space, which is exponentially large. We actually attempted this approach, but it did not work at all—the algorithm couldn't learn anything, and the win rates were always zero. Therefore, we did not include this approach in the comparison. In our approach, we compute the global $V$ using local $Q$ values (which only require sampling over the local action space, making it much more feasible) and then aggregate the global $V$ using the mixing network. \n\n> Why is it necessary to compute Q^tot via Q^tot = -M (-Q)? \n\nThe main reason for this double negation is not only for the monotonicity, as $Q^{tot} = M(Q)$ is sufficient for that purpose. We use this approach to ensure that the global objective function is concave in $Q$ (Theorem 4.5), \n\n>Did the authors consider continuous action space settings? \n\nSo far, our algorithm is generally not suitable for continuous action spaces. However, all the environments under consideration have discrete action spaces and are taken from prior SOTA MARL works. Extending the approach to continuous action spaces would require further  investigation. We plan to explore this in future work.\n\n*We hope that the above responses address your concerns. If you have any other comments or concerns, we are more than happy to address them*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E0xMJaXQkc", "reviewer": "Reviewer_9wvM", "summary": "This paper extends the IQ-Learn method to cooperative multi-agent settings. The main insight is to use mixing networks to enable centralized training via decentralized Q functions.", "questions": "- How do the agent have access to the global state information. If this is the case, why does the paper even define observations? Is the global state information available only in training or after deployment, too? In what settings is this applicable?\n- How could one adapt this algorithm for non-cooperative settings? Is there a straightforward way or does it require completely new approaches?", "limitations": "The paper does not discuss the broader impacts. I disagree that there is no potential societal impact. I invite the authors to think about the applications their algorithm may have and then consider how their algorithm would affect those applications (both positively and negatively).", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is quite relevant to NeurIPS and it is indeed important to extend IQ-Learn (or similar inverse learning algorithms) to multi-agent systems.", "weaknesses": "- The major concern that I have is that, if my understanding is correct, the paper assumes access to the global state information. This is not realistic. In real application, this will never be the case. So the algorithm does not seem useful in practice.\n- Typo: In line 62, it should be \"generalization\" instead of \"generation\",\n- In line 72, \\citet should be used instead of \\cite or \\citep so that the author names will become a part of the sentence.\n- In line 162, \\eqref should be used instead of \\ref so that the parenthesis will appear around the equation number.\n- The architecture figure is in page 7. It would significantly increase the readability if it came earlier. \n- By the time the reader reads line 191, the IGC principle is still undefined. This makes reading very difficult.\n- The same thing is true at line 203, too.\n- Typo: In line 241, it should be \"makes\" instead of \"make\".\n- Typo: In line 242, it should be \"yields\" instead of \"yield\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> How do the agent have access to the global state information. If this is the case, why does the paper even define observations? Is the global state information available only in training or after deployment, too? In what settings is this applicable? \n\nThank you for the question! We would like to clarify that our model does *not* assume access to global state information of the entire environment. As briefly described in Section 4.2.1, each agent in our MARL setting only  has local observations of other agents (enemies or allies) in the agent’s neighborhood. *The state information notion S in our MIFQ model is simply a combination of these local observations of our agents.* We will make this distinction clearer in the revised version of the paper. We note that this setting of local observations is standard in many previous MARL studies. Such local observations are available in both training and deployment and, we believe, are highly realistic in practical applications. \n\n>How could one adapt this algorithm for non-cooperative settings? Is there a straightforward way or does it require completely new approaches? \n\nWe thank the reviewer for the question. Theoretically, our approach can be applied to non-cooperative settings. However, in practice, it would require a completely new algorithm. With conflicting rewards, the non-cooperative setting is much more challenging to train compared to the cooperative setting. We will definitely explore this in future work.\n \n We also thank the reviewer for pointing out the typos, which we highly appreciate and will correct.\n\n*We hope that the above responses address your concerns. If you have any other comments or concerns, we are more than happy to address them.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for carefully reading our paper and providing constructive feedback and questions, which we have been happy to consider and clarify. Please find a summary of our responses below.

**Reviewer GGqd** raised a concern about the fact that equation (2) does not hold under our mixing architecture. In response, we have clarified that the relationship between V and Q cannot be satisfied at both local and global levels simultaneously. Therefore, we chose to keep the equation valid at the local level, making our algorithm practical. In contrast, maintaining the V-Q equation at the global level (Eq. 2) would require sampling over the joint action space, which is impractical, especially for large-scale tasks such as SMAC_v2.

The reviewer also mentioned a soft-actor-critic (SAC) IQ-learn as a missing baseline. In response, we argued that such an SAC algorithm is neither suitable nor practical in our multi-agent setting. To support our arguments, we have conducted an additional experiment, detailed in the attached 1-page PDF, where we compare a SAC IQ-learn adapted to multi-agent tasks. The results generally show that SAC-IQ performs worse than our algorithm, MIFQ. We will include these results in the paper.

We have also provided detailed responses to other questions regarding why DAGGER is not used, why MIFQ outperforms IQVDN, and why QPLEX is not suitable. We will update our paper to clarify these points.

**Reviewer vZKY** has a clarification question about the curves in Figure 2 and pointed out a typo. We have provided a response to address this.

**Reviewer v3qG** raised several questions and requested clarification on the following points: (i) whether the state-only approach in the IQ-learn paper is applicable, (ii) why independent IQ-learn is limited, (iii) why MIFQ cannot achieve expert performance as in the single-agent IQ-learn setting, (iv) why MIFQ performs worse than BC on MPE, (v) how the method performs with demonstrations not sourced from MAPPO, (vi) why our method needs more demonstrations than single-agent IQ Learn, (vii) why it is necessary to maintain Q and V networks separately, and (viii) whether the authors considered continuous action space settings. In response, we have provided detailed answers to each question.

The reviewer also requested clarification regarding the number of trials for our results, the number of demonstrations used in the experiments, and the standard deviations reported in Figure 2. We will clarify these points in the paper 

**Reviewer 9wvM** raised a concern about the use of global state information in our training algorithm, which makes it impractical. In response, we clarified that we only assume access to local observations of neighboring  agents. These local observations are available in both training and deployment and are highly realistic in practical applications. The use of such information is also standard in previous multi-agent reinforcement learning algorithms. We will clarify this point in the updated paper.

The reviewer also asked whether our algorithm can be applied to non-cooperative settings. In response, we believe that the non-cooperative setting would be much more challenging and would require a new MARL algorithm, which we will explore in future work.The reviewer also pointed out some typos, which we highly appreciate and will correct.

*We thank all the reviewers for their comments and feedback, which we have tried to address and clarify. If you have any further questions, we are more than happy to discuss and clarify them.*

Meta-review: This paper presents a novel algorithm called multi-agent inverse factored Q-learning for cooperative multi-agent imitation learning. It extends inverse soft Q-learning from single-agent to multi-agent settings by a mixing network under the CTDE paradigm. Experimental evaluation across several environments demonstrates the effectiveness of the proposed algorithm. The proposed algorithm is a significant and contribution to the imitation learning field. The reviewers are largely satisfied with the author's rebuttal. I trust the authors will further improve the paper's organization and address clarity issues in the camera-ready version.
Decision: Accept (poster)