Inverse Factorized Soft Q-Learning for Cooperative Multi-agent Imitation Learning

Authors: viet  the
  bui
tien  mai
hong  thanh
  nguyen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:17:34

Abstract: This paper concerns imitation learning (IL) in cooperative multi-agent systems.
The learning problem under consideration poses several challenges, characterized by high-dimensional state and action spaces and intricate inter-agent dependencies. In a single-agent setting, IL was shown to be done efficiently via an inverse soft-Q learning process. However, extending this framework to a multi-agent context introduces the need to simultaneously learn both local value functions to capture local observations and individual actions, and a joint value function for exploiting centralized learning.
In this work, we introduce a new multi-agent IL algorithm designed to address these challenges. Our approach enables the
centralized learning by leveraging mixing networks to aggregate  decentralized Q functions.
We further establish conditions for the mixing networks under which the multi-agent IL objective function exhibits convexity within the Q function space.
We present  extensive experiments conducted on some challenging multi-agent game environments, including an advanced version of the Star-Craft multi-agent challenge (SMACv2), which demonstrates the effectiveness of our algorithm.

Main paper: 
 introduction: 
 Imitation learning (IL) is a powerful approach for sequential decision making in complex environments in which agents can learn desirable behavior by imitating an expert. There are applications of IL in several real-world domains, including healthcare [40, 34] and autonomous driving [43, 17, 23] . A rich body of IL literature focuses on simple single-agent settings [1, 18, 13, 44, 32, 14] while many recent works develop new multi-agent IL methods that are tailored to either cooperative or noncooperative settings [36, 42, 22, 4, 19, 37] . Leading methods in [36, 42] explore solution concepts for Markov games such as Nash equilibria. Findings on underlying properties of these solution concepts are integrated to extend the single-agent IL models to multi-agent settings. While these methods show promising results, they face difficulties in training, as the underlying adversarial optimization process involves biased and high variance gradient estimators, leading to a highly unstable learning process.Our work studies IL in cooperative multi-agent settings. We leverage recent advanced findings in both single-agent IL and multi-agent reinforcement learning (MARL) to build a unified multi-agent IL algorithm, named Multi-agent Inverse Factorized Q-learning (MIFQ). MIFQ is built upon inverse soft Q-learning (IQ-Learn) [14] , a leading single-agent IL method of which advantage is to learn a single Q-function that implicitly defines both reward and policy functions, thus avoiding adversarial training. To adapt this idea for multi-agent settings, following the well-known paradigm of centralised training with decentralised execution (CTDE) in MARL [26, 21] , we develop a centralized learning approach based on the concepts of mixing and hyper-networks [16] . This approach facilitates the integration of individual state-value and reward functions into a unified learning objective, enabling the training of decentralized imitation policies for agents in a centralized manner.In this work, we formulate the IL problem as a multi-agent inverse soft-Q learning task, where the objective is to match the occupancy distribution of the joint learning policy with that of the expert. To make the learning process practical and facilitate the CTDE paradigm, we propose factorizing both the global Q and V functions using mixing networks with non-negative weights and convex activation functions (such as ReLU and ELU). We further examine the Individual-Global-Max (IGM) principle [29, 35] , a core concept in factorization learning, which suggests that the optimal joint actions across agents are equivalent to the collection of individual optimal actions for each agent. However, we argue that this principle does not apply for our problem setting since our objective is to recover soft policies based on max-entropy reinforcement learning. Therefore, we introduce a generalized version of IGM, called Individual-Global-Consistency (IGC), which requires that the distribution of joint actions produced by the optimal joint policy matches the collection of local action distributions produced by the individual optimal policies. We then demonstrate that IGC holds in our approach.A key advantage of single-agent inverse soft Q-learning (IQ-Learn) is that the original training problem, which is a max-min optimization, can be conveniently converted into a (non-adversarial) concave maximization problem [14] . This conversion ensures that the optimization objective is well-behaved, contributing to the effectiveness of the original IQ-Learn. Interestingly, we provide theoretical results showing that under our factorization approach, the multi-agent training objective can also be converted into a (non-adversarial) concave maximization problem. This advantage holds with any multi-layer feed-forward mixing networks with non-negative weights and convex activations.Finally, we conduct extensive experiments in three domains: SMACv2 [9] , Gold Miner [12] , and MPE (Multi Particle Environments) [25] . We show that our MIFQ outperforms other baselines in all these environments. To the best of our knowledge, our experiments with SMACv2 mark the first time IL algorithms are employed and evaluated on such a challenging multi-agent environment.Concretely, we make the following main contribution: (i) We introduce a new multi-agent IL method based on inverse soft-Q learning and factorization.(ii) We show that our approach satisfies the IGC, a generation of the IGM principle.(iii) We show that the max-min learning objective can be converted into a concave maximization problem, under any mixing networks of non-negative weights and convex activations, which helps avoid adversarial training and ensure well-behaved learning within the Q-space.(iv) We empirically show state-of-art results on several challenging multi-agent game tasks, including an advanced version of the Star-Craft multi-agent challenge (SMACv2). 
related work: 
 Single-Agent Imitation Learning. There is a rich body of existing works focusing on generating policies that mimic an expert's behavior given data of expert demonstrations in single-agent settings. A classic approach is behavioral cloning which casts IL as a supervised learning problem, attempting to maximize the likelihood of the expert's trajectories [32, 28] . This approach, while simple, requires a large amount of data to work well due to its compounding error issue. An alternative approach is to recover the reward function (either implicitly or explicitly) for which the expert's policy is optimal [13, 18, 30, 20, 10] . Leading methods [18, 13] follow an adversarial optimization process (which is similar to GAN [15] ) to train the imitation policy and reward function alternatively. These adversarial training-based methods, however, suffer instability. A more recent work by [14] overcomes this instability issue by introducing an inverse soft-Q learning process. For a comprehensive review of literature on this topic, we refer readers to the survey article in [2] .Multi-Agent Imitation Learning. Most of single-agent IL works, however, do not apply directly to multi-agent settings. Literature on multi-agent IL is rather limited. A few works study multi-agent IL either in cooperative environments [3, 22, 37, 5] or competitive environments [24, 31, 36, 42] . Recent leading methods employ equilibrium solution concepts in Markov games to extend some existing single-agent IL methods to the multi-agent settings [36, 42] . However, these methods still suffer the instability challenge during training as they still rely on adversarial training. Our work focuses on multi-agent IL in a cooperative Dec-POMDP environment. We utilize the idea of inverse soft-Q learning in single-agent IL [14] to avoid adversarial training. We extend the idea to the multi-agent settings under the paradigm of centralized training decentralized execution (CTDE) from MARL, allowing an efficient and stable learning process.MARL. The literature of MARL encompasses a number of advanced methods; many follow the well-known CTDE paradigm. In [25, 11] , they employ actor-critic architectures and train a centralized critic that utilizes global information. Value-decomposition (VD) methods represent the joint Qfunction as a function of agents' local Q-functions [38, 29] . QMIX [29] offers a more advanced VD method for consolidating agents' individual local Q-functions through the utilization of mixing and hyper-network [16] concepts. Later value function factorization approaches, such as QTRAN [35] and QPLEX [39] , introduce new factorization methods for MARL based on the IGM principle, which necessitates consistency between joint and local action selections. There are also policy gradient based MARL algorithms. Independent PPO (IPPO), a decentralized MARL, can achieve high success rates in several challenging SMAC maps [7] . MAPPO, a PPO version for MARL, achieves SOTA results on several tasks [41] . Our work utilizes MAPPO to train our expert and generate expert demonstrations.We also employ a hyper-network architecture [16] with mixing networks of non-negative weights and convex activations to facilitate our CTDE. While our mixing network techniques are similar to those used in QMIX [29] , we show that this configuration offers several unique advantages in the context of inverse Q-learning. Specifically, the training objective can be transformed from a max-min problem into a (non-adversarial) concave maximization problem, and it upholds the IGC principle.Other methods, such as QTRAN and QPLEX [35, 39] , are primarily based on the IGM principle and are designed for different objective structures, making them unsuitable for our context. 
preliminaries: 
 A multi-agent cooperative system can be described as a decentralized partially observable Markov decision process (Dec-POMDP), defined by a tuple {S, O, N , A, P, R} [27] , where S is the set of global states, O is the set of local observations of agents, and N is all the agents. In addition, A = i∈N A A i is the set of joint actions of all the agents, A i is the set of actions of an agent i ∈ N , and P is the transition dynamics of the multi-agent environment. Finally, in Cooperative MARL (Coop-MARL), all agents share the same reward function, R, that can take inputs as global states and actions of all the agents and return the corresponding rewards.At each step, given a global state S, each ally agent i takes an action a i ∈ A i based on his policy π i (a i | o i ), where o i is his local observation. The joint action is defined as A = {a i | i ∈ N } and the joint policy is defined accordingly:Π(A | S) = i∈N π i (a i | o i ).After all the agent actions are executed, the global state is updated to a new state S ′ ∈ S with the transition probability P (S ′ | A, S). The objective of Coop-MARL is to find a joint policyΠ(• | S) = i π i (• | o i ) thatmaximizes the expected long-term joint reward, formulated as follows:max Π E Π ∞ t=0 γ t R(A t , S t ) 4 Multi-agent Inverse Q-LearningIn IL, the objective is to recover an expert reward or expert policy from some expert demonstration data. Our new multi-agent IL algorithm is built upon an integration of recent advanced findings in single-agent IL (i.e., the new leading IQ-Learn [14] ) and in cooperative MARL. In the following, we first present the main idea of IQ-Learn, with some direct centralized and decentralized adaptations to multi-agent settings. We then discuss key shortcomings of such simple adaptations in the context of a Dec-POMDP. Finally, we present our new algorithm which tackles all those shortcomings.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The data we used, along with our source code, has been uploaded with the main paper. We have also provided sufficient instructions for their use.Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We provides all the details in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
inverse soft: 
 Q-Learning 4.1.1 Centralized Inverse Q-LearningIn general, IQ-Learn can be directly adapted for IL in a fully-observable cooperative multi-agent setting. Given expert samples D E = {τ = {(A t , S t ), t = 0, 1...}}, IQ-Learn aims to solve the following maximin problem, which is also the objective of adversarial learning-based IL approaches [18, 13] :EQUATIONwhere ρ Π (S, A) is the occupancy measure of visiting state S and joint action A, under the policy Π:ρ Π (S, A) = (1 -γ)Π(A | S) ∞ t=0 γ t P (S t = S | Π),ρ E is the occupancy measure of the expert policy π E and E ρ Π ln Π(S, A) is the entropy regularizer.A typical method to solve (1) is to run an adversarial optimization process over rewards and policies, of which idea is similar to generative adversarial networks [18, 13] . However, such an approach suffers instability, a well-known challenge of adversarial training. Thus, IQ-Learn avoids adversarial training by learning a single soft Q-function, as defined in the following. Definition 4.1 (Soft Q-function [14] ). Given a policy Π, the soft Bellman operator B Π is defined as:EQUATIONB Π is contractive and defines a unique soft Q-function for the reward function R, i.e., Q = B Π Q.Essentially, in [14] , they show that ( 1) is equivalent to the following single minimization problem which only requires optimizing over the soft Q-function:EQUATIONwhereQ : S × A → R is the soft Q-function, S 0 is the initial state, and V Q (S) is computed as V Q (S) = log( A∈A exp(Q(S, A))). A shortcoming of centralized IQ-Learn described above is that the computation of V Π,Q (S) or V Q (S)is not tractable as it requires to sample joint actions, an the joint action space grows exponentially in the number of agents. Furthermore, it requires full observations for agents, which is not applicable in a Dec-POMDP with local partial observations.An alternative approach to overcome the shortcomings of centralized IQ-Learn is to consider a separate IL problem for each individual agent, considering its local observations. That is, one can set the objective to recover a local Q function Q i (o i , a i ), as a function of a local observation o i and local action a i , for each agent i ∈ N . The local IQ-Learn loss function can be formulated as follows:EQUATIONHere, o ′ i ∼ P (• | S, A) means o ′ i is the local observation of agent i corresponding to the new state S ′ ∼ P (• | S, A). The value function can be computed as [14] :EQUATIONThis approach is more tractable than the centralized method, making it suitable for a Dec-POMDP environment by enabling decentralized policies for agents. However, it has limitations in addressing the interdependence between agents and the global information available during the training process.We now present our new multi-agent IL algorithm designed to recover Q functions in the multi-agent setting, following the CTDE paradigm. A possible straightforward approach is to directly factorize the centralized objective function in 3. This factorization can be achieved by decomposing the global Q-function Q(S, A) into local Q-functions Q i (o i , a i ), and computing the global V-functions using the formula V Q (S) = log( A∈A exp(Q(S, A))). However, this means that the global policy has to be computed based on a softmax of the global Q-function as Π(S, A) = exp(Q(S,A))A ′ exp(Q(S,A ′ )) . This requirement of a global policy computation violates the decentralized execution under CTDE. According to this analysis, we will instead start from the original max-min formulation in (1) , keeping in mind the necessity of having π i to support CTDE.Our key ideas involve creating (i) agent local Q-value networks that output local V-values V Q,π i (o i , a i ) and (ii) mixing the networks that utilize global state information to combine local values of agents into joint values that comprise the objective of the inverse soft Q-learning; and (iii) hyper-networks that provide a rich representation for the weights of the mixing networks, allowing us to govern their value ranges. There are two important aspects driving our mixing architecture: the consistency between local and global policies, and whether we can leverage the advantages of single-agent IQ-learn [14] , specifically non-adversarial training and convexity within the Q-space. We will first describe our factorization approach, followed by an analysis of these two aspects.Overall, our network architecture comprises of three different types of networks, described below.Agent local Q networks. Let Q(S, A) = {Q i (o i , a i )} mi=1 of local soft Q-values of agents where m = |N | is the number of agents and (o i , a i ) ∈ (S, A), i ∈ N . For an abuse of notations, we denote by Q i (o i , a i ; θ i ) as the local Q-value network of the agent i of which learnable parameter is θ i . 2 Given (Q, Π), we then use V Q,Π (S, A) to represent the corresponding state-value vector (5) . Q and V Q,Π will be passed to the corresponding mixing networks to induce the joint values Q tot and V tot Q,Π . These joint values will be then incorporated into computing the objective of the inverse soft Q-learning. Here, we do not assume any specific model for the policies π i . As we show later, there exists a closed-form solution to compute optimal policies, which allows us to eliminate the variable π i from the training problem.V Q,Π (S) = {V Q,π i (o i )} m i=1 where V Q,π i (o i ) is computed inValue mixing networks. We create two mixing networks to combine local Q and V values into the joint values Q tot and V tot , respectively. Let's denote these networks as M ψ Q (•) and M ψ V (•). Here ψ Q and ψ V are corresponding weights of these two networks. In particular, we have:Q tot (S, A) = -M ψ Q -Q(S, A) V tot Q,Π (S) = M ψ V V Q,Π (S)Note that, instead of directly mixing Q(S), we mix the negative of this vector to achieve the IGC principle and the convexity. We can now formulate the objective function of our multi-agent inverse factorized Q-learning w.r.t the local Q-values and polices, and these mixing networks, as follows:EQUATIONWe further assume that the two mixing networks are multi-layer feed-forward networks, constructed with non-negative weights and convex activation functions (e.g., ReLU, ELU, and Maxout). This configuration is widely-used in the literature and, as shown in the next section, is sufficient to ensure consistency between the global and local policies (a form of the IGM property, as shown later) and, importantly, the convexity of the training objective function within the Q-space.Hyper-networks. Finally, we create two hyper-networks corresponding the two mixing networks. These hyper-networks take the global state S as an input and generate the weights ψ V and ψ Q of the mixing networks accordingly. The creation of such hyper-networks allows us to have a rich representation of the weights that can be governed to ensure the convexity of the objective J(Q, Π, ψ Q , ψ V ) as well as the IGC property (as described below) . We can write ψ V = ψ V (S; ω V ) and ψ Q = ψ Q (S; ω R ) where ω V and ω R denote trainable parameters of the state-value and reward hyper-networks. We can alternatively write the objectiveJ(Q, Π, ψ Q , ψ V ) as J(θ, ω R , ω V ) when the local soft Q-values Q are parameterized by θ = {θ 1 , θ 2 , • • • , θ m }and the weights of the mixing networks (ψ V , ψ Q ) are parameterized by ω V and ω R respectively. The optimal policy, as shown later, can be computed as soft-max of Q. We note that we will use eitherJ(Q, Π, ψ Q , ψ V ) or J(θ, ω R , ω V )depending on the context of our analysis.One of the key aspects of CTDE is ensuring consistency between global learning and local policy execution. In previous MARL approaches based on mixing Q-functions, this is often expressed through the IGM principle [35, 29, 39] , implying that the optimal joint actions across agents (obtained by maximizing the joint Q-function) are equivalent to the collection of individual optimal actions of each agent (obtained by maximizing their local Q-functions). In our context, there are significant differences that prevent direct application of this principle. First, our recovered policy is not derived from directly maximizing the Q-function. Instead, it is obtained by solving a distribution-matching objective, which results in soft policies computed as the soft-max of Q-values. To adapt this principle in the context of inverse Q-learning, we introduce the concept of Individual-Global-Consistency (IGC), which is a generalization of the IGM, defined in the following.A factorized learning approach is said to adhere to the IGC principle if and only if the optimal joint policy (obtained from solving the global training problem) is equivalent to the collection of individual optimal policies for each agent (obtained by solving their respective local training objectives).The following result states that our factorized inverse Q-learning approach satisfies the IGC principle.Theorem 4.3. Let Π * = argmin Π {J(Q, Π, ψ Q , ψ V )}, then there are a set of optimal local policies {π * i } such that π * i = argmin πi J i (Q i , π i ), ∀i ∈ N , and Π * = {π * i , i ∈ N }.The theorem implies that the distribution of joint actions, produced by the optimal joint policy, is the same as the collection of local action distributions produced by the local optimal policies. Moreover, as an additional note, the IGM principle with respect to the joint Q-function also holds under our mixing network architecture, i.e., for all S ∈ S:argmax A {Q tot (S, A)} = {argmax ai {Q i (o i , a i )}, i ∈ N }.One of the main advantages of the single-agent inverse Q-learning algorithm is that the training problem can be equivalently transformed into a concave maximization problem over the Q-space, making the training process highly stable and well-behaved. We demonstrate below that these features still hold under our mixing network architecture. Proposition 4.4. The max-min problem in (6) is equivalent to the following maximization problem:EQUATIONwhereV tot Q (S) = M ψ V V Q (S) , and V Q (S) = V Q i (o i ) def = log( ai exp(Q i (o i , a i ))), i ∈ N .Moreover, let Q * be optimal to (7), then the global optimal policy can be recovered as follows:EQUATIONFurthermore, it can be shown that the objective in (7) is concave in Q which is an essential property that make the Q-learning procedure well-behaved and stable [14] .Theorem 4.5. J(Q, ψ Q , ψ V ) is (strictly) concave in Q.As a result, (7) always yield a unique solution within the Q-space.Theorem (4.5) indicates that the global training objective J(Q, ψ Q , ψ V ) is concave in Q when using any multi-layer feed-forward mixing networks with non-negative weights and convex activation functions. This result is highly general and notably non-trivial due to the nonlinearity and complexity of V Q i (as functions of Q i ) and the mixing networks. Prior work often relies on a two-layer mixing structure [29] , noting that such a two-layer structure is sufficient for the mixing network to approximate any monotonic function arbitrarily closely in the limit of infinite width [8] .We now present our practical algorithm, MIFQ, and details of our implemented network architecture.Mixing and Hyper Networks. We employ the following two-layer feed-forward network structure:EQUATIONfor mixing networks, whereψ Q = {W Q 1 , W Q 2 , b Q 1 , b Q 2 , } and ψ V = {W V 1 , W V 2 , b V 1 , b V 2 }are the weight and bias vectors. The absolute operations | • | ensure that all the weights are non-negative. Moreover, ψ Q and ψ V are an output of a hyper-network taking the global state S as input. Each hypernetwork consists of two fully-connected layers with a ReLU activation. Finally, ELU is employed to mitigate the issue of gradient vanishing and to ensure that negative inputs remain negative. Indeed, ELU is convex and the two mixing networks M ψ Q and M ψ V have non-negative weights, implying that the loss functionJ(Q, ψ Q , ψ V ) is concave in Q and the IGC holds.Practical Implementation. Similar to [14] , we use a χ 2 -regularizer ϕ(x) = x + 1 2 x 2 for the first terms of the loss function in (6) . This convex regularizer is useful to ensure that this is lowerbounded even when Q i , for some i ∈ N , go to -∞, which is crucial to keep the learning process stable. In addition, instead of directly estimating E S0 [V tot (S 0 )], we utilize the following equation to approximate E S0 [V tot Q (S 0 )] which can stabilize training:(1 -γ)E[V tot Q (S)] = E (S,A)∼ρ V tot Q (S) -γE S ′ ∼P (•|S,A) [V tot Q (S ′ )] for any value function V tot Q (•) and occupancy measure ρ [14], we can estimate E S0 [V tot Q (S 0 )] by sampling (S, A) from replay buffer and estimate E (S,A,S ′ )∼replay [V tot Q (S) -γV tot Q (S ′ )] instead.In summary, we will employ the following practical loss function:EQUATIONFig. 1 shows our network architecture and the details of our MIFQ can be found in the appendix. Agent 
experiments: 
 Environments. We use three environments in which enemy agents' policies are fixed and controlled by a simulator. We collect expert trajectories from well-trained ally agents and build IL to mimic them. The resulting imitating agents are evaluated by letting them play against the simulator's enemies.SMACv2 [9] . SMAC is a well-known multi-agent environment built based on StarCraft II. We employ SMACv2 [9] , an enhanced version of SMACv1 [33] , that introduces a more formidable environment for evaluating MARL algorithms. In SMACv2, scenarios are procedurally generated, compelling agents to adapt to previously un-encountered situations. This benchmark has 6 sub-tasks, including Protoss, Terran, and Zerg which feature 5 to 10 agents. These agents have the flexibility to engage with opponents of differing difficulty levels.0.5M 1M 0.5M 1M 0.5M 1M 0.5M 1M 0.5M 1M Expert BC IIQ IQVDN MASQIL MAAIRL MAGAIL MIFQ (Det) MIFQ (Soft)Gold Miner [12] . This game originates from a MARL competition, in which two teams, ally and enemy, navigate through a 2D terrain to find gold. A team win if they mined a larger amount of gold than the other. Winning this game is challenging since the allied agents must compete against exceptionally well-developed heuristic-based enemies. We consider three sub-tasks, each involves two ally and two enemy agents: (i) Easy: The enemies employ a simple shortest-path strategy to reach the gold deposits; (ii) Medium: One enemy agent follows a greedy approach, while the other follows the algorithm developed by the second-ranking team in the competition; and (iii) Hard: the enemies consists of the first-and second-ranking teams.Multi Particle Environments (MPE) [25] . MPE contains multiple communication-oriented deterministic multi-agent environments. We use three cooperative scenarios available in MPE for evaluating, including: (i) Simple spread: three agents learn to avoid collisions while covering all of the landmarks;(ii) Simple reference: two agents learn to get closer to the target landmarks. Each target landmark is known only by the other agents, so all agents have to communicate to each others; and (iii) Simple speaker listener: similar to simple reference, but one agent (speaker) can speak but cannot move, and one agent (listener) can move but cannot speak.Expert Demonstrations. For each task, we trained an expert policy by MAPPO with large-scale hyper-parameters (e.g., multi layers, higher dimensions, longer training steps, and using recurrent neural network, etc). In term of expert buffer collection, we test each method with different numbers of expert trajectories: up to 128 trajectories for MPEs and up to 4096 trajectories for Miner and SMAC-v2. Note that MPEs are not dynamic environments, so we do not need a large number of expert demonstrations for evaluation. For each collected trajectory, we used a different random seed. For a fair comparison, each method uses the same saved expert demonstrations for the training.Baselines. We compare our MIFQ against other multi-agent IL algorithms, which either originate from the multi-agent IL literature, or be adapted from SOTA single-agent IL algorithms. These baselines include: (i) Behavior Cloning (BC); (ii) Independent IQ-Learn (IIQ) -this is a simple adaption of IQ-Learn for a multi-agent setting, described in Section 4.1.2; (iii) IQ-Learn with Value Decomposition Network (IQVDN) -this is another adaptation of IQ-Learn, but instead of using mixing and hyper networks to aggregate agent Q functions, we employ Value Decomposition (VDN) [38] ; (iv) MASQIL -we adapt SQIL for multi-agent settings [30] . MASQIL shares some similar advantages with our MIFQ, such as being non-adversarial and enabling decentralized learning through centralized learning; (v) MAGAIL -a multi-agent IL algorithm introduced by [36] ; and finally (vi) MAAIRL -this algorithm is proposed by [42] . Moreover, since our MIFQ algorithm relies on soft policies while most previous factorized Q-learning algorithms [29, 35, 39] use deterministic policies, we include a deterministic-policy version of MIFQ for comparison purposes. In this version, the deterministic optimal local actions are determined as a * i = argmax ai {Q i (o i , a i )}, and the local V functions areV Q i (o i ) = max ai Q i (o i , a i ).We denote our main MIFQ algorithm as MIFQ (Soft), and the deterministic-policy version as MIFQ (Det).Comparison Results. We train each task with our algorithms and the different baselines, using 128 trajectories for MPE, and 4096 trajectories for SMACv2 and Miner. As a standard practice, we report winrates for SMACv2 & Gold Miner, and reward scores for MPEs. Figure 2 shows the learning curves and Table 1 reports the final winrates and average scores (the full table with std values can be found in appendix). Our methods MIFQ (Soft) significantly outperforms all other baselines on SMACv2 & Gold Miner tasks, and has competitive performance on MPEs, with a note that MPEs is a deterministic environment and its tasks are considerably easier than SMACv2 and Gold Miner. We also observe that MIFQ (Det) significantly outperforms other baselines, but remarkably worse than MIFQ (Soft), indicating the advantage of using soft-policy learning in our context. Moreover, to evaluate the efficiency of our method with different numbers of expert demonstrations, we compare our MIFQ with the baselines on two dynamic tasks: SMACv2 & Miner. Figure 3 shows box and whisker plots of the average winning rate of each method on each task for data summarising analysis. It shows that MIFQ (Det and Soft) offer higher winrates, and it converges better with smaller standard errors. Moreover, to evaluate how the convexity help enhance our algorithm, we compares MIFQ with those that rely on non-convex mixing networks, which are obtained by replacing the convex activation ELU with non-convex ones such as sigmoid or tanh. More details can be found in the appendix. 
conclusion and limitations: 
 Conclusion. We developed a multi-agent IL algorithm based on the inversion of soft-Q functions. By employing mixing and hyper-network architectures, our algorithm, MIFQ, is non-adversarial and enables the CTDE approach. We demonstrated that, with some commonly used two-layer mixing network structures, our IL loss function is convex within the Q-function space, making learning convenient. Extensive experiments conducted across several challenging multi-agent tasks demonstrate the superiority of our algorithm compared to existing IL approaches.Limitations. Some limitations of the work includes: (i) while MIFQ achieves impressive performance across various tasks, it falls short of reaching the expertise levels (increasing the number of expert demonstrations could solve the issue, but it would introduce additional computational challenges, as the replay buffer will become too large and the training process will be very costly); (ii) MIFQ, along with other baseline methods, struggles when confronted with very large-scale tasks, such as some of the largest games in SMACv2; and (iii) our multi-agent IL (and other baselines) still requires a large amount of (clean) expert demonstrations, which would be not always available in practice. These limitations could pave the way for future work.Broader Impacts. Our research focuses on imitation learning in multi-agent systems, it may have potential applications similar to areas where imitation learning has been impactful, such as autonomous driving, healthcare, and game theory. There are also potential negative impacts. For instance, imitation learning could be used for surveillance purposes, following and monitoring individuals in public spaces, or for developing autonomous weapons. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: [TODO]Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification: The paper develops a general imitation learning algorithm for multi-agent games, which we have tested only in simulated environments. As such, we do not foresee any direct societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [TODO]Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have provided clear citations to the source code and data we used in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Our source code is submitted alongside the paper, accompanied by sufficient instructions. We will share the code publicly for re-producibility or benchmarking purposes.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Proof of Theorem 4.3Theorem. Let Π * = argmin Π {J(Q, Π, ψ Q , ψ V )}, then there are a set of optimal local policies {π * i } such that π * i = argmin πi J i (Q i , π i ), ∀i ∈ N , and Π * = {π * i , i ∈ N }.Proof. We first consider the local objection function:J i (Q i , π i ) = E D E Q i (o i , a i ) -γE o ′ i ∼P (•|S,A) V Q,π i (o ′ i ) -(1 -γ)E o 0 i V Q,π i (o 0 i )From this we can see that thatJ i (Q i , π i ) is minimized (over π i ) if each value function V Q,π i (o ′ i ) is maximized (over π i ). Moreover, from the definition V Q,π i (o i ) = E ai∼πi(oi|ai) [Q i (o i , a i ) - log π i (a i | o i )], we see that the value of V Q,π i (o i ) is only dependent of Q i (o i , a i ) and π i (a i |o i ) for all a i ∈ A i . It follows that V Q,π i (o i )achieves its maximization at:EQUATIONNow let Π * = {π * i , i ∈ N } be a joint policy (or a set of local policies) defined as in (12) . It follows thatV Q,π i (o i ) ≤ V Q,π * i (o i ) for any i ∈ N .We now look at the joint objective function J(Q, Π, ψ Q , ψ V ), defined asJ(Q, Π, ψ Q , ψ V ) = (S,A)∈D E Q tot (S, A) -γE S ′ ∼P (•|S,A) V tot Q,Π (S ′ ) -(1 -γ)E S 0 V tot Q,Π (S 0 ) Given V tot Q,Π (S) = M ψ V (V Q,Π (S)), our mixing structure ensures that M ψ V (V Q,Π (S)) is monotonically increasing with respect to each element of V Q,Π (S). Additionally, for any joint policyΠ = {π i , i ∈ N }, we have V Q,π i (o i ) ≤ V Q,π * i (o i ) for any i ∈ N .Combining this with the monotonicity of the mixing network, we obtain:V tot Q,Π (S) ≤ V tot Q,Π * (S), which implies: J(Q, Π, ψ Q , ψ V ) ≥ J(Q, Π * , ψ Q , ψ V )for any joint policy Π. This also impliesΠ * = argmin Π {J(Q, Π, ψ Q , ψ V )}, as desired.A.2 Proof of Proposition 4.4Proposition. The max-min problem in (6) is equivalent to the following maximization problem:max Q J(Q, ψ Q , ψ V ) = (S,A)∈D E Q tot (S, A) -γE S ′ ∼P (•|S,A) V tot Q (S ′ ) -(1 -γ)E S 0 V tot Q (S 0 )whereV tot Q (S) = M ψ V V Q (S) , and V Q (S) = V Q i (o i ) def = log( ai exp(Q i (o i , a i ))), i ∈ N .Moreover, let Q * be optimal to (7), then the global optimal policy can be recovered as:Π * = π * i (o i , a i ) = exp(Q i (o i , a i ))) a ′ i exp(Q i (o i , a ′ i )) i ∈ NProof. We can leverage the result stated in Theorem 4.3 to prove the proposition. That is, Theorem 4.3 tells us thatJ(Q, Π, ψ Q , ψ V ) is minimized at Π * = {π * i , i ∈ N } such that Π * = π * i (o i , a i ) = exp(Q i (o i , a i ))) a ′ i exp(Q i (o i , a ′ i )) i ∈ NTherefore, the max-min problem can be converted to a maximization one within the Q-space:max Q min Π {J(Q, Π, ψ Q , ψ V )} = max Q J(Q, Π * , ψ Q , ψ V )Moreover, it is well-known the maximal value V Q,π * i (o i ) can be computed as a log-sum-exp function of Q i (o i , a i ) [14] as:V Q,π * i (o i )= log ai exp(Q i (o i , a i )) = V Q i (o i )which confirm our claims.A.3 Proof of Theorem 4.5Theorem J(Q, ψ Q , ψ V ) is (strictly) concave in Q.As a result, (7) always yield a unique solution within the Q-space.Proof. To prove the concavity, we will prove that each component of the objective function, i.e. Q tot (S, A) and V tot Q (S) are concave in Q, keeping in mind that the mixing networks have nonnegative weights and convex activations. We will need the following lemmas to validate the main claim.Lemma A.1. Any feed-forward mixing networks M(X) constructed with non-negative weights and non-decreasing convex activation functions (such as ReLU or ELU or Maxout) is convex and non-decreasing in X.Proof. Any N -layer feed-forward network with input X can be defined recursively asEQUATIONwhere σ n is a set of activation functions applied to each element of vector f n-1 (X), and W n and b n are the weights and biases, respectively, at layer n. Therefore, we will prove the result by induction, i.e., f n (X) is convex and non-decreasing in X for n = 0, . . .. Here we note that f n (X) is a vector, so when we say "f n (X) is convex and non-decreasing in X", it means each element of f n (X) is convex and non-decreasing in X.We first see that claim indeed holds for n = 0 . Now let us assume that, f n-1 (X) is convex and non-decreasing in X, we will prove that f n (X) is also convex and non-decreasing in X. The nondecreasing property can be easily verified as we can see, given two vector X and X ′ such that X ≥ X ′ (element-wise comparison), then we have the following chain of inequalitiesf n-1 (X) (a) ≥ f n-1 (X ′ ) σ n (f n-1 (X)) (b) ≥ σ n (f n-1 (X ′ )) σ n (f n-1 (X)) × W n + b n (c) ≥ σ n (f n-1 (X ′ )) × W n + b nwhere (a) is due to the induction assumption that f n-1 (X) is non-decreasing in X, (b) is because σ n are also non-decreasing, and (c) is because the weights W n is non-negative.To verify the convexity of f n (X),we will show that for any X, X ′ , and any scalar α ∈ (0, 1), the following holdsEQUATIONTo this end, we writeαf n (X) + (1 -α)f n (X ′ ) = ασ n (f n-1 (X)) + (1 -α)σ n (f n-1 (X ′ )) × W n + b n (d) ≥ σ n αf n-1 (X) + (1 -α)f n-1 (X ′ ) × W n + b n (e) ≥ σ n f n-1 (αX + (1 -α)X ′ ) × W n + b n = f n (αX + (1 -α)X ′ )where (d) is due to the assumption that activation functions σ n are convex and W n ≥ 0, and(e) is because αf n-1 (X) + (1 -α)f n-1 (X ′ ) ≥ f n-1 (αX + (1 -α)X ′ ) (because f n-1 (X) is convex in X,by the induction assumption) and the activation functions σ n is non-decreasing and W n ≥ 0. So, we haveαf n (X) + (1 -α)f n (X ′ ) ≥ f n (αX + (1 -α)X ′ )implying that f n (X) is convex in X. We then complete the induction proof and conclude that f n (X) is convex and non-decreasing in X for any n = 0, ..., N .Lemma A.2. For any i ∈ N ,V Q i (o i ) = log ai exp(Q i (o i , a i ))are convex in Q.Lemma A.2 can be easily verified, asV Q i (o i ) has the log-sum-exp form, so it is convex in Q [6]. Lemma A.3. Let X(Q) = (X i (Q), . . . , X m (Q)) be a vector of size m where each element is a convex function of Q, i.e., X i (Q) is convex in Q for any i ∈ N , then M(X(Q)) is convex in Q, where M(X(Q))is a mixing network of non-negative weights and convex activations.Proof. We will make used general properties of convexity to verify the claim, i.e., we will prove that, for any two vector Q and Q ′ , and any scalar α ∈ (0, 1), the following inequality always holdsEQUATIONAccording to Lemma A.1, we know that M(X) is convex in X, thus:EQUATIONMoreover, sinceX i (Q) is convex in Q for all i ∈ N , we have αX i (Q) + (1 -α)X i (Q ′ ) ≥ X i (αQ + (1 -α)Q ′ ).In addition, from the monotonicity of M(X) in each element X i , we haveEQUATIONCombine ( 18) and ( 19) we getαM(X(Q)) + (1 -α)M(X(Q)) ≥ M(X(αQ + (1 -α)Q ′ ))which validates (16) , completing our proof for Lemma A.3.We now go back to the main result. First, considering M ψ Q (-Q(S, A)), we can see that each element of -Q(S, A) is linear in Q. Thus, Lemma A.3 tells us thatM ψ Q (-Q(S, A)) is convex in Q. As a result, Q tot (S, A) is concave in Q. Similarly, V Q (S) is also a vector where each element V Q i (o i ) is convex in Q (Lemma A.2), thus M ψ V (V Q (S)) (and V tot Q (S)) is also convex in Q . Therefore, the objective function of our IL, formulated asJ(Q, ψ Q , ψ V ) = (S,A)∈D E Q tot (S, A) -γE S ′ ∼P (•|S,A) V tot Q (S ′ ) -(1 -γ)E S 0 V tot Q (S 0 ) , should be concave in Q. We further see that V Q i (o i ), ∀o i , are strictly convex in Q, implying that J(Q, ψ Q , ψ V ) is also strictly concave in Q, as desired.It is important to note that the above result only holds if Q i are independent. It is not the case if Q i , for some i, share a common network structure. This is also the case of the IQVDN considered in the main paper, i.e., the global reward and value function R tot and V tot are sums of the corresponding local functions, but Q i share the same neural network structure. θ ← θ -λ θ ∇ θ J(θ, ω R , ω V ); (ω R , ω V ) ← (ω R , ω V ) -λ ω ∇ ω R ,ω V J(θ, ω R , ω V );until a certain number of loops;We discuss equations that describe the relationship between the Q and V functions, which we refer to as Value equations. Specifically, we consider Eq. 5 for local value functions and Eq. 2 for global ones. So far, we have assumed that the Value equations hold for all local agents (i.e., Eq. 5 holds), ensuring that distribution-matching is executed well at the individual level. An alternative approach to the factorized learning is to assume that the Value equations hold at the global level, i.e.,EQUATIONRecall that the local value equations can be written asV Q,π i (o i ) = E πi(ai|oi) [Q i (o i , a i ) -log π i (a i |o i )]We then can see that the local and global Value equations, generally, cannot hold simultaneously, in particular when V tot and Q tot have different mixing structures. Thus, we can only ensure the validity of either the local Value equations in (5) or the global one in (20) . It can be seen that, while adhering to ( 5) is highly tractable due to several advantages shown above (i.e., IGC, non-adversarial training, and convexity), using (20) to compute V tot is highly impractical as due to several reasons. First, it cannot help avoid adversarial training as there is no closed form solution for the minimization over Π, so one needs to learn both Q and Π adversarially. Second, it requires sampling over joint actions approximate V tot , which shares the same computational issue as the centralized approach discussed in Section 4.1.1, namely, the exponentially large joint action space make the training process highly inefficient.Each environment has different a observation space, state dimension, and action space. Therefore, we use different hyper-parameters on each task to try to make all algorithms work stably. Moreover, due to limitations in computing resources, especially random access memory, we reduce the buffer size to 5000 on two hardest tasks, Miner and SMACv2, to be more efficient in running time with parallel workers. More details are available in Table 2 . We use four High-Performance Computing (HPC) clusters for training and evaluating all tasks. Specifically, each HPC cluster has a workload with an NVIDIA L40 GPU 48 GB GDDR6, 32 Intel-CPU cores, and 100GB RAM. In terms of model architecture, Figure 1 shows our proposed model structure with mixing networks based on QMIX algorithm [29] . Recall that when convex activation functions are used in building mixing networks, our Theorem 4.5 shows that the objective function of the multi-agent inverse soft Q-learning is convex as long as the mixing networks are constructed with non-negative weights and convex activations. This convexity property guarantees the optimization objective is well-behaved and has an unique optimization solution with Q-space, leading to the effectiveness of the IL process. Therefore, in the practical implementation of our algorithm, we chose the ELU activation function. In this section, we empirically demonstrate the impact of convex activation functions by examining the performance of our algorithm when non-convex activation functions such as sigmoid or tanh are used in mixing networks instead of convex once. Table 3 reports the winrates and rewards, and Figure 4 shows the learning curves of different approaches. These results clearly show that the ELU activation significantly outperform those using non-convex activations (i.e., sigmoid, tanh), for both versions MIFQ (Soft) and MIFQ (Det).In this section, we explore the performance of SAC-IQ, an adaptation of the Soft Actor-Critic (SAC) algorithm combined with IQ-Learn, tailored for multi-agent settings. SAC-IQ employs a centralized Q function alongside decentralized critics, maintaining adherence to the original objective of IQ-Learn. This approach is evaluated across 9 distinct tasks within two environments: SMACv2 and Gold Miner. For SMACv2, we consider scenarios with varying team sizes, specifically 5vs5 and 10vs10. In the Gold Miner environment, we assess performance across three difficulty levels: easy, medium, and hard. Miner: hardThe results, detailed in Table 4 , illustrate that SAC-IQ demonstrates commendable performance when compared to other baselines such as Behavior Cloning (BC), Independent IQ-Learn (IIQ), and Multi-Agent Generative Adversarial Imitation Learning (MAGAIL). However, it does not surpass the effectiveness of our proposed methods. Figure 5 provides a visual representation of the learning curves, highlighting the training dynamics of SAC-IQ in comparison to other approaches. This visualization underscores the competitive nature of SAC-IQ while also affirming the superior performance of our methods in these challenging multi-agent environments.In this section, we present the experimental details for the SMACv2, Miner, and MPE tasks, with varying numbers of expert demonstrations. These details complement the box plots shown in Figure 3 in the main paper. Tables 4-9 report win rates (as percentage values) for SMACv2 and Miner tasks with the number of trajectories ranging from 128 to 4096. Table 10 reports final rewards for MPE tasks with the number of trajectories ranging from 1 to 128. We also plot the learning curves of these experiments in Figure 7 . Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.38th Conference on Neural Information Processing Systems (NeurIPS 2024).We convert the maximization formulation in[14] into minimization to facilitate our later theoretical analyses.In practical implementation, we consider a shared local Q-value network for all agents. 


Decision: This paper presents a novel algorithm called multi-agent inverse factored Q-learning for cooperative multi-agent imitation learning. It extends inverse soft Q-learning from single-agent to multi-agent settings by a mixing network under the CTDE paradigm. Experimental evaluation across several environments demonstrates the effectiveness of the proposed algorithm. The proposed algorithm is a significant and contribution to the imitation learning field. The reviewers are largely satisfied with the author's rebuttal. I trust the authors will further improve the paper's organization and address clarity issues in the camera-ready version.

Reviews: review: paper_summary: paper_summary: The paper addresses the imitation problem in cooperative Multi-Agent Reinforcement Learning (MARL). It extends inverse soft-Q learning to the multi-agent domain by leveraging value factorizations under the Centralized Training with Decentralized Execution (CTDE) paradigm. Experimental results demonstrate the effectiveness of the proposed approach across several environments.
strength_weakness: strength_weakness: - The paper's organization could be improved. The current structure alternates between theory and architecture without a clear flow.

- The similarity between IGC and IGO[1] requires further clarification.

- The objective function (6) introduces sub-optimality compared to the original objective (3) due to the restriction that $Q^{tot}$ and $V^{tot}$ must be monotonic. Additionally, since $Q^{tot}$ and $V^{tot}$ use different mixing networks, the relationship between them violates Equation (2). This indicates that Equation (6) does not represent the same objective as Equation (3), even without considering the sub-optimality introduced by factorization. These issues need further theoretical exploration and discussion.

- Although the experimental results are promising, the superior performance seems to stem from the QMIX algorithm's advantage over other MARL algorithms. An important missing baseline is the soft actor-critic version of IQ-Learn, which uses a centralized Q function with decentralized critics and does not seem to violate the original objective.

[1] Zhang, et al., FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning, ICML 2021.
questions: questions: 1.	Is BC trained online, given that it shows learning curves with environment steps? If so, why not use DAGGER?
2.	Could the authors explain why MIFQ significantly outperforms IQVND? Is it solely due to the factorization structure?
3.	Why does the paper state that QPLEX is unsuitable for the proposed method? QPLEX also has $\partial Q/\partial Q_i>0$.
limitations: limitations: Limitations are discussed in conclusion section.
paper_summary: paper_summary: This paper presents a novel algorithm, Multi-agent Inverse Factorized Q-learning (MIFQ), for cooperative multi-agent imitation learning (IL). It extends the inverse soft-Q learning framework to multi-agent settings by introducing a mixing network architecture for centralized training with decentralized execution. This enables learning local and joint value functions effectively. The authors conducted extensive experiments across multiple challenging environments, demonstrating that their approach outperforms existing methods.
strength_weakness: strength_weakness: As someone who is not an expert in the field of imitation learning, I perceive no significant weaknesses in this paper from my perspective.
questions: questions: - In Figure 2, the semi-transparent curves are not standardly explained. If these do not represent standard deviations, what statistical measure do they depict?
- Minor Error: On Line 62, the term "generation" is used where "generalization" might be intended. Could the authors clarify or correct this in the context?
limitations: limitations: Yes
paper_summary: paper_summary: This paper addresses the problem of extending a single-agent imitation learning algorithm, inverse soft-Q learning (IQ-learn, Garg et al. Neurips 21) to the multi-agent cooperative setting. The proposed algorithm, MIFQ, leverages the ideas of mixing networks and the individual-global-max (IGM) principle, to perform the extension. Experimental evaluations of MIFQ are conducted on SMAC-v2, MPE, and Gold Miner, and demonstrate that MIFQ improves over baselines across various domains and with varying numbers of demonstrations.
strength_weakness: strength_weakness: 1. Some aspects of the method do not seem fully justified to me: 
    - The authors claim in lines 143-148 that a shortcoming of the IQ learn method is that the objective depends on the centralized state and joint action. However, Section 5.4 of the IQ Learn paper presents a state-only objective (independent from the actions). I wonder if the authors could discuss whether a simple state-only extension of IQ Learn, where critic depends on the centralized state as usual, but the actor depends on the observations, would be sufficient to sidestep many of the concerns addressed by IQ Learn? 
    - The authors also claim in Section 4.1.2 that the straightforward Independent Inverse Q-learning is not a satisfactory solution because the method "…has limitations in addressing the interdependence between agents and the global information available during the training process.". Can the authors more explicitly discuss what the shortcomings of an independent version of IQ-learn is not satisfactory? Does it suffer from convergence problems? 
					
2. The current experimental analysis is somewhat shallow, and essentially amounts to a description of the plots. The authors could improve the analysis of MIFQ by considering the following additional questions: 
    - The original IQ learn paper plots the rewards to validate that their method recovers the ground truth reward. Can the same be done here? 
    - Why does MIFQ perform worse than BC on MPE, particularly the reference and spread tasks?
3. There are some issues with how the experimental results have been reported. 
    - What is the number of trials for each of the results? Please include this in the main paper. 
    - The caption of Figure 2 is missing key information to understand the figure. What is the number of demonstrations used to train each of the methods? What does the shaded region mean? Based on the std devs reported in the Appendix, I assume it is the standard deviation; please see the note below and instead compute 95% confidence intervals. 
    - No measurements of uncertainty are provided in Table 2, and standard deviations are provided only in the Appendix. Standard deviations reflect the underlying variance in models learned by the algorithm, rather than providing a measure of statistical significance. Please also compute 95% confidence intervals to enable readers to judge the statistical significance of the gaps in mean test returns -- ideally, bootstrapped confidence intervals. See this paper for a reference on best practices:  https://arxiv.org/abs/2304.01315
3. There are also some minor clarity issues: 
    - IGC is used in line 192, but is only explained in the following Section 4.2.2
    - Definition 4.2 - this definition is not specific enough to be useful. It handwaves  by only requiring that the joint policy be 'equivalent' to the collection of individual optimal policies. Equivalent in what sense?
questions: questions: 1. Questions about experiments: 
    - What are some reasons why MIFQ does not achieve expert level performance? While the other methods also do not achieve expert level performance, the original IQ learn algorithm does have this ability. 
    - How does the method perform with demonstrations not sourced from MAPPO (an algorithm that learns gaussian policies)? For example, demonstrations sourced from QMIX, which learns 'hard max' policies? 
    - Why does the method need an order of magnitude more demonstrations than IQ Learn needs on complex single-agent tasks? 
   
2. Method: 
    - Why is it necessary to maintain Q and V networks separately? Why not derive the global V function by computing the softmax of the Q functions as described in line 163-164? 
    - Why is it necessary to compute Q^tot via Q^tot = -M (-Q)? What is the purpose of the double negation? The stated justification is that this enable the method "to achieve the IGC principle and the convexity", but why exactly is this? Requiring the networks to be multi-layer feedforward w/nonnegative weights and convex activation functions (lines 194-195) is enough to ensure that Q^tot is monotonic w.r.t. the local Q functions, thus ensuring the IGC principle and convexity. 
    - Would major changes be necessary to enable this algorithm to operate on continuous action spaces? Did the authors consider continuous action space settings?
limitations: limitations: yes
paper_summary: paper_summary: This paper extends the IQ-Learn method to cooperative multi-agent settings. The main insight is to use mixing networks to enable centralized training via decentralized Q functions.
strength_weakness: strength_weakness: - The major concern that I have is that, if my understanding is correct, the paper assumes access to the global state information. This is not realistic. In real application, this will never be the case. So the algorithm does not seem useful in practice.
- Typo: In line 62, it should be "generalization" instead of "generation",
- In line 72, \citet should be used instead of \cite or \citep so that the author names will become a part of the sentence.
- In line 162, \eqref should be used instead of \ref so that the parenthesis will appear around the equation number.
- The architecture figure is in page 7. It would significantly increase the readability if it came earlier. 
- By the time the reader reads line 191, the IGC principle is still undefined. This makes reading very difficult.
- The same thing is true at line 203, too.
- Typo: In line 241, it should be "makes" instead of "make".
- Typo: In line 242, it should be "yields" instead of "yield".
questions: questions: - How do the agent have access to the global state information. If this is the case, why does the paper even define observations? Is the global state information available only in training or after deployment, too? In what settings is this applicable?
- How could one adapt this algorithm for non-cooperative settings? Is there a straightforward way or does it require completely new approaches?
limitations: limitations: The paper does not discuss the broader impacts. I disagree that there is no potential societal impact. I invite the authors to think about the applications their algorithm may have and then consider how their algorithm would affect those applications (both positively and negatively).

Comments: title: Thank you for the reply!
Thank you for reading our responses!
Limitations
comment: We highly appreciate the reviewer for taking the time to read our responses and for the positive feedback on our work. We will definitely improve our discussion on global state information and include a discussion on the broader impacts of our work.
I thank the authors for their response. After reading their answers, I did another pass of the paper and I believe I now have a better understanding of their algorithm. I will update my score and trust the authors that they will (1) clarify the confusion about global state information and the meaning of $$S$$, and (2) add the discussion about broader impacts in their camera-ready version.
We thank the reviewer for reading our responses and for maintaining a positive outlook on our paper.

> The authors argued that sampling actions from the joint policies is intractable due to the size of the joint action space. However, the paper only addresses scenarios where the number of agents is relatively limited (up to 10 agents). Further, this cost would only be incurred during the training phase. Since sample efficiency is not a primary objective of this paper, I don't think this is a key issue.

At this point, we have found that this approach (computing V directly via Q) does not work in our context. There might be ways to overcome this, and we will explore them in the future. Thank you for your feedback!

> Since the authors performed the experiment, perhaps they can add the results of directly computing V through Q to the appendix.

Thank you for the suggestion. We will definitely include these additional experiments (directly computing V through Q) in our paper.
Thanks for addressing most of my concerns and questions. The only questions whose answers I wasn't completely satisfied with why to maintain Q and V networks separately. 

The authors argued that sampling actions from the joint policies is intractable due to the size of the joint action space. However, the paper only addresses scenarios where the number of agents is relatively limited (up to 10 agents). Further, this cost would only be incurred during the training phase. Since sample efficiency is not a primary objective of this paper, I don't think this is a key issue. 

Since the authors performed the experiment, perhaps they can add the results of directly computing V through Q to the appendix. 

In any case, I am largely satisfied with the author's rebuttal and will raise my score.
We thank the reviewer for reading our responses and for the prompt reply!
Regarding the broader impact of our work, since our research focuses on imitation learning in multi-agent systems, it may have potential applications similar to areas where imitation learning has been impactful, such as autonomous driving, healthcare, and game theory. There are also potential negative impacts. For instance, imitation learning could be used for surveillance purposes, following and monitoring individuals in public spaces, or for developing autonomous weapons. We thank the reviewer for bringing this up and will elaborate such impacts in detail in our revised version.
Thanks for the authors' response. I have increased my initial score.