Title: Multi-Agent Imitation by Learning and Sampling from Factorized Soft Q-Function
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Agent Imitation Learning

Energy-Based Models

Inverse Reinforcement Learning
Abstract: Learning from multi-agent expert demonstrations, known as Multi-Agent Imitation Learning (MAIL), provides a promising approach to sequential decision-making. However, existing MAIL methods including Behavior Cloning (BC) and Adversarial Imitation Learning (AIL) face significant challenges: BC suffers from the compounding error issue, while the very nature of adversarial optimization makes AIL prone to instability. In this work, we propose \textbf{M}ulti-\textbf{A}gent imitation by learning and sampling from \textbf{F}actor\textbf{I}zed \textbf{S}oft Q-function (MAFIS), a novel method that addresses these limitations for both online and offline MAIL settings. Built upon the single-agent IQ-Learn framework, MAFIS introduces the value decomposition network to factorize the imitation objective at agent level, thus enabling scalable training for multi-agent systems. Moreover, we observe that the soft Q-function implicitly defines the optimal policy as an energy-based model, from which we can sample actions via stochastic gradient Langevin dynamics. This allows us to estimate the gradient of the factorized optimization objective for continuous control tasks, avoiding the adversarial optimization between the soft Q-function and the policy required by prior work. By doing so, we obtain a tractable and \emph{non-adversarial} objective for both discrete and continuous multi-agent control. Experiments on common benchmarks including the discrete control tasks StarCraft Multi-Agent Challenge v2 (SMACv2), Gold Miner, and Multi Particle Environments (MPE), as well as the continuous control task Multi-Agent MuJoCo (MaMuJoCo), demonstrate that MAFIS achieves superior performance compared with baselines. Our code is available at https://github.com/LAMDA-RL/MAFIS.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new framework for multi-agent imitation learning that avoids the instability of adversarial methods and the compounding errors of behavior cloning. MAFIS extends the inverse soft Q-learning paradigm (IQ-Learn) to multi-agent settings by introducing a value decomposition network that factorizes the soft Q-function across agents. This factorized soft Q-function implicitly defines an energy-based policy from which actions are sampled using stochastic gradient Langevin dynamics (SGLD), enabling tractable imitation learning in both discrete and continuous domains. The method is evaluated on SMACv2, Gold Miner, MPE, and Multi-Agent MuJoCo, showing superior performance to existing imitation learning baselines in both online and offline settings.\n\nSTRENGTHS AND WEAKNESSES:\n### **strength**\nI think the paper introduces a well-motivated and technically sound MAIL framework. Extending IQ-Learn to the multi-agent setting by adapting both the policy and soft Q-function into joint forms is a logical step, but the key contribution is the agent-level factorization of the global soft Q-function. The insight of interpreting the soft Q-function as an energy-based model elegantly avoids the need for logsumexp over continuous action spaces. Unlike many MAIL methods that are restricted to either online or offline settings, MAFIS offers a single, unified solution. The experiments are broad and convincing.\n\n### **Weakness**\n1. I think the paper could better discuss the trade-offs of using value decomposition. While value decomposition makes the learning objective factorable, it may limit expressivity in tasks requiring high-order coordination. I would have liked to see either ablations with different decomposition architectures (e.g., QMIX, Qplex) or discussion of when the factorization assumption might break down.\n2. While I appreciate that MAFIS is positioned within imitation learning, I believe it should be evaluated against recent offline MARL methods such as OMAR or ICQ-MA, which are also designed to learn from fixed datasets without online exploration. Since MAFIS shares many traits with offline RL approaches—such as value-based training from demonstrations, scalability, and stability—I would have liked to see whether it matches or exceeds these general-purpose baselines. Including such comparisons would help clarify the relative strengths of MAFIS in broader offline settings.\n\nQUESTIONS:\n1. Could you elaborate on the limitations of value decomposition in high-coordination tasks?\n2. Why was there no comparison to recent offline MARL methods such as OMAR or ICQ-MA?\n3. How sensitive is MAFIS to the number and quality of expert trajectories?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MAFIS, a novel approach for multi-agent imitation learning that works for both online and offline settings in discrete and continuous control tasks. The authors adapt the single-agent IQ-Learn framework to multi-agent scenarios by introducing a value decomposition network that factorizes the joint soft Q-function as a weighted sum of individual agent Q-functions. This factorization enables the imitation objective to be decomposed at the agent level, allowing for scalable training and decentralized execution. Unlike existing methods that rely on adversarial training, MAFIS converts the adversarial objective into a non-adversarial one through the factorization approach. The optimal joint policy can be expressed as a product of individual agent policies, eliminating the need for adversarial optimization between the Q-function and policy. For continuous action spaces where computing the $\\texttt{logsumexp}$ over actions is intractable, the authors make a key observation that the soft Q-function implicitly defines the optimal policy as an energy-based model. They use stochastic gradient Langevin dynamics (SGLD) to sample actions from this distribution, enabling gradient estimation without explicitly computing the intractable normalization terms. The experimental evaluation demonstrates MAFIS's effectiveness across multiple benchmarks including SMAC v2, MPE, Gold Miner, and MAMuJoCo, showing superior performance compared to existing baselines including Behavioral Cloning, MA-GAIL, and MIFQ.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: The paper provides rigorous mathematical derivations, particularly in Propositions 3.1 and 3.2, showing how the joint soft Q-function factorization leads to a tractable non-adversarial objective. The experiments span multiple challenging benchmarks (SMACv2, MPE, Gold Miner, MaMuJoCo) covering both discrete and continuous control tasks, with both online and offline settings. The sensitivity analysis (Section 4.2 and Appendix D) provides insights into key hyperparameters like the number of samples $N$ and entropy weight $\\alpha$. Figure 1 effectively demonstrates the instability of adversarial training in multi-agent continuous control, motivating the proposed approach. The paper tackles real issues in MAIL - BC's compounding errors and AIL's training instability. Providing a single approach that works for discrete/continuous and online/offline settings is valuable. The factorization approach enables decentralized execution, which is crucial for real-world multi-agent systems. Using SGLD to sample from the soft Q-function as an EBM for continuous control is creative and well-executed.\n\n**Weaknesses**: The paper only compares against BC, MA-GAIL, and MIFQ, other recent MAIL methods could strengthen the evaluation. The paper collects only 20 trajectories for continuous tasks and 100 for discrete tasks; the sensitivity to the amount of expert data is not explored. While the method requires SGLD sampling with $N=20$ samples and $K=25$ steps, the computational overhead compared to baselines is not discussed. While Appendix B attempts to clarify differences from MIFQ, the distinction could be clearer in the main text. MAFIS creates a fundamental mismatch between training and execution policies.\n\nQUESTIONS:\n1. Your training objective (Equation 7) optimizes Q-functions such that the induced policy $\\pi_i(a_i|\\tau_i) \\propto \\exp{\\frac{1}{\\alpha} k_i(s) Q_i(\\tau_i, a_i)}$ matches expert behavior. However, during decentralized execution, agents use $\\tilde{\\pi}_i(a_i|\\tau_i) \\propto \\exp{\\frac{1}{\\alpha} Q_i(\\tau_i, a_i)}$. While you correctly note these share the same argmax for greedy action selection, I have concerns about: a) In scenarios requiring exploration or stochasticity (e.g., multi-modal action distributions), the two policies will sample different actions with different probabilities. Has this been considered? b) Did you consider learning Q-functions directly for the execution policy $\\tilde{\\pi}_i$? For instance, by reparameterizing $Q_i$ to absorb the effect of $k_i(s)$?\n2. Can you provide analysis on why MAFIS still lags behind expert performance, particularly in continuous control? Have you tried: Varying the number of expert demonstrations (currently only 20 for continuous)? Alternative sampling methods (e.g., HMC)?\n3. Have you considered comparing with other recent MAIL methods, particularly: SQIL which also builds on soft Q-learning or any offline MARL methods that can work without rewards (e.g., multi-agent behavior cloning with transformers)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper studies multi-agent imitation learning by extending the single-agent IQ-Learn method (a well-known imitation learning approach in the single-agent setting).\n\n- By employing value decomposition (where the global Q-function is factorized as a linear combination of local Q-functions), the authors propose a CTDE (Centralized Training with Decentralized Execution) approach to learn local policies through a centralized loss function adapted from the IQ-Learn objective.\n\n- Experiments conducted on recent and strong benchmarks, such as SMAC\\_v2 and MAMuJoCo, show that the proposed method, MAFIS, appears to perform well.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n - Imitation learning in the multi-agent setting is less explored than in the single-agent case, so investigating this direction is worthwhile.\n- The proposed approach aims to ensure some fundamental properties in MARL, such as consistency between global and local value functions.\n- The experimental results seem solid — the proposed MAFIS method, as reported in the paper, appears to outperform some recent baselines such as MFIQ.\n\n**Weaknesses**\n- The main weakness is that the idea of extending IQ-Learn to the multi-agent setting is not novel. This has already been thoroughly explored in recent work [1]. Compared to MFIQ in [1], the proposed method appears to follow the same strategy (extending the IQ-Learn loss function to the multi-agent setting using CTDE and value factorization to decompose the global Q-function). Moreover, MFIQ even explores a more general setting and architecture — it discusses both 1-layer and 2-layer mixing networks, while MAFIS focuses only on a simple linear combination. In addition, properties such as global-local consistency and convexity have been more thoroughly investigated in [1].\n- There is a brief discussion on the difference between MFIQ and MAFIS, but the difference seems marginal, and the core idea remains essentially the same.\n- Some theoretical results are trivial and can be easily deduced from prior work. For instance, Proposition 3.1 is just a special case of the local policy formulation mentioned in [1], where a 1-layer network is used. Proposition 3.2 also appears to be trivial.\n- Compared to [1], several relevant baselines are missing, such as VDN and value factorization with non-linear mixing networks.\n\nIn summary, the contribution is relatively weak given that [1] has already extended IQ-Learn to the multi-agent setting with a more in-depth investigation, stronger theoretical results, and richer experimental studies\n\n[1] The Viet Bui, Tien Mai, and Thanh Hong Nguyen. \"Inverse Factorized Soft Q-Learning for Cooperative Multi-agent Imitation Learning.\" In Advances in Neural Information Processing Systems (NeurIPS), 2024.]\n\nQUESTIONS:\n- How does your factorization method perform when using non-linear mixing structures, such as a 2-layer mixing network?\n- Can you say anything about convexity in the Q-function space — an important property of IQ-Learn that has been discussed in both the original IQ-Learn paper and in MFIQ?\n- How does your method compare with other standard baselines, such as VDN or independent IQ-Learn (where you perform IQ-Learn independently for each agent)?\n\nLIMITATIONS:\n- Yes, the authors discuss some limitations of their work, which is reasonable.\n- I do not see any significant negative social impact that needs to be discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MAFIS, a novel algorithm for Multi-Agent Imitation Learning (MAIL) that based on IQ-learn. Inspired by QMIX, the authors propose a value decomposition method that factorizes the joint soft Q-function into a weighted sum of individual agents' Q-functions. This factorization yields a tractable and non-adversarial training objective. It uses Stochastic Gradient Langevin Dynamics (SGLD) to extract actions from the soft q function for continuous action. The authors conduct comprehensive experiments across a variety of discrete (SMACv2, MPE, Gold Miner) and continuous (MaMuJoCo) multi-agent benchmarks, showing that MAFIS consistently outperforms baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. The combination of IQ-learn and factorized Q function is novel and elegant.\n2. The paper provides clear derivations for its factorized objective (Proposition 3.1) and its gradient (Proposition 3.2). \n3. The method is evaluated on multiple challenging multi-agent benchmarks, covering discrete and continuous domains and online and offline settings.\n\nWeakness:\n\n1. My primary concern lies in the factorization of the Q-function and the policy. Since each agent selects actions and evaluates its Q-function almost independently, coordination between agents may be hindered at each timestep. Please correct me if I’m mistaken, but consider the following example: if two agents receive a high reward only when they choose the same action, then sampling actions independently would prevent them from achieving optimality. This suggests that the current factorization may fail to capture essential inter-agent dependencies.\n2. Another concern arises from the use of the state-dependent coefficients k^i(s), which are used to compose the joint Q-function. As stated in Lines 183–184, these coefficients—as well as the state itself—are not observable by individual agents during decentralized execution. This raises a potential issue: due to the presence of entropy regularization, the policy that is optimal with respect to an individual Q-function may not align with the policy optimal for the joint Q-function. While this misalignment may not significantly affect practical performance, it suggests a discrepancy between the training and deployment objectives that could impact theoretical guarantees or worst-case behavior.\n3. The use of SGLD for sampling in continuous action spaces can be computationally expensive, potentially limiting the method’s scalability. Moreover, employing SGLD for Q-function sampling is a well-established technique and, by itself, does not constitute a significant contribution.\n\nQUESTIONS:\n1. though factorization is standard, I want to understand the design choice that why not considering learning a centralized q function and then factorized it by learning the expectation of each agent’s Q value?\n2. it seems that if we want multi-agent collaborates in the beginning, it would be necessary to introduce some hidden variables before agents make decision. I’d like to hear authors’ though on this.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}