NeurIPS 2025

Multi-Agent Imitation by Learning and Sampling
from Factorized Soft Q-Function
Yi-Chen Li1,2, Zhongxiang Ling1,2, Tao Jiang1,2,4, Fuxiang Zhang3, Pengyuan Wang1,2,
Lei Yuan1,2,4, Zongzhang Zhang1,2, Yang Yu1,2,4∗
1 National Key Laboratory for Novel Software Technology, Nanjing University, China,
2 School of Artificial Intelligence, Nanjing University, Nanjing, China,
3 Nanyang Technological University, Singapore,
4 Polixir Technologies, Nanjing, China,
liyc@lamda.nju.edu.cn, lingzx@smail.nju.edu.cn, fuxiang001@e.ntu.edu.sg
{wangpy, yuanl}@lamda.nju.edu.cn, zzzhang@nju.edu.cn, yuy@nju.edu.cn
Abstract
Learning from multi-agent expert demonstrations, known as Multi-Agent Imitation
Learning (MAIL), provides a promising approach to sequential decision-making.
However, existing MAIL methods including Behavior Cloning (BC) and Adver-
sarial Imitation Learning (AIL) face significant challenges: BC suffers from the
compounding error issue, while the very nature of adversarial optimization makes
AIL prone to instability. In this work, we propose Multi-Agent imitation by learn-
ing and sampling from FactorIzed Soft Q-function (MAFIS), a novel method that
addresses these limitations for both online and offline MAIL settings. Built upon
the single-agent IQ-Learn framework, MAFIS introduces the value decomposition
network to factorize the imitation objective at agent level, thus enabling scalable
training for multi-agent systems. Moreover, we observe that the soft Q-function
implicitly defines the optimal policy as an energy-based model, from which we
can sample actions via stochastic gradient Langevin dynamics. This allows us to
estimate the gradient of the factorized optimization objective for continuous control
tasks, avoiding the adversarial optimization between the soft Q-function and the pol-
icy required by prior work. By doing so, we obtain a tractable and non-adversarial
objective for both discrete and continuous multi-agent control. Experiments on
common benchmarks incl...