Title: Learning Equilibria from Data: Provably Efficient Multi-Agent Imitation Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Multi-Agent Reinforcement Learning

Imitation Learning

Multi-Agent Imitation Learning
Abstract: This paper provides the first expert sample complexity characterization for learning a Nash equilibrium from expert data in Markov Games.
We show that a new quantity named the *single policy deviation concentrability coefficient* is unavoidable in the non-interactive imitation learning setting, and we provide an upper bound for behavioral cloning (BC) featuring such coefficient. BC exhibits substantial regret in games with high concentrability coefficient, leading us to utilize expert queries to develop and introduce two novel solution algorithms: MAIL-BRO and MURMAIL. The former employs a best response oracle and learns an $\varepsilon$-Nash equilibrium with $\mathcal{O}(\varepsilon^{-4})$ expert and oracle queries. The latter bypasses completely the best response oracle at the cost of a worse expert query complexity of order  $\mathcal{O}(\varepsilon^{-8})$. Finally, we provide numerical evidence, confirming our theoretical findings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with the setting of multi-agent RL where the agent tries to learn in a Markov game. In particular, the authors investigate the possibility of doing imitation learning in this setting.\n\nOne one hand, the paper derives a fundamental lower bound for the Behavioral Cloning algoritm. This shows that, even in the ideal scenario of inifite data and access to a generative model, there are environments where the learned policy can be exploited by the adversary. Therefore, it is not possible to learn an $\\varepsilon-$Nash equilibrium.\n\nAlso the authors design two alsorithms, one that is oracle-efficient and the other that gets rid of the oracle at a cost of a much higher sample complexity. the oracel here is a Best Response Oracle (BRO), who is assumed to output the best response for a given strategy.\n\nSTRENGTHS AND WEAKNESSES:\n**strenghts**\n\nThe paper defines the Concentrability Coefficient $C$ and shows that this is a fundamental quantity to control the error of BC in this setting. To do this, the authors construct a Markov game in which no non-interactive algorithm, even with infinite data and full transition knowledge, can learn a near-Nash equilibrium, when $C$ is infinity. This result establishes a fundamental impossibility in offline MAIL, highlighting the need for interaction.\n\nIf I understand correctly, this is the main take-home message of the paper: in order to learn $\\varepsilon-$Nash equilibria for arbitrarily small $\\varepsilon$ from expert demonstrations, one must be able to interact with the expert to query demonstrations at specific points.\n\nThe negative result is completed by two positive result saying that 1) MAIL-BRO, which requires a best response oracle, achieves $\\varepsilon-$Nash with O($\\varepsilon^{-4}$) queries in Polynomial-time, with no dependence on concentrability 2) MURMAIL does the same without the need of an oracle at a price of an higher sample complexity.\n\n**weaknesses**\n\nThe only weakness that I see is the bad dependence of the upper bounds for the sample complexity in both $\\varepsilon$ and the constants. It is not clear if these results are optimal.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nUnfortunately, my very limited knowledge of the topic prevents me to understanding how groundbreaking this paper is w.r.t. the literature.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-09"}

comments: SUMMARY:
The paper studies the problem of learning an approximate Nash Equilibrium (aNE) in two player, zero sum Markov games. The authors first study the non-interactive variation where the learners have access to samples from the game, but cannot simulate it. They prove that by using Behavior Cloning, they can compute an aNE, where the approximation factor depends on a novel coefficient, which is necessary to be bounded, otherwise obtaining the aNE is infeasible. The authors also provide positive results in the interactive setting; given best response oracles they provide an algorithm which finds an aNE with polynomial, in the parameters of the game, samples. Moreover, they provide an algorithm that computes an aNE without the need of Best Response oracles but with worse sample complexity.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well written, easy to read, and has nice illustrations that are intuitive.
2. The authors provide strong theoretical results that are coupled with experiments which verify them.
3. The authors provide a comprehensive study of finding equilibria from data in two player Markov Chains. They study both the interactive and non-interactive setting, and they provide efficient algorithms in terms of sample complexity that compute aNE in each case. In the non-interactive setting, they identify exactly the cases where learning an aNE is infeasible.

QUESTIONS:
1. Is the single policy deviation concentrability coefficient a property of the game or the data trajectories? Can you explain more intuitively what having a very large coefficient means?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates a cutting-edge research topic, namely Multi-Agent Imitation Learning, for which theoretical results are currently scarce. It is mostly closely related to [Tang 2024] in the reference. The first result (Theorem 3.1) says that the Nash-Gap is close to zero if we use a behavior-cloning algorithm to learn the behavior very well, and a constant C(\\hat{\\mu},\\hat{\\nu}) is bounded. If the behavior are welled learned, C(\\hat{\\mu},\\hat{\\nu}) is expected to be close to C(\\mu^E,\\nu^E), where \\mu^E and \\nu^E are expert policies that achieve the Nash equilibrium. However, even if the nash equilibrium is unique, it may still be possible that C(\\hat{\\mu},\\hat{\\nu})=\\infty, since fixing one player's policy other player may still have non-unique policy. This observation is formalized in Theorem 3.2, which says that there are examples of offline two-player games where the Nash gap is not vanishing even when the transition probability matrix is exactly known. This highlights an interesting difference between multi-agent and single-agent imitation learning, since for the latter it was known that behavior cloning can be optimal. Then Section 4 discusses alternative strategies and information that can be used to again close the gap.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper is very well written and provides necessary background, literature review, and proof sketch for people who are not intimately familiar with the topic. I haven't checked the proofs, but the intuitions seem to make sense. The paper only deals with the tabular setting and some of the results are expected, so I suppose that the proofs are not very technical sophisticated/challenging.\n\nQUESTIONS:\nJust to make sure I understand the setup, the reason for the counterexample in Theorem 3.2 is that even if the algorithm knows the (action,state)-> state transition exactly, it does not know the reward function r, so that if the trajectory deviates from the equilibrium, the algorithm will not know how to act, is that correct?\n\nLIMITATIONS:\nThe experimental results seem to be artificially constructed Markov chains. Is there any practical settings where the framework and theoretical results in this paper can be applied? Also, the present results are limited to the tabular settings, and there is a gap to the widely used deep learning models.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work concerns multi-agent imitation learning and, more specifically, learning equilibria of zero-sum Markov games (MG) using a dataset of trajectories. I.e., an algorithm is provided with a number of trajectories sampled from a Nash equilibrium, and it needs to output a policy for each agent s.t. they jointly form a Nash equilibrium. The authors further demonstrate the impossibility of this task when the dataset does not satisfy a rather strict condition. To circumvent this, the authors allow *interactive* imitation learning, meaning that the algorithms is allowed to query the expert (who collected the dataset in the first place) for further demonstrations. In the latter setting, the authors manage to offer two algorithmic solutions for the task of equilibrium learning from data with only polynomial sample complexity. Further, behavior cloning and its shortcomings is extensively studied in this paper. Finally, the authors provide some experimental verification of their theoretical claims.\n\nBehavior cloning is a common heuristic to tackle the problem in single-agent Markov decision processes (and also a previous multi-agent work Tang et al 2024). In behavior cloning, one needs to compute a maximum likelihood estimate of the two player policies for that particular dataset of trajectories. Under a bounded concentration coefficient, this works well and the authors prove this in Theorem 3.1. \n\nThe contributions further include:\n* a clear separation between single-agent and multi-agent imitation learning without interactions (Theorem 3.2). The authors construct a MG where behavior cloning will output a pair of policies with exploitability lower bounded by a constant of the game. \n\n* using an interactive expert assumption to design two algorithms for Nash equilibrium learning. The first algorithm assumes access to a policy best-response oracle and the second one circumvents by tweaking the first algorithm.\n\n\nIn Algorithm 1,\n* at every iteration $k$, the best-response oracle is queried for a best response to minimizer's policy $\\mu_k$ and maximizer's policy $\\nu_k$.\n\n* then, the algorithm queries the expert to provide trajectory samples.\n\n* the samples are used to construct a feedback vector which will in turn be used with a policy optimization update (exponential weight updates).\n\n* the algorithm samples a joint strategy from all iteration and outputs it.\n\nThe guarantee is that in expectation, the output is an approximate Nash equilibrium. (Theorem 4.1)\n\n\nAlgorithm 2 essentially goes beyond the need of a best response oracle by using an upper confidence bound value iteration subroutine. This extension requires non-trivial technical work.\n\nSTRENGTHS AND WEAKNESSES:\nAmong the strengths of this paper are its clear-cut improvement from previous work (Tang et al 2024). They set apart single-agent and multi-agent imitation learning with a clean and simple MG example. The separation is robust to even the strong assumption of knowledge of the game dynamics; the corresponding result by (Tang et al 2024) was not as strong as knowledge of the dynamics was enough to ameliorate things.\n\nHaving proven the impossibility of learning a policy pair whose exploitability does not depend on the *single policy deviation concentrability coefficient*, they consider the setting of interactive imitation learning. Then, they provide a strong algorithmic solution to the problem of learning a Nash equilibrium. The algorithm and its guarantees' proof make use of a broad array of the theoretical toolbox of RL and learning theory.\n\nTo the paper's weaknesses, I would list, the weak empirical evaluation without any code. There is no benchmark. Further, the setting the authors consider is only tabular MGs and they did not offer any function approximation guarantees.\n\nQUESTIONS:\n* Can you reduce MAIL to a (potentially) general-utility Markov decision process? (single agent) What are the connections to convex MDPs or convex Markov games? (Gemp et al 2024)\n\n* In algorithm 2 for uncertain states, why did you not pick the KL divergence between $\\mu_E$ and $\\mu_k$ (lines 266-271)?\n\n* Would you be able to extend your results to games with some low rank interaction assumption, e.g. (Kalogiannis and Panageas 2023, Park et al 2023, Zhan et al 2024)? \n\n* Does the uniqueness of the equilibrium of zero-sum games play an crucial role in your results so much so that it would not extend your results as asked by previous question?\n\n* What would be some further benchmarks for this setting?\n\n\n\nGemp, I., Haupt, A.A., Marris, L., Liu, S. and Piliouras, G., Convex Markov Games: A New Frontier for Multi-Agent Reinforcement Learning\n\nZhan, W., Fujimoto, S., Zhu, Z., Lee, J.D., Jiang, D.R. and Efroni, Y., 2024. Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank. \n\nPark, C., Zhang, K. and Ozdaglar, A., 2023. Multi-player zero-sum markov games with networked separable interactions.\n\nKalogiannis, F. and Panageas, I., 2023. Zero-sum polymatrix markov games: Equilibrium collapse and efficient computation of nash equilibria\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}