Title: On Feasible Rewards in Multi-Agent Inverse Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Multi-Agent Reinforcement Learning

Inverse Reinforcement Learning

Multi-Agent Inverse Reinforcement Learning
Abstract: Multi-agent inverse reinforcement learning (MAIRL) aims to recover agent reward functions from expert demonstrations. We characterize the feasible reward set in Markov games, identifying all reward functions that rationalize a given equilibrium. However, equilibrium-based observations are often ambiguous: a single Nash equilibrium can correspond to many reward structures, potentially changing the game's nature in multi-agent systems. We address this by introducing entropy-regularized Markov games, which yield a unique equilibrium while preserving strategic incentives. For this setting, we provide a sample complexity analysis detailing how errors affect learned policy performance. Our work establishes theoretical foundations and practical insights for MAIRL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical analysis of reward recovery feasibility in multi-agent inverse reinforcement learning (MAIRL) within Markov games, using two behavioral game theory equilibrium concepts: (i) Nash equilibrium and (ii) Quantal Response Equilibrium (QRE). It demonstrates that under the Nash setting, access to a single equilibrium policy may be insufficient for accurate reward recovery, as the resulting policy can exhibit a large optimality gap (Proposition 3.4). In contrast, under the QRE setting, the paper establishes sample complexity bounds for estimating transition dynamics and agents' QRE policies, ensuring reward recovery meets optimality guarantees (Theorem 3.9). Additionally, Theorem 4.1 identifies conditions under which a player’s average reward can be recovered from observations of two distinct opponent policies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The paper offers a thorough theoretical analysis of the feasibility, ambiguity, and complexity of reward recovery in Markov games. The analysis is grounded in well-established equilibrium concepts—Nash and Quantal Response Equilibrium (QRE)—from competitive game theory.\n+ The writing is clear and well-organized.\n\nWeaknesses:\n− The connection between the Nash and QRE analyses lacks consistency. While the Nash analysis applies to general Markov games with two or more players, the QRE results are restricted to the two-player setting.\n\n− In Theorem 4.1, it is unclear under what conditions the assumption—that two distinct opponent policies yield the same average reward for the player—can be realistically satisfied.\n\n− In Theorem 3.9, all equations and definitions referred should be included in the main paper, not in the appendix.\n\nQUESTIONS:\n1. Can you please justify the assumption of the same average reward function in Theorem 4.1?\n\n2. Can you elaborate on how your analysis on QRE can be extended to more than 2 players?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the Inverse RL problem in the multi-agent setting (MAS) where they formalize the problem of learning the joint reward function (feasible reward set) from the expert demonstrations (at some equilibrium). They formalize the added complexities that arise due to the multi-agent setting, in particularly, they present an important result that states a observations from a singe Nash equilibrium can be insufficient for a identifying feasible reward set. This is due to the nature of multi-agent settings where there can be multiple Nash-equilibria and it can result in a Nash of order $(1-\\gamma)^-1$ (Prop 3.4). \n\nThe paper then presents a method to mitigate this issue via building on the recent work in single-agent entropy-regularized IRL in the multi-agent IRL setting and Quantile Response Equilibria (QRE). For the QRE, they provide the analysis in terms of error and sample-complexity bounds for finding feasible rewards (Theorem 3.7, Thm 3.9). \n\nThen the authors ask if the reward identifiability even feasible in MAS and propose that for general-sum Markov Games, it is only possible for the average reward case where the reward structure has a particular form, i.e. when they are linearly separable (Prop 4.3).\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths \n- The paper is clearly written and does the important task of characterizing the MAIRL problem which is important given the agentic trends. \n- The paper is rigorous - assumptions are laid clearly, and the results and analysis does a good job of characterizing the findings. \n- The main result shows the hardness of the MAIRL problem and lays open the ground for future opportunities and problems in this domain. \n\n# Weakness\n- The main weakness of the paper is Sec 3.2 where the paper switches the setting to Entropy-regularized MGs and QRE. While the authors provide some motivation of working with QRE, it is unclear why wasn't this introduced in the beginning (instead of NE). That would make the paper more consistent. For instance, how does QRE changes the optimality gape (Definition 3.3) that was defined earlier? \n- A similar concern also lies with switching from regular rewards to average rewards based criterions (Eq 2). Why we need this? How does it impact the general setting? Any other implications?\n\nQUESTIONS:\n- My suggestion will be to bringing the QRE and average reward definitions in technical background, and then work with consistent setting throughout the paper. If that seems quite a big delta from the current version, a more detailed section describe the need of switching the setting will help the reader and make paper more accessible.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies feasible‐reward sets in Multi‐Agent Inverse Reinforcement Learning (MAIRL), showing that inferring rewards from a single observed Nash equilibrium is fundamentally ambiguous, different reward assignments can admit novel, exploitable equilibria. To address this, the authors first define a multi‐equilibrium feasible‐reward set (intractable in general) and then introduce an entropy‐regularized framework yielding a unique quantal‐response equilibrium (QRE). They propose a new optimality criterion, the Nash Imitation Gap, to mirror the value‐gap objective in single‐agent IRL, and under the QRE they derive an explicit characterization of all reward functions consistent with the expert policy. For linearly decomposable rewards, identification is possible up to additive constants (as in single‐agent IRL), and they prove polynomial sample‐complexity bounds for recovering these rewards from finite data. Finally, they also establish a negative result: without entropy regularization, reward identifiability is not possible in the general MAIRL setting.\n\nSTRENGTHS AND WEAKNESSES:\nBesides some room for improved clarity, as noted above, the paper is well written and pleasant to read. The authors explain the topic with attention to detail and make room to elaborate on the underlying intuition. They do a great job capturing the essence of the MAIRL problem and explaining if, when, and how feasible-reward identification is possible -- delivering what a reader would expect from a paper with this title. They also introduce valuable tools, such as their optimality criterion, that future work can build upon. I believe this work makes a valuable contribution by strengthening the foundations of MAIRL and will be a relevant read for anyone entering the field. That is why I have emphasized clarity and the placement of key information in the main text: clear, unambiguous presentation is particularly important when building a theoretic foundation, and especially helpful for newcomers. However, few of the issues I have raised detract from the paper’s contributions; if the content-sensitive typos are corrected or clarified, I consider this work a valuable contribution.\n\nQUESTIONS:\n### Remarks and Questions \n1. Content-Sensitive Typos\n\t- In the \"Mathematical Background\" paragraph, the definition of the Hausdorff (pre)metric contains two $\\inf_{\\sup_{y\\in\\mathcal{Y}}}$ which are supposed to be $\\inf_{y\\in\\mathcal{Y}}$.\n\t- In line 180 \"...player 2 plays action $a_{2}$ ...\" should be \"action $b$\" instead to align with the following expressions.\n\t- In lines 109 and 181 the $\\mathcal{A}_{i}$ should be $\\mathcal{A}^{i}$ to be consistent with the notation introduced in the Preliminaries/Markov Games paragraph.\n\t- In line 110 the expectation is missing the condition on the initial state $s$.\n\t- In line 192 the closing parenthesis on the superscript of the last $V$ is aligned with the $Nash$ instead of the $\\pi$. \n\t- In line 362 the cross-reference to \"Section 4\"  should refer to the appendix?\n2. Some elements would benefit from some notational clarity. To give some examples:\n\t - On line 88, the probability $\\mathbb{P}$ should include a time index (e.g. $\\mathbb{P}^t$) to indicate that it transitions from the initial state to $(s,\\mathbf{a})$ by executing the policy for $t$ steps.\n\t - E.g. in lines 98 and 110 it would increase clarity to add a time index to variables where applicable ($s$, $a$, $b$, etc.) or a separate notation for random variables and realizations.\n\t - The paragraph \"Entropy Regularized Markov Games\" would benefit from adding that it is described in the setting of a two-player game.\n\t - The action set of player 2 is sometimes referred to as $\\mathcal{A}^{2}$ and sometimes as $\\mathcal{B}$, while both refer to the same set, I would suggest using the same notation as synonyms might spark a first thought of -- something is different here? -- when its in fact the same object.\n\t While most of these issues can be understood in the context and it is mostly clear from the context what is meant by the expressions or symbols, I think that it would improve clarity and ease of access.\n3. Figure 2 needs a more thorough description to give context to the values shown as well as the \"Stag Hunt MG\". Another option would be to move this figure to E.1.\n4. There are some consistency related issues. The two things that stand out the most are the mixed usage of \"Markov Games\" (capitalized) and \"Markov games\" (lowercase) as well as the inconsistent capitalization in section and paragraph titles. Lastly, the use of acronyms versus their expanded counterparts is inconsistent. Fixing these little issues would further improve the otherwise well written text.\n5. In Theorem 3.7, is $TV(\\nu, \\hat{\\nu})$ the total variation distance?\n6. The insights from the future-work and limitations sections should be integrated into Section 5 of the main paper, as they offer valuable and relevant information on the topic. \n\n##### Typos:\n- Description of Figure 1: Plural \"..sets of Nash equilibira...\"\n- Line 120: considers -> consider\n- Line 128: Typo in equilibrium\n- Line 309: agent' 1 -> agent 1's ?\n- Line 367: showed -> shown\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the problem of inverse reinforcement learning (IRL) in Markov games (MG). They start with defining a feasible reward set for a given MG as the set of reward functions under which a single observed Nash equilibrium (NE) policy has higher value than the policies that can be obtained by unilateral deviations. The authors observe the existence of degenerate MGs for which a NE policies of a non-constant reward function in the feasible reward set might have Nash Imitation Gap of order $(1 - \\gamma)^{-1}$. This stems from the fact that the feasible reward set captures many reward functions, some of which allows undesirable scenarios where the nature of the game is changed. The authors argue that, although one can address this issue by requiring a feasible reward function to align with every NE of the original game, this would lead to tractability issues. In order to mitigate these issues, they investigate the entropy-regularized 2-player MGs. In this setting, they show that the sample complexity of the problem is of order $\\mathcal{O}\\left(\\gamma^2 (1 - \\gamma)^{-4} |\\mathcal{S}| |\\mathcal{A}| |\\mathcal{B}| \\Delta_{\\min}^{-4} \\right)$, where $\\mathcal{A}$ and $\\mathcal{B}$ are the action spaces of agents, $\\mathcal{S}$ is the state space and $\\gamma$ is the discount factor, and $\\Delta_{\\min}$ is a lower bound on the lowest probability of an action. While this shows an exponential dependence on the number of players, the authors point out that this matches the complexity of learning an NE, which is known to have such exponential dependence in the worst case. Finally, the authors focus on the identifiability problem and note that when the other agent's policy is fixed, the 2-agent game reduces to an MDP, which allows the single-agent IRL theory to be used. The authors thus provide a condition under which the average reward can be identified up to a constant. Finally, the authors show that reward identification is possible in linearly separable Markov games.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality and Clarity:** The paper presents a strong and methodologically sound contribution. The results and the ideas are clearly explained. Some experimental results were presented; as an addition, it would be nice to see an experiment designed to observe how identifiability differs in general games and linearly separable games.\n\n**Significance and Originality:** This paper extends the theoretical line of work on multi-agent inverse reinforcement learning (MAIRL) under varied game-structure assumptions. It demonstrates the inherent reward ambiguity that arises when only a single equilibrium is observed, and analyses the identifiability of reward functions in entropy-regularized games, both with and without the assumption of linearly separable rewards.\n\n**Minor issues:**\n1. typo in Definition 3.1: equilirbium\n2. the dependence of $\\mathcal{E}$ on $R$ is rather unclear at first sight, e.g. in the inequalities in Definition 3.3. I would consider making the dependence explicit.\n3. There are some related works that were not cited in the work, including: [1], [2]\n\n[1] X. Lin, S. C. Adams, and P. A. Beling, ‘Multi-agent Inverse Reinforcement Learning for Certain General-sum Stochastic Games’, _Journal of Artificial Intelligence Research_, vol. 66, pp. 473–502, Oct. 2019.  \n[2] J. Liao, Z. Zhu, E. X. Fang, Z. Yang, and V. Tarokh, ‘Decoding Rewards in Competitive Games: Inverse Game Theory with Entropy Regularization’, in _Forty-second International Conference on Machine Learning_, 2025.\n\nQUESTIONS:\n1. As of Section 3.2, the authors focuses two-player MGs for better readability (line 234). Is it possible to straightforwardly extend the analysis to n-agent case?\n2. Theorem 3.7 provides an upper-bound on the estimated rewards for two MAIRL problems. This allows the quantification of the difference between the original and recovered MAIRL problems. Could your analysis lead to a lower bound on the difference of estimated rewards as well to understand the fundamental limitations of the problem?\n3. Does your results require knowing $\\lambda$ for entropy-regularized Markov games? If so, could you discuss how restrictive is this assumption? Is it possible to obtain a bound similar to Theorem 3.7 for when there is a mismatch between the true $\\lambda$ and the one used for solving the MAIRL problem?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}