Title: Multi-Agent Imitation Learning: Value is Easy, Regret is Hard
Venue: NeurIPS 2024
Accepted: yes
Keywords: imitation learning

multi-agent reinforcement learning
Abstract: We study a multi-agent imitation learning (MAIL) problem where we take the perspective of a learner attempting to *coordinate* a group of agents based on demonstrations of an expert doing so. Most prior work in MAIL essentially reduces the problem to matching the behavior of the expert *within* the support of the demonstrations. While doing so is sufficient to drive the *value gap* between the learner and the expert to zero under the assumption that agents are non-strategic, it does not guarantee robustness to deviations by strategic agents. Intuitively, this is because strategic deviations can depend on a counterfactual quantity: the coordinator's recommendations outside of the state distribution their recommendations induce. In response, we initiate the study of an alternative objective for MAIL in Markov Games we term the *regret gap* that explicitly accounts for potential deviations by agents in the group. We first perform an in-depth exploration of the relationship between the value and regret gaps. First, we show that while the value gap can be efficiently minimized via a direct extension of single-agent IL algorithms, even *value equivalence* can lead to an arbitrarily large regret gap. This implies that achieving regret equivalence is harder than achieving value equivalence in MAIL. We then provide a pair of efficient reductions to no-regret online convex optimization that are capable of minimizing the regret gap *(a)* under a coverage assumption on the expert (MALICE) or *(b)* with access to a queryable expert (BLADES).

Multi-Agent Imitation Learning: Value is Easy, Regret is Hard

Abstract

We study a multi-agent imitation learning (MAIL) problem where we take the perspective of a learner attempting to coordinate a group of agents based on demonstrations of an expert doing so. Most prior work in MAIL essentially reduces the problem to matching the behavior of the expert within the support of the demonstrations. While doing so is sufficient to drive the value gap between the learner and the expert to zero under the assumption that agents are non-strategic, it does not guarantee robustness to deviations by strategic agents. Intuitively, this is because strategic deviations can depend on a counterfactual quantity: the coordinator's recommendations outside of the state distribution their recommendations induce. In response, we initiate the study of an alternative objective for MAIL in Markov Games we term the regret gap that explicitly accounts for potential deviations by agents in the group. We first perform an in-depth exploration of the relationship between the value and regret gaps. First, we show that while the value gap can be efficiently minimized via a direct extension of single-agent IL algorithms, even value equivalence can lead to an arbitrarily large regret gap. This implies that achieving regret equivalence is harder than achieving value equivalence in MAIL. We then provide a pair of efficient reductions to no-regret online convex optimization that are capable of minimizing the regret gap (a) under a coverage assumption on the expert ( MALICE ) or (b) with access to a queryable expert ( BLADES ).

1 Introduction

We consider the problem of a mediator learning to coordinate a group of strategic agents via recommendations of actions to take without knowledge of their underlying utility functions (e.g. routing a group of drivers through a road network). Given the difficulty of manually specifying the quality of a recommendation in such situations, it is natural to provide the mediator with data of desired coordination behavior, turning our problem into one of multi-agent imitation learning (MAIL, [27, 6, 19, 26, 11]). In our work, we explore the nuances of a fundamental MAIL question:

What is the right objective for the learner in a multi-agent imitation learning problem?

We can begin to answer this question by exploring the following scenario: consider developing a routing application to provide personalized route recommendations ( σ ) to a group of users with joint policy π (e.g. the routing policy that underlies the recommendations provided in Google Maps [3]). As usual in imitation learning (IL), we assume we are given access to demonstrations from an expert σ E (e.g. a past iteration of the application). We can imagine two kinds of users of our application (i.e.

agents ): non-strategic users who blindly follow the recommendations of our routing application and strategic users who will deviate from our recommendations if they have the incentive to do so under their (unknown) personal utility function (e.g. we recommend a long route to a busy driver). We use J i ( π σ ) below to denote the value of the mediator's learned policy σ under the i th agent's utility.

Case 1: No Strategic Agents. In the idealized situation where all agents in the population are perfectly obedient, we can essentially treat a MAIL problem as a single-agent IL (SAIL) problem over joint policies. It is therefore natural to use a direct extension of the well-studied value gap criterion from the SAIL literature [1, 28, 21, 24, 22, 23, 25, 16] to the multi-agent setting:

Intuitively, driving the value gap to 0 (i.e. achieving value equivalence in the terminology of [10]) implies that along as long as all agents blindly follow our recommendations, we have learned a policy that performs at least as well as that of the expert from the perspective of any agent in the population. In our running routing application example, this means that if no driver deviates from the previous behavior, all drivers will be at least as happy as they were with the prior iteration of the application.

Case 2: Strategic Agents. Of course for any MAIL problem where agents actually have agency , we need to account for the fact that agents may deviate from our recommendations if it appears beneficial to do so from their subjective perspective. Let us denote the class of deviations (i.e. policy modifications) for agent i as Φ i . Then, we can define the regret induced by the mediator's policy as

where ϕ i is a strategic deviation of agent i and π σ,ϕ i is the joint agent policy induced by all agents other than i following σ 's recommendations . Intuitively, regret captures the maximum incentive any agent in the population has to deviate from the mediator's recommendations. We can then compare this metric between the expert and learner policies to arrive at the notion of a regret gap [27]:

Driving the regret gap to zero (i.e. achieving regret equivalence ) implies that even if agents are free to deviate, our learned policy is at least as good as the expert's from the perspective of an arbitrary agent in the population . In our preceding example, this means that despite the fact that they are not forced to follow our application's recommendations, all agents would have no more incentive to take an alternate route than they did under the previous iteration of the application.

A simple decomposition allows us to show that a small value gap does not in general imply a small regret gap. Consider the performance difference between the learner's policy under all obedient ( J i ( π σ ) ) and a deviating i th agent ( J i ( π σ,ϕ i ) ). We can decompose this quantity into the following:

where we use π σ E ,ϕ i to denote agent joint behavior under expert recommendations and deviation ϕ i . Term III is the standard single-agent value gap (i.e. the performance difference under the assumption that no agents deviate). Term II is the expert's regret under deviation ϕ i (i.e. a quantity we cannot control). Thus, the difference between the regret gap and value gap objectives can be boiled down to Term I: J i ( π σ,ϕ i ) -J i ( π σ E ,ϕ i ) . Observe that because of the state distribution shift induced by deviation ϕ i , minimizing Term III doesn't give us any guarantees with respect to Term 1. This underlies our key insight: regret is hard in MAIL as it requires knowing what the expert would have done in response to an arbitrary agent deviation. More explicitly, our contributions are three-fold:

We initiate the study of the regret gap for MAIL in Markov Games. Unlike the value gap - the standard objective in single-agent IL - the regret gap captures the fact that agents in the population may choose to deviate from the mediator's recommendations. The shift from value to regret gap captures what is fundamentally different about the SAIL and the MAIL problems.

2. We investigate the relationship between regret gap and the value gap. We show that under the assumption of complete reward and deviation function classes, regret equivalence implies value equivalence. However, we also prove that value equivalence provides essentially no guarantees on the regret gap, establishing a fundamental limitation of applying SAIL algorithms to MAIL problems.

3. We provide a pair of efficient algorithms to minimize the regret gap under certain assumptions. While regret equivalence is hard to achieve in general as it depends on counter-factual expert recommendations, we derive a pair of efficient reductions for minimizing the regret gap that operate under different assumptions: MALICE (which operates under a coverage assumption) and BLADES (which requires access to a queryable expert). We prove that both algorithms can provide O ( H ) bounds on the regret gap, where H is the horizon, matching the strongest known results for the value gap in single-agent IL. See Table 1 for a summary of our regret gap bounds.

2 Related Work

Single-Agent Imitation Learning. Much of the theory of imitation learning focuses on the singleagent setting [14]. Offline approaches like behavioral cloning (BC, [15]) reduce the problem of imitation to mere supervised learning. Ignoring the covariate shift in state distributions between the expert and learner policies can cause compounding errors [17, 21] and associated poor performance. In response, interactive IL approaches like inverse reinforcement learning (IRL, [1, 28]) allow the learner to observe the consequences of their actions during the training procedure, preventing compounding errors [21]. However, such approaches can be rather sample-inefficient due to the need to repeatedly solve a hard RL problem [25, 16]. Alternative approaches include interactively querying the expert to get action labels on the learner's induced state distribution (DAgger, [17]) or, assuming full coverage of the demonstrations, using importance weighting to correct for the covariate shift (ALICE, [20]). Our BLADES and MALICE algorithms can be seen as the regret gap analog of the value gap-centric DAgger and ALICE algorithms, operating under the same assumptions.

Multi-Agent Imitation Learning. The concept of the regret gap was first introduced in the exceptional work of Waugh et al. [27], though their exploration was limited to Normal Form Games (NFGs), in contrast to the more general Markov Games (MGs) we focus on. Fu et al. [7] briefly consider the regret gap in Markov Games (MGs) but do not explore its properties nor provide algorithms for efficient minimization. Most empirical MAIL work [19, 12, 4, 26, 11] is value gap-based, while we take a step back and ask what the right objective is for MAIL in the first place.

Inverse Game Theory. Another line of work focuses on inverse game theory in Markov Games [13, 8], where the goal is to recover a set of utility functions that rationalize the observed agent behavior, rather than learning to coordinate from demonstrations. A detailed comparison between the goals of our work at that of inverse game theory provided in Appendix F.

3 Preliminaries

We begin with the notation we will use in our paper. Throughout, we use ∆( X ) denote the space of probability distribution over a set X . We will use ℓ to denote the loss function each algorithm optimizes, which should be thought of as a convex upper bound on the total variation distance TV . We use ℓ TV when the loss function is exactly the TV distance.

Markov Games. We use MG ( H, S , A , T , { r i } i m =1 , ρ 0 ) to denote a Markov Game (MG) between m agents. Here, H is the horizon, S is the state space, and A = A 1 × ... ×A m is the joint action space for all agents. We use T : S × A → ∆( S ) to denote the transition function. Furthermore, the reward (utility) function for agent i ∈ [ m ] is denoted by r i : S × A → [ -1 , 1] . Lastly, we use ρ 0 to denote the initial state distribution from which the initial state s 0 ∼ ρ 0 is sampled.

Learning to Coordinate. Rather than considering the problem of learning individual agent policies in the MG, we take the perspective of a mediator who is giving recommendations to each agent to help them coordinate their behavior (e.g. a smartphone mapping application providing directions to a set of users). At each time step, the mediator gives each agent i a private action recommendation a i to take at the current state s . Critically, no agent observes the recommendations the mediator provides to another agent. We can represent the mediator as a Markovian joint policy σ ∈ Σ , where σ : S → ∆( A ) . We use σ ( ⃗ a | s ) to denote the probability of recommending joint action ⃗ a in state s . We use π : S → ∆( A ) to denote the joint policy that agents play in response to the mediator's policy. When agents exactly follow the mediator's recommendations, we denote their joint policy as π σ .

A trajectory ξ ∼ π = { s h , ⃗ a h } h =1 ,...,H refers to a sequence of state-action pairs generated by starting from s 0 ∼ ρ 0 and repeatedly sampling joint action ⃗ a h and next states s h +1 from π and T for H -1 time steps. Let d π h denote the state visitation distribution at timestep h following π and let d π = 1 H ∑ H h =1 d π h be the average state distribution. Let ρ π h ( s h , ⃗ a h ) denote the occupancy measure -i.e., probability of reaching state s and then taking action ⃗ a at time step h . By definition, we know that ∀ h , ∑ s,⃗ a ρ π h ( s,⃗ a ) = 1 . Let ρ π ( s,⃗ a ) = 1 H ∑ H h =1 ρ π h ( s,⃗ a ) be the average occupancy measure.

We use V π i,h to denote the expected cumulative reward of agent i under this policy from time step h , i.e. V π i,h ( s ) = E ξ ∼ π [ ∑ H t = h r i ( s t , ⃗ a t ) | s h = s ] . We define Q-value function of agent i as Q π i,h ( s,⃗ a ) = E ξ ∼ π [ ∑ H t = h r i ( s t , ⃗ a t ) | s h = s,⃗ a h = ⃗ a ] . We define advantage of an agent i to be the difference between its Q-value on a selected action and the V-value on the state, i.e. A π i,h ( s,⃗ a ) = Q π i,h ( s,⃗ a ) -V π i,h ( s ) . We also define the performance of a policy π from the perspective of agent i as J i ( π ) = E s 0 ∼ ρ 0 [ E ξ ∼ π [ ∑ H t =1 r i ( s t , ⃗ a t ) | s = s 0 ]] . Observe that performance is the inner product between the occupancy measure and the agent's reward function, i.e. J i ( π ) = H ∑ s,⃗ a ρ π ( s,⃗ a ) r i ( s,⃗ a ) .

̸

Correlated Equilibria. We now introduce the notion of a correlated equilibrium (CE, Aumann [2]). First, we define a strategy deviation ϕ i for the i -th agent as a map ϕ i : S × A i → A i . Intuitively, a strategy deviation captures how the agent responds to the current state of the world and the recommendation of the mediator - they can either obey (in which case ϕ i ( s, a ) = a ) or defect (in which case ϕ i ( s, a ) = a ). Let Φ i be the set of deviations for agent i , which is a subset of all possible deviations. We use Φ := { Φ i } i m =1 to denote deviations for all agents. We assume that for all i , the identity mapping ϕ i ( s, a ) ≡ a is in Φ i . We use π σ,ϕ i to denote ( ϕ i ◦ π σ,i ) ⊙ π σ, -i : the joint agent policy induced by mediator policy σ being over-ridden by deviation ϕ i . We can now define a CE.

Definition 3.1 (Regret and CE in General-Sum MGs) . Let σ ∈ Σ be the mediator's policy in a Markov Game, and Φ i , i ∈ [ m ] be the deviation classes for each agent. Then,

We define the regret of a mediator policy σ to be

2. We say a mediator with policy σ induces an ϵ -approximate Correlated Equilibrium (CE) if

Intuitively, regret captures the maximum utility any agent can gain by defecting from the mediator's recommendation. A CE is an induced joint policy where no agent has a large incentive to deviate.

4 On the Relationship between the Value Gap and the Regret Gap

As sketched above, we consider two potential objectives for the learner in MAIL:

Definition 4.1 (Value Gap) . We define the value gap between the expert's policy σ E and the learner's policy σ ∈ Σ as

Definition 4.2 (Regret Gap) . We define the regret gap between the expert's policy σ E and the learner's policy σ ∈ Σ as

Regret

Equivalence

Complete Reward / Deviation Classes

Value

Equivalence

We say that the learner's policy satisfies value / regret equivalence when the value / regret gap is 0 . We now explore the relationship between the value and regret gap in MAIL, 1 summarized in Figure 1. We use J i ( π σ , f ) and R Φ ( σ, f ) to denote the value/regret of policy σ under the reward function f .

4.1 Regret Equivalence + Complete Reward / Deviation Class = ⇒ Value Equivalence

First, we show that if the reward function class and deviation class are both complete , then regret equivalence implies value equivalence. We say that the reward function class is complete when F = {S × A → [ -1 , 1] } (i.e. all convex combinations of state-action indicators), and that the deviation class is complete if for every agent i , Φ i = {S × A i →A i } (i.e. all possible deviations).

Theorem 4.1 (Complete Classes) . If the reward function class F and deviation class Φ are complete and regret equivalence is satisfied (i.e. sup f ∈F ( R Φ ( σ, f ) -R Φ ( σ E , f )) = 0 ), then value equivalence is also satisfied: sup f ∈F max i ∈ [ m ] ( J i ( π σ E , f ) -J i ( π σ , f )) = 0 . [Proof]

Next, we prove that large classes are needed for this implication to hold true.

̸

Theorem 4.2 (Incomplete Classes) . There exists an MG, an expert policy σ E , and a trained policy σ such that even though the regret equivalence is satisfied under the true reward function r , i.e. R Φ ( σ, r ) -R Φ ( σ E , r ) = 0 , the value gap max i ∈ [ m ] ( J i ( π σ E , r ) -J i ( π σ , r )) = 0 . [Proof]

Together, these results tell us that with an expressive enough class of reward functions / deviations, regret equivalence is stronger than value equivalence. We now turn our attention to the converse.

̸

4.2 Value Equivalence = ⇒ Regret Equivalence

We now show a surprising result: value equivalence does not directly imply a low regret gap! In the worst case, value equivalence fails to provide any meaningful guarantees on the regret gap. This reveals a critical distinction between SAIL and MAIL not fully addressed in the prior work.

Theorem 4.3. There exists a Markov Game, an expert policy σ E , and a learner policy σ , such that even occupancy measure of π σ exactly matches π σ E , i.e. ∀ ( s,⃗ a ) , ρ π σ ( s,⃗ a ) = ρ π σ E ( s,⃗ a ) (i.e. we have value equivalence under all rewards), the regret gap R Φ ( σ ) -R Φ ( σ E ) ≥ Ω( H ) . [Proof]

We leave the details of the proof for this theorem in Appendix E.3. As visualized in Figure 2, both the expert and learner policies only visit the states in the lower path s 2 , s 4 , ..., s 2 H -2 . The trained policy perfectly matches the occupancy measure of the expert by taking identical actions in visited states s 2 , s 4 , ..., s 2 H -2 . However, expert demonstrations lack coverage of state s 1 as it is unreachable by executing π E . This omission becomes critical when agent 1 deviates from the original policy, making s 1 unreachable with high probability. Consequently, the trained policy may perform poorly in s 1 , in stark contrast to the expert playing a CE under the true reward function. This example highlights the key difference between value equivalence and regret equivalence: the former only depends on states actually visited by the policy, while the latter depends on the counterfactual recommendations the learner would make at unvisited states in response to an agent deviations.

Remark 4.1. As shown in Theorem 4.3, even if the learner has access to infinite samples on the equilibrium path from expert demonstrations, it is possible that the learner remains unaware of the expert's behavior in states unvisited by the expert (but reachable by the deviated agents joint policy). Thus, from an information theoretic perspective, it is impossible for the learner to minimize the regret gap without knowing how the expert would behave on those states. This demonstrates the fundamental difficulty of minimizing the regret gap, and thus, regret is 'hard' in MAIL. We therefore need a fundamentally new paradigm of MAIL algorithm to minimize the regret gap.

...

s

0

s

1

s

2

s

3

s

4

s

5

s

6

s

2

H

-

1

s

2

H

a

2

a

1

else

all

else

a

2

a

1

all

all

...

̸

4.3 Low Regret Gap = ⇒ CE, Low Value Gap = ⇒ CE

Given the deep connections between regret and correlated equilibrium discussed above, it is perhaps intuitive that if the expert σ E is playing a CE, a low regret gap means the learner is as well.

Theorem 4.4 (Regret Gap Implies CE) . If the expert policy σ E induces a δ 1 -approximate CE, and the learner policy σ satisfies R Φ ( σ ) -R Φ ( σ E ) ≤ δ 2 , then σ induces a δ 1 + δ 2 -approximate CE. [Proof]

Then, by combining our preceding result with Theorem 4.3, it follows that a low value gap does not imply that the learner is playing a CE.

Corollary 4.5. There exists a Markov Game, an expert policy σ E , and a learner policy σ , such that σ E induces a δ 1 -approximate CE, and σ satisfies max i ∈ [ m ] ( J i ( π σ E ) -J i ( π σ )) = δ 2 , σ induces a Ω( H ) -approximate CE.

Together, these results imply that if we are interested in inducing a CE amongst the agents in the population, the regret gap is a more suitable objective.

4.4 Efficient Algorithms for Minimizing the Value Gap

Although we have shown that the value gap is a 'weaker' objective in some sense, in many real-world scenarios, the agents may be non-strategic. In these scenarios, minimizing value gap can be a reasonable learning objective. As we will demonstrate here, the natural multi-agent generalization of single-agent IL algorithms can efficiently minimize the value gap-hence, value is 'easy' in MAIL.

Behavior Cloning (BC) and Inverse Reinforcement Learning (IRL) are two single-agent IL algorithms aimed at minimizing the value gap. By running these algorithms over joint policies, we can apply BC and IRL to the multi-agent setting, which we call Joint Behavior Cloning (J-BC) and Joint Inverse Reinforcement Learning (J-IRL). Doing so results in the same value gap bounds as in the single-agent setting. More details on of J-BC and J-IRL can be found in Appendix B.

Theorem 4.6 (J-BC Value Gap Upper Bound) . If J-BC returns a policy σ that satisfies E s ∼ d πσ E [ ℓ ( σ E ( s ) , σ ( s ))] ≤ ϵ , then the value gap max i ∈ [ m ] ( J i ( π σ E ) -J i ( π σ )) ≤ O ( ϵH 2 ) . [Proof]

Theorem 4.7 (J-IRL Value Gap Upper Bound) . If J-IRL outputs a policy σ with moment-matching error

then the value gap max i ∈ [ m ] ( J i ( π σ E ) -J i ( π σ )) ≤ O ( ϵH ) . [Proof]

As argued by Swamy et al. [21], satisfying the conditions for either of the above theorems can be achieved oracle-efficiently via a reduction to no-regret online learning. We now turn our attention to sufficient conditions for there to exist efficient algorithms for minimizing the regret gap.

5 Efficient Algorithms for Minimizing the Regret Gap

In our following analysis, we will make a recoverability assumption: that a single-step agents deviation could at most cost the expert a fixed constant.

Assumption 5.1 ( u -recoverability) . We say that an MG is u -recoverable if the expert advantage function is bounded for all deviations, i.e. ∀ s,⃗ a, h, i, ϕ i , ∣ ∣ ∣ A π σ E ,ϕ i i,h ( s,⃗ a ) ∣ ∣ ∣ ≤ u .

Intuitively, a small value of u means that we're not in a problem where a single agent can deviate and a joint mistake happens that even the expert couldn't recover from for the rest of the episode. In the worst case, u is O ( H ) . However, we believe that in many cases u is small. For instance, in the route planning example, at some point many cars may miss their turns/intersections/exits, but this can be recovered within a constant time, even when a single driver chooses not to follow its recommendation, rather than u increasing as H increases.

This assumption can be thought of natural multi-agent generalization of the standard recoverability assumption in SAIL [18, 21, 20] which is necessary and sufficient to avoid compounding errors while maintaining computational efficiency. While we define recoverability with respect to the actual reward function for implicitly, one can instead easily define it with respect to the worst-case reward function in a class ( sup f ∈F ) -moment recoverability - as in [21] to avoid the need to know the ground truth set of agent reward functions r to bound u .

In Section 4.2, we proved that for general MGs, J-BC and J-IRL don't give any guarantees on the regret gap. Fundamentally, without the ability to observe how the expert would have responded in the counter-factual state induced by a deviation, the learner cannot ensure that they match the expert's regret. We now explore two different sets of assumptions that give us this ability.

5.1 Assumption 1: Full Coverage of Expert Demonstrations

In this section, we introduce a coverage assumption on the expert's state distribution d π σ E ( s ) which states that the expert visits every state with a positive probability. We will show that this assumption is sufficient to give a regret gap guarantee. The state coverage assumption is a common theoretical assumption in the analysis of learning in MDPs/MGs [5] and has been explored in SAIL [20].

Assumption 5.2 ( β -coverage) . There exists a constant β > 0 such that for the expert's policy σ E , it holds that d π σ E ( s ) ≥ β for all s .

Intuitively, this assumption implies that in the infinite sample limit, there are no states where we are unsure what the expert would recommend. As discussed in Remark 4.1, without the ability to interactively query the expert, a coverage assumption is necessary because we cannot minimize the regret gap without knowing the expert mediator's actions in counter-factual states.

We first show that under Assumption 5.2, J-BC and J-IRL get a (relatively weak) regret gap guarantee.

5.1.1 Regret Gaps of J-BC and J-IRL under Full Demonstration Coverage

We begin by analyzing joint behavioral cloning (J-BC).

Theorem 5.1 (J-BC Regret Gap Upper Bound) . Under Assumption 5.1 and Assumption 5.2, if the J-BC algorithm returns a policy σ that satisfies E s ∼ d πσ E [ ℓ ( σ E ( s ) , σ ( s ))] ≤ ϵ , then

[Proof]

We leave the proof in Appendix E.7. It is worth to note that although the dependency of H is linear under our recoverability assumption, we still need to pay for the term 1 β in our regret gap bound. In general, this term can grow exponentially with the horizon, making this guarantee relatively weak. We can show its tightness by slightly modifying the example in Theorem 4.3 to satisfy the assumptions.

Theorem 5.2 (J-BC Regret Gap Lower Bound) . There exists a Markov Game, an expert policy σ E , and learner policy σ such that σ E satisfies Assumption 5.1 and Assumption 5.2, σ achieves BC error

[Proof]

We now prove analogous results for joint inverse reinforcement learning (J-IRL).

Theorem 5.3 (J-IRL Regret Gap Upper Bound) . Under Assumption 5.2 and Assumption 5.1 and with a complete reward function class F , if J-IRL returns a policy σ with moment-matching error

There are two interesting features of this theorem. The first is that we needed to assume that the reward function class is complete - otherwise, a small value gap can still translate to a large regret gap. The second is that the upper-bound for J-IRL matches that for J-BC, which is in stark contrast to the single-agent setting, where IRL enjoys linear-inH guarantees with respect to the value gap [21]. We now show this is not an artifact of our analysis by providing a matching lower bound.

Corollary 5.4 (J-IRL Regret Gap Lower Bound) . There exists a Markov Game, an expert policy σ E , and a policy σ such that σ E satisfies Assumption 5.1 and Assumption 5.2, the trained policy σ gets moment-matching error

This result implies another fundamental distinction between SAIL and MAIL: in contrast to the value gap, interactive training alone is not sufficient to effectively minimize the regret gap.

5.1.2 MALICE : Multi-agent Aggregation of Losses to Imitate Cached Experts

Observe that the upper bounds for both J-BC and J-IRL include a dependence on the inverse of the coverage coefficient 1 β , which can be rather large for problems with long horizons or large action spaces. We now present an efficient algorithm that is able to avoid this dependence by extending the ALICE algorithm [20] to the multi-agent setting. ALICE is an interactive algorithm that, at each round, uses importance sampling to re-weight the behavior cloning (BC) loss based on the density ratio between the current learner policy and that of the expert. Accordingly, ALICE requires a full demonstration coverage assumption to ensure that these importance weights are finite. ALICE uses a no-regret algorithm to learn a policy that minimizes reweighed on-policy error, which guarantees a linear-inH bound on the value gap under a recoverability assumption [20].

In Algorithm 1, we describe Multi-agent ALICE ( MALICE ), where adapt ALICE to the multi-agent setting (i.e. minimizing the regret gap). Specifically, we modify the ALICE loss function to include a maximum over all deviations. This gives us

Since E s ∼ d π E [ d π ˆ σ,ϕ i ( s ) d πσ E ( s ) ℓ ( σ E ( s ) , σ ( s )) ] is a convex loss function, and the maximum of convex functions is still a convex function, we know that ℓ MALICE ( σ, D E , ˆ σ ) is a valid convex loss function with scales in [0 , 1] . As a result, we can run an (arbitrary) no-regret online convex optimization (OCO) algorithm to efficiently optimize it, giving us an efficient reduction from regret gap minimization to no-regret online convex optimization under demonstration coverage .

We now provide regret gap guarantees on the policy returned by MALICE .

Algorithm 1 MALICE (Multi-agent Aggregation of Losses to Imitate Cached Experts)

Theorem 5.5 ( MALICE Regret Gap Upper Bound) . Let σ be a policy such that ℓ MALICE ( σ, D E , σ ) ≤ ϵ . Under Assumption 5.1 and Assumption 5.2, we have

[Proof]

As promised, observe that adapting the importance sampling technique of Spencer et al. [20] to the multi-agent setting allows us to efficiently minimize the regret gap while avoiding an upper bound that depends on the coverage coefficient of the expert demonstrations.

We now show that the bound in Theorem 5.5 is tight by constructing a matching lower bound.

Theorem 5.6 ( MALICE Regret Gap Lower Bound) . There exists a Markov Game, an expert policy σ E that satisfies Assumption 5.1, and a trained policy σ that gets error ℓ TV , MALICE ( σ, D E , σ ) ≤ ϵ , and

[Proof]

We now turn our attention to an alternate assumption and the corresponding regret gap algorithm.

5.2 Assumption 2: Access to a Queryable Expert

For many problems, full coverage of expert demonstrations is not a reasonable assumption. Thus, we explore another natural assumption that allows us to observe expert recommendations at counterfactual states: access to a queryable expert. In their classic DAgger algorithm, Ross et al. [18] showed that access to a queryable expert allows one to eliminate the covariate shift that results from the difference between expert and learner induced state distributions. When we transition to the multi-agent setting, we can again use access to a queryable expert to handle yet another source of covariate shift: potential strategic deviations by agents in the population that push the learner outside of the support of the expert. We refer to our multi-agent extension of DAgger as BLADES .

In each iteration of BLADES , we request the expert to provide recommendations under all possible agent deviations, before training on the aggregated data. More formally, we minimize the following sequence of loss functions:

Similar to MALICE , we know that the loss ℓ BLADES is also a valid convex loss function, and thus we can use a no-regret algorithm to efficiently minimize it. This gives us an efficient reduction from regret gap minimization to no-regret online convex optimization with access to a queryable expert. We now derive and upper and lower bounds on the regret gap of a policy returned by BLADES .

Theorem 5.7 ( BLADES Regret Gap Upper Bound) . Under Assumption 5.1, if a policy σ satisfies ℓ BLADES ( σ, σ ) ≤ ϵ , then

[Proof]

Theorem 5.8 ( BLADES Regret Gap Lower Bound) . There exists a Markov Game, an expert policy σ E , and a trained policy σ such that σ E satisfies Assumption 5.1, σ achieves error ℓ TV , BLADES ( σ, σ ) ≤ ϵ , and

[Proof]

In short, under either a demonstration coverage assumption or with access to a queryable expert, we are able to efficiently minimize the regret gap on a recoverable MAIL problem.

6 Conclusion

Our work focuses on the core question of what fundamentally distinguishes multi-agent IL problems from single-agent ones. In short, our answer is that on problems with strategic agents that are not mere puppets, we need to deal with another source of distribution shift: deviations by agents in the population. This new source of distribution shift cannot be efficiently controlled with environment interaction (i.e. inverse RL). Instead, we need to be able to estimate how the expert would act in counter-factual states. Based on this core insight, we derive two reductions that are able to minimize the regret gap under a coverage or queryable expert assumption. We leave the development and implementation of practical approximations of our idealized algorithms to future work.

8 Contribution Statements

JT uncovered the distinction between the regret and value gaps, proved all of the core results in the paper, and drafted the initial version of the paper.

GS initially proposed the project, came up with the sufficient conditions and associated algorithms for minimizing the regret gap, and wrote most of the final version of the paper.

FF and ZSW advised the project.

References

P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , page 1, 2004.

R. J. Aumann. Correlated equilibrium as an expression of bayesian rationality. Econometrica: Journal of the Econometric Society , pages 1-18, 1987.

M. Barnes, M. Abueg, O. F. Lange, M. Deeds, J. Trader, D. Molitor, M. Wulfmeier, and S. O'Banion. Massively scalable inverse reinforcement learning in google maps. arXiv preprint arXiv:2305.11290 , 2023.

R. P. Bhattacharyya, D. J. Phillips, B. Wulfe, J. Morton, A. Kuefler, and M. J. Kochenderfer. Multi-agent imitation learning for driving simulation. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 1534-1539. IEEE, 2018.

L. Erez, T. Lancewicki, U. Sherman, T. Koren, and Y. Mansour. Regret minimization and convergence to equilibria in general-sum markov games. In International Conference on Machine Learning , pages 9343-9373. PMLR, 2023.

J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248 , 2017.

J. Fu, A. Tacchetti, J. Perolat, and Y . Bachrach. Evaluating strategic structures in multi-agent inverse reinforcement learning. Journal of Artificial Intelligence Research , 71:925-951, 2021.

D. Goktas, A. Greenwald, S. Zhao, A. Koppel, and S. Ganesh. Generative adversarial inverse multiagent learning. In The Twelfth International Conference on Learning Representations , 2023.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

C. Grimm, A. Barreto, S. Singh, and D. Silver. The value equivalence principle for model-based reinforcement learning. Advances in Neural Information Processing Systems , 33:5541-5552, 2020.

C. Gulino, J. Fu, W. Luo, G. Tucker, E. Bronstein, Y. Lu, J. Harb, X. Pan, Y. Wang, X. Chen, et al. Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research. Advances in Neural Information Processing Systems , 36, 2024.

H. M. Le, Y. Yue, P. Carr, and P. Lucey. Coordinated multi-agent imitation learning. In International Conference on Machine Learning , pages 1995-2003. PMLR, 2017.

X. Lin, S. C. Adams, and P. A. Beling. Multi-agent inverse reinforcement learning for certain general-sum stochastic games. Journal of Artificial Intelligence Research , 66:473-502, 2019.

T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends® in Robotics , 7(1-2):1-179, 2018.

D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems , 1, 1988.

J. Ren, G. Swamy, Z. S. Wu, J. A. Bagnell, and S. Choudhury. Hybrid inverse reinforcement learning. arXiv preprint arXiv:2402.08848 , 2024.

S. Ross and D. Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 661-668. JMLR Workshop and Conference Proceedings, 2010.

S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011.

J. Song, H. Ren, D. Sadigh, and S. Ermon. Multi-agent generative adversarial imitation learning. Advances in neural information processing systems , 31, 2018.

J. Spencer, S. Choudhury, A. Venkatraman, B. Ziebart, and J. A. Bagnell. Feedback in imitation learning: The three regimes of covariate shift. arXiv preprint arXiv:2102.02872 , 2021.

G. Swamy, S. Choudhury, J. A. Bagnell, and S. Wu. Of moments and matching: A gametheoretic framework for closing the imitation gap. In International Conference on Machine Learning , pages 10022-10032. PMLR, 2021.

G. Swamy, S. Choudhury, D. Bagnell, and S. Wu. Causal imitation learning under temporally correlated noise. In International Conference on Machine Learning , pages 20877-20890. PMLR, 2022.

G. Swamy, S. Choudhury, J. Bagnell, and S. Z. Wu. Sequence model imitation learning with unobserved contexts. Advances in Neural Information Processing Systems , 35:17665-17676, 2022.

G. Swamy, N. Rajaraman, M. Peng, S. Choudhury, J. Bagnell, S. Z. Wu, J. Jiao, and K. Ramchandran. Minimax optimal online imitation learning via replay estimation. Advances in Neural Information Processing Systems , 35:7077-7088, 2022.

G. Swamy, D. Wu, S. Choudhury, D. Bagnell, and S. Wu. Inverse reinforcement learning without reinforcement learning. In International Conference on Machine Learning , pages 33299-33318. PMLR, 2023.

E. Vinitsky, N. Lichtlé, X. Yang, B. Amos, and J. Foerster. Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world. Advances in Neural Information Processing Systems , 35:3962-3974, 2022.

K. Waugh, B. D. Ziebart, and J. A. Bagnell. Computational rationalization: The inverse equilibrium problem. arXiv preprint arXiv:1308.3506 , 2013.

B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.

A Broader Impacts

As the algorithms we proposed are theoretical, we do not foresee any direct societal concerns resulting from this work. However, these theoretical algorithms can serve as a foundation for developing practical algorithms or provide guidance for designing practical algorithms in MAIL, which could be applied to real world problems in the future.

B Extending Single-Agent IL Algorithms to Minimize the Value Gap

B.1 Multi-Agent Joint Behavior Cloning

Behavioral Cloning (BC, Pomerleau [15]) treats the problem of imitation learning as supervised learning and performs maximum likelihood estimation with expert states as inputs and expert actions as labels. Unfortunately, as first analyzed by Ross and Bagnell [17], the covariate shift between the training (expert states) and test (learner states) distributions can lead to compounding errors - i.e. a value gap that increases quadratically as a function of the horizon H . We note that this is not an artifact of the particular objective used in BC - as argued by Swamy et al. [21], the same can be said for any offline imitation learning algorithm. J-BC extends BC to a multi-agent setting by learning a map from the state space S to the joint action space A . By adapting the analysis of Ross and Bagnell [17] and Swamy et al. [21] to the multi-agent setting, we establish a similar compounding error result for multi-agent behavior cloning in Theorem 4.6. There exists an example of MDP/MG that matches this bound, which shows that the bound is tight [21].

B.2 Multi-Agent Inverse Reinforcement Learning

A popular family of online techniques for imitation learning is inverse reinforcement learning (IRL). Intuitively, IRL can be thought of as being similar to a GAN [9] but in the space of trajectories: the generator is the learner's policy coupled with a world model to actually give us trajectories, while the discriminator is trained between expert and learner trajectories and is used as a reward function for policy updates. More formally, IRL can be viewed as a two-player zero-sum game between a reward player and a policy player [21]. In each round, the reward player picks a reward function from F that maximizes the value gap between σ E and σ , while the policy player uses a reinforcement learning algorithm to learn a new policy in Σ that maximizes the performance under this reward function.

Intuitively, as the learner can see policy rollouts during training procedure, they cannot be 'surprised' by where their policy ends up at test time, removing the covariate shift issue that lies at the heart of compounding errors. More formally, Swamy et al. [21] proved that value gap for single-agent IRL algorithm is O ( ϵH ) . We now generalize this result to the multi-agent setting. Accordingly, our policy class Σ becomes one of joint policies. We use a reward function class F that is identical for all agents (i.e. we assume the the game is common payoff). Then, by following the proof in Swamy et al. [21], we prove a O ( ϵH ) value gap bound for multi-agent IRL algorithm in Theorem 4.7.

Algorithm 3 J-IRL

1: Input: expert demonstration D , Policy class Σ , Reward class F

E 2: Set σ (1) ∈ Σ 3: for n = 1 to N do 4: f ( n ) ← arg max J ( π σ E , f ) -J ( Unif ( π σ (1: n ) ) , f ) + R ( f ) // Treat it as a single-agent RL problem over joint action space under reward function f ( n ) 5: σ ( n +1) ← MaxEntRL ( r = f ( n ) ) 6: end for 7: Return best σ ( n ) on validation

C Useful Lemmas

We introduce a lemma which will be very useful in the analysis under the recoverability assumption. It is used in the analysis in the single-agent DAgger [18] and ALICE [20], and we will also use

it in the analysis for MAIL. It shows that if the policy achieves small on-policy error, then, with recoverability assumption, the value gap is linear over H .

Lemma C.1. [Ross et al. [18]] For agent joint policy π 1 and π 2 , if the advantage of π 1 is bounded under the true reward function ∀ i, h, s,⃗ a, | A π 1 i,h ( s,⃗ a ) | ≤ u , and π 2 get on-policy error E s ∼ d π 2 [ ℓ ( π 1 ( s ) , π 2 ( s ))] ≤ ϵ , then | J i ( π 1 ) -J i ( π 2 ) | ≤ ϵuH, ∀ i ∈ [ m ] .

Proof. Via the performance difference lemma, ∀ i ∈ [ m ] , we have

For our analysis of MALICE and BLADES , we will let π 1 be any deviated expert policy π σ E ,ϕ i and π 2 be the deviated trained policy π σ,ϕ i under the same deviation.

D Equivalence of Regret Gap and Value Gap in Single-Agent IL

For single-agent IL we prove that the regret gap and the value gap are equivalent.

Theorem D.1 (Equivalence in Single-Agent IL) . For single-agent MDP, regret gap and value gap are equivalent to each other

Proof. For single-agent MDP, we ignore the index i in the following proof. A strategy deviation in single-agent MDP is equivalent to taking another policy, because there are no other agents affecting the dynamics of the agent. We have

where π ∗ is the optimal policy under the true reward function. Similarly, we have

Therefore,

In single-agent MDPs, the dynamics are fixed because no other agents affect the agent's dynamics, and therefore, the regret gap is equivalent to the value gap.

E Proofs

Contents

E.1 Proof of Theorem 4.1

Proof. We prove the lemma by showing that the occupancy measures of π σ and π σ E exactly match, i.e. ρ π σ ( s,⃗ a ) = ρ π σ E ( s,⃗ a ) for every ( s,⃗ a ) . Consider a cooperative reward function f s ′ ,⃗ a ′ = -1 ( s = s ′ , ⃗ a = ⃗ a ′ ) .

Under f s,⃗ a , we have J ( π σ ) = -Hρ π σ ( s,⃗ a ) , J ( π σ E ) = -Hρ π σ E ( s,⃗ a ) . The maximum value performance the expert/learner can get after deviation is 0 because the reward function is non-positive. ( 0 can be achieved by simply not taking ⃗ a on s ).

Since R Φ ( σ ) -R Φ ( σ E ) = 0 , we know that ρ π σ ( s,⃗ a ) = ρ π σ E ( s,⃗ a ) . This implies that the occupancy measures of two policies exactly match. As a result,

E.2 Proof of Theorem 4.2

Proof. We can construct an example in normal form games, in which there are mulitple CEs with different pay-offs. We can let the σ E plays CE 1 and σ plays CE 2. Therefore, although the regret gap R Φ ( σ ) -R Φ ( σ E ) = 0 , the value gap max i ∈ [ m ] ( J i ( π σ E ) -J i ( π σ )) = 0 . The NFG in Figure 5 is an example, where ( a 1 , a 1 ) and ( a 2 , a 2 ) are two CEs with different values.

E.3 Proof of Theorem 4.3

Proof. We prove the theorem by constructing such a Markov Game and policies that can get Ω( H ) regret gap. For simplicity, we construct a two-player cooperative game where the reward is identical for all agents. Agents can not visit the same state at different time steps. These allow us to omit the index i in the reward function in the proof. The notation a i a j is used to represent the action pair ( a i , a j ) .

̸

The transition dynamics are illustrated in Figure 2, and the rewards are action free. The reward function r ( s 3 ) = r ( s 5 ) = ... = r ( s 2 H -3 ) = 1 , with all other states yielding a reward of 0 . Each agent has an action space A i = { a 1 , a 2 , a 3 } .

The expert policy σ E satisfies σ E ( a 1 a 1 | s 0 ) = 1 .σ E ( a 3 a 3 | s 1 ) = 1 . Action on all other states don't matter because the transition and the reward would be the same. The trained policy σ satisfies σ ( a 1 a 1 | s 0 ) = 1 , σ ( a 1 a 1 | s 1 ) = 1 , and plays the same as the expert in all other states.

It is not hard to verify that σ E plays a CE under this reward function, which means

The worst deviation for σ is to deviate action of agent 1 from playing a 1 to a 2 on both s 0 and s 1 . We get

E.4 Proof of Theorem 4.4

Proof. From the definition of CE, we know R Φ ( σ E ) ≤ δ 1 . Therefore,

Thus, we know that σ induces a δ 1 + δ 2 -approximate CE.

E.5 Proof of Theorem 4.6

Proof. For any i , we can view multi-agent problem as a single agent MDP over the joint action space under reward function r i . Following the proof in Ross and Bagnell [17], Swamy et al. [21], we can prove J i ( π σ E ) -J i ( π σ ) ≤ O ( ϵH 2 ) . Therefore, max i ∈ [ m ] ( J i ( π σ E ) -J i ( π σ )) ≤ O ( ϵH 2 ) .

E.6 Proof of Theorem 4.7

Proof. For any i ,

i ∈ [ m ] i σ E i σ

E.7 Proof of Theorem 5.1

Proof. With Assumption 5.2, we know that

For any deviation ϕ i ,

Therefore,

s

0

s

1

s

2

s

3

s

4

s

5

s

6

s

2

u

′

-

1

s

2

u

′

s

2

H

-

1

s

2

H

a

2

a

1

else

all

else

a

2

a

1

all

all

...

...

...

...

Taking the maximum over i, ϕ i , we get

E.8 Proof of Theorem 5.2

Proof. We prove the theorem by constructing such a Markov Game policies that can get Ω( 1 β ϵuH ) regret gap. We consider the two-player cooperative game similar to the example in Theorem 4.3. What we need to do is to slightly modify the MG and the policy to satisfy Assumption 5.1 and Assumption 5.2. The rewards are action free. Let u ′ = ⌊ u ⌋ , the reward function r ( s 3 ) = r ( s 5 ) = ... = r ( s 2 u ′ -3 ) = 1 , with all other states yielding a reward of 0 . The transition of the MG is shown in Figure 3. We know that the value is between [0 , u ′ ] for any policy, which means Assumption 5.1 is satisfied.

Let σ E be the policy that σ E ( a 1 a 1 | s 0 ) = 1 -2 β, σ E ( a 2 a 1 | s 0 ) = 2 β, σ E ( a 2 a 1 | s 1) = 1 2 , σ E ( a 3 a 3 | s 1 ) = 1 2 . Action at all other states doesn't matter because the transition and the reward would be the same. σ E satisfies Assumption 5.2.

Let trained policy σ be the policy that σ ( a 1 a 1 | s 0 ) = 1 -2 β, σ ( a 2 a 1 | s 0 ) = 2 β, σ ( a 2 a 1 | s 1) = 1 2 , σ ( a 1 a 1 | s 1) = ϵH 2 β , σ ( a 3 a 3 | s 1 ) = 1 2 -ϵH 2 β . σ and σ E only differs at s 1 .

Behavior cloning error of σ satisfies

It is not hard to verify, the worst deviation for π σ E is to deviate action of agent 1 at s 0 from playing a 1 to a 2 , and thus

The worst deviation of π σ is to deviate action of agent 1 from playing a 1 to a 2 at s 0 and s 1 .

Therefore, the regret gap R Φ ( σ ) -R Φ ( σ E ) = ϵH 2 β ( u ′ -2) = Ω( 1 β ϵuH ) .

E.9 Proof of Theorem 5.3

Proof. We prove it by showing that under complete reward function class F , low IRL error will imply low BC error, and then apply Theorem 5.1.

When F = [ -1 , 1] |S||A| ,

Therefore, we have ∑ s,⃗ a | ρ π σ E ( s,⃗ a ) -ρ π σ ( s,⃗ a ) | ≤ ϵ .

Therefore, we get

E.10 Proof of Corollary 5.4

Proof. Consider the same example in proof of Theorem 5.2 with parameter ϵ ′ . In the example, the only difference between the occupancy measures of two policies are ρ ( s,⃗ a ) at state s 1 . Therefore,

E.11 Proof of Theorem 5.5

Proof. From the definition of ℓ MALICE, we know

From Lemma C.1, we know that for all i, ϕ i , we have

And

Therefore, we get

Taking the maximum over i, ϕ i , we get

E.12 Proof of Theorem 5.6

Proof. We prove the theorem by constructing such a Markov Game policies that MALICE can get Ω( ϵuH ) regret gap. We consider a single-agent MDP shown in Figure 4. The rewards are action free. Let u ′ = ⌊ u ⌋ , the reward function r ( s 1 ) = r ( s 3 ) = ... = r ( s 2 u ′ -3 ) = 1 , with all other states yielding a reward of 0 . The transition of the MDP is shown in Figure 4. We know that the value is between [0 , u ′ ] for any policy, and thus Assumption 5.1 is satisfied.

Let σ E be the policy that σ E ( a 1 | s 0 ) = 1 -β, σ E ( a 2 | s 0 ) = β . Action at all other states doesn't matter because the transition and the reward would be the same. It is easy to verify that σ E satisfies Assumption 5.2.

Let trained policy σ be the policy that σ ( a 1 | s 0 ) = 1 -β -Hϵ,σ ( a 2 | s 0 ) = β + Hϵ . σ and σ E only differ at s 0 .

Now we verify that ℓ TV , MALICE ( σ, D E , σ ) ≤ ϵ .

Since σ and σ E only differ at state s 0 , and d π σ,ϕ i ( s 0 ) = 1 for any i, ϕ i , we have that

Therefore,

It is not hard to verify, the worst deviation for π E is to deviate action on s 0 from playing a 2 to a 1 , and thus

the worst deviation for π E is also to deviate action on s 0 from playing a 2 to a 1 .

Therefore, the regret gap R Φ ( π ) -R Φ ( π E ) = ϵ ( u ′ -1) H = Ω( ϵuH ) .

And

Therefore, we get

Taking the maximum over i, ϕ i , we get

s

0

s

1

s

2

s

3

s

4

s

5

s

6

s

2

u

′

-

1

s

2

u

′

s

2

H

-

1

s

2

H

a

1

a

2

all

all

all

all

...

...

...

...

E.13 Proof of Theorem 5.7

Proof. From the definition of ℓ BLADES, we know

From Lemma C.1, we know that for all i, ϕ i , we have

E.14 Proof of Theorem 5.8

Proof. Let MDP, expert policy σ E and the trained policy σ be the same example in the proof of Theorem 5.6.

Since σ and σ E only differ at state s 0 , and d π σ,ϕ i ( s 0 ) = 1 for any i, ϕ i , we have

Therefore, the trained policy π satisfies

The regret gap R Φ ( σ ) -R Φ ( σ E ) = ϵ ( u -1) H = Ω( ϵuH ) .

F Comparison with Goktas et al. [8]

Recent work Goktas et al. [8] worked on similar problem as ours. We will highlight some of the difference between two works.

First, the learning goals are different. They focus on a problem of inverse game theory, where the goal is to recover a reward function to rationalize the expert's behavior, i.e. the expert policy plays

an equilibrium under such a reward function. However, in our setting, instead of recovering a singe reward function, our goal is to learn a robust policy that get similar regret performance under a class of reward functions. We will show later that if the ultimate goal is to learn this robust policy, simply recovering a single reward function is not enough.

Second, the solution concepts are different. they work on Nash equilibrium, while in our setting, we focus on correlated equilibrium. We note that our algorithms also work for learning independent policies, by restricting the policy class to be a class of independent policies.

Third, in finite demonstration setting, their objective is to find a reward function which the learned policy plays a local NE, under the constraints that ℓ 2 difference of the observations for behaviors of two learned policy is small. We note that in general simply matching this difference is not enough to guarantee that the learned policy play an equilibrium. From Theorem 4.3, we know that even if the occupancy measures of two policies exactly match, the regrets can still be significantly different under the same reward function.

In conclusion, they work on a inverse game theory style problem where the goal is to recover a single reward function to rationalize the agents behavior. We work on imitation learning problem, where the goal is not recovering a single reward function but learning a policy that matches the regret performance of the expert under a class of reward functions.

We will give examples in normal form games (NFG) to show that recovering a single reward function is not enough to learn a policy that minimizing the regret gap for a large class of reward functions. NFG can be viewed as an MG in which H = 1 and |S| = 1 .

Lemma F.1. For an expert policy σ E , there may exist multiple reward functions that rationalize it.

Proof. We show this by an example of normal form games in Figure 5. Consider the policy to be σ E ( a 1 a 1 ) = 1 , then the expert plays CE/NE under both reward functions r and r ′ , which means both reward functions rationalize σ E .

Lemma F.2. For a fixed reward function, There may exist multiple CE/NEs.

Proof. For reward function r in Figure 5, we can construct such two policies σ 1 , σ 2 . For σ 1 , let σ 1 ( a 1 , a 1 ) = 1 . Let σ 2 ( a 1 , a 1 ) = 4 9 , σ 2 ( a 1 , a 2 ) = σ 2 ( a 2 , a 1 ) = 2 9 , σ 2 ( a 2 , a 2 ) = 1 9 . Tt is not hard to verify that both σ 1 and σ 2 play CE/NE under the reward function r .

Therefore, since there is no one-to-one mapping between the equilibria and the pay-off structures, simply recovering a single reward function might not help recover a policy that gets small regret gap.

For example, the true reward function is r in Figure 5, and expert policy σ E satisfies σ E ( a 1 , a 1 ) = 1 . The algorithm may recover r ′ in Figure 5, and a trained policy σ that plays NE/CE under recovered reward function r ′ would be σ ( a 1 , a 1 ) = σ ( a 1 , a 2 ) = σ ( a 2 , a 1 ) = σ ( a 2 , a 2 ) = 1 4 . However, this trained policy σ does not play NE/CE under the true reward function r .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We discuss the relationship between the two objectives, and list our contributions in the introduction. We include the assumptions when introducing the algorithms.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include limitations in ?? , where we discuss the computational and sample complexity of the algorithms, the assumptions made, and the fact that the algorithms are designed for tabular MGs and have not yet been implemented as practical algorithms.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are provided in Section 5 and full proofs are in Appendix E.6. We do not include proof sketches in the main paper. However, we provide links between the theorem statements and their corresponding proofs for the readers' convenience.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This is a theory paper. We do not include experiments for the algorithms.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This is a theory paper. We do not include experiments for the algorithms.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This is a theory paper. We do not include experiments for the algorithms.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This is a theory paper. We do not include experiments for the algorithms.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This is a theory paper. We do not include experiments for the algorithms.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss broader impacts in Appendix A.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper is a theory paper and does not pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This is a theory paper and it does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of an mediator coordinating the actions of a collection of agents by learning from the demonstrations of an expert. Unlike previous work the paper considers strategic deviations of the agents given the recommended actions instead of just matching the experts recommendations. It is shown the minimizing the standard performance measure (value gap) can lead to high regret in this setting. Finally, two algorithms are shown to minimize the regret under some assumptions.\n\nSTRENGTHS:\n1-The paper makes a nice extension of prior work. It is reasonable to expect the agents to deviate to maximize their utility instead of just following the recommended actions. \n\n2-I am not familiar with this area and the related work, but the paper has a nice logical flow. It starts by showing the difference between the performance measure of value and regret. Then it gives two algorithms to minimize the regret.\n\nWEAKNESSES:\n1-The paper's presentation can be improved. So many theorems are followed by [proof] in blue which looks very sloppy. \n\n2-It seems that the motivation of the paper has some similarities with incentivized exploration where the principle recommends an action but the agent is not forced to follow it, but I did not see it mentioned in the related work. Perhaps, there is even an extension of this work that connects to incentivized exploration.\n\nQUESTIONS:\n1-In lines (123-124) it says \"Critically, no agent observes the recommendations the mediator provides to another agent\", but in the definition of regret between lines 50 and 51 we have the agent maximizing utility based on $\\pi_{\\sigma,\\phi_i}$ which is the \"joint agent policy induced by all agents other than i following σ’s recommendations\". I don't understand how agent $i$ can find the strategy to maximize utility if he does not know the recommended actions and therefore  $\\pi_{\\sigma,\\phi_i}$ ? \n\n\n2-Please see the points above under Weaknesses.\n\nLIMITATIONS:\nI think the limitations were properly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors 1.initiate the study of an alternative objective for multi-agent imitation learning termed the regret gap; 2. investigate the relationship between regret gap and the value gap; 3. provided 2 provably efficient algorithms to minimize the regret gap under coverage assumption on the expert or access to a queryable expert respectively.\n\nSTRENGTHS:\nThis work proposes to study a new alternative objective for multi-agent imitation learning, which is one step closer to downstream applications like routing a group of drivers.\n\nThe paper contains both upper and lower bounds for regret gap optimization, along with detailed proofs.\n\nThe presentation is generally clear, and the appendix is well-organized.\n\nWEAKNESSES:\nSome assumptions are not explicitly made clear, such as the realizable expert, which can be relatively strong in practice. Also, in line 123, the requirement that no agent observes the recommendations the mediator provides to another agent sounds restrictive. Additionally, in line 146, it is required that identity mapping is in the class. It would be beneficial to have an extensive summary of assumptions in the appendix.\n\nProposed algorithms seems like straight forward applications of existing algorithms, it would be nice to list the challenges when applying existing results.\n\nNo experimentation or code is included with the work, making it difficult to examine whether the proposed algorithms are efficient in practice.\n\nQUESTIONS:\nQ1: The notion of regret gap seems confusing to me. Is it related to the regret in online learning? Could you please explain the reasoning behind this naming?\n\nQ2: I assume realizability (i.e., the expert policy being contained in the learner's policy class) is required for your theorems to hold. Is that true?\n\nQ3: It would be helpful if the authors could state the challenges encountered when obtaining new theoretical results based on previous works.\n\nQ4: It would be helpful if there is an experimental plan to verify the algorithms, such as in simulated driver routing games.\n\nMinor suggestions:\nIs there a typo in line 147 that begins with \"identity mapping ...\"? Should that sentence end with \"Φi\"?\n\nLIMITATIONS:\nSame as weakness. Realizability is a limitation. As stated in limitations, this work focus on tabular MGs, while the computation and sample/query complexity is linear over the cardinality of the deviation class.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of a mediator coordinating a group of strategic agents by recommending actions. Unlike previous work that focuses on non-strategic users who follow recommendations blindly, this study explores strategic users who may deviate based on their personal utility functions. The authors find that the traditional approach of minimizing the value gap is insufficient for handling deviations by strategic agents, as it fails to account for recommendations outside the induced state distribution. To address this, the paper introduces the concept of the regret gap in multi-agent imitation learning (MAIL) within Markov Games, which explicitly considers potential deviations by agents. The authors demonstrate that under the assumption of complete reward and deviation function classes, achieving regret equivalence ensures value equivalence. However, value equivalence does not guarantee a small regret gap. To minimize the regret gap, the paper presents two efficient algorithms: MALICE, which operates under coverage assumptions, and BLADES, which requires access to a queryable expert. These contributions offer a novel perspective and tools for improving robustness in multi-agent coordination scenarios involving strategic agents.\n\nSTRENGTHS:\n1. **Practical Relevance**: The paper addresses a significant real-world problem in multi-agent imitation learning, applicable in scenarios such as routing a group of drivers through a road network.\n2. **Comprehensive Literature Review**: The literature review is thorough, contrasting the prevalent value gap-based approaches in empirical MAIL work with the novel focus on regret gap-based methods.\n3. **Strong Motivation**: The paper is well-motivated, extending the existing framework to include strategic users and demonstrating the limitations of previous methods, thereby justifying the need for their proposed approach.\n4. **Clarity and Readability**: The paper is well-written and easy to follow, making complex concepts accessible to the reader.\n5. **Discussion of Assumptions and Limitations**: The authors provide a clear discussion of the assumptions and limitations of their work, adding depth to their analysis.\n\nWEAKNESSES:\n1. **Algorithmic Novelty**: The proposed algorithms, MALICE and BLADES, may lack novelty. MALICE modifies behavior cloning with an importance sampling term, relying on the expert visiting every state with positive probability. BLADES involves querying the expert for action recommendations based on possible deviations. These approaches are not entirely new.\n2. **Lack of Empirical Evaluation**: The algorithms have not been empirically evaluated, even in tabular Markov games. Their actual performance remains unknown, and the requirement to enumerate all user deviations may be impractical for real-world applications.\n\nQUESTIONS:\n1. What is the core difference between Algorithm 1 and Algorithm 2? They both enumerate all possible user deviations. Then Algorithm 1 requires a full demonstration coverage assumption to ensure that the importance weights are finite, and Algorithm 2 requests the expert to provide recommendations under all possible user deviations. These two requirements seem to be similar. Could the authors provide more discussions about their core differences?\n\n2. How strict is the Assumption 5.1 (u-recoverability)? This assumption states that all single-agent deviations only incur bounded advantage fluctuations, which seems to be too strict. \n\n3. Is Figure 1 a good example? The Markov game in Figure 1 assumes that the expert only visits the states in the lower path, which contradicts Assumption 5.2 ($\\beta$-coverage), I understand that Figure 1 is used to show why \"regret is hard\", but it is still confusing.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of the proposed algorithms.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of multi-agent imitation learning (MAIL), with a focus on the differences and relationships between the value gap objective and the regret gap objective.\n\nThe set-up consists of a multi-agent Markov game, where a central mediator attempts to coordinate the behaviors of the agents. Two potential objectives are compared: (1) Value gap objective: the coordinator tries to minimize the maximum deficiency among the agents in the overall value under the (joint) learning policy compared to the expert policy; (2) Regret gap objective: the coordinator minimizes instead the excess regret with respect to the expert policy, with regret defined as the maximum utility gain among the agents by deviating from the recommendation.\n\nThe authors show that, under certain completeness assumptions of both the reward and the deviation classes, regret equivalence implies value equivalence; In contrast, value equivalence in general does not imply regret equivalence. These are in sharp contrast to the single-agent imitation learning setting, where the two notions become equivalent.\n\nWhile regret equivalence is hard in general, the authors provide two algorithms, MALICE and BLADES, under different assumptions on recoverability, policy coverage, and/or query-ability of expert. Both algorithms provide a linear (in horizon) bound on the regret gap, which matches the single agent case.\n\nSTRENGTHS:\n[Originality and significance] The paper brings novel insights on MAIL in the sense that it characterizes how MAIL differs from SAIL in the regret perspective. Both the value-regret relationship results and the algorithms demonstrate the hardness in the multi-agent case due to lack of access to expert recommendations in counter-factual settings. In addition to the hardness results, the authors also provide algorithms that can be potentially useful to tackle the regret gap problem; this is valuable despite the still limited conditions.\n\n[Quality and clarity] The paper is generally well written and easy for me to follow. The results and their proofs seem correct.\n\nWEAKNESSES:\nThe main limitations of this paper lie in the significance of the results, both from the theoretical and the practical front. This is by no means to diminish the value of the results. What concerns me is two folds: (1) The main claims on the hardness of value gap and regret gap see mostly standard. The hardness due to inability to observe certain information in the multi-agent setting seems generally intuitive and unsurprising, and I think prior works (e.g., [1]) have to some extent demonstrated similar insights in related (albeit different) settings. The contribution seems more marginal in this respect. (I do perceive the contribution on the algorithmic side, though, as more substantial.) (2) On the practical side, I am not sure how the proposed methodology shall be applied to solving real world problems. Despite the theoretical nature of this paper, additional discussion on this can help make the paper more approachable to a larger audience.\n\n[1] Zhao, G., Zhu, B., Jiao, J., & Jordan, M. Online learning in Stackelberg games with an omniscient follower. International Conference on Machine Learning 2023.\n\nQUESTIONS:\n- Line 172: I am not sure why Theorem 4.2 “proves that the completeness of the classes is *necessary*”. I suspect the word “necessary” is not in its mathematical meaning, since the theorem seems to only show that some condition on the classes is necessary.\n- Line 179: Why is this the italicized statement “surprising”? I in fact find it quite expected, especially given the discussion in Line 176-177 on the regret equivalence as a stronger notion. This really makes me think that I’m missing on some key intuition regarding why one could hope for the direct implication.\n- I would like to see more discussion on how restrictive the assumptions (in particular, the recoverability Assumption 5.1) actually are, e.g., when one could expect them to hold. This would help better bridge the gap between the theoretical results and the practice.\n\nMinor comments:\n- Line 146: I find this notation a bit unstandard $\\Psi := \\{\\Psi_i\\}_{I=1}^m$ (as a sequence of sets). Maybe the product is more appropriate?\n- I think $r$ and $f$ are used interchangeably when denoting the reward function. Nothing wrong, but I suppose they could be made consistent.\n- Line 183: Typo “even” => “even if”\n\nLIMITATIONS:\nAddressed in Section 6 and Appendix A. The paper is of a theoretical nature.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}