Title: Sub-optimal Experts mitigate Ambiguity in Inverse Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Inverse Reinforcement Learning

Sub-optimal Experts

Sample Complexity
Abstract: Inverse Reinforcement Learning (IRL) deals with the problem of deducing a reward function that explains the behavior of an expert agent who is assumed to act *optimally* in an underlying unknown task. Recent works have studied the IRL problem from the perspective of recovering the *feasible reward set*, i.e., the class of reward functions that are compatible with a unique optimal expert. However, in several problems of interest it is possible to observe the behavior of multiple experts with different degree of optimality (e.g., racing drivers whose skills ranges from amateurs to professionals). For this reason, in this work, we focus on the reconstruction of the feasible reward set when, in addition to demonstrations from the optimal expert, we observe the behavior of multiple *sub-optimal experts*. Given this problem, we first study the theoretical properties showing that the presence of multiple sub-optimal experts, in addition to the optimal one, can significantly shrink the set of compatible rewards, ultimately mitigating the inherent ambiguity of IRL.
Furthermore, we study the statistical complexity of estimating the feasible reward set with a generative model and analyze a uniform sampling algorithm that turns out to be minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to that of the optimal expert.

Sub-optimal Experts mitigate Ambiguity in Inverse Reinforcement Learning

Abstract

Inverse Reinforcement Learning (IRL) deals with the problem of deducing a reward function that explains the behavior of an expert agent who is assumed to act optimally in an underlying unknown task. Recent works have studied the IRL problem from the perspective of recovering the feasible reward set , i.e., the class of reward functions that are compatible with a unique optimal expert. However, in several problems of interest it is possible to observe the behavior of multiple experts with different degree of optimality (e.g., racing drivers whose skills ranges from amateurs to professionals). For this reason, in this work, we focus on the reconstruction of the feasible reward set when, in addition to demonstrations from the optimal expert, we observe the behavior of multiple sub-optimal experts . Given this problem, we first study the theoretical properties showing that the presence of multiple sub-optimal experts, in addition to the optimal one, can significantly shrink the set of compatible rewards, ultimately mitigating the inherent ambiguity of IRL. Furthermore, we study the statistical complexity of estimating the feasible reward set with a generative model and analyze a uniform sampling algorithm that turns out to be minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to that of the optimal expert.

1 Introduction

Inverse Reinforcement Learning [IRL, 26] deals with the problem of recovering a reward function that explains the behavior of an expert agent who is assumed to act optimally in an underlying unknown task. Over the years, the IRL problem has consistently captured the attention of the research community (see, for instance, [4] and [1] for in-depth surveys). Indeed, this general scenario, where the reward function needs to be learned, emerges in numerous real-world applications. A example arises from human-in-the-loop settings [25], where the expert is a human solving a task, and an explicit specification of the human's goal in the form of a reward function is often unavailable. Notably, humans encounter difficulty in expressing their intentions in a numerical form, preferring instead to demonstrate what they perceive as the correct behavior. Once we retrieve a reward function, (i) we obtain explicit information for understanding the expert's choices, and, furthermore, (ii) we can utilize it to train reinforcement learning agents, even under shifts in the underlying system.

Since the seminal work [26], IRL has emerged as a significantly complex task. One of its primary challenges lies in the intrinsic ill-posed nature of the problem, as multiple reward functions compatible with the expert's behavior exist. Recently, a promising avenue of research [24, 19, 22] has tackled this ambiguity issue from an intriguing perspective. Specifically, this strand of works focuses on estimating all the reward functions that are compatible with the observed demonstration, thereby postponing the selection of the reward and directing their focus solely on the expert's intentions.

Nevertheless, these approaches based on recovering the feasible set [24, 19, 22] fall short in modeling more articulated situations that arise in the real world. In several problems of interest, it is possible to observe the behavior of multiple agents with different degrees of expertise . As an illustrative example, consider the human-in-the-loop settings, mentioned above, in which we are interested in recovering reward functions that explain the intent behind racing drivers. In this scenario, racing car companies typically have access to a variety of drivers with diverse skills, including professionals, test drivers, and emerging talents from developmental programs. In this context, while the focus is typically on the reward function of professional drivers, we expect a proficient IRL method to effectively leverage demonstrations provided by drivers with lower expertise. From an intuitive perspective, if we have some information on the degree of expertise of other drivers, we expect that, by exploiting their demonstrations, we can reduce the ambiguity of IRL problem.

With these motivations, in this work, we extend the IRL formulation as the reconstruction of the feasible reward set to settings where, in addition to demonstrations from an optimal expert, we observe the behavior of multiple sub-optimal experts, of which we know a bound on their sub-optimality . More specifically, we will primarily focus in answering the following theoretical questions:

(Q1) How does the presence of sub-optimal experts affect the set of reward functions that are compatible with the observed behaviors? Can they mitigate the intrinsic ambiguity of IRL?

(Q2) What is the statistical complexity of estimating the set of reward functions that are compatible with the given experts? How does it compare against the one of single-experts IRL problems?

Contributions and Outline. After providing the necessary background, we introduce the novel problem of Inverse Reinforcement Learning with multiple and sub-optimal experts (IRL-SE, Section 2). We then proceed by studying the theoretical properties of the class of reward functions that are compatible with a given set of experts under the assumption that an upper bound on the performance between a sub-optimal expert and the optimal expert is available to the designer of the IRL system (Section 3). Our findings indicate that having multiple sub-optimal experts can significantly shrink the set of compatible rewards, thereby mitigating the ambiguity issue that affects IRL. Leveraging our previous results, we continue by studying the statistical complexity of estimating the feasible reward set with a generative model (Section 4). To this end, after formally introducing a Probabilistic Approximately Correct [PAC, 9] framework, we derive a novel lower bound on the number of samples that are required to obtain an accurate estimate of the feasible reward set. Then, we present a uniform sampling algorithm and analyze its theoretical guarantees. Our results show that (i) the IRL problem with sub-optimal experts is statistically harder than the single expert IRL setting, and (ii) that the uniform sampling algorithm is minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to the one of the optimal expert. Finally, we conclude with a discussion on existing works (Section 5) and by highlighting potential avenues for future research (Section 6).

2 Preliminaries

In this section, we provide the notation and essential concepts employed throughout this document. Appendix A contains tables of symbols and a summary of the notation.

Notation. Let X be a finite set, we denote with ∆ X the set of probability measures over X . Let Y be a set, we denote with ∆ X Y the set of functions f : Y → ∆ X . Given f ∈ R n , we denote with ‖ f ‖ ∞ the infinite norm of f . Let X and X ′ be two non-empty subsets of a metric space ( Y , d ) , we define the Hausdorff distance [31] between X and X ′ as H d ( X , X ′ ) = max { sup x ∈X inf x ′ ∈X ′ d ( x, x ′ ) , sup x ′ ∈X ′ inf x ∈X d ( x, x ′ ) } . The Hausdorff distance directly depends on the metric d . We denote with 1 n the n -dimensional vector given by (1 , . . . , 1) glyph[latticetop] .

Markov Decision Processes. A Markov Decision Process without a reward function (MDP\R) is a tuple M = ( S , A , p, γ ) , where S is the set of states, A is the set of actions, p ∈ ∆ S S×A is

the transition probability kernel, and γ ∈ [0 , 1) is the discount factor. We consider finite state and action spaces, namely |S| = S and |A| = A . A Markov Decision Process [MDP, 27] is obtained by combining an MDP\R M with a reward function r ∈ R S×A . Without loss of generality, we assume reward functions bounded in [0 , 1] . We denote with M∪ r the resulting MDP. The behavior of an agent is described by a policy π ∈ ∆ A S , that, for each state, prescribes a distribution over actions.

Operators. Let f ∈ R S and g ∈ R S×A . We denote with P and π the operators induced by the transition model p and the policy π respectively. 1 Specifically, Pf ( s, a ) = ∑ s ′ ∈S p ( s ′ | s, a ) f ( s ′ ) , and πg ( s ) = ∑ a ∈A π ( a | s ) g ( s, a ) . Moreover, we introduce the operators E and ¯ B π defined as: Ef ( s, a ) = f ( s ) and ( ¯ B π g ) ( s, a ) = ✶ { π ( a | s ) = 0 } g ( s, a ) . Finally, we define d π f as the expectation of f under the discounted occupancy measure: d π f = ( I S -γπP ) -1 f = ∑ + ∞ t =0 ( γπP ) t f .

Value Functions and Optimality. Given an MDP M∪ r and a policy π , the Q-function Q π M∪ r represents the expected discounted sum of rewards collected in M∪ r starting from ( s, a ) and following policy π . Formally:

where the expectation is taken w.r.t. the stochasticity of the policy and the environment, i.e., s t +1 ∼ p ( ·| s t , a t ) and a t ∼ π ( ·| s t ) . The V-function V π M∪ r is the expectation of the Q -function over the action space, namely V π M∪ r = πQ π M∪ r . The advantage function A π M∪ r = Q π M∪ r -EV π M∪ r is the immediate gain of taking a given action, rather than following policy π . A policy π ∗ is optimal if it has non-positive advantage in each-state action pair; namely A π ∗ M∪ r ≤ 0 holds element-wise.

Inverse Reinforcement Learning. An Inverse Reinforcement Learning [IRL, 26] problem is a tuple B = ( M , π E ) , where M is an MDP\R and π E ∈ ∆ A S is the expert policy. Given a reward function r ∈ R S×A , we say that r is feasible for B if it is compatible with the behavior of the expert, namely π E is an optimal policy for the MDP M∪ r . We denote with R B the set of feasible reward functions:

The set R B is named feasible reward set [24, 19, 22]. To characterize the set R B , [24] have shown that a reward function r belongs to R B if and only if there exists ζ ∈ R S×A ≥ 0 and V ∈ R S such that:

Thus, each reward function in R B , is the sum of two components. The first one, -¯ B π E ζ , which is non-zero only when π E ( a | s ) = 0 , can be interpreted as the advantage function A π E M∪ r . The second one, ( E -γP ) V , instead, can be interpreted as a reward-shaping via function V , which maintains the optimality of the expert's policy [26]. It follows that ‖ V ‖ ∞ ≤ (1 -γ ) -1 and ‖ ζ ‖ ∞ ≤ (1 -γ ) -1 .

3 Sub-Optimal Experts and the Feasible Reward Set

In this section, we extend the IRL formulation to problems where, in addition to demonstrations from an optimal expert, we observe the behaviors of multiple and sub-optimal experts. After having formulated and motivated the problem (Section 3.1), we delve into the theoretical properties of the induced feasible reward set, by providing both an implicit (Section 3.2) and explicit (Section 3.3) descriptions. Our results indicate that the presence of sub-optimal experts can significantly shrink the feasible set of compatible rewards, thus, mitigating the ambiguity issue of IRL.

3.1 Problem Formulation

We define the Inverse Reinforcement Learning problem with multiple and Sub-optimal Experts (IRLSE) as a tuple ¯ B = ( M , π E 1 , ( π E i ) n +1 i =2 , ( ξ i ) n +1 i =2 ) , where M is an MDP\R, π E 1 is the policy of an

optimal expert, 2 ( π E i ) n +1 i =2 are a collection of n sub-optimal experts policies, and ( ξ i ) n +1 i =2 are the corresponding sub-optimality bounds. More concretely, ξ i represents a known upper bound on the performance gap between the optimal expert and the i -th sub-optimal expert. Consequently, a reward function r ∈ R S×A is feasible for ¯ B if π E 1 is an optimal policy for the MDP M∪ r and if:

Thus, a feasible reward r must make the value function V π Ei M∪ r ( s ) of the i -th expert smaller than that of the optimal expert V π E 1 M∪ r ( s ) = V ∗ M∪ r ( s ) by no more than the threshold ξ i , uniformly over s ∈ S . We denote by R ¯ B the set of feasible rewards for ¯ B , i.e., r ∈ [0 , 1] S×A belongs to R ¯ B if (i) A π E 1 M∪ r ≤ 0 and (ii) Equation (3) holds. Notice that, whenever no sub-optimal expert is present, we recover the definition of the feasible set for single-expert IRL problems, i.e., R B in Equation (1).

We remark that ξ i can even be a crude overestimate (i.e., an upper bound) of the sub-optimality of expert i . Nevertheless, as we shall see, the ability of the sub-optimal policy π E i in mitigating the IRL ambiguity, i.e., shrinking the feasible reward set, will be tightly connected on the magnitude of ξ i .The following examples illustrate how an expression ξ i can be obtained in common scenarios with no knowledge of the (possible) reward function optimized by the expert policy π E 1 . Formal proof for these statements are reported in Appendix C.

Example 3.1. Suppose that the i -th sub-optimal expert π E i is optimal for the same reward function that π E 1 is optimizing for, but under different transition model P i . In this case, ‖ V π E 1 M∪ r -V π Ei M∪ r ‖ ∞ ≤ 2 γ (1 -γ ) 2 max ( s,a ) ∈S×A ‖ P ( ·| s, a ) -P i ( ·| s, a ) ‖ 1 =: ξ i holds for all rewards r .

Example 3.3. Suppose that the i -th sub-optimal expert π E i is sufficiently close to the optimal policy π E 1 , namely that max s ∈S ‖ π E 1 ( ·| s ) -π E i ( ·| s ) ‖ 1 ≤ glyph[epsilon1] . In this case, ‖ V π E 1 M∪ r -V π Ei M∪ r ‖ ∞ ≤ 1 1 -γ glyph[epsilon1] =: ξ i holds for all rewards r .

Example 3.2. Suppose that the i -th sub-optimal expert π E i is optimal for the same reward function that π E 1 is optimizing for, but using a different discount factor γ ′ . In this case, ‖ V π E 1 M∪ r -V π Ei M∪ r ‖ ∞ ≤ 2 | γ -γ ′ | (1 -γ )(1 -γ ′ ) =: ξ i holds for all rewards r .

3.2 Implicit Formulation of R ¯ B

In this section, we analyze the implicit description of the feasible reward set R ¯ B . From its definition (Equations (1) and (3)), a reward function r ∈ [0 , 1] belongs to R ¯ B if and only if the following conditions are satisfied:

(i) Q π E 1 M∪ r ( s, a ) = V π E 1 M∪ r ( s ) ∀ ( s, a ) ∈ S × A : π E 1 ( a | s ) > 0 ,

(iii) V π E 1 M∪ r ( s ) ≤ V π Ei M∪ r ( s ) + ξ i ∀ s ∈ S , ∀ i ∈ { 2 , . . . , n +1 } .

(ii) Q π E 1 M∪ r ( s, a ) ≤ V π E 1 M∪ r ( s ) ∀ ( s, a ) ∈ S × A : π E 1 ( a | s ) = 0 ,

Specifically, conditions (i) and (ii) directly encode the optimality of policy π E 1 for M∪ r , i.e., the advantage function A π E 1 M∪ r is non-positive. Condition (iii), on the other hand, arises from the presence of sub-optimal experts, and directly follows from Equation (3). At this point, by closely examining these conditions, it is possible to gain insight into the advantages and limitations associated with the presence of multiple and sub-optimal experts. Consider, the following illustrative examples.

Example 3.4. Suppose that π E i = π E 1 holds for all i ∈ { 2 , . . . , n +1 } . Condition (iii) is clearly satisfied for any reward function r . Thus, the feasible reward set R ¯ B is determined by the requirement that the advantage function of π E 1 is non-negative, and, as a consequence, the set R ¯ B coincides with the one of the single-expert IRL problem, namely R ¯ B = R B . Analogously, if ξ i ≥ (1 -γ ) -1 holds for all sub-optimal experts, condition (iii) is vacuous, and, similarly, R ¯ B reduces to R B .

Example 3.5. Consider the MDP with 2 states depicted in Figure 1, and suppose that only one additional sub-optimal expert is present. The optimal and the sub-optimal experts follow different policies in S 0 . From the definition of R ¯ B , we can see that, in addition to the constraint r ( S 0 , A 1 ) ≥ r ( S 0 , A 2 ) (i.e., π E 1 is optimal), condition (iii) enforces a further relationship between r ( S 0 , A 1 ) and r ( S 0 , A 2 ) , i.e., r ( S 0 , A 1 ) -r ( S 0 , A 2 ) ≤ ξ i . If ξ i is sufficiently small (i.e., ξ i < 1 ), the presence of the sub-optimal experts significantly reduces the set of compatible rewards (Figure 2).

S

0

S

1

A

1

|

π

E

1

(

A

1

|

S

0

)=1

A

2

π

E

i

(

A

2

S

0

)=1

A

|

|

0

0

.

25

0

.

5

0

.

75

1

0

0

.

25

0

.

5

0

.

75

1

r

(

S

0

, A

1

)

r

(

S

0

, A

2

)

0

0

.

25

0

.

5

0

.

75

1

0

0

.

25

0

.

5

0

.

75

1

r

(

S

0

, A

1

)

Abstracting away from the examples, we deduce that whenever (a) the sub-optimal experts exhibit behavior significantly different from that of the optimal expert and (b) their performance level is sufficiently close to being optimal, R ¯ B can notably shrink compared to R B . In the next section, through an explicit formulation of the feasible set, we analyze this phenomenon quantitatively.

3.3 Explicit Formulation of R ¯ B

In this section, we continue by providing an explicit formulation of the feasible set R ¯ B . The following result (proof in Appendix D) summarizes our findings.

Theorem 3.6. Let ¯ B be an IRL problem with sub-optimal experts. Let r ∈ [0 , 1] S×A . Then, r ∈ R ¯ B if and only if there exists ζ ∈ R S×A ≥ 0 and V ∈ R S such that the following conditions are satisfied:

and, for all i ∈ { 2 , . . . , n +1 } :

Theorem 3.6 deserves some comments. First, from Equation (4), we see that a necessary condition for having r ∈ R ¯ B is that it can be expressed as the sum of two components, namely -¯ B π E 1 ζ and ( E -γP ) V . This result is a direct consequence of the fact that π E 1 is an optimal policy for M∪ r , and, in this sense, it recovers the existing condition of single expert IRL settings (Equation 2) [24]. 3

Sub-optimal experts constrain the sub-optimality gaps. The role of the sub-optimal experts is apparent in Equation (5) enforcing a set of linear constraints on the values that ζ can take. 4 Since, as discussed in Section 2, -¯ B π E 1 ζ represents the advantage function of the optimal policy π E 1 , Equation (5) limits how much sub-optimal the values of actions not played by π E 1 can be. Indeed, the resulting Q -function of the optimal expert can be expressed as Q π E 1 M∪ r = -¯ B π E 1 ζ + EV [24].

Sub-optimal experts affect only states in which they behave sub-optimally. The linear constraints in Equation (5) are expressed in terms of π E i ¯ B π E 1 ζ . As a consequence, they will only affect stateaction pairs ( s, a ) ∈ S × A that are played by the sub-optimal experts (i.e., π E i ( a | s ) > 0 ) and that are not played by the optimal expert (i.e., π E 1 ( a | s ) = 0 ). Therefore, as previously highlighted with the implicit formulation of R ¯ B , a sub-optimal expert π E i should behave differently w.r.t. the optimal expert π E 1 in order to provide meaningful information and reduce the feasible reward set. Furthermore, the constraints over ζ are expressed w.r.t. expected discounted occupancy of π E i .

Therefore, Equation (5) has provided a precise mathematical description (i.e., if and only if conditions ) of the effect produced by the presence of sub-optimal experts on the feasible rewards r .

While in classical IRL problems, we have ‖ ζ ‖ ∞ ≤ (1 -γ ) -1 , the maximum value ζ can take in the sub-optimal experts case may be smaller. Let us fix a state s ′ ∈ S and a sub-optimal expert i ∈ { 2 , . . . , n +1 } ; then, the constraint associated to state s ′ in Equation (5) can be written as:

where d π Ei s ′ ( s ) is the discounted expected number of times that policy π E i visits state s ∈ S starting from state s ′ . From Equation (6), we obtain necessary conditions on the values of ζ . More specifically, let X ( s, a ) ⊆ { 2 , . . . , n +1 } be the subset of sub-optimal experts such that π E i ( a | s ) > 0 . Then, for each state-action pair ( s, a ) ∈ S × A such that π E 1 ( a | s ) = 0 and π E i ( a | s ) > 0 , we have that:

The term k ( s, a ) directly follows from Equation (6), while (1 -γ ) -1 is the maximum value that any ζ ( s, a ) can take, and arises, as in the classical IRL setting, from the fact that advantage functions are bounded by (1 -γ ) -1 for any reward function. Thus, confirming our previous observations, Equation (7) implies a significant potential reduction in the maximum values that the advantage function can take, i.e., how much sub-optimal an action not played by π E 1 can be in terms of Q -function values. 5

Example 3.7. Consider an IRL problem with only one additional sub-optimal expert. Suppose that π E 1 and π E i are deterministic. For all state-action pairs in which π E 1 ( a | s ) = 0 and π E i ( a | s ) = 1 , Equation (7) implies that ζ ( s, a ) ≤ min { ξ i , (1 -γ ) -1 } . If ξ i is significantly smaller than (1 -γ ) -1 , we obtain a notable restriction on the set of feasible reward functions.

Remark 3.8 (About alternative Sub-optimality Formulations) . We have described the sub-optimal experts by means of upper bounds on the value functions gaps; i.e., V π E 1 M∪ r -V π Ei M∪ r ≤ 1 S ξ i . In principle, we may consider lower bounds on the value functions gaps; i.e., V π E 1 M∪ r -V π Ei M∪ r ≥ 1 S ξ i . Intuitively, in the former case, we know that an expert is sub-optimal at most ξ i , whereas, in the latter, that an expert is sub-optimal at least ξ i . Our theory smoothly generalizes to these latter scenario obtaining an analogous of Theorem 3.6 where the constraints of Equation (5) are replaced with d π Ei π E i ¯ B π E 1 ζ ≥ 1 S ξ i . We refer the reader to Theorem D.5 for a formal statement.

4 Learning the Feasible Set with Sub-Optimal Experts

In this section, we address the statistical complexity of estimating R ¯ B with a generative model . Specifically, we first introduce a Probabilistic Approximately Correct (PAC) framework (Section 4.1). Then, we study the statistical complexity of the problem by presenting lower bounds on the sample complexity any algorithm requires in order to correctly identify the feasible set (Section 4.1). Finally, we propose a uniform sampling algorithm and analyze its theoretical guarantees (Section 4.3). 6

4.1 PAC Framework

We define a learning algorithm for an IRL-SE problem ¯ B as a tuple A = ( τ, ν ) where τ is a stopping and ν = ( ν t ) t ∈ N is a history-dependent sampling strategy, i.e., ν t ∈ ∆ S×A D t -1 , where D t = ( S × A × S × ( A ) n +1 ) t is the set of samples collected up to time step t ∈ N . At each time step t , the algorithm selects a state-action pair ( s t , a t ) ∼ ν t , and observes a sample s ′ t ∼ p ( ·| s t , a t ) from the environment, together with actions sampled from the experts' policy, namely ( a ( i ) t ) n +1 i =1 , where a ( i ) t ∼ π E i ( ·| s t ) . The observed realizations D t are then used to update the sampling strategy ν t , and the process goes on until the stopping rule τ is satisfied. Then, the algorithm leverages the

collected data D τ to output the estimate of the feasible reward set ̂ R τ . We are interested in designing algorithms that, for any desired accuracy glyph[epsilon1] ∈ (0 , 1) and any risk parameter δ ∈ (0 , 1) , guarantee that:

We refer to these algorithms as ( glyph[epsilon1], δ ) -correct identification strategies and we define their sample complexity as the total number of interactions with the generative model before stopping, i.e., τ .

4.2 Lower Bound

In this section, we present lower bounds on the sample complexity that any ( glyph[epsilon1], δ ) -correct algorithm needs to perform to learn R ¯ B . The following theorem (proof in Appendix E) reports our result.

Theorem 4.1. Let A be a ( glyph[epsilon1], δ ) -correct algorithm for the IRL problem with sub-optimal experts. There exists a problem instance ¯ B such that the expected sample complexity is lower bounded by:

where Ω( · ) hides constant dependencies. Let π min > 0 be:

and let q 0 := π -1 min max i ∈{ 2 ,...,n +1 } ξ i . Then, there exists an instance ¯ B ′ in which q 0 < 1 such that:

Theorem 4.1 provides two distinct lower bounds (i.e., Equations 9 and 11) for IRL problems with suboptimal experts. Whenever q 0 < 1 holds, the lower bound for the IRL-SE setting can be expressed as the maximum between Equation (9) and (11). We now comment in-depth on these two equations.

Sub-optimal experts do not reduce the statistical complexity of IRL. Concerning Equation (9), as our analysis reveals, it directly arises from the problem of estimating rewards functions that are compatible with π E 1 (i.e., with Equation (4) in Theorem 3.6). In this sense, it represents the complexity of single-expert IRL problems. 7 As a consequence of the structure of the feasible set we derived in Theorem 3.6, this leads to a lower bound also for the multiple sub-optimal experts setting. Therefore, Equation (9) formally shows that the sub-optimal expert setting is always at least as difficult as the single expert IRL problem.

Sub-optimal experts have introduce further statistical complexities. Equation (11), on the other hand, is tightly related to the presence of sub-optimal experts. More precisely, under the assumption that q 0 < 1 (e.g., for sufficiently small values of ξ i ), it shows a dependency in the lower bound of a factor π -1 min , where π min is the minimum probability with which sub-optimal experts plays their actions. From an intuitive perspective, its presence is related to the difficulty in estimating reward functions that are compatible with Equation (5) in Theorem 3.6. As we have shown in Section 3, the presence of sub-optimal experts can limit the value of ζ with a relationship that involves π -1 min (i.e., Equation (7)). As our analysis will reveal, the proof of Equation (11) is directly related to these worst-case upper-bounds on ζ . We remark that, according to the value of π min, Equation (11) can be significantly larger than Equation (5), thus showing an increased difficulty in the statistical complexity that is related to the stochasticity of sub-optimal experts.

We remark that τ represents the number of calls to the generative model and that each call, in our setting, provides demonstrations from each (sub-)optimal expert. It can be shown that, by slightly modifying the learning formalism, Equation (11) actually represents a lower bound to the number of samples that should be gathered from each sub-optimal expert . 8 In this sense, the statistical complexity increases significantly in the sub-optimal expert setting compared to the single expert one.

Therefore, to conclude, we notice that, in order to gain the reduction in the feasible reward set that we discussed in Section 5, we need to gather additional sampled demonstrations from the sub-optimal experts. This unavoidable trade-off is a direct consequence of the structure of the feasible set R ¯ B that we derived in Theorem 3.6, and, it arises from the statistical complexity of estimating reward functions that are compatible with the linear constraints of Equation (5).

4.3 Uniform Sampling Algorithm

In this section, we present the Uniform Sampling algorithm for Inverse RL with Suboptimal Experts (US-IRL-SE).

Algorithm. The pseudo-code can be found in Algorithm 1 US-IRL-SE takes as input the number of samples m that will be queried to the generative model in each state-action pair. Then, it uniformly gathers data across the entire state-action space, and it updates the empirical estimates ˆ p t and (ˆ π t,E i ) n +1 i =1 of the transition model and expert's policies. Let D t be a dataset of t ∈ { 1 , . . . , m } tuples D t = { ( s j , a j , s ′ j , ( a ( i ) j ) n +1 i =1 ) } t j =1 , where s ′ j ∼ p ( ·| s j , a j ) , and a ( i ) j ∼ π E i ( ·| s j ) . Given D t , we

Algorithm 1 US-IRL-SE Algorithm

Require: Number of samples for each ( s, a ) pair m

1: for t ∈ { 1 , . . . , m } do

3: Update ˆ p t and (ˆ π t,E i ) n +1 i =1 according to Equation (12)

2: Collect ( s ′ , ( a ( i ) ) n +1 i =1 ) where s ′ ∼ p ( ·| s, a ) and a ( i ) ∼ π E i ( ·| s ) from each ( s, a ) ∈ S × A and i ∈ { 1 . . . , n +1 }

4: end for

5: return R ̂ ¯ B m

define the empirical transition model ˆ p t and the empirical experts' policy ˆ π t,E i as follows:

where N t ( s, a, s ′ ) = ∑ t j =1 1 { ( s j , a j , s ′ j ) = ( s, a, s ′ ) } , N t ( s, a ) = ∑ s ′ ∈S N t ( s, a, s ′ ) , N t ( s ) = ∑ ( a,s ′ ) ∈A×S N t ( s, a, s ′ ) , and N ( i ) t ( s, a ) = 1 { ( s j , a ( i ) j ) = ( s, a ) } . Then, we denote with ̂ ¯ B t the empirical IRL problem induced by ˆ p t and (ˆ π t,E i ) n +1 i =1 . Finally, the algorithm returns the feasible set R ̂ ¯ B m corresponding to the estimated IRL-SE problem ̂ ¯ B m defined in terms of ˆ p m and (ˆ π m,E i ) n +1 i =1 .

Sample Complexity Upper Bound The following theorem (proof in Appendix E), describes the theoretical guarantees of US-IRL-SE.

Theorem 4.2. Let q 1 = min { π -1 min max i ∈{ 2 ,...,n +1 } ξ i , (1 -γ ) -1 } , and q 2 = max { 1 , q 1 } . Then, with a total budget of:

US-IRL-SE is ( glyph[epsilon1], δ ) -correct and ˜ O ( · ) hides constant and logarithmic dependencies.

Theorem 4.2 deserves some comments. First of all, it shows that when the total number of queries to the generative is sufficiently large, US-IRL-SE is ( glyph[epsilon1], δ ) -correct, and its sample complexity is provided in Equation (13). Since m represents the number of calls to the generative model in each state-action pair, its expression can simply be calculated dividing Equation (13) by SA . 9 As a consequence, we remark that, in order to compute the value of m , the algorithm requires knowledge of the minimum probability π min with which the sub-optimal experts play their actions.

US-IRL-SE is minimax optimal when π -1 min ξ i ≤ 1 . Equation (13) is the maximum between two terms whose expressions closely resemble the lower bound of Theorem 4.1. The only difference arises in the definition of q 0 , q 1 and q 2 . We remark that, whenever the sub-optimal expert's performance level is sufficiently close to the one of the optimal expert, i.e., π -1 min ξ i ≤ 1 for all i ∈ { 2 , . . . , n +1 } ,

Equation (13) exactly recovers the lower bound that we presented in Theorem 4.1. Indeed, under this condition, it holds that q 0 = q 1 , and q 2 = 1 . We recall that, according to Theorem 3.6, as the values of ξ i 's decrease, the feasible reward set is substantially reduced. In this sense, US-IRL-SE enjoys minimax optimality in the most interesting scenarios where the presence of sub-optimal experts is particularly useful for mitigating the intrinsic ambiguity that affects IRL problems .

Technical challenges of Theorem 4.2. We highlight that the proof of Theorem 4.2 poses notable technical challenges beyond the ones of the analysis for standard IRL [24]. In our setting, studying how the Hausdorff distance between R ¯ B and R ̂ ¯ B t decreases as we collect more samples requires taking into account that these feasible reward sets are subject to the peculiar structure identified in Theorem 3.6, namely the set of constraints of Equation (5) arising from the presence of sub-optimal experts. We study, with probabilistic arguments, error terms of the form ‖ ζ -proj ̂ Ω t ζ ‖ ∞ , where ζ is feasible for the exact problem ¯ B , ̂ Ω t denotes the set of ζ that are compatible with the empirical problem ̂ ¯ B t , and proj ( · ) denotes the infinite norm projection. Analyzing these terms to obtain nearly optimal rates requires careful considerations on the geometry of feasible reward set that the sub-optimal experts induce. For further details, we refer the interested reader to Appendix E.

5 Related Works

In this section, we survey the works about IRL and the presence of multiple (sub-optimal) experts that are related to our proposal.

Inverse Reinforcement Learning. Historically, solving an IRL problem [1] involves determining a reward function that is compatible with the behavior of an optimal expert. Since the seminal work of [26], the problem has been recognized as ill-posed, as multiple reward functions that satisfies this requirement exists [34]. For this reason, over the years, several algorithmic criteria have been introduced to address this ambiguity issue. These criteria includes maximum margin [30], Bayesian approaches [28], maximum entropy [37], and many others [e.g., 21, 23, 36]. More recently, a new line of works have circumvented the ambiguity issue by redefining the IRL task as the problem of estimating the entire feasible reward set [24, 19, 22]. In our work, we take this novel perspective, and, in this sense, this recent research strand is the most related to our document. Specifically, of particular interests is the work of [22]. In their work, the authors study, for the first time, lower bounds for the single-expert IRL problem in finite horizon settings. Furthermore, they show that uniform sampling algorithm is minimax optimal for this task. Nevertheless, it has to remarked that this recent strand of research focuses entirely on single expert problems. As we have shown, however, the extension to the multiple and sub-optimal experts setting requires non-trivial effort. The reason is that the feasible reward set significantly differ (see, e.g., Theorem 3.6), and the problem is harder from a statistical perspective (see, e.g., Theorem 4.1).

Multiple and/or Sub-optimal Experts. The presence of multiple/sub-optimal experts has garnered attention in the Imitation Learning [IL, 11] community. In IL problems, contrary to IRL, the goal lies in directly leveraging demonstrations of optimal behavior to accelerate the training process of reinforcement learning algorithms. In this context, works that are close in spirit to ours are [17, 13, 6, 20]. Here, the authors extend the IL formulation to account for the fact that demonstrations are provided from multiple and/or sub-optimal experts. However, unlike our specific focus, their emphasis is on understanding how to effectively exploit imperfect demonstrations to improve training of RL agents. In our work, instead, we exploit the presence of sub-optimal experts to reduce the intrinsic ambiguity that affects the IRL formulation. In this sense, our work is complementary to several studies that analyzed how to improve the identifiability of the reward function in IRL problems by making additional structural assumptions. These include the possibility of observing an optimal expert interacting with several MDPs [e.g., 30, 3, 2] and focusing on peculiar types of MDPs that allows for strong theoretical guarantees [e.g., 8, 15, 5]. Along this line of work, the most related to ours is [32]. Here, the authors study how the presence of multiple experts impact the identifiability of the reward function. Contrary to our work, however, the authors assume each expert to follow an entropy regularized objective and, furthermore, they focus on the case in which all experts act optimally in the underlying environment. In this sense, our work encompasses a wider spectrum of applications, as we do not require optimality for each of the expert, nor an entropy

regularized objective. Furthermore, it has to be remarked that the multiple expert setting and IRL have been studied in [18] with the goal of providing practical algorithms that can be used in realworld applications. Also in this scenario, each expert is assumed to act optimally in the underlying domain. Finally, our work is related to approaches that aimed at extracting a single reward function by leveraging possibly sub-optimal demonstrations [e.g., 35, 16, 12, 29]. In our work, instead, we take a different theoretical perspective, and focus on the set of reward functions that are compatible with multiple sub-optimal experts.

6 Conclusions

In this work, we studied the novel problem of Inverse RL where, in addition to demonstrations from an optimal expert, we can observe the behavior of multiple and sub-optimal experts. More precisely, we first investigated the theoretical properties of the class of reward functions that are compatible with a given set of experts, i.e., the feasible reward set. Our results formally show that, by exploiting this additional structure, it is possible to significantly reduce the intrinsic ambiguity that affects the IRL formulation. Secondly, we have tackled the statistical complexity of estimating the feasible reward set from a generative model. More precisely, we have shown that a uniform sampling algorithm is minimax optimal whenever the performance level of the sub-optimal expert is sufficiently close to the one of the optimal expert.

Our research opens up intriguing avenues for future studies. In the following, we highlight several possibilities.

Closing the Theoretical Gap The results that we presented in Section 4 do not completely match. When the performance of the sub-optimal experts are not sufficiently close to the one of the optimal agent, the upper bound differs from the lower bound. Closing this gap, either by (i) developing tighter lower bounds or (ii) proposing novel algorithms and/or refining the analysis of US-IRL to achieve tighter upper bounds, is an interesting future direction. This would allow for a complete understanding of the statistical complexity of the problem.

Offline IRL with Sub-Optimal Experts Secondly, we note that this work leverages the presence of a generative model in the algorithm design. In the future, it would be interesting to remove this assumption by considering an offline setting where only a dataset of collected demonstrations is available to the learning system. This formulation is more practical since, in several real-world IRL applications [18], only a dataset of pre-collected demonstrations is available to the designer of the IRL system. We also note that this setting is, in principle, more challenging, as the data coverage of the dataset is not under the control of the IRL system.

Large State-Action Spaces For instance, since we have shown that sub-optimal experts can improve the identifiability of the reward function, future research should focus on building practical algorithms that can exploit this additional structure. To this end, as an intermediate step, it might be interesting to extend our results to the case in which the reward function is expressed as a linear combination of features. This approach would enable addressing infinite state-spaces [e.g., 26].

Acknowledgments and Disclosure of Funding

Funded by the European Union - Next Generation EU within the project NRPP M4C2, Investment 1.,3 DD. 341 - 15 march 2022 - FAIR - Future Artificial Intelligence Research - Spoke 4 - PE00000013 D53C22002380006. AI4REALNET has received funding from European Union's Horizon Europe Research and Innovation programme under the Grant Agreement No 101119527. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.

References

Stephen Adams, Tyler Cody, and Peter A Beling. A survey of inverse reinforcement learning. Artificial Intelligence Review , 55(6):4307-4346, 2022.

Kareem Amin, Nan Jiang, and Satinder Singh. Repeated inverse reinforcement learning. Advances in neural information processing systems , 30, 2017.

Kareem Amin and Satinder Singh. Towards resolving unidentifiability in inverse reinforcement learning. arXiv preprint arXiv:1601.06569 , 2016.

Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence , 297:103500, 2021.

Haoyang Cao, Samuel N Cohen, and Lukasz Szpruch. Identifiability in inverse reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Ching-An Cheng, Andrey Kolobov, and Alekh Agarwal. Policy improvement via imitation of multiple oracles. Advances in Neural Information Processing Systems , 33:5587-5598, 2020.

Joseph L Doob. Measure theory , volume 143. Springer Science & Business Media, 2012.

Krishnamurthy Dvijotham and Emanuel Todorov. Inverse optimal control with linearly-solvable mdps. In Proceedings of the 27th International conference on machine learning (ICML-10) , pages 335-342, 2010.

Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Pac bounds for multi-armed bandit and markov decision processes. In Computational Learning Theory: 15th Annual Conference on Computational Learning Theory, COLT 2002 Sydney, Australia, July 8-10, 2002 Proceedings 15 , pages 255-270. Springer, 2002.

Torben Hagerup and Christine Rüb. A guided tour of chernoff bounds. Information processing letters , 33(6):305-308, 1990.

Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR) , 50(2):1-35, 2017.

Alexis Jacq, Matthieu Geist, Ana Paiva, and Olivier Pietquin. Learning from a learner. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2990-2999. PMLR, 09-15 Jun 2019.

Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Chao Yang, Bin Fang, and Huaping Liu. Reinforcement learning from imperfect demonstrations under soft expert guidance. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 5109-5116, 2020.

Anders Jonsson, Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Edouard Leurent, and Michal Valko. Planning in markov decision processes with gap-dependent sample complexity. Advances in Neural Information Processing Systems , 33:1253-1263, 2020.

Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward identification in inverse reinforcement learning. In International Conference on Machine Learning , pages 5496-5505. PMLR, 2021.

Edouard Klein, Matthieu Geist, and Olivier Pietquin. Batch, off-policy and model-free apprenticeship learning. In European Workshop on Reinforcement Learning , pages 285-296. Springer, 2011.

Andrey Kurenkov, Ajay Mandlekar, Roberto Martin-Martin, Silvio Savarese, and Animesh Garg. Ac-teach: A bayesian actor-critic method for policy learning with an ensemble of suboptimal teachers. In Proceedings of the Conference on Robot Learning , 2020.

Amarildo Likmeta, Alberto Maria Metelli, Giorgia Ramponi, Andrea Tirinzoni, Matteo Giuliani, and Marcello Restelli. Dealing with multiple experts and non-stationarity in inverse reinforcement learning: an application to real-life problems. Machine Learning , 110:2541-2576, 2021.

David Lindner, Andreas Krause, and Giorgia Ramponi. Active exploration for inverse reinforcement learning. Advances in Neural Information Processing Systems , 35:5843-5853, 2022.

Xuefeng Liu, Takuma Yoneda, Chaoqi Wang, Matthew Walter, and Yuxin Chen. Active policy improvement from multiple black-box oracles. In International Conference on Machine Learning , pages 22320-22337. PMLR, 2023.

Anirudha Majumdar, Sumeet Singh, Ajay Mandlekar, and Marco Pavone. Risk-sensitive inverse reinforcement learning via coherent risk models. In Robotics: science and systems , volume 16, page 117, 2017.

Alberto Maria Metelli, Filippo Lazzati, and Marcello Restelli. Towards theoretical understanding of inverse reinforcement learning. arXiv preprint arXiv:2304.12966 , 2023.

Alberto Maria Metelli, Matteo Pirotta, and Marcello Restelli. Compatible reward inverse reinforcement learning. Advances in neural information processing systems , 30, 2017.

Alberto Maria Metelli, Giorgia Ramponi, Alessandro Concetti, and Marcello Restelli. Provably efficient learning of transferable rewards. In International Conference on Machine Learning , pages 7665-7676. PMLR, 2021.

Eduardo Mosqueira-Rey, Elena Hernández-Pereira, David Alonso-Ríos, José Bobes-Bascarán, and Ángel Fernández-Leal. Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review , 56(4):3005-3054, 2023.

Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning , volume 1, page 2, 2000.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI , volume 7, pages 2586-2591, 2007.

Giorgia Ramponi, Gianluca Drappo, and Marcello Restelli. Inverse reinforcement learning from a gradient-based learner. Advances in Neural Information Processing Systems , 33:2458-2468, 2020.

Nathan D Ratliff, J Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In Proceedings of the 23rd international conference on Machine learning , pages 729-736, 2006.

R Tyrrell Rockafellar and Roger J-B Wets. Variational analysis , volume 317. Springer Science &Business Media, 2009.

Paul Rolland, Luca Viano, Norman Schürhoff, Boris Nikolov, and Volkan Cevher. Identifiability and generalizability from multiple experts in inverse reinforcement learning. Advances in Neural Information Processing Systems , 35:550-564, 2022.

Igal Sason. On reverse pinsker inequalities. arXiv preprint arXiv:1503.07118 , 2015.

Joar Max Viktor Skalse, Matthew Farrugia-Roberts, Stuart Russell, Alessandro Abate, and Adam Gleave. Invariance in policy optimisation and partial identifiability in reward learning. In Proceedings of the 40th International Conference on Machine Learning , 2023.

Umar Syed and Robert E Schapire. A game-theoretic approach to apprenticeship learning. Advances in neural information processing systems , 20, 2007.

Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. Maximum-likelihood inverse reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems , 35:10122-10135, 2022.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.

The structure of the supplementary materials is organized as follows:

Appendix A provide tables that summaries the main symbols used in this document.

Appendix B describes how to extend our result to the multiple optimal expert setting.

Appendix C provides proof of Examples 3.1-3.3.

Appendix D provides formal proofs of the theoretical claims of Section 3, together with discussions on alternative formulations of the IRL-SE problem.

Appendix E provides formal proofs of the theoretical claims of Section 4.

Appendix F discusses how to extend our result to the setting in which π E 1 is stochastic.

Appendix G discusses how to change the learning formalism to derive a lower-bound that directly depends on the number of samples that are needed from each sub-optimal expert.

Appendix H provides additional details on US-IRL-SE (i.e., exact description of m ) and computational complexity analysis.

Appendix I presents a simple numerical experiments that highlights the reduction of the feasible reward set due to the presence of multiple and sub-optimal experts.

A Symbols and Notation

To begin, we provide tables that summaries the main symbols used in this document. Table 1 reports a summary on the notation used throughout the paper. Table 2 reports a precise definition of the operators used.

SYMBOL MEANING

B Additional Multiple Optimal Expert Setting

In this section, we discuss the extension of the IRL-SE setting to the case in which multiple optimal policies are available. More specifically, we define this IRL-SE setting as a tuple

S

→

S

˜ B = ( M , ( π ∗ E i ) n 1 i =1 , ( π E i ) n 2 i =1 , ( ξ i ) n 2 i =1 ) , where ( π ∗ E i ) n 1 i =1 is a set of n 1 optimal policies, ( π E i ) n 2 i =1 is a set of n 2 sub-optimal policies with known degree of sub-optimality ( ξ i ) n 2 i =1 .

At this point, one can verify that the shape of the feasible set R ˜ B , is exactly the one of Theorem 3.6, where Equations (4) an (5) are obtained by replacing π E 1 with each optimal policy π ∗ E i with i ∈ { 1 , . . . , n 1 } .

Concerning learning the feasible reward set, it is sufficient to extend the generative model so that samples are gathered from all optimal and sub-optimal experts. The proof of Section 4 holds almost unchanged.

C Proof of Examples 3.1-3.3

We now give formal proofs of Examples 3.1-3.3.

Proof of Example 3.1. Consider two MDPs \R M 1 and M 2 that differs only in the transition kernel, which we denote by p 1 and p 2 . Suppose that π E 1 and π E 2 are optimal for M 1 and M 2 respectively. For any state s , it holds that:

where we added and subtracted V π E 1 M 2 ∪ r , and we used V π E 1 M 1 ∪ r ≤ V π E 2 M 2 ∪ r due to the optimality of π E 2 in M 2 . Then, focus on V π E 1 M 1 ∪ r ( s ) -V π E 1 M 2 ∪ r ( s ) (but an identical reasoning can be applied for the second difference as well):

which, in turn, can be further bounded by:

Unrolling the summation to iterate the aforementioned argument, and using the fact that ∑ + ∞ t =0 γ t = 1 1 -γ concludes the proof.

Proof of Example 3.2. In this proof, we explicit the relationship of the value function V with the discount factor γ by writing V π,γ M∪ r . Then, it holds that:

where we added and subtracted V π E 1 ,γ ′ M∪ r ( s ) , and we used V π E 1 ,γ ′ M∪ r ( s ) ≤ V π E 2 ,γ ′ M∪ r ( s ) due to the optimality of π E 2 for the discount factor γ ′ . Then, focus on V π E 1 ,γ M∪ r ( s ) -V π E 1 ,γ ′ M∪ r ( s ) . By the

definition of the value function, together with the fact that rewards are bounded in [0 , 1] , we can rewrite this difference as:

An identical argument holds for the second difference, thus concluding the proof.

Proof of Example 3.3. By using the definition of value function, we can rewrite V π E 1 M∪ r ( s ) -V π E 2 M∪ r ( s ) as ∑ a ( π E 1 ( a | s ) -π E 2 ( a | s )) r ( s, a )+ γ ∑ s ′ p ( s ′ | s, a )( V π E 1 M∪ r ( s ′ ) -V π E 2 M∪ r ( s ′ ))) . At this point, since rewards are bounded in [0 , 1] and by the policy similarity assumption, we have that the difference in value functions can be upper bounded by glyph[epsilon1] + γ ∑ s ′ p ( s ′ | s, a )( V π E 1 M∪ r ( s ′ ) -V π E 2 M∪ r ( s ′ )) . Unrolling the summation to iterate the aforementioned argument, and using the fact that ∑ + ∞ t =0 γ t = 1 1 -γ concludes the proof.

D Proofs and Derivations of Section 3

In this section, we provide formal proofs of the theoretical results of Section 3. We begin by reporting for completeness some results from [24] that will be used in our analysis.

Lemma D.1. Let B = ( M , π E ) be a single-agent IRL problem. Let r ∈ [0 , 1] S × A , then r is a feasible reward if and only if for all ( s, a ) ∈ ( S, A ) it holds that:

Lemma D.2. Let B = ( M , π E ) be a single-agent IRL problem. A Q-function satisfies condition of Lemma D.1 if and only if there exist ζ ∈ R S × A ≥ 0 and V ∈ R S such that:

Lemma D.3. Let B = ( M , π E ) be a single-agent IRL problem. Let r ∈ R S × A , then r is a feasible reward, if and only if there exist ζ ∈ R S × A ≥ 0 and V ∈ R S such that:

At this point, we proceed by proving a Lemma that implicitly describes the set of reward functions that are compatible with a IRL-SE problem.

Lemma D.4. Let ¯ B be an IRL problem with sub-optimal experts. Let r ∈ [0 , 1] S×A . Then, r ∈ R ¯ B if and only if the following conditions are satisfied:

Proof. Condition (i) and (ii) are necessary conditions for the claim to hold. This directly follows by the definition of R ¯ B and from Lemma D.1. At this point, what remains to be proven is condition (iii).

We conclude by showing that if (i)-(iii) are satisfied, then r ∈ R B . Since (i) and (ii) holds, then, by Lemma D.1, we know that π E 1 is optimal for M∪ r . It remains to be proven that V π E 1 M∪ r -V π Ei M∪ r ≤ ξ i holds as well. However, since (i) holds, (iii) implies that V π E 1 M∪ r ( s ) -V π Ei M∪ r ( s ) ≤ ξ i ( s ) holds for all s ∈ S , thus concluding the proof.

We begin by showing that, if r ∈ R B then condition (iii) is satisfied. Since (i) and (ii) holds, it sufficies to plug V π E 1 M∪ r -V π Ei M∪ r ≤ ξ i within (i) and (ii) to obtain the desired result.

We continue by proving Theorem 3.6.

Theorem 3.6. Let ¯ B be an IRL problem with sub-optimal experts. Let r ∈ [0 , 1] S×A . Then, r ∈ R ¯ B if and only if there exists ζ ∈ R S×A ≥ 0 and V ∈ R S such that the following conditions are satisfied:

and, for all i ∈ { 2 , . . . , n +1 } :

Proof. First of all, Equation (4) is a necessary condition for the claim to hold. Indeed, from Lemma D.3 and Lemma D.4, Equation (4) is a necessary and sufficient condition to make π E 1 optimal in M∪ r .

At this point, we proceed conditioning on the fact that Equation (4) holds. We need to show that, if r ∈ R ¯ B , then Equation (5) holds, and if Equation (5) holds, then r ∈ R ¯ B .

From Lemma D.4, we know that a necessary and sufficient condition for the previous statements to hold is that:

holds for any state-action pair ( s, a ) and for all experts i ∈ { 2 , . . . , n +1 } . Given Equation (4) and Lemma D.2, the previous equation can be conveniently written as:

At this point, consider ( s, a ) such that π E 1 ( a | s ) > 0 . Then, Equation (14) reduces to:

Conversely, consider ( s, a ) such that π E 1 ( a | s ) = 0 , then Equation (14) reduces to:

Since ζ ( s, a ) ≥ 0 by assumption, Equation (16) is directly implied by Equation (15). Therefore, it suffices to study:

At this point, we can plug Equation (4) within the previous Equation. More specifically, we obtain:

We now proceed by rewriting the right hand side of Equation (17). More precisely, we notice that:

Plugging this result within Equation (17), we arrive at:

which can be rewritten as:

which concludes the proof.

D.1 Other Assumptions on the Behavior of the Sub-optimal Experts

In this section, we investigate the generality of the results presented in Theorem 3.6. Specifically, we remark that all the results that we derived in Section 3 can be easily extended to other assumptions on the sub-optimal experts. More specifically, the results of Theorem 3.6 can easily be extended to the following cases:

or

Equation (18) encodes the fact that that a given sub-optimal expert i is at least ξ i sub-optimal w.r.t. the optimal policy π E 1 , while Equation (19) encodes the the fact that the sub-optimal expert i is exactly ξ i sub-optimal w.r.t. the optimal agent. In these cases, it is possible to derive the following generalizations of Theorem 3.6.

Theorem D.5. Let ¯ B be an IRL problem with sub-optimal experts where V π E 1 M∪ r -V π Ei M∪ r ≥ 1 S ξ i holds for all sub-optimal experts i . Let r ∈ [0 , 1] S×A . Then, r ∈ R ¯ B if and only if there exists ζ ∈ R S×A ≥ 0 and V ∈ R S such that the following conditions are satisfied:

and, for all i ∈ { 2 , . . . , n +1 } :

Theorem D.6. Let ¯ B be an IRL problem with sub-optimal experts where V π E 1 M∪ r -V π Ei M∪ r = 1 S ξ i holds for all sub-optimal experts i . Let r ∈ [0 , 1] S×A . Then, r ∈ R ¯ B if and only if there exists ζ ∈ R S×A ≥ 0 and V ∈ R S such that the following conditions are satisfied:

and, for all i ∈ { 2 , . . . , n +1 } :

The proofs of Theorem D.5 and D.6 are identical to the one of Theorem 3.6. In terms of results, the only difference lies in the fact that the set of linear constraints introduces a different type of relationship between ζ and ξ i . 10

At this point, we remark that Theorem D.5 follows a very similar interpretation of the one we presented in Section 3. In other words, the sub-optimal experts introduces lower bounds on the values of the advantage function that are played by the sub-optimal experts π E i and that are not played by the optimal expert. 11 This result is as expected. Indeed, if we know that a given policy is sub-optimal at least by a given quantity ξ i , then, intuitively, we can extrapolate knowledge, expressed as lower bounds, on how sub-optimal certain actions are.

Concerning Theorem D.6, instead, we notice that the result is much more stronger w.r.t. to the case in which inequalities are involved in the problem (i.e., Theorem 3.6 and Theorem D.5). More specifically, we notice that in this case, starting from Equation (23), it is possible to obtain the following result on the values of ζ :

Developing this constraint for a specific state s , we obtain the following linear constraint:

In other words, it provides a set of hard constraints that the values of ζ should satisfy. At this point, we remark on two important observations. The first one is that Equation (25) might not be satisfied for any choice of ζ ∈ R S × A . In particular, suppose that there is only one sub-optimal expert. In this case, if π E 1 = π E i , Equation (25) reduces to:

which is clearly false for any strictly positive value of ξ i . We notice that this result is as expected. If the two policies are identical, then there should be no gap in the performance of π E 1 and π E i . As a consequence, the feasible reward set of this IRL problem is empty. 12 Secondly, instead, suppose that all the experts are deterministic, and suppose that they all behave differently to π E 1 in each state-action pair (so that the feasible reward set is non-empty). Then, focus on the i -th expert and consider a state-action pair ( s, a ) such that π E i ( a | s ) = 1 . Then, Equation (25) reduces to:

In this sense, the presence of the sub-optimal expert implies a unique value that ζ ( s, a ) can take. 13 It follows that, is for each state-action pair ( s, a ) such that π E 1 ( a | s ) = 0 , there exists a single sub-optimal expert i such that π E i ( a | s ) = 1 , we are able to recover entirely a unique vector ¯ ζ that is compatible with the underlying IRL problem. In other words, in this scenario, we are able to exactly recover the values of the advantage function that express the sub-optimality gaps of actions that are not played by the optimal agent.

D.2 Measuring Volumes of the Feasible Values of ζ

Finally, we conclude this section by reporting an additional analysis that quantitatively measure the reduction in the feasible values of ζ that are compatible with the presence of the sub-optimal experts.

To begin, we first introduce some notation. Let X be a measurable subset of R n , we denote with Vol ( X ) the Lebesgue measure of X [7]. In other words, V ol ( X ) represents the n -dimensional volume of X .

At this point, from Theorem 3.6, we know that the presence of sub-optimal experts can effectively limit the values that ζ can assume. Consequently, in order to measure the reduction of the feasible reward set, we will compute upper bounds on the volume of the region of ζ that induces at least one feasible reward function in R ¯ B . In the remainder of this section, for a generic IRL problem B , we will denote with Z ( B ) such set. More specifically, we define:

Notice that, we are directly restricting the analysis to state-action pairs for which π E 1 ( a | s ) = 0 holds. Indeed, fix ( s, a ) such that π E 1 ( a | s ) = 0 holds. Then, changing the value of ζ ( s, a ) does not affect class of compatible reward functions.

At this point, the following proposition provides upper bounds on the volume of the feasible values of ζ for IRL and IRL-SE problems.

Proposition D.7. Let B and ¯ B be an IRL and an IRL-SE problem. Let g ( s, a ) be defined as follows:

where k ( s, a ) is defined as in Equation (7) , and X ( s, a ) denotes the subset of sub-optimal experts such that π E i ( a | s ) > 0 . Then, it holds that:

Proof. The proof of Equation (26) follows directly by noticing that ‖ ζ ‖ ∞ ≤ (1 -γ ) -1 . Therefore, as a consequence:

Equation (27), instead, follows from worst-case upper bounds on ζ ( s, a ) that arise from Equation (5). Specifically, as discussed in Section 3.3, it is possible to show that:

As a consequence, we have that:

which concludes the proof.

At this point, we recall that k ( s, a ) is given by:

Therefore, Proposition D.7 highlights that, for sufficiently small values of ξ i , we obtain a notable reduction in the upper bounds of Vol ( Z ( ¯ B )) .

E Proofs and Derivations of Section 4

In this section, we provide formal proofs for the statements of Section 4. We first proceed with the proof of the lower bound (Theorem 4.1), and then we continue with the analysis US-IRL-SE (Theorem 4.2).

E.1 Proof of Theorem 4.1

In this section, we prove Theorem 4.1. We recall that Theorem 4.1 is composed of two parts. The first one is related to learning reward functions that are compatible with the behavior of the optimal expert (i.e., Equation (9)), while the second one directly arises from the structure of the sub-optimal experts (i.e., Equation (11)).

We begin by proving Equation (9). As we have anticipated in Section 4.2, Equation (9) is directly connected with the problem of learning reward functions that are compatible with the behavior of the optimal expert π E 1 . In this sense, we recall that, recently, [22] have provided lower bounds for the single-agent IRL problem in finite-horizon MDPs. In our work, we provide an extension of these results for the infinite-horizon IRL formulation.

Theorem E.1. Let A be a ( glyph[epsilon1], δ ) -correct algorithm for the IRL problem with sub-optimal experts. There exists a problem instance ¯ B such that the expected sample complexity is lower bounded by:

Proof. Similarly to [22], this results follows from two different lower bounds (i.e., Theorem E.2 and Theorem E.3), and by assuming to observe instances like the ones of Theorem E.2 with probability 1 2 and instances like the ones of Theorem E.3 with probability 1 2 .

As discussed, Theorem E.1 follows from two intermediate results that leverage two different constructions (i.e., Theorem E.2 and Theorem E.3). We now delve into the proofs of these intermediate theorems.

s

root

...

...

s

j

s

1

s

S

s

+

s

-

w.p.

1

S

w.p.

1

S

w.p.

1

S

w.p.

1

2

w.p.

1

2

w.p.

1

2

w.p.

1

2

w.p.

1

2

+

glyph[epsilon1]

′

w.p.

1

2

-

glyph[epsilon1]

′

w.p.

1

2

w.p.

1

2

Theorem E.2. Let A be a ( glyph[epsilon1], δ ) -correct algorithm for the IRL problem with sub-optimal experts. There exists a problem instance ¯ B such that the expected sample complexity is lower bounded by:

Proof. The proof is split into several steps. Since most of the arguments are borrowed from the proof of Theorem B.2 in [22], we only provide a short sketch that involves the main differences between the finite-horizon and the infinite horizon IRL model.

Step 1: Base Instance and Alternative Instances We consider MDP\R instances that are presented in Figure 3. More specifically, the state space is given by S = { s root , s 1 , . . . , s ¯ S , s -, s + } , the action space is composed of k actions A = { a 1 , a 2 , . . . , a k } , the transition model is described below, and γ ∈ (0 , 1) .

We then consider a set of ¯ S × A alternative instances by varying the behavior of the transition model in state-action pairs ( s j , a k ) ∈ { s 1 , . . . s ¯ S } × A . Specifically, by taking a k , the agent will end up, with probability 1 2 + glyph[epsilon1] ′ in s + , and, with probability 1 2 -glyph[epsilon1] ′ in s -.

In state s root all actions have the same effect, and they lead, with probability 1 ¯ S to a state in { s 1 , . . . , s ¯ S } . Similarly, in state s -and s + all actions have deterministic effect, and they all lead to s -and s + respectively. All the experts are deterministic and take action a 1 with probability 1 in all states. We then consider a base instance, where, in each state s j ∈ { s 1 , . . . , s ¯ S } , all actions lead, with probability 1 2 to s -and s + .

Step 2: Feasible Reward Set and Hausdorff Distance lower bound At this point, we study the structure of the feasible reward set that is compatible with the instances we described. Specifically, we are interested in the behavior of the reward function related to actions taken in states s j ∈ { s 1 , . . . s ¯ S } .

Specifically, for the base instance, we have:

which can be rewritten as:

For the alternative instance in which we varied the behavior of the state-action pair ( s j , a k ) , instead, we obtain:

thus leading to:

Given this construction, we can lower bound the Hausdorff distance between the feasible reward set of the base and the alternative instance. Specifically, we first pick a reward function r ′ compatible with the alternative instance as follows: r ′ ( s -, a 1 ) = 1 , r ′ ( s + , a 1 ) = 0 , r ′ ( s j , a k ) = 1 and r ′ ( s j , a 1 ) = 1 -glyph[epsilon1] ′ γ 1 -γ . Then, we study which reward function compatible with the base instance minimizes the infinite norm w.r.t. r ′ . Given these choices, similarly to [22], it is possible to obtain the following lower-bound to the Hausdorff distance:

where R denotes the feasible reward set of the base instance, while R ′ denotes the feasible reward set of the alternative instance. We enforce the following constraint on the previous equation:

which, in turns, leads to the following requirement on glyph[epsilon1] ′ :

Step 3: Lower bounding the sample complexity At this point, the rest of the proof follows identical to Theorem B.2 in [22], and leads to the desired result.

We now continue with the proof of second intermidiate result that is needed for the proof of Theorem E.1.

Theorem E.3. Let A be a ( glyph[epsilon1], δ ) -correct algorithm for the IRL problem with sub-optimal experts. There exists a problem instance ¯ B such that the expected sample complexity is lower bounded by:

Proof. The proof is split into several steps. Since most of the arguments are borrowed from the proof of Theorem B.3 in [22], we only provide a short sketch that involves the main differences between the finite-horizon and the infinite horizon IRL model.

Step 1: Base Instance and Alternative Instances We consider MDP\R instances that are presented in Figure 4. More specifically, the state space is given by S = { s root , s 1 , . . . , s ¯ S , s ′ 1 , . . . , s ′ ¯ S } , the action space is composed of k actions A = { a 1 , a 2 , . . . , a k } , the transition model is described below, and γ ∈ (0 , 1) . In the following, we will assume ¯ S to be divisible by 16 .

In state s root all actions have the same effect, and they lead, with probability 1 ¯ S to a state in { s 1 , . . . , s ¯ S } . In state s ∈ { s ′ 1 , . . . , s ′ ¯ S } all actions have deterministic effect, and they all lead to the same state in which the action is taken. All the experts are deterministic and take action a 1

s

root

...

...

s

1

s

S

s

′

1

s

′

2

s

′

S

...

w.p.

1

S

w.p.

1

S

a

j

w.p.

1+

glyph[epsilon1]

′

v

(

s S

,a j ,h

)

1

S

a

j

w.p.

1+

glyph[epsilon1]

′

v

(

s S

,a j ,h

)

2

S

a

j

w.p.

1+

glyph[epsilon1]

′

v

(

s S

,a j ,h

)

S

S

a

j

w.p.

1+

glyph[epsilon1]

′

v

(

s

1

,a j ,h

)

1

S

a

j

w.p.

1+

glyph[epsilon1]

′

v

(

s

1

,a j ,h

)

2

S

a

j

w.p.

1+

glyph[epsilon1]

′

v

(

s

1

,a j ,h

)

S

S

with probability 1 in all states. Finally, in all states s ∈ { s 1 , . . . s ¯ S } , action a 1 leads w.p. 1 ¯ S to any state in s ′ ¯ S ∈ { s ′ 1 , . . . , s ′ ¯ S } .

We then consider a class of instances by varying the behavior of the transition model in state-action pairs ( s j , a k ) ∈ { s 1 , . . . s ¯ S } × A \ { a 1 } . Specifically, for each triplet ( s j , s ′ i , a k ) , p ( s ′ i | s j , a k ) = 1+ glyph[epsilon1] ′ v ( s j ,a k ) i 2 , where v ( s j , k ) = ( v ( s i ,a j ) 1 , . . . v ( s i ,a j ) ¯ S ) ∈ V , and V = { {-1 , 1 } ¯ S : ∑ ¯ S i v j = 0 } and glyph[epsilon1] ′ ∈ [ 0 , 1 2 ] .

Step 2: Feasible Reward Set and Hausdorff Distance lower bound We now proceed with lower bounding the Hausdorff distance between the feasible reward set that is induced by two instances that belongs to V , namely R ¯ B v and R ¯ B w . To this end, first of all, we notice that our class of reward functions admits elements that are bounded in [0 , 1] . Let us denote with ¯ R ¯ B v the class of compatible rewards functions that are bounded in [ -1 , 1] . Then, we have that:

The proof of Equation (31) follows by the following reasonings. First of all, we can see that, for any v , ¯ r ∈ ¯ R ¯ B v holds if and only if ¯ r +1 2 ∈ R ¯ B v holds. 14 Then, with simple algebraic manipulations, we have that:

from which it follows Equation (31).

At this point, our analysis follows by lower bounding the Hausdorff distance using reward functions that are bounded in [ -1 , 1] . To this end, first of all, we analyze the feasible reward set for a single instance ¯ B v . In this case, since a 1 is played by the optimal expert, we have that:

s

root

...

...

s

j

s

1

s

S

s

sink

w.p.

1

S

w.p.

1

S

regardless the action w.p.

1

S

a

2

a

1

a

2

a

1

π

E

1

(

a

1

|

s

j

)=1

π

E

i

(

a

1

|

s

j

)=1

-

π

min

π

E

i

(

a

2

|

s

j

)=

π

min

which, in turns, leads to:

At this point, following the same steps of [22], we obtain:

on which we enforce the following constraint:

that is glyph[epsilon1] ′ ≥ 128 glyph[epsilon1] 1 -γ γ .

Step 3: Lower-bounding the sample complexity At this point, the rest of the proof follows identical to Theorem B.3 in [22], and leads to the desired result.

We continue by proving the novel statement related to Equation (11), which captures the role of the estimating reward functions that are compatible with the behavior of the sub-optimal experts.

Theorem E.4. Let A be a ( glyph[epsilon1], δ ) -correct algorithm for the IRL problem with sub-optimal experts. For sufficiently small values of glyph[epsilon1] , there exists an instance ¯ B in which q 0 < 1 such that the expected sample complexity is lower bounded by:

Proof. The proof follows as the combination of several steps. We first describe the construction considering the case in which there is a single sub-optimal expert. Then, the results will follow by iterating the procedure for each sub-optimal expert in the given set.

Step 1: Base Instance and Alternative Instance We consider MDP\R instances that are presented in Figure 5. More specifically, the state space is given by S = { s root , s 1 , . . . , s ¯ S , s sink } , the action space is composed of two actions A = { a 1 , a 2 } , the transition model is described below, and γ ∈ (0 , 1) .

In state s root all actions have the same effect, and they lead, with probability 1 ¯ S to a state in { s 1 , . . . , s ¯ S } . In each state s i ∈ { s 1 , . . . , s ¯ S } , again, all actions have the same effect, and they lead, deterministically to state s sink . In state s sink all actions are deterministic and they all lead to s sink .

We then consider a base instance, where, in states s root and state s sink all experts take action a 1 with probability 1 . In each state s j ∈ { s 1 , . . . , s ¯ S } , instead, π E 1 ( a 1 | s j ) = 1 , while π E i ( a 1 | s j ) = 1 -π min and π E i ( a 2 | s j ) = π min for the sub-optimal expert.

We then consider a set of ¯ S alternative instances by varying the behavior of the sub-optimal expert in state s j ∈ { s 1 , . . . s ¯ S } . Specifically, the sub-optimal expert will act accordingly to the following policy: π E i ( a 1 | s j ) = 1 -απ min, and π E i ( a 2 | s j ) = απ min for some α > 1 (to be defined later). We will denote our set of instances as M = { π ( l ) E i : l ∈ [ ¯ S ] } , and 0 denotes the behavior of the base instance we described above.

Step 2: Feasible Reward Set At this point, we study the structure of the feasible reward set that is compatible with the each instance we described. Specifically, we are interested in the properties of feasible reward functions in states s j ∈ { s 1 , . . . , s ¯ S } .

For the base instance, in each state s j ∈ { s 1 , . . . , s ¯ S } , we have that, since a 1 is the action taken by the optimal expert:

which reduces to:

Moreover, due to the presence of the sub-optimal expert, we have that V π (0) E 1 M∪ r ( s j ) ≤ V π Ei M∪ r ( s j ) + ξ i , which results in:

Similarly, for each alternative instance in M , we obtain:

Step 3: Lower-bounding the Hausdorff Distance At this point, we proceed by lower bounding the Hausdorff distance between the feasible reward set of the base instance, namely R B (0) , and that of alternative instance j in which the policy of the sub-optimal expert is modified in state s j , namely R B ( j ) .

To this end, we notice that it is sufficient to pick any r ∈ R B (0) , and then study the following optimization problem:

which can be rewritten as:

We pick the following choice for r . For action a 1 in state s j we pick r ( s j , a 1 ) = ξ i π min . Instead, for all the other state-action pair, we pick r ( · , · ) = 0 . 15 In this case, one can verify that the optimization problem in (37) can be rewritten as follows:

where x corresponds to r ′ ( s j , a 1 ) , and y to r ′ ( s j , a 2 ) . The optimization problem in (38) can be further lower-bounded by:

We study the optimization problem in (39) by cases (i.e., ξ i π min -1 < x , and ξ i π min -1 ≥ x ). First, we suppose that ξ i π min -1 < x . However, in this case, we would have y ≤ x -ξ i π min -1 , that is x ≥ y + ξ i π min -1 . Chaining this inequality with x ≤ y + ξ i ( απ min ) -1 yields:

which is impossible since α > 1 . Concerning the case in which ξ i π min -1 ≥ x , instead, we have that y ≤ ξ i π min -1 -x . It follows that, since x ≤ ξ i π -1 min + y , the maximum value of x will be attained for the maximum value of y . Therefore, we obtain:

Plugging this result into the objective function, we obtain the solution to optimization problem (39), which in turn provides a lower bound to the Hausdorff distance. More specifically, we have that:

We enforce the following constraint on this quantity, that is:

We notice that in this sense, we need to impose α = ξ i ( ξ i -4 glyph[epsilon1]π min ) -1 , which results being greater than 1 for all values of ξ i > 4 glyph[epsilon1]π min. Furthermore, since απ min < 1 , 16 we obtain the following condition on the values of glyph[epsilon1] which guarantes that the construction is valid:

Thus leading to:

Step 4: Lower-bounding Probability At this point, consider an ( glyph[epsilon1], δ ) -correct algorithm A that outputs the correct feasible set R ̂ ¯ B for each IRL-SE problem ¯ B . For A it holds that:

where P ¯ B , A denotes the joint probability distribution of all events realized by the execution of algorithm A in the IRL-SE problem ¯ B , that is:

At this point, following the same reasonings of, e.g., Theorem B.2 in [22], one can see that, for all the alternative instances in l ∈ M , the previous equation can be further lower-bounded to obtain:

Step 5: KL-divergence As a final step, we will proceed by analyzing Equation (46). We will start by analyzing the KL divergence between P ¯ B (0) , A and P ¯ B (0) , A . Specifically, we have that:

where TV ( p 1 , p 2 ) represents the total variation distance between distribution p 1 and p 2 . The first inequality step follows from the reverse Pinkser's inequality [33], and the second one, instead, from the definition of the total variation distance.

At this point, however, from Equation (41), we obtain that:

thus leading to:

Plugging this result into Equation (46), and rearranging the terms, we obtain:

Finally, to conclude, we notice that:

Iterating this procedure over all the possible experts yields the desired result.

Finally, we are now ready to prove Theorem 4.1.

Theorem 4.1. Let A be a ( glyph[epsilon1], δ ) -correct algorithm for the IRL problem with sub-optimal experts. There exists a problem instance ¯ B such that the expected sample complexity is lower bounded by:

where Ω( · ) hides constant dependencies. Let π min > 0 be:

and let q 0 := π -1 min max i ∈{ 2 ,...,n +1 } ξ i . Then, there exists an instance ¯ B ′ in which q 0 < 1 such that:

Proof. The proof follows directly by combining Theorem E.1 and Theorem E.4.

E.2 Proof of Theorem 4.2

In this section, we provide a formal proof for Theorem 4.2.

First of all, we begin by defining a good event E that holds with probability at least 1 -δ . To this end, we begin by reporting, for the sake of completeness, the concentration tools that are used in controlling the probability with which E holds.

Lemma E.5 (Multiplicative Chernoff bound [10]) . Consider t independent random variables X 1 , . . . , X t taking values in [0 , 1] . Suppose that E [ X i ] = µ holds for all i ∈ { 1 , . . . , t } . Consider α ∈ (0 , 1) . Then, we have that:

Lemma E.6 (Proposition 1 in [14]) . Let p be a categorical distribution over the simplex of dimension y , and let ˆ p be the maximum likelihood estimate of p obtained with t ≥ 1 independent samples. Then, for all δ ∈ (0 , 1) it holds that:

where KL ( q 1 , q 2 ) denotes the Kullback-Leibler divergence between distributions q 1 and q 2 .

At this point, we proceed by defining our good event. In the following, the subscript t is used to denote the iteration number of the US-IRL-SE algorithm.

Lemma E.7 (Good event) . Consider t such that:

holds for all i ∈ { 2 , . . . , n +1 } and for all ( s, a ) ∈ S × A such that π E i ( a | s ) > 0 . Let us define the following events:

Consider E = E p ∩ E π ∩ E π min . Then, it holds that P ( E ) ≥ 1 -δ

Proof. The proof follows by controlling the probability of the complementary event E c . More specifically, we have that:

where in the last passage we have used Boole's inequality. At this point, we will use Lemma E.5 and Lemma E.6 to control P ( E c p ) + P ( E c π ) + P ( E c π min ) . More specifically, concerning E c p , we have that:

where, in the first step we have applied Boole's inequality, and in the second one Lemma E.6. From identical reasoning, we can upper bound P ( E c π ) , thus obtaining:

Finally, concerning E c π min , we have that:

where, in the first step we have used Boole's inequality, and in the second one Lemma E.5 together with Equation (49). At this point, combining these results, we obtain P ( E c ) ≤ δ , thus concluding the proof.

At this point, it has to be remarked that Lemma E.7 holds whenever:

We remark that this term is due to the requirement that Lemma E.5 requires α in (0 , 1) . As we shall see, however, this will be sufficient to carry out the theoretical analysis of US-IRL-SE.

We now proceed by presenting error propagation results that study the Hausdorff distance between R ¯ B and R ̂ ¯ B . Before diving into the details, we introduce the following notation. We denote with Ψ the set of ζ 's that are compatible with the linear constraints of Equation (5). More specifically:

Similarly, we denote with ̂ Ψ , the set of ζ 's that are compatible with the linear constraints of Equation (5) that are induced by the empirical IRL problem ̂ ¯ B . More precisely:

where ˆ d π denotes the discounted expected occupancy of policy π under the transition model ˆ p .

At this point, we provide a preliminary Lemma that will be used to study the Hausdorff distance.

Lemma E.8 (Error Propagation) . Let ¯ B be an IRL-SE problem and let ̂ ¯ B be its empirical estimate. Then, for any r ∈ R ¯ B such that r = -¯ B π E 1 ζ +( E -γP ) V , and ( I -γπ E i P ) -1 π E i ¯ B π E 1 ζ ≤ ξ i , there exists ˆ r ∈ R ̂ ¯ B such that element-wise it holds that:

where proj ̂ Ψ ( · ) denotes the infinite norm projection on the set ̂ Ψ . More formally, for ζ ∈ R S×A :

Proof. From Theorem 3.6, we know that we can express the reward functions as:

where V and ˆ V belongs to R S , and ζ , ˆ ζ ∈ R S×A ≥ 0 satisfies the following equations for all i ∈ { 2 , . . . , n +1 } :

At this point, we recall that, for any r ∈ R B , we are interested in a specific reward function ˆ r ∈ R ̂ B that is sufficiently close to r . For this reason, we pick ˆ V = V , and ˆ ζ = ¯ B π E 1 proj ̂ Ψ ( ζ ) . 17 Plugging these choices within | r -ˆ r | , and applying the triangular inequality, we obtain:

Let us focus now on | -( ¯ B π E 1 ζ + ¯ B ˆ π E 1 ˆ ζ ) | .

where in the last step we have used B π + ¯ B π = I S×A . 18 We now focus on the last term:

Plugging these results within Equation (53) concludes the proof.

Before providing some interpretation to Lemma E.8, we recall the definition of the Hausdorff distannce we are interested in:

Given this definition, we can appreciate how Lemma E.8 can be used to upper-bound the error of the first component of the Hausdorff distance, namely:

Nevertheless, it is possible to obtain a symmetric version of Lemma E.8 that analyzes the existence of an r ∈ R ¯ B that is sufficiently close to any ˆ r ∈ R ̂ ¯ B . At this point, we notice that Lemma E.8 decomposes the error on the recovered reward set as the sum of three different components. The first one is related to the error in the estimation of the optimal policy π E 1 , and, to zero it out, it is sufficient to estimate for each state one action that is played by the optimal expert. The second one, instead, is related to the errors in the estimation of the transition model, and, finally, the last one is related to the presence of multiple and sub-optimal experts. More precisely, we notice that the first two components are present also for the single-agent setting [24], and, in this sense, they arise from the difficulty of estimating reward functions that are compatible with Equation (4). The last term, instead, is related to the presence of sub-optimal experts, and it denotes the infinite norm between any ζ ∈ Ψ and its infinite-norm projection to the set ̂ Ψ . As one can imagine, and as our proofs will reveal, studying how this last source of error decreases with the iterations of US-IRL-SE introduces significant challenges in the analysis w.r.t. the single agent setting. Precisely, this complexity will be tackled within the proof of the following Lemma, which analyzes how the error of Equation (52) decreases with the number of iterations of US-IRL.

Lemma E.9 (High-Probability Error Propagation) . Let t be the iteration of US-IRL. Suppose that N t ( s, a ) ≥ 1 for all ( s, a ) ∈ S × A and, furthermore, suppose that t is such that:

Then let ¯ B be an IRL-SE problem and let ̂ ¯ B be its empirical estimate after t iteration of US-IRL. Then, with probability at least 1 -δ , for any r ∈ R ¯ B , there exists ˆ r ∈ R ̂ ¯ B such that:

where:

Proof. The proof follows by analyzing in greater detail the result of Lemma E.8. Indeed, from Lemma E.8, we know that there exists ˆ r ∈ R ̂ ¯ B such that:

We split the analysis of Equation (64) into two three parts. First of all, we will focus on the two first terms, that are ‖ ¯ B π E 1 B ˆ π E 1 ζ ‖ ∞ and γ ‖ ( P -ˆ P ) V ‖ ∞ , and then we will tackle the most challenging aspect, that is ‖ ζ -proj ̂ Ψ ( ζ ) ‖ ∞ .

Concerning ‖ ¯ B π E 1 B ˆ π E 1 ζ ‖ ∞ , we notice that, whenever N t ( s, a ) > 1 holds, we have that:

for any possible value of ζ . This is a direct consequence of the fact that π E 1 is deterministic.

Secondly, let us focus on γ ‖ ( P -ˆ P ) V ‖ ∞ . First of all, we notice that, since N t ( s ) > 3 log ( 3 SAnδ -1 ) π min holds for all s ∈ S , then E holds with probability at least 1 -δ (Lemma E.7).

At this point, conditioning on E , we have that:

where the first inequality follows the fact that ‖ V ‖ ∞ ≤ 1 1 -γ , the third one by Pinksker's inequality, and the last one from Lemma E.7. At this point, since US-IRL gathers samples uniformly from each state-action pair, we have that N t ( s, a ) = t for all ( s, a ) ∈ S × A . Thus leading to:

Finally, we focus on ‖ ζ -proj ̂ Ψ ( ζ ) ‖ ∞ . To upper-bound this last term we proceed in several steps.

Step 1: Relationship between Ψ and ̂ Ψ First of all, we begin by finding a more explicit relationship between Ψ and ̂ Ψ . To this end, we recall that ζ ∈ R S × A ≥ 0 belongs to ̂ Ψ if the following condition is satisfied for all i ∈ { 2 , . . . , n +1 } :

Under the assumption that N t ( s, a ) ≥ 1 , since the expert policy π E 1 is deterministic, the previous Equation can be equivalently rewritten as:

At this point, we proceed with some algebraic manipulations of the left-hand side of the previous Equation. Specifically:

At this point, focus on ˆ d ˆ π Ei -d π Ei :

At this point, plugging this result into the previous Equation, we obtain that ζ belongs to ̂ Ψ if and only if the following holds for all experts i ∈ { 2 , . . . , n +1 } :

As we can appreciate, Equation (65) relates the sets Ψ and ̂ Ψ by an explicit relationship. Indeed, by renaming glyph[epsilon1] 1 ( ζ ) := γd π Ei [ ˆ π E i ˆ P -π E i P ] ˆ d π ˆ π E i ¯ B π E 1 ζ and glyph[epsilon1] 2 ( ζ ) := d π Ei (ˆ π E i -π E i ) ¯ B π E 1 ζ , Equation (65) can be rewritten as:

which closely resambles the definition of Ψ .

Step 2: Restricting the set ̂ Ψ We continue our proof by defining a new set ˜ Ψ such that ˜ Ψ ⊆ ̂ Ψ with probability at least 1 -δ . Indeed, by definition of the projection according to the infinite norm, this allows to upper-bound ‖ ζ -proj ̂ Ψ ( ζ ) ‖ ∞ . More specifically, for any ˜ Ψ ⊆ ̂ Ψ , we have that:

More specifically, in order to define ˜ Ψ , we will first proceed by upper bounding glyph[epsilon1] 1 ( ζ ) and glyph[epsilon1] 2 ( ζ ) . Precisely, consider ζ ∈ ̂ Ψ . For glyph[epsilon1] 1 ( ζ ) we have that:

where, given f ∈ R S , and g ∈ R S × A , ∣ ∣ ∣ ˆ π -π ∣ ∣ ∣ denotes the operator defined as ∣ ∣ ∣ ˆ π -π ∣ ∣ ∣ g ( s, a ) = ∑ a ∣ ∣ ∣ ˆ π ( a | s ) -π ( a | s ) ∣ ∣ ∣ g ( s, a ) , and, similarly ∣ ∣ ∣ ˆ P -P ∣ ∣ ∣ f ( s ) = ∑ s ′ ∣ ∣ ∣ ˆ p ( s ′ | s, a ) -p ( s ′ | s, a ) ∣ ∣ ∣ f ( s ′ ) . At this point, we notice that, since, for all ζ ∈ ̂ Ψ , it holds that ˆ d ˆ π Ei ˆ π E i ¯ B π E 1 ≤ 1 S ξ i . Therefore, we can further upper bound the previous equation to obtain:

At this point, let us focus on γ ∣ ∣ ∣ ∣ ∣ ∣ [ d π Ei ∣ ∣ ∣ ˆ π E i -π E i ∣ ∣ ∣ ] 1 S×A ξ i ∣ ∣ ∣ ∣ ∣ ∣ ∞ :

where in the third step we have used Pinkser's inequality, in the fourth we have used Lemma E.7, and in the fifth one we have used the fact that, in US-IRL-SE, N t ( s ) = ∑ a ∈A N t ( s, a ) = tA . At this

where in the second step we have used Pinkser's inequality, in the third one Lemma E.7, and in the fourth one we have used the fact that, in US-IRL-SE, N t ( s ) = t for all states s ∈ S . Therefore, plugging these results within Equation (67), we obtain an high-probability upper bound on glyph[epsilon1] 1 ( ζ ) , that is:

We now proceed with similar reasoning to obtain an upper bound on glyph[epsilon1] 2 ( ζ ) . Nevertheless, contrary to glyph[epsilon1] 1 ( ζ ) , here we proceed with an element-wise upper bound on glyph[epsilon1] 2 ( ζ ) . Specifically, for each state s ′ ∈ S :

where the third step follows from the fact that, since N t ( s ) ≥ 1 , ˆ π E i ( a | s ) = 0 for all ( s, a ) such that π E i ( a | s ) = 0 , the fifth one, instead, from Lemma E.7, and the last one from the fact that, in US-IRLSE, N t ( s ) = ∑ a N t ( s, a ) = tA . 19 Therefore, we have obtained an element-wise upper-bound on glyph[epsilon1] 2 ( ζ ) of the following form:

We are finally ready to define the set ˜ Ψ . More specifically:

As one can verify, the definition of ˜ Ψ follows from upper-bounding glyph[epsilon1] 1 ( ζ ) and glyph[epsilon1] 2 ( ζ ) with Equations (68) and (69). As a direct consequence, whenever ζ ∈ ˜ Ψ , we have that ζ belongs to ̂ Ψ as well.

Step 3: Ensuring that the feasible region of ˜ Ψ is non-empty At this point, one might be tempted to directly study the projection of ‖ ζ -proj ˜ Ψ ( ζ ) ‖ ∞ . Nevertheless, we notice that, for sufficiently large values of α t and β t , ˜ Ψ might be empty.

Sufficient conditions to guarantee that ˜ Ψ is not empty are the following ones:

Using Lemma 12 in [14], it is possible to verify that Equations (56) and (58) are sufficient conditions for Equations (71)-(72) to hold.

Step 4: Picking ˜ ζ ∈ ˜ Ψ to upper-bound ‖ ζ -proj ̂ Ψ ( ζ ) ‖ ∞ At this point, we are ready to conclude our proof. As we have previously verified, the set ˜ Ψ is non-empty. We can now study ‖ ζ -proj ̂ Ψ ( ζ ) ‖ ∞ . More precisely, we have that:

To further upper bound this Equation, we notice that we can always pick, by definition of the infinite norm projection, any ˜ ζ ∈ ˜ Ψ . In other words, we have that:

More specifically, we choose ˜ ζ in the following way. If for all i ∈ { 2 , . . . , n +1 } , π E i ( a | s ) = 0 , then we pick ˜ ζ ( s, a ) = ζ ( s, a ) ; 20 otherwise, we pick ζ ( s, a ) in the following way:

First of all, we verify that this choice of ˜ ζ belongs to ˜ Ψ . Plugging Equation (73) into the definition of ˜ Ψ we obtain that:

However, since ζ ∈ Ψ , we have that d π Ei π E i ¯ B π E 1 ζ ≤ ξ i , thus leading to:

which is always true, and, consequently ˜ ζ ∈ ˜ Ψ .

To conclude the proof, it remains to analyze ‖ ζ -˜ ζ ‖ ∞ . At this point, we notice that:

Indeed, for a state-action pair ( s, a ) such that π E 1 ( a | s ) = 0 , we can notice that any value ζ ( s, a ) will not affect the resulting reward function. 21 Furthermore, whenever there is no sub-optimal expert such that π E i ( a | s ) > 0 , then ˜ ζ ( s, a ) = ζ ( s, a ) holds by definition. To conclude, with simple algebraic manipulations we can obtain the following result:

where in the last step we have upper-bounded ζ ( s, a ) with an upper bound that follows directly from Equation (7), thus concluding the proof.

We can appreciate as Lemma E.9 provides, under certain conditions on the time-step t , a highprobability upper bound on the difference inf ˆ r ∈R ̂ ¯ B ‖ r -ˆ r ‖ ∞ for any choice of r ∈ R ¯ B . Furthermore, as we can see, the proof is fairly involved due to the necessity of upper bounding the error term ‖ ζ -proj ̂ Ψ ( ζ ) ‖ ∞ . At this point, by deriving symmetric results (i.e., Lemma E.8 and Lemma E.9), it is possible to derive an identical upper-bound for inf r ∈R ¯ B ‖ r -ˆ r ‖ ∞ for any choice of ˆ r ∈ R ̂ ¯ B . As a consequence, Lemma E.9 provides the following high-probability upper-bound on the Hausdorff distance between R ̂ ¯ B and R ¯ B :

Theorem 4.2. Let q 1 = min { π -1 min max i ∈{ 2 ,...,n +1 } ξ i , (1 -γ ) -1 } , and q 2 = max { 1 , q 1 } . Then, with a total budget of:

US-IRL-SE is ( glyph[epsilon1], δ ) -correct and ˜ O ( · ) hides constant and logarithmic dependencies.

Proof. Let us start from Lemma E.9 and consider t such that the condition of Lemma E.9 are satisfied. As previously discussed, due to Lemma E.9, we have that, at time t , the algorithm US-IRL-SE induces an estimated feasible reward set such that the following holds with high-probability:

To conclude the proof, we need to find a sufficiently large t such that the following holds:

for any glyph[epsilon1] ∈ (0 , 1) . To this end, it is sufficient to find the smallest t that satisfies the following conditions:

At this point, let us focus on ρ t min { π -1 min max i ξ i , (1 -γ ) -1 } ≤ glyph[epsilon1] 4 . By simple algebraic manipulations we obtain the following sufficient condition for t :

For 2 √ 2 γ (1 -γ ) β t ≤ glyph[epsilon1] 4 , instead, we obtain the following condition on t : 22

Similarly, for 2 √ 2 γ (1 -γ ) min { π -1 min max i ξ i , (1 -γ ) -1 } β t ≤ glyph[epsilon1] 4 , we need at least the following number of samples:

Finally, for 2 √ 2 γ (1 -γ ) min { π -1 min max i ξ i , (1 -γ ) -1 } α t ≤ glyph[epsilon1] 4 , we obtain:

At this point, we notice that the total number of samples gathered from the generative at step t is given by tSA . Therefore, from the previous equations, together with the conditions of Lemma E.9, we obtian the following sufficient condition to guarantee that US-IRL-SE is ( glyph[epsilon1], δ ) -correct.

where the first term is due to Equation (75), the second one comes from Equations (76)-(77), and the last term arises from Equation (55), and is independent w.r.t. the desired accuracy glyph[epsilon1] . 23 Specifically, we notice that for sufficiently small values of glyph[epsilon1] (i.e., glyph[epsilon1] < q 1 ), the last term is always dominated by the first one, which concludes the proof.

F Stochastic Optimal Expert

In this section, we discuss how to extend our analysis to the case in which the optimal expert plays a stochastic policy. First of all, let us define π min ,E 1 as the minimum probability with which the optimal expert plays its actions. 24

At this point, we notice that, for obtaining sample-complexity guarantees of the US-IRL-SE algorithm, Lemma E.8 implies that we are interested in learning which are the actions that are played with positive probability by the optimal expert. In other words, for any non-zero vector x ∈ R S × A , we want the following to hold with high-probability:

To this end, one can apply Lemma D.3 of [22], which implies that with a number of samples that is:

Equation (80) holds w.p. at least 1 -δ . Once this condition is verified, the rest of the proof of the complexity of US-IRL-SE follows identically to the one of Theorem 4.2. 25 We notice that this introduces an additional maximum in the result of the sample-complexity. However, it is also possible to extend the proof of the lower bound of [22] (see Theorem D.1 in [22]), to show that the dependency resulting from (81) is unavoidable. In this sense, US-IRL-SE remains minimax optimal whenever the performance of the sub-optimal experts are sufficiently close to the ones of the optimal agent.

G Per-expert Complexity of IRL-SE

In this section, we provide a description of how to change the PAC learning formalism to show that Equation (11) actually represents a lower bound to the number of data that is necessary to gather from each of the different sub-optimal experts.

Specifically, we define a learning algorithm for an IRL problem ¯ B as a tuple A = ( τ, ν ) , where τ is a stopping time that controls the end of the data acquisition phase, and ν = ( ν t ) t ∈ N is a historydependent sampling strategy over S × A × ( S ) n +1 . More precisely, ν t ∈ ∆ S×A× ( S ) n +1 D t , where

(i) a state-action pair ( S t , A t ) and observes a sample S ′ t ∼ p ( ·| S t , A t ) from the environment

The observed realizations are then used to update the sampling strategy ν t , and the process goes on until the stopping rule is satisfied. At the end of the data acquisition phase, the algorithm leverages the collected data to output the estimate of the feasible reward set R ̂ ¯ B τ that is induced by the resulting

empirical IRL problem ̂ ¯ B τ . Given this formalism, we are interested in designing learning algorithms that, for any desired accuracy glyph[epsilon1] ∈ (0 , 1) and any risk parameter δ ∈ (0 , 1) , guarantee that:

glyph[negationslash]

10

8

Upper bound on Z(s, a)

6

2

5

10 11 12 13 14 15 16 17 18 19

(s,a) index

10

8

Upper bound on &(s, a)

6

2

4

6

7

10 11 12 13 14 15 16 17 18 19

(s,a) index

We refer to these algorithms as ( glyph[epsilon1], δ ) -correct identification strategies. For ( glyph[epsilon1], δ ) -correct strategies, we define their sample complexity as the sum of the number of unitary samples gathered from the generative model. In other words, let us denote with N t ( s, a ) the number of samples gathered, at step t , from the environment, and let N ( i ) t ( s ) denotes the number of samples gathered at step t from the i -th expert at state s . Then, the sample complexity is given by:

Given this learning formalism, it is straightforward to extend the results of Theorem 4.1 to this setting. More specifically, Theorem E.1 can be used to lower bound E [ ∑ s N ( i ) τ ( s ) ] for each sub-optimal expert, thus showing a significant increase in the sample complexity (i.e., linear in the number of sub-optimal experts).

H Further details on US-IRL-SE

In this section, we provide further details on US-IRL-SE. More precisely, we specify the exact expression of the parameter m that is used to provide ( glyph[epsilon1], δ ) -correc

Expression of m From the proof of Theorem 4.2, it is possible to derive an exact expression of m that can be used in US-IRL-SE. More specifically, since t = N t ( s, a ) , 26 it is sufficient to take m as the minimum t that satisfies Equations (75)-(78) and Equations (55)-(58).

Computational Complexity of US-IRL-SE At each iteration, US-IRL-SE will query the generative model in all state-action pair ( s, a ) ∈ S × A . Notice that the generative model samples (i) the environment (ii) all the expert's policy. Assuming a unitary cost for generating each sample, a single query to the generative model results in a computational complexity of O ( n ) . Therefore, the total computational complexity of US-IRL-SE is given O ( SAmn ) .

I Numerical Validation

We designed an experiment that aims at visualizing the reduction of the feasible reward set.

Computing Resources The experiments have been run on a laptop with 8 Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz and 8GB of RAM. Total time to gather the results is less then a few hours.

Experiment Setup We considered as environment the forest management scenarios with 10 states and 2 actions that is available in the "pymdptoolbox" library. 27 We considered a discount factor γ = 0 . 9 . We have run policy iteration on this domain to recover a set of expert policies (i.e., the policy returned at each iteration by the policy iteration algorithm is a sub-optimal expert), and we have considered as ξ the infinite norm between the value functions of the optimal policy and the sub-optimal ones computed using the true reward function.

Results To appreciate the reduction in the feasible set that is introduced by the sub-optimal experts, we have plotted the maximum value that ζ ( s, a ) can achieve according to the theoretical bound of Eq. (7) (notice that this is way easier to visualize rather than an entire set of reward functions). For the sake of visualization, we have flattened the matrix that contained upper bounds on ζ , and we plotted the result in Figure 6 (left). As one can notice, the presence of the sub-optimal experts can significantly limit the value of the advantage function in many state-action pairs (notice that the maximum value for ζ is given by (1 -γ ) -1 ≈ 10 ; this theoretical threshold represents the upper-bound on that was derived in [24] for the single-agent problem).

We have then run our algorithm for 20 times using glyph[epsilon1] = 0 . 1 and δ = 0 . 1 , and computed the (empirical) theoretical upper bound on ζ . The reviewer can find the empirical value of the upper bounds on in the Figure 6 (right). We have reported only the empirical mean, as the 95% confidence intervals are in the order of 1e-5. As one can verify, the results are almost identical to the exact case.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and the introduction reflect the original contribution of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are noted as future directions in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the statements are correlated with proofs. The definitions and the quantity used are clearly presented.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Appendix I provides detail on how to reproduce the experiments. We will also make the code public for reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The codebase to reproduce the results will be public.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experiment details are reported in Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Figure 6 reports only the mean as the 95% confidence intervals are in the interval of 1e-5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Experiment results and details are provided in Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We checked the guidelines and we are compliant with them.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: As a theoretical research paper, the authors do not see any negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We rely on the pymdptoolbox library, and we make an explicit reference to it in Appendix I.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce the IRL-SE problem, which which uses multiple experts, including sub-optimal ones, to enhance reward function learning. Given a bounded performance gap between experts, the authors provide a theoretical analysis on shrinking the feasible set of compatible reward functions. Using a PAC framework, the authors provide a lower bound on the sample complexity for estimating the feasible reward set \n\nThe authors propose a uniform sampling algorithm with theoretical guarantees, demonstrating its minimax optimality under certain conditions.\n\nSTRENGTHS:\nThe authors provide interesting novel insights in a more general sitaution compared to previously cited work\n\nThe submission is technically sound and well supported by extremely detailed theoretical analysis. The work is complete and the authors evaluate their strenghts and provide future research avenues. The techincal notation is dense and can be difficult to follow, but the authors did well in writing precisely. \n\nI think the overall proposal in this paper is quite applicable in many RL settings, so this is a strong submission with many potential use cases.\n\nWEAKNESSES:\nThe submission could be written more clearly. Some results could be expanded instead of just staing the results. Particularly, the use of \"it's easy to see\" should be removed and expanded upon for ease to the reader. In some cases it may also aid readability to expand out expression instead of writing it in the most concise way possible. \n\nThe paper has a strong theoretical focus, which is strong and necessary. However, it would be powerful to see it actually employed in a case study to see how such a technique would be used. Even outside of empirical evidence, it would aid in readability of the paper to see how the notations used in the theoretical sections are actually used in practice. To that end, even a case study with a toy example would go a long way. The authors do a good job in providing technical examples through the paper, but these could be be futher enhanced by connecting them to a toy example of some kind. For a such a dense technical paper, it would make a world of difference in readability.\n\nQUESTIONS:\nExamples 3.1,3.2,3.3. Maybe it's \"easy to show\", but can you please show where these come from? \n\nSuggestion: The word \"Indeed\" is used too many times. It is quite repetitive. \n\nSuggestion: For line 109, can you explain why the infinite norm is limited to the particular range? \nOperators: $\\pi$ is overloaded to be both an operator and a policy\n\nLIMITATIONS:\nFuture research directions are provided, but a dicussion on technical limitations would be also be good to have. No discussion on potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to mitigate the intrinsic reward ambiguity in IRL problems.  The authors propose that incorporating sub-optimal expert demonstrations can lead to a more accurate estimate of the feasible reward set. Initially, they formulate the IRL with Sub-Optimal Experts (IRL-SE) problem and explore the theoretical properties of the resulting feasible reward set, offering both implicit and explicit descriptions. Additionally, they establish the lower bound of the statistical complexity for IRL-SE problems within the Probably Approximately Correct (PAC) framework. Finally, the authors introduce a uniform sampling algorithm that achieves minimax optimality for IRL-SE.\n\nSTRENGTHS:\n- This paper is well-written and easy to follow. The examples provided in the paper effectively illustrate the role of sub-optimal experts.\n- This paper provides a clear and comprehensive theoretical analysis of the IRL-SE problem. The approach of identifying, modeling, evaluating, and solving the problem is highly commendable and worth emulating. First, the authors formalize the problem and demonstrate the theoretical feasibility of resolving reward ambiguity. Then, they determine the difficulty of the problem by analyzing the lower bound of statistical complexity within the PAC framework. Finally, the authors propose an algorithm that achieves minimax optimality to solve the IRL-SE problem.\n\nWEAKNESSES:\n- The paper does not provide a method for determining a suitable performance gap $\\zeta$ for sub-optimal experts. Additionally, in examples such as 3.1 and 3.3, when $\\gamma=0.99$ (a commonly used value), the estimated performance gap $\\zeta$ tends to be quite large. This may limit the effectiveness of sub-optimal experts in resolving ambiguity in reward sets, which could be a significant obstacle for algorithm deployment.\n- The paper lacks experimental results to validate the role of sub-optimal experts in resolving reward ambiguity.\n\nQUESTIONS:\nBased on the weaknesses, my questions are as follows.\n\n- How can we utilize the feasible reward set in a real-world scenario? I understand that IRL enables learning of reward models from expert demonstrations, which can then optimize policies through RL. However, if we obtain a reward set instead, how should we make use of it?\n- Can you provide more insights into determining the performance gap of sub-optimal policies?\n- Can you experimentally verify if sub-optimal experts can mitigate reward ambiguity? For example, in certain MDPs, as more sub-optimal experts are provided, the cardinality of the feasible reward set tends to decrease.\n\nLIMITATIONS:\nThis paper does not discuss the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies unregularized IRL involving one optimal expert and $n$ suboptimal experts. The authors show that the additional suboptimal experts can help mitigate the reward ambiguity in IRL that arises due to the unknown suboptimality of unvisited state-action pairs. They present an analytical expression for the feasible reward set in this setting and demonstrate that the reward ambiguity can be reduced when learning from not-too-suboptimal experts yet playing differently from the optimal expert. Furthermore, assuming a generative model and sample-based access to experts' policies, the authors provide a PAC lower bound for the sample complexity of IRL and propose an algorithm that is minimax optimal for small suboptimalities of the experts.\n\nSTRENGTHS:\n1. The paper makes a notable contribution to IRL by showing that leveraging suboptimal experts can effectively reduce reward ambiguity.\n2. Theorems 4.1 and 4.2, presenting lower and upper bounds, are particularly appreciated.\n3. The mathematical notation is clearly introduced, and full proofs are available in the appendix.\n\nWEAKNESSES:\n1. The paper assumes access to a generative model of the experts' policies. However, in practical IRL a finite data set of precollected expert demonstrations is the norm. This makes Algorithm 1 difficult to implement in practice. \n2. Minor points: a) In my opinion the claims made in line 109 and in Examples 3.1-3.3 need a reference or a brief proof. b) I believe there is a typo in Eq. (5): Since we are taking the expectation with respect to the state-action occupancy measure, we don't need the right-hand side should be a scalar, right? Moreover, in Eq. (10), I assume it should be $\\min$ instead of $\\max$.\n\nQUESTIONS:\n1. As pointed out, we would generally like the suboptimalities $\\xi_i$ to be small and the $\\pi_{\\min}$ to be large. However, these may be conflicting goals, given that we also want the suboptimal experts to deviate from $\\pi_{E_1}$. Could this problem be mitigated by having experts that are more suboptimal (e.g. a completely uniform expert) but with lower and upper bounds on their suboptimality?\n2. What if we were to drop the optimal expert $\\pi_{E_1}$? Can we say anything about the feasible reward set in that setting?\n\nLIMITATIONS:\nSome limitations are discussed in the conclusion section. However, it would be good to be more clear about the practical limitations of the provided results.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper develops a theory to address inverse reinforcement learning (IRL) from sub-optimal expert demonstrations. The authors assume a set of experts with known degrees of sub-optimality. Rather than learning a single reward model, as is often done in standard IRL, they provide an explicit characterization of all plausible reward models compatible with the experts. Moreover, they give lower bounds on the number of generated samples from the experts for a PAC guarantee on the set of reward models. Finally, they propose a uniform sampling procedure that can result in near-optimal estimation of the reward models in terms of sample complexity.\n\nSTRENGTHS:\n1. Although this is not the first work for IRL from sub-optimal experts, I haven't seen characterizing all plausible reward models for such a setting, so I would call the work original in this sense.\n2. The authors provide good intuition on why having more sub-optimal experts can shrink the set of feasible reward models in Figures 1 and 2.\n3. I haven't fully validated the proofs for the theory. But it seems both Theorem 4.1 (the lower bound on the required sample size for the $(\\epsilon,\\delta)$-correct identification of the reward set) and Theorem 4.2 (the upper bound for the uniform sampling algorithm) are mainly based on the theoretical results in Metelli et al., 2023. Moreover, the characterization of plausible reward models in Theorem 3.3 is mainly developed by Metelli et al., 2021 (especially the eq. (5), which characterizes the feasible rewards for the case of optimal experts). In that sense, the novelty of this work is introducing more constraints based on the sub-optimality index of additional demonstrators, which is characterized in eq. (3) and point (iii) in line 162 of page 3. Having said that, those extensions based on the extra constraints seem non-trivial. So, I'd call the theoretical contribution novel.\n\nWEAKNESSES:\nThe main weakness of the setting is that it assumes having access to a generative model for the optimal and sub-optimal experts during sampling and that there is a performance gap between the sub-optimal and optimal experts. However, given that the main contribution of the paper is theoretical, I still vote for acceptance.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-30"}