Title: How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach
Venue: NeurIPS 2024
Accepted: yes
Keywords: Inverse Reinforcement Learning

Reward-Free Exploration

Linear MDPs
Abstract: In online Inverse Reinforcement Learning (IRL), the learner can collect samples about the dynamics of the environment to improve its
estimate of the reward function. Since IRL suffers from identifiability issues, many theoretical works on online IRL focus on estimating the entire set of rewards that explain the demonstrations, named the *feasible reward set*. However, none of the algorithms available in literature can scale to problems with large state spaces. In this paper, we focus on the online IRL problem in Linear Markov Decision
Processes (MDPs). We show that the structure offered by Linear MDPs is not sufficient for efficiently estimating the feasible set when the state space is large. As a consequence, we introduce the novel framework of *rewards compatibility*, which generalizes the notion of feasible set, and we develop CATY-IRL, a sample efficient algorithm whose complexity is independent of the size of the state space in Linear MDPs. When restricted to the tabular setting, we demonstrate that CATY-IRL is minimax optimal up to logarithmic factors. As a by-product, we show that Reward-Free Exploration (RFE) enjoys the same worst-case rate, improving over the state-of-the-art lower bound. Finally, we devise a unifying framework for IRL and RFE that may be of independent interest.

How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach

Abstract

In online Inverse Reinforcement Learning (IRL), the learner can collect samples about the dynamics of the environment to improve its estimate of the reward function. Since IRL suffers from identifiability issues, many theoretical works on online IRL focus on estimating the entire set of rewards that explain the demonstrations, named the feasible reward set . However, none of the algorithms available in the literature can scale to problems with large state spaces. In this paper, we focus on the online IRL problem in Linear Markov Decision Processes (MDPs). We show that the structure offered by Linear MDPs is not sufficient for efficiently estimating the feasible set when the state space is large. As a consequence, we introduce the novel framework of rewards compatibility , which generalizes the notion of feasible set, and we develop CATY-IRL , a sample efficient algorithm whose complexity is independent of the cardinality of the state space in Linear MDPs. When restricted to the tabular setting, we demonstrate that CATY-IRL is minimax optimal up to logarithmic factors. As a by-product, we show that Reward-Free Exploration (RFE) enjoys the same worst-case rate, improving over the state-of-the-art lower bound. Finally, we devise a unifying framework for IRL and RFE that may be of independent interest.

1 Introduction

Inverse Reinforcement Learning (IRL) is the problem of inferring the reward function given demonstrations of an optimal behavior, i.e., from an expert agent. [49, 42]. Since its formulation, much of the research effort has been put into the design of efficient algorithms for solving the IRL problem [6, 4]. Indeed, the solution of the IRL problem opens the door to a variety of interesting applications, including Apprenticeship Learning (AL) [2, 1], reward design [16], interpretability of the expert's behavior [17], and transferability to new environments [15].

Nowadays, the factor that most negatively impacts the adoption of IRL solutions in real-world applications is the intrinsic ill-posedness of its formulation. The IRL problem has been historically defined as the problem of recovering the reward function underlying the demonstrations [49, 42], even though mere demonstrations can be equivalently explained by a variety of rewards. In other words, the IRL problem is underconstrained, even in the limit of infinite demonstrations [42, 39].

To overcome this weakness and to come up with a single reward function, three main approaches are commonly adopted in the literature. ( i ) The first approach consists of the use of a heuristic to select a specific reward function from the set of all the rewards that explain the demonstrations. Implicitly, these works re-define IRL as the problem of recovering the reward function explaining the demonstrations and complying with the heuristic. As an example, [42, 48] select the reward that maximizes some notion of margin, and [70] implicitly chooses the reward returned by the optimization algorithm among those that maximize the likelihood. However, these approaches may generate issues in applications [56, 15]. ( ii ) In the second approach, additional constraints beyond

Politecnico di Milano Milan, Italy

mere demonstrations are enforced to guarantee the uniqueness of the reward function to recover. In 'reward identifiability' works, the additional information commonly concerns some structure of the environment [25], or multiple demonstrations across various environments [5, 10]. In Reward Learning (ReL) works [19], demonstrations of optimal behavior are combined with other kinds of expert feedback, like comparisons [65]. ( iii ) As a third approach, recently, [39, 38] proposed the alternative formulation of IRL as the problem of recovering all the reward functions compatible with the demonstrations, i.e., the feasible reward set . In this manner, we are not subject to the limitations of the first approach, and we do not depend on additional information like in the second approach.

In practical applications, the chosen IRL formulation has to be tackled by algorithms that use a finite number of demonstrations and a limited knowledge of the dynamics of the environment. In the common online IRL scenario, the learner explores the (unknown) environment, and exploits this additional information to improve its performance on the IRL task [e.g., 39, 33, 38, 68, 31]. On this basis, the IRL approach ( iii ) based on the feasible set [39, 38] displays desirable properties since 'postpones' the choice of the heuristic and/or enforcement of additional constraints, with the advantage of analyzing the intrinsic complexity of the IRL problem only, without being obfuscated by other factors. In other words, this recent formulation of the IRL problem paves the way for the design and analysis of provably efficient IRL algorithms, endowed with solid theoretical guarantees.

However, the algorithms designed for learning the feasible set currently available in the literature [e.g., 39, 33, 38, 68, 31] struggle when attempting to scale them to IRL problems with large state spaces . This is apparent because their sample complexity exhibits an explicit dependence on the cardinality of the state space. This inevitably represents a major limitation since most real-world scenarios concern problems with large, or even continuous, state spaces [e.g., 15, 7, 40, 14].

In this context, function approximation represents an essential tool to tackle the curse of dimensionality and enforce generalization [54, 41]. Linear Markov Decision Processes (MDPs) [23, 67] offer a simple but powerful structure, in which we assume the reward function and the transition model can be expressed as linear combinations of known features, that permits theoretical analysis of the sample complexity. Even though many extensions have been developed [64, 22, 13], the Linear MDPs framework typically represents one of the first function approximation settings to analyze when focusing on a novel problem, before moving to more complex settings [e.g., 63, 61].

In this paper, we aim to shed light on the challenges of scaling the feasible reward set to large-scale problems. Motivated by its limitations when dealing with large state spaces, we introduce the novel Rewards Compatibility framework. Being a generalization of the notion of feasible set, it allows us to define the new IRL Classification Problem , a fourth approach to cope with the ill-posedness of the IRL formulation. This permits the development of CATY-IRL ( C omp AT ibilit Y for IRL ), a provably efficient IRL algorithm for Linear MDPs characterized by large or even continuous state spaces.

Original Contributions. The main contributions of the current work can be summarized as follows:

We prove that the notion of feasible set can not be learned efficiently in MDPs with large/continuous state spaces, even under the structure enforced by Linear MDPs. Nevertheless, we show that this problem disappears under the assumption that the expert's policy is known, by providing a sample efficient algorithm for such setting (Section 3).

To overcome the need for knowing the expert's policy exactly, we propose Rewards Compatibility , a novel framework that formalizes the intuitive notion of compatibility of a reward function with expert demonstrations. It generalizes the feasible set and allows us to define an original learning setting, IRL classification , based on a new formulation of IRL classification task (Section 4).

For the newly-devised framework, we develop CATY-IRL ( C omp AT ibilit Y for IRL ), a new sample and computationally efficient IRL algorithm for both tabular and Linear MDPs. Remarkably, this CATY-IRL does not require the additional assumption that the expert's policy is known (Section 5).

In the tabular setting, we prove a tight minimax lower bound to the sample complexity of the IRL classification problem of Ω ` H 3 SA ϵ 2 p S ` log 1 δ q ˘ episodes, where S and A are the cardinalities of the state and action spaces, H is the horizon, ϵ the accuracy and δ the failure probability. This bound is matched by CATY-IRL , up to logarithmic factors. Exploiting a similar construction, we show that a lower bound with the same rate holds also for the Reward-Free Exploration (RFE) problem, improving by an H factor over the RFE state-of-the-art lower bound [21] (Section 6.1).

Finally, we formulate a novel Objective-Free Exploration (OFE) setting that isolates the challenges of exploration beyond Reinforcement Learning (RL), by unifying RFE and IRL (Section 6.2).

Additional related works and the proofs of all the results are reported in Appendix A and B -E.

2 Preliminaries

Notation. Given an integer N P N , we define J N K : ' t 1 , . . . , N u . Given sets X and Y , we denote H d p X , Y q : ' max t sup x P X inf y P Y d p x, y q , sup y P Y inf x P X d p y, x qu their Hausdorff distance with inner distance d . We denote by ∆ X the probability simplex over X , and by ∆ X Y the set of functions from Y to ∆ X . Sometimes, we denote the dot product between vectors x, y as x x, y y : ' x ⊺ y . We employ O , Ω , Θ for the common asymptotic notation and O , Ω , Θ to omit logarithmic terms.

Linear MDPs. Based on [23], we say that an MDP M ' p S , A , H, d 0 , p, r q is a Linear MDP with a (known) feature map ϕ : S ˆ A Ñ R d , if for every h P J H K , there exist d P N unknown (signed) measures µ h ' r µ 1 h , . . . , µ d h s ⊺ over S and an unknown vector θ h P R d , such that for every p s, a q P S ˆ A , we have p h p¨| s, a q ' x ϕ p s, a q , µ h p¨qy and r h p s, a q ' x ϕ p s, a q , θ h y . Without loss of generality, we assume } ϕ p s, a q} 2 ď 1 for all p s, a q P S ˆ A , and max t} θ h } 2 , }| µ h |p S q} 2 u ď ? d . 1 M is a Linear MDP without reward if its transition model satisfies the assumption described above.

r r r Markov Decision Processes. A finite-horizon Markov Decision Process (MDP) without reward [45] is defined as a tuple M : ' p S , A , H, d 0 , p q , where S and A are the measurable state and action spaces, H P N is the horizon, d 0 P ∆ S is the initial-state distribution, and p P P : ' ∆ S S ˆ A ˆ J H K is the transition model. Given a (deterministic) reward function r P R : ' r´ 1 , 1 s S ˆ A ˆ J H K , we denote by M : ' M Yt r u the MDP obtained by pairing M and r . Each policy π P Π : ' ∆ A S ˆ J H K induces in M a state-action probability distribution d p,π : ' t d p,π h u h P J H K (we omit d 0 for simplicity) that assigns, to each subset Z Ď S ˆ A , the probability of being in Z at stage h P J H K when playing π in M . We denote with S p,π h the set of states supported by d p,π h for any action at stage h , and with S p,π the disjoint union of sets t S p,π h u h P J H K . The Q -function of policy π in MDP M is defined at every p s, a, h q P S ˆ A ˆ J H K as Q π h p s, a ; p, r q : ' E p,π r ř H t ' h r t p s t , a t q| s h ' s, a h ' a s , and the optimal Q -function as Q ˚ h p s, a ; p, r q : ' sup π P Π Q π h p s, a ; p, r q , where the expectation E p,π is computed over the stochastic process generated by playing policy π in the MDP M . Similarly, we define the V -function of policy π at p s, h q as V π h p s ; p, r q : ' E p,π r ř H t ' h r t p s t , a t q| s h ' s s , and the optimal V -function as V ˚ h p s ; p, r q : ' sup π P Π V π h p s ; p, r q . We define the utility of π as J π p r ; p q : ' E s ' d 0 r V π 1 p s ; p, r qs , and the optimal utility as J ˚ p r ; p q : ' E s ' d 0 r V ˚ 1 p s ; p, r qs . A forward (sampling) model of the environment permits to collect samples starting from s ' d 0 and following some policy. A generative (sampling) model consists in an oracle that, given an arbitrary state-action-stage triple s, a, h in input, returns a sampled next state s 1 ' p h p¨| s, a q .

BPI and RFE. In both Best-Policy Identification (BPI) [37] and Reward-Free Exploration (RFE) [21], the learner has to explore the unknown MDP to optimize a certain reward function. In BPI, the learner observes the reward function r during exploration, and its goal is to output a policy p π such that, in the true MDP with transition model p we have P ` J ˚ p r ; p q ´ J p π p r ; p q ď ϵ ˘ ě 1 ´ δ for every ϵ, δ P p 0 , 1 q . RFE considers the setting in which the reward to optimize is revealed a posteriori of the exploration phase. Thus the goal of the agent in RFE is to compute an estimate p p of the true dynamics p so that P ` sup r P R t J ˚ p r ; p q ´ J p π r p r ; p qu ď ϵ ˘ ě 1 ´ δ for every ϵ, δ P p 0 , 1 q , where p π r is the optimal policy in the MDP with p as transition model and r as reward function.

p Online IRL. We consider the online 2 IRL setting [39, 33, 68, 66, 53] in which, similarly to the online AL setting [53, 66], we are given a dataset D E ' tp s i 1 , a i 1 , . . . , s i H ´ 1 , a i H ´ 1 , s i H qu i P J τ E K of τ E P N trajectories collected by executing the expert's policy π E in a certain (unknown) MDP M ' M Yt r E u . We make the assumption that π E is optimal under the true (unknown) reward r E in M . Since the dynamics of M is unknown, we are allowed to actively explore the environment through a forward model to collect a new state-action dataset D . The goal is to use the latter and demonstrations in D E to estimate a reward function that makes the expert's policy π E optimal. Sometimes, we will denote an IRL instance as M Yt π E u , and a Linear IRL instance with recovered reward r as an IRL instance in which M Yt r u is a Linear MDP.

3 Limitations of the Feasible Set

In this section, after having characterized the feasible set formulation in Linear MDPs, we show that it suffers from statistical (and computational ) inefficiency in problems with large state spaces, even under the Linear MDP assumption. We will provide a solution to these issues in Section 4.

The Feasible Set. According to the standard definition [e.g., 39, 33, 38, 68, 31], the feasible set contains the rewards that make the expert's policy π E optimal, as defined below.

Definition 3.1 (Feasible Set [31]) . Let M be an MDP without reward and let π E be the expert's policy. The feasible set R p,π E of rewards compatible with π E in M is defined as:

Without function approximation, the feasible set contains a variety of rewards for any deterministic policy. In Linear MDPs, due to the feature map, the feasible set might exhibit some degeneracy. 3 Definition 3.1 can be adapted to Linear MDPs with feature map ϕ as: R ϕ,p,π E : ' t r P R | J π E p r ; p q ' J ˚ p r ; p q ^ D θ : J H K Ñ R d , @p s, a, h q P S ˆ A ˆ J H K : r h p s, a q ' x ϕ p s, a q , θ h yu . We omit ϕ in R ϕ,p,π E for notational simplicity.

Proposition 3.1. Let M be a Linear MDP without reward with a finite state space, and let ϕ be a feature mapping. Let t Φ π E h u h P J H K and t Φ h u h P J H K be the sets of expert's and non-expert's features, defined for every h P H as:

where A E h p s q : ' t a P A | π E h p¨| s q ą 0 u for every s P S . If for none of the H pairs of sets p Φ π E h , Φ h q there exists a separating hyperplane, then R p,π E ' t r u , with r h p s, a q ' 0 @p s, a, h q P S ˆ A ˆ J H K i.e., the feasible set with linear rewards in ϕ contains only the reward function that assigns zero reward everywhere.

Intuitively, expert's actions must have the largest optimal Q -value among all actions, and linearity imposes the 'separability' requirement. The result holds also for MDPs with linear rewards only. We exemplify Proposition 3.1 in Appendix B.1.

Learning the Feasible Set. In order to highlight the challenges of learning the feasible set with large-scale MDPs, based on [38, 31], we devise the following PAC requirement.

Definition 3.2 (PAC Algorithm) . Let ϵ, δ P p 0 , 1 q , and let A be an algorithm that collects τ E samples about π E using a generative model, and τ episodes from a Linear MDP without reward M ' p S , A , H, d 0 , p q using a forward model. Let p R be the estimate of the feasible set R p,π E outputted by A . Then, A is p ϵ, δ q -PAC for IRL if P M , A ` H d p R p,π E , p R q ď ϵ ˘ ě 1 ´ δ , where P M , A is the probability measure induced by A in M , and d p r, p r q9 sup π P Π ř h P J H K E p s,a q' d p,π h p¨ , ¨q | r h p s, a q´ p r h p s, a q| . 4 The sample complexity is the pair p τ E , τ q .

It is worth noting that in Definition 3.2, we are considering a generative model for collecting samples from the expert's policy, which represents the easiest learning scenario. The following result shows that, even in this convenient setting, estimating the feasible set is statistically inefficient.

Theorem 3.2 (Statistical Inefficiency) . Let M Yt π E u be a Linear IRL instance with finite state space S and deterministic expert's policy, and let ϵ, δ P p 0 , 1 q . If an algorithm A is p ϵ, δ q -PAC, then τ E ' Ω p S q , where S : ' | S | is the cardinality of the state space.

In other words, even under the easiest learning conditions (i.e., generative model and deterministic expert), the sample complexity scales directly with the cardinality of the state space S , thus, it is infeasible when S is large or even infinite. Observe that this result extends to any class of MDPs that contains Linear MDPs. In Appendix B.2, we analyze if additional assumptions can drop the Ω p S q dependence. Nevertheless, if π E is known , it is possible to construct sample efficient algorithms. Algorithm 1 (whose pseudocode is presented in Appendix B.3), under the assumption that π E is known, makes use of an inner RFE routine (Algorithm 1 of [62]) to recover the feasible set.

Theorem 3.3. Assume that π E (along with its support S p,π E ) is known. Then, for any ϵ, δ P p 0 , 1 q , Algorithm 1 is p ϵ, δ q -PAC for IRL with a number of episodes τ upper bounded by:

Limitations of the Feasible Set. We can now conclude that the feasible set suffers from two main limitations. ( i ) Sample Inefficiency : If π E is unknown, it requires a number of samples that depends on the cardinality of the state space (Theorem 3.2). ( ii ) Lack of Practical Implementability : It contains a continuum of rewards, thus, no practical algorithm can explicitly compute it. We will discuss in the next section how to overcome both these issues.

4 Rewards Compatibility

In this section, we present the main contribution of this work: Rewards Compatibility , a novel framework for IRL that allows us to conveniently rephrase the learning from demonstrations problem as a classification task. We anticipate that the presentation of the framework is completely general and independent of structural assumptions of the MDP (e.g., Linear MDP).

4.1 Compatible Rewards

In the following, for ease of presentation, we consider the exact setting, i.e., when d 0 , p , and π E are known. In addition, we will drop the dependence on p when clear from the context.

In IRL, an expert agent demonstrates policy π E assumed optimal under some (unknown) reward function r E , i.e., J ˚ p r E q ' J π E p r E q . The task is to recover a reward r such that J ˚ p r q ' J π E p r q . By definition, IRL tells us that r E makes the demonstrated policy π E optimal, but what about other policies? We do not and cannot know. Since there are (infinite) rewards making π E optimal (but they differ in the performance attributed to other policies) we realize that there are many rewards equally 'compatible' with π E . 5 Clearly, wih no additional information, we are unable to identify r E .

The feasible set considers only these rewards, i.e., r P R for which J ˚ p r q ' J π E p r q , and it refuses all the others. This can be interpreted as the feasible set carrying out a classification of rewards based on a 'hard' notion of compatibility with demonstrations. In other words, rewards r satisfying condition J ˚ p r q ' J π E p r q are compatible with π E , and the others are not. Nevertheless, our insight is that some rewards are 'more' compatible with π E than others.

Example 4.1. Consider an MDP with one state and H ' 1 in which the expert has three actions: Eating a muffin (M), a cake (C), or some (bad) vegetable soup (S). The true reward r E assigns r E p M q ' ` 1 , r E p C q ' ` 0 . 99 and r E p S q ' ´ 1 , i.e., the expert has a (weak) preference for the muffin over the cake, while she hates the soup; thus, she will demonstrate π E ' M . Let r g , r b be:

Intuitively, r g is 'more' compatible with π E than r b , because it establishes that M and C are much better than S, while reward r b reverses the preferences. Clearly, we make a small error if we model the preferences of the expert with r g instead of the true reward r E . However, the notion of feasible set is completely blind to the difference between r g and r b at modeling r E , and it refuses both of them.

We propose the following 'soft' definition of (non)compatibility to capture this intuition. 6

Definition 4.1 (Rewards (non)Compatibility) . Let M Yt π E u be an IRL instance, and let r P R be any reward. We define the (non)compatibility C p,π E : R Ñ R ě 0 of reward r w.r.t. M Yt π E u as:

In words, the (non)compatibility of reward r w.r.t. policy π E in problem M quantifies the suboptimality of π E in the MDP M Yt r u . By definition, rewards r belonging to the feasible set (i.e., r P R p,π E ) satisfy C p,π E p r q ' 0 , i.e., they have zero non-compatibility with π E in M . 7

Example 4.1 (Continued) . (Non)compatibility discriminates between r g and r b . Indeed, we have that C p,π E p r E q ' 0 , C p,π E p r g q ' 0 . 01 , and C p,π E p r b q ' 2 . In words, reward r g suffers from very small (non)compatibility, while r b suffers from large (non)compatibility, thus we say that reward r g is more compatible with π E than r b , as expected.

By definition of IRL, the true reward r E makes the observed π E optimal, but reveals no information about the other policies. Thus, it is meaningful that C p,π E considers the suboptimality of π E only, because demonstrations from π E do not provide information about other policies, as illustrated below.

Example 4.2. Let r 1 b be such that r 1 b p M q ' ` 0 . 99 , r 1 b p C q ' ´ 1 , r 1 b p S q ' ` 1 . Clearly, r 1 b is much worse than r g at modeling r E , because it does not capture the fact that the expert appreciates the cake but she hates the soup. However, demonstrations from π E alone do not provide information about C or S, but only about π E ' M (i.e., the expert always eats the muffin). Thus, we have that C p,π E p r g q ' C p,π E p r 1 b q ' 0 . 01 , i.e., r g and r 1 b are equally compatible with the given demonstrations.

For a discussion on comparing the (non)compatibility of different rewards, see Appendix C.4.

4.2 The IRL Classification Formulation

Our goal is to overcome the limitations of the feasible set highlighted in Section 3. Drawing inspiration from the notion of 'membership checker' algorithm in [31], we propose a novel formulation of IRL.

Definition 4.2 (IRL Classification Problem and IRL Algorithm) . An IRL Classification Problem instance is made of a tuple p M , π E , R , ∆ q , where M is an MDP without reward, π E is the expert's policy, R Ď R is a set of rewards to classify, and ∆ P R ě 0 is some threshold. The goal is to classify all and only the rewards r P R based on their (non)compatibility with π E in M w.r.t. ∆ . In symbols:

An IRL algorithm takes in input a reward r P R and outputs a boolean saying whether C p,π E p r q ď ∆ .

Given r P R , we output whether it makes the expert's policy π E at most ∆ -suboptimal or not. Intuitively, we classify rewards in R based on how good π E performs w.r.t. them. A ∆ -(non)compatible reward guarantees that, among its ∆ -optimal policies, there is π E , but the optimal policy might be different from π E (see Appendix C.3 for how this relates to (forward) RL). Note that we allow for R ‰ R to manage scenarios in which we have some prior knowledge on r E , i.e., r E P R Ă R .

Remark 4.1. Permitting non-zero (non)compatibility is equivalent to enlarging the feasible set. Let R ' R , and define the set of rewards positively classified as R ∆ , i.e., R ∆ : ' t r P R | C p,π E p r q ď ∆ u . For any ∆ , ∆ 1 s.t. 0 ď ∆ ď ∆ 1 ď 2 H , we have: R p,π E ' R 0 Ď R ∆ Ď R ∆ 1 Ď R 2 H ' R .

Discussion on Reward Compatibility. It should be remarked that:

The limits of the rewards compatibility framework are the same as the limits of the feasible set . We cannot identify r E from the feasible set or among the rewards with small (non)compatibility. As aforementioned, this is an inherent limit of IRL and cannot be overcome with a more refined objective formulation, unless further information on r E is available (e.g., preferences).

Rewards compatibility offers advantages over feasible set . Differently from the feasible set, as we will see in Section 5, it is possible to practically implement algorithms that solve the IRL classification problem, with guarantees of sample efficiency even when the state space is large.

4.3 A Learning Framework for Online IRL Classification

In this section, we combine the online IRL setting presented in Section 2 with the IRL classification problem of Definition 4.2. Intuitively, the performance of an algorithm depends on its accuracy at estimating the (non)compatibility of the rewards, as formalized by the following PAC requirement.

0

R

∆

C

p

r

q

´

ϵ

`

ϵ

(a) Reward r is classified cor- rectly.

0

R

∆

C

p

r

q

´

ϵ

`

ϵ

(b) Reward r can be mis- classified.

0

R

∆

´

ϵ

`

ϵ

(c) Range of uncertain (non)compatibility values.

Figure 2: The axis represents (estimated) (non)compatibility values. (a) Rewards r whose true (non)compatibility C p r q : ' C p,π E p r q is far from threshold ∆ by at least ϵ , are correctly classified, while (b) in the opposite case, rewards can be mis-classified. (c) The red interval r ∆ ´ ϵ, ∆ ` ϵ s exemplifies the set of rewards t r P R | | C p r q ´ ∆ | ď ϵ u that are (potentially) mis-classified. The length of the interval reduces with ϵ .

Definition 4.3 (PAC Framework) . Let ϵ, δ P p 0 , 1 q , and let D E be a dataset of τ E expert's trajectories. An algorithm A exploring for τ episodes is p ϵ, δ q -PAC for the IRL classification problem if:

where P M ,π E , A is the joint probability measure induced by π E and A in M , and p C is the estimate of C p,π E computed by A . The sample complexity is defined by the pair p τ E , τ q .

Intuitively, our goal is to estimate the (non)compatibility of the rewards in R with sufficient accuracy, so that, given a threshold ∆ ě 0 , we are able to classify 'most' of them correctly w.h.p. (with high probability). The concept is exemplified in Figure 2. Note that the estimation problem is independent of the threshold ∆ , which can be appropriately selected to cope with noise in the demonstrations, (unknown) expert suboptimality, or to manage the amount of 'false negatives' and 'false positives'.

Remark 4.2. For η ě 0 , let R η : ' t r P R | C p,π E p r q ď η u and p R η : ' t r P R | p C p r q ď η u denote the sets of rewards positively classified using, respectively, the true (non)compatibility C p,π E and the estimate p C constructed by an p ϵ, δ q -PAC algorithm. Then, with probability 1 ´ δ , it holds that: p R ∆ ´ ϵ Ď R ∆ Ď p R ∆ ` ϵ . Thus, we can trade-off the amount of 'false negatives' (resp. 'false positives') by, e.g., choosing the threshold ∆ Ð ∆ ` ϵ (resp. ∆ Ð ∆ ´ ϵ ).

5 CATY-IRL :

A Provably Efficient Algorithm for IRL

In this section, we present CATY-IRL ( C omp AT ibilit Y for IRL ), a provably efficient algorithm for solving the online IRL classification problem. We consider three different kinds of structure for the MDPs: tabular MDPs, tabular MDPs with linear rewards, and Linear MDPs. Similarly to RFE, our online IRL classification setting is made of two phases: ( i ) an exploration phase, in which the algorithm explores the environment using the knowledge of R and

Exploration phase

Classification phase

Input

:

expert dataset

D

E

,

threshold ∆, set of

rewards to classify

R

Collect exploration dataset

D

Input

:

reward to clas-

sify

r

P

R

Estimate

ˆ

J

E

p

r

q «

J

π

E

p

r

;

p

q

using

D

E

Estimate

ˆ

J

˚

p

r

q «

J

˚

p

r

;

p

q

via planning with

D

Classify reward

p

C

p

r

q '

ˆ

J

˚

p

r

q ´

ˆ

J

E

p

r

q

p

C

p

r

q ď

∆?

class

Ð

True

class

Ð

False

Output

:

class

yes

no

of the expert's dataset D E to collect samples about the dynamics of the MDP, and ( ii ) a classification phase, in which it performs the classification of a reward r P R without interactions with the environment. A flow-chart is reported in Figure 1 (pseudocode in Appendix D).

Exploration phase. The exploration phase collects a dataset D in a way that depends on the structure of the MDP and of the set of rewards R to be classified. Specifically, for Linear MDPs,

CATY-IRL executes RFLin [62]. Instead, for tabular MDPs (with or without linear reward), CATY-IRL instantiates either BPI-UCBVI [37] for each reward r P R (when | R | ' Θ p 1 q , i.e., a 'small' constant w.r.t. to the size of the MDP, where 'small' depends on the size of the state space, see Appendix D.2) or RF-Express [37]. Note that CATY-IRL in this phase does not use the expert's dataset D E .

Classification phase. The classification performs the estimation p C p r q of the (non)compatibility term C p,π E p r q for the single input reward r P R by splitting it into two independent estimates: p J E p r q « J π E p r ; p q , which is computed with D E only, and p J ˚ p r q « J ˚ p r ; p q , which is computed with D only. Concerning p J E p r q , when the reward is linear r h p s, a q ' x ϕ p s, a q , θ h y , CATY-IRL uses D E to construct an empirical estimate p ψ E « ψ p,π E of the expert's expected feature count [6]. Otherwise, it directly estimates p d E « d p,π E the expert's occupancy measure. Such estimates can be used to derive p J E p r q straightforwardly. Regarding p J ˚ p r q , CATY-IRL exploits the planning phase of the corresponding RFE (or BPI) algorithm adopted at exploration phase. 8 Finally, CATY-IRL applies the (potentially negative) input threshold ∆ to the difference p J ˚ p r q ´ p J E p r q to perform the classification. See Appendix D for the full pseudo-code. Clearly, CATY-IRL can be implemented in practice, since it considers a single reward at a time instead of computing the full feasible set, and it is computationally efficient in linear MDPs, since it uses a computationally efficient algorithm as subroutine (see [62]).

Sample Efficiency. The next result analyzes the sample complexity (Definition 4.3) of CATY-IRL .

Theorem 5.1 (Sample Complexity of CATY-IRL ) . Let ϵ, δ P p 0 , 1 q . Then CATY-IRL is p ϵ, δ q -PAC for IRL with a sample complexity upper bounded by:

Some observations are in order. We conjecture that the d 2 dependence when | R | ' Θ p 1 q is unavoidable in Linear MDPs because of the lower bound for BPI in [62]. In tabular MDPs with deterministic expert, one might use the results in [66] to reduce the rate of τ E from r O p SAH 3 log p δ ´ 1 q{ ϵ 2 q to r O p SH 3 { 2 log p δ ´ 1 q{ ϵ 2 q . Finally, note that the choice ∆ ' ϵ allows us to positively classify all the rewards in the feasible set R p,π E w.h.p. and, in this case, other rewards positively classified have true (non)compatibility at most 2 ϵ w.h.p. In light of this result we conclude that rewards compatibility framework allows the practical development of sample efficient algorithms (e.g., CATY-IRL ) in Linear MDPs with large/continuous state spaces.

6 Statistical Barriers and Objective-Free Exploration

In this section, we show that CATY-IRL is minimax optimal for the number of exploration episodes in tabular MDPs, and that RFE and IRL share the same theoretical sample complexity. This allows us to formulate Objective-Free Exploration , a unifying setting for exploration problems.

6.1 The Theoretical Limits of IRL (and RFE) in the Tabular Setting

In CATY-IRL , we use a minimax optimal RFE algorithm for exploration. However, this does not entail that CATY-IRL is minimax optimal for the IRL classification problem. There might exist another PAC algorithm with a sample complexity smaller than CATY-IRL . The following result states that, in the tabular setting, the bound in Theorem 5.1 is tight for the number of episodes τ .

Theorem 6.1 (IRL Classification - Lower Bound) . Let A be an p ϵ, δ q -PAC algorithm for the IRL classification in tabular MDPs. Let τ be the number of exploration episodes. Then, there exists an IRL classification instance such that:

In both cases, the lower bound is matched by CATY-IRL , up to logarithmic factors. Note that CATY-IRL explores without using D E , thus, minimax optimality for τ can be achieved without the knowledge of D E at exploration phase. As a by-product, we observe that a similar lower bound construction can be made also for RFE, leading to the following result.

Theorem 6.2 (RFE - Refined Lower Bound) . Let A be an p ϵ, δ q -PAC algorithm for RFE in tabular MDPs. Let τ be the number of exploration episodes. Then, there exists an RFE instance such that:

This bound improves the state-of-the-art RFE lower bound Ω p H 3 SA ϵ 2 p S H ` log 1 δ qq (obtained combining the bounds in [21] and [12]) by one H factor, and it is matched by RF-Express [37].

6.2 Objective-Free Exploration (OFE)

What is the most efficient exploration strategy that can be performed in an unknown environment? It depends on the subsequent task that shall be solved. However, if the task is unknown at the exploration phase, we need a strategy that suffices for all the tasks that one might be interested in solving. Let us denote by F the set of RL and IRL classification tasks. Since CATY-IRL is a sample efficient algorithm for the IRL classification problem, and it uses RFE as a subroutine, we conclude that the RFE exploration strategy is sufficient (and also minimax optimal in tabular MDPs) to obtain guarantees for class F . Are there other problems for which RFE exploration suffices when the specific problem instance is revealed a posteriori of the exploration phase? We believe so, and in Appendix E, we identify two additional problems, i.e., Matching Performance (MP) and Imitation Learning from Observations alone (ILfO) [34], that represent potential candidates to belong to F .

More in general, we formulate the Objective-Free Exploration (OFE) problem as follows:

Definition 6.1 (Objective-Free Exploration) . Given a tuple p M , F , p ϵ, δ qq , where M is an unknown environment (e.g., MDP without reward), and F is a certain class of tasks (e.g., all RL and IRL problems), the Objective-Free Exploration (OFE) problem aims to find an exploration of the environment M (e.g., RFE exploration) that permits to solve any task f P F in an p ϵ, δ q -correct manner.

This problem is called 'objective-free' because it does not require the knowledge of the specific 'objective' f P F to be solved. In Appendix F, we describe a use case for OFE. We believe this is an interesting problem to be studied in future.

7 Conclusions

In this paper, we have shown that the feasible set cannot be learned efficiently in problems with large/continuous state spaces even under the strong structure provided by Linear MDPs. For this reason, we have introduced the powerful framework of compatible rewards , which formalizes the intuitive notion of compatibility of a reward function with expert demonstrations, and it allows us to formulate the IRL problem as a classification task. In this context, we have devised CATY-IRL , a provably efficient IRL algorithm for Linear MDPs with large/continuous state spaces. Furthermore, in tabular MDPs, we have demonstrated the minimax optimality of CATY-IRL at exploration by presenting a novel lower bound to the IRL classification problem. As a by-product, our construction improves the current state-of-the-art lower bound for RFE. Finally, we have introduced OFE, a unifying problem setting for exploration problems, which generalizes both RFE and IRL.

Limitations. A limitation of our contributions concerns the adoption of the Linear MDP model, whose assumptions are overly strong to be consistently applied to real-world applications. Nevertheless, while the rewards compatibility framework is general and not tied to Linear MDPs, we believe that Linear MDPs represent an important initial step toward the development of provably efficient IRL

algorithms with more general function approximation structures. Although a lower bound for Linear MDPs is missing, we believe that it represents an interesting direction for future works. Finally, we note that the empirical validation of the proposed algorithm is out of the scope of this work.

Future Directions. Promising directions for future works concern the extension of the analysis of the rewards compatibility framework beyond Linear MDPs to general function approximation and to the offline setting. In addition, it might be fascinating to extend the notion of reward compatibility to other kinds of expert feedback (in the context of ReL), and to other IRL settings (e.g., suboptimal experts). Finally, we believe that OFE should be analysed in-depth given its practical importance.

Acknowledgments and Disclosure of Funding

AI4REALNEThas received funding from European Union's Horizon Europe Research and Innovation programme under the Grant Agreement No 101119527. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.

Funded by the European Union - Next Generation EU within the project NRPP M4C2, Investment 1.,3 DD. 341 - 15 march 2022 - FAIR - Future Artificial Intelligence Research - Spoke 4 - PE00000013 D53C22002380006.

References

Pieter Abbeel, Adam Coates, Morgan Quigley, and Andrew Ng. An application of reinforcement learning to aerobatic helicopter flight. In Advances in Neural Information Processing Systems 19 (NeurIPS) , 2006.

Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning 21 (ICML) , 2004.

Pieter Abbeel and Andrew Y. Ng. Exploration and apprenticeship learning in reinforcement learning. In International Conference on Machine Learning 22 (ICML) , 2005.

Stephen Adams, Tyler Cody, and Peter A. Beling. A survey of inverse reinforcement learning. Artificial Intelligence Review , 55:4307-4346, 2022.

Kareem Amin and Satinder Singh. Towards resolving unidentifiability in inverse reinforcement learning, 2016.

Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence , 297:103500, 2018.

Matt Barnes, Matthew Abueg, Oliver F. Lange, Matt Deeds, Jason Trader, Denali Molitor, Markus Wulfmeier, and Shawn O'Banion. Massively scalable inverse reinforcement learning in google maps, 2024.

P. Dimitri Bertsekas. Convex Optimization Theory . Athena Scientific, 2009.

Michael Bowling, John D. Martin, David Abel, and Will Dabney. Settling the reward hypothesis. In International Conference on Machine Learning 40 (ICML) , 2023.

Haoyang Cao, Samuel Cohen, and Lukasz Szpruch. Identifiability in inverse reinforcement learning. In Advances in Neural Information Processing Systems 34 (NeurIPS) , pages 1236212373, 2021.

Gregory Dexter, Kevin Bello, and Jean Honorio. Inverse reinforcement learning in a continuous state space with formal guarantees. In Advances in Neural Information Processing Systems 34 (NeurIPS) , pages 6972-6982, 2021.

Omar Darwiche Domingues, Pierre Ménard, Emilie Kaufmann, and Michal Valko. Episodic reinforcement learning in finite mdps: Minimax lower bounds revisited. In International Conference on Algorithmic Learning Theory 32 (ALT) , volume 132, pages 578-598, 2021.

Simon Du, Sham Kakade, Jason Lee, Shachar Lovett, Gaurav Mahajan, Wen Sun, and Ruosong Wang. Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning 38 (ICML) , volume 139, pages 2826-2836, 2021.

Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning 33 (ICML) , volume 48, pages 49-58, 2016.

Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. In International Conference on Learning Representations 5 (ICLR) , 2017.

Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Systems 30 (NeurIPS) , 2017.

Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. In Advances in Neural Information Processing Systems 29 (NeurIPS) , 2016.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems 29 (NeurIPS) , 2016.

Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. In Advances in Neural Information Processing Systems 33 (NeurIPS) , pages 4415-4426, 2020.

Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E. Schapire. Contextual decision processes with low Bellman rank are PAC-learnable. In International Conference on Machine Learning 34 (ICML) , volume 70, pages 1704-1713, 2017.

Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In International Conference on Machine Learning 37 (ICML) , volume 119, pages 4870-4879, 2020.

Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. In Advances in Neural Information Processing Systems 34 (NeurIPS) , pages 13406-13418, 2021.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory 33 (COLT) , volume 125, pages 2137-2143, 2020.

Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Edouard Leurent, and Michal Valko. Adaptive reward-free exploration. In International Conference on Algorithmic Learning Theory 32 (ALT) , volume 132, pages 865-891, 2021.

Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward identification in inverse reinforcement learning. In International Conference on Machine Learning 38 (ICML) , pages 5496-5505, 2021.

Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imitation learning. In International Conference on Machine Learning 37 (ICML) , volume 119, pages 5286-5295, 2020.

Edouard Klein, Matthieu Geist, Bilal Piot, and Olivier Pietquin. Inverse reinforcement learning through structured classification. In Advances in Neural Information Processing Systems 25 (NeurIPS) , 2012.

Abi Komanduru and Jean Honorio. On the correctness and sample complexity of inverse reinforcement learning. In Advances in Neural Information Processing Systems 32 (NeurIPS) , 2019.

Abi Komanduru and Jean Honorio. A lower bound for the sample complexity of inverse reinforcement learning. In International Conference on Machine Learning 38 (ICML) , volume 139, pages 5676-5685, 2021.

David M. Kreps. Notes On The Theory Of Choice . Westview Press, 1988.

Filippo Lazzati, Mirco Mutti, and Alberto Maria Metelli. Offline inverse rl: New solution concepts and provably efficient algorithms. In International Conference on Machine Learning 41 (ICML) , 2024.

Gen Li, Yuling Yan, Yuxin Chen, and Jianqing Fan. Minimax-optimal reward-agnostic exploration in reinforcement learning, 2023.

David Lindner, Andreas Krause, and Giorgia Ramponi. Active exploration for inverse reinforcement learning. In Advances in Neural Information Processing Systems 35 (NeurIPS) , pages 5843-5853, 2022.

YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In IEEE International Conference on Robotics and Automation (ICRA) , pages 1118-1125, 2018.

Zhihan Liu, Yufeng Zhang, Zuyue Fu, Zhuoran Yang, and Zhaoran Wang. Learning from demonstration: Provably efficient adversarial policy imitation with linear function approximation. In International Conference on Machine Learning 39 (ICML) , volume 162, pages 14094-14138, 2022.

Manuel Lopes, Francisco Melo, and Luis Montesano. Active learning for reward estimation in inverse reinforcement learning. In Machine Learning and Knowledge Discovery in Databases (ECML PKDD) , pages 31-46, 2009.

Pierre Menard, Omar Darwiche Domingues, Anders Jonsson, Emilie Kaufmann, Edouard Leurent, and Michal Valko. Fast active learning for pure exploration in reinforcement learning. In International Conference on Machine Learning 38 (ICML) , volume 139, pages 7599-7608, 2021.

Alberto Maria Metelli, Filippo Lazzati, and Marcello Restelli. Towards theoretical understanding of inverse reinforcement learning. In International Conference on Machine Learning 40 (ICML) , volume 202, pages 24555-24591, 2023.

Alberto Maria Metelli, Giorgia Ramponi, Alessandro Concetti, and Marcello Restelli. Provably efficient learning of transferable rewards. In International Conference on Machine Learning 38 (ICML) , volume 139, pages 7665-7676, 2021.

Bernard Michini, Mark Cutler, and Jonathan P. How. Scalable reward learning from demonstration. In IEEE International Conference on Robotics and Automation (ICRA) , pages 303-308, 2013.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013.

Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning 17 (ICML) , pages 663-670, 2000.

Donald Ornstein. On the existence of stationary optimal strategies. Proceedings of the American Mathematical Society , 20(2):563-569, 1969.

Riccardo Poiani, Gabriele Curti, Alberto Maria Metelli, and Marcello Restelli. Inverse reinforcement learning with sub-optimal experts, 2024.

Martin Lee Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley & Sons, Inc., 1994.

Nived Rajaraman, Yanjun Han, Lin Yang, Jingbo Liu, Jiantao Jiao, and Kannan Ramchandran. On the value of interaction and function approximation in imitation learning. In Advances in Neural Information Processing Systems 34 (NeurIPS) , volume 34, pages 1325-1336, 2021.

Nived Rajaraman, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. Toward the fundamental limits of imitation learning. In Advances in Neural Information Processing Systems 33 (NeurIPS) , pages 2914-2924, 2020.

Nathan D. Ratliff, J. Andrew Bagnell, and Martin A. Zinkevich. Maximum margin planning. In International Conference on Machine Learning 23 (ICML) , pages 729-736, 2006.

Stuart Russell. Learning agents for uncertain environments (extended abstract). In Conference on Computational Learning Theory 11 (COLT) , pages 101-103, 1998.

Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach . Prentice Hall, 3 edition, 2010.

Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca Dragan. On the feasibility of learning, rather than assuming, human biases for reward inference. In International Conference on Machine Learning 36 (ICML) , volume 97, pages 5670-5679, 2019.

Mehran Shakerinava and Siamak Ravanbakhsh. Utility theory for sequential decision making. In International Conference on Machine Learning 39 (ICML) , volume 162, pages 19616-19625, 2022.

Lior Shani, Tom Zahavy, and Shie Mannor. Online apprenticeship learning. In AAAI Conference on Artificial Intelligence 36 (AAAI) , pages 8240-8248, 2022.

David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature , 529:484-503, 2016.

Joar Skalse and Alessandro Abate. Misspecification in inverse reinforcement learning. In AAAI Conference on Artificial Intelligence 37 (AAAI) , pages 15136-15143, 2023.

Joar Max Viktor Skalse, Matthew Farrugia-Roberts, Stuart Russell, Alessandro Abate, and Adam Gleave. Invariance in policy optimisation and partial identifiability in reward learning. In International Conference on Machine Learning 40 (ICML) , volume 202, pages 32033-32058, 2023.

Wen Sun, Anirudh Vemula, Byron Boots, and Drew Bagnell. Provably efficient imitation learning from observation alone. In International Conference on Machine Learning 36 (ICML) , volume 97, pages 6036-6045, 2019.

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . A Bradford Book, 2018.

Gokul Swamy, Nived Rajaraman, Matt Peng, Sanjiban Choudhury, J. Bagnell, Steven Z. Wu, Jiantao Jiao, and Kannan Ramchandran. Minimax optimal online imitation learning via replay estimation. In Advances in Neural Information Processing Systems 35 (NeruIPS) , volume 35, pages 7077-7088, 2022.

Umar Syed and Robert E Schapire. A game-theoretic approach to apprenticeship learning. In Advances in Neural Information Processing Systems 20 (NeurIPS) , 2007.

Luca Viano, Stratis Skoulakis, and Volkan Cevher. Imitation learning in discounted linear mdps without exploration assumptions, 2024.

Andrew J Wagenmaker, Yifang Chen, Max Simchowitz, Simon Du, and Kevin Jamieson. Reward-free RL is no harder than reward-aware RL in linear Markov decision processes. In International Conference on Machine Learning 39 (ICML) , volume 162, pages 22430-22456, 2022.

Ruosong Wang, Simon S Du, Lin Yang, and Russ R Salakhutdinov. On reward-free reinforcement learning with linear function approximation. In Advances in Neural Information Processing Systems 33 (NeurIPS) , pages 17816-17826, 2020.

Ruosong Wang, Ruslan Salakhutdinov, and Lin F. Yang. Reinforcement learning with general value function approximation: provably efficient approach via bounded eluder dimension. In Advances in Neural Information Processing Systems 34 (NeurIPS) , pages 6123-6135, 2020.

Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research , 18:1-46, 2017.

Tian Xu, Ziniu Li, Yang Yu, and Zhimin Luo. Provably efficient adversarial imitation learning with unknown transitions. In Conference on Uncertainty in Artificial Intelligence 39 (UAI) , volume 216, pages 2367-2378, 2023.

Lin Yang and Mengdi Wang. Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning 36 (ICML) , volume 97, pages 6995-7004, 2019.

Lei Zhao, Mengdi Wang, and Yu Bai. Is inverse reinforcement learning harder than standard reinforcement learning? In International Conference on Machine Learning 41 (ICML) , 2024.

Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory 34 (COLT) , pages 4532-4576, 2021.

Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence 23 (AAAI) , pages 1433-1438, 2008.

A Related Works

In this appendix, we report and describe the literature that most relates to this paper. Theoretical works concerning the online IRL problem can be grouped in works that concern the feasible set, and works that do not.

Let us begin with works related to the feasible set. While the notion of feasible set has been introduced implicitly in [42], the first paper that analyses the sample complexity of estimating the feasible set in online IRL is [39]. Authors in [39] adopt the simple generative model in tabular MDPs, and devise two sample efficient algorithms. [33] focuses on the same problem as [39], but adopts a forward model in tabular MDPs. By adopting RFE exploration algorithms, they devise sample efficient algorithms. However, as remarked in [68], paper [33] suffers from a limitation in the definition of the dissimilarity between feasible sets. [38] builds upon [39] to construct the first minimax lower bound for the problem of estimating the feasible set using a generative model. The lower bound is in the order of Ω ` H 3 SA ϵ 2 p S ` log 1 δ q ˘ , where S and A are the cardinality of the state and action spaces, H is the horizon, ϵ is the accuracy and δ the failure probability. In addition, [38] develops US-IRL, an efficient algorithm whose sample complexity matches the lower bound. [44] analyze a setting analogous to that of [38], in which there is availability of a single optimal expert and multiple suboptimal experts with known suboptimality. [31] analyse the problem of estimating the feasible set when no active exploration of the environment is allowed, but the learner is given a batch dataset collected by some behavior policy π b . Interestingly, [31] focuses on two novel learning targets that are suited for the offline setting, i.e., a subset and a superset of the feasible set. Authors in [31] demonstrate that such sets are the tightest learnable subset and superset of the feasible set, and propose a pessimistic algoroithm, PIRLO, to estimate them. [68] analyses the same offline setting as [31], but instead of focusing on the notion of feasible set directly, it considers the notion of reward mapping, which considers reward functions as parametrized by their value and advantage functions, and whose image coincides with the feasible set.

With regards to online IRL works that do not consider the feasible set, we mention [36], which analyses an active learning framework for IRL. However, [36] assumes that the transition model is known, and its goal is to estimate the expert policy only. Works [28] and [29] provide, respectively, an upper bound and a lower bound to the sample complexity of IRL for β -strict separable problems in the tabular setting. However, both the setting considered and the bound obtained are fairly different from ours. Analogously, [11] provides a sample efficient IRL algorithm for β -strict separable problems with continuous state space. However, their setting is different from ours since they assume that the system can be modelled using a basis of orthonormal functions.

A.1 Additional Related Works

In this section, we collect additional related works that deserve to be mentioned.

Identifiability and Reward Learning. As aforementioned, the IRL problem is ill-posed, thus, to retrieve a single reward, additional constraints shall be imposed. [5] analyses the setting in which demonstrations of an optimal policy for the same reward function are provided across environments with different transition models. In this way, authors can reduce the experimental unidentifiability, and recover the state-only reward function. [10] and [26] concern reward identifiability but in entropy-regularized MDPs [70, 15]. Such setting is in some sense easier than the common IRL setting, because entropy-regularization permits a unique optimal policy for any reward function. [10] uses expert demonstrations from multiple transition models and multiple discount factors to retrieve the reward function, while [26] analyses properties of the dynamics of the MDP to increase the constraints. With regards to the more general field of Reward Learning (ReL), we mention [19], which introduces a framework that formalizes the constraints imposed by various kinds of human feedback (like demonstrations or preferences [65]). Intuitively, multiple feedbacks about the same reward represent additional constraints beyond mere demonstrations. [56] characterizes the partial identifiability of the reward function based on various reward learning data sources.

Linear MDPs and Extensions. As explained for instance in [23], since lower bounds to the sample complexity of various RL tasks in tabular MDPs depend explicitly on the cardinality state space S , then we need to add structure to the problem if we want to develop efficient algorithms that scale to large state spaces. For this reason, the works [67, 23] analyze the Linear MDP model, which enforces

some linearity constraints to the common MDP model. In this way, authors are able to provide efficient algorithms for RL in problems with large/continuous state spaces. However, there are other settings beyond Linear MDPs that are analysed in the RL literature. [20] introduces the notion of Bellman rank as complexity measure, and provides a sample efficient algorithm for problems with small Bellman rank. [64] analyzes general value function approximation when the function class has a low eluder dimension. [22] generalizes both the eluder dimension and Bellman rank complexity measures by defining the Bellman eluder dimension and providing a provably efficient algorithm. [13] introduces bilinear classes, a structural framework that, among the others, generalizes Linear MDPs.

Reward-Free Exploration (RFE) in Tabular and Linear MDPs. The RFE problem was introduced in [21], where authors provided a sample efficient algorithm and a lower bound for tabular MDPs. Later on, the state-of-the-art sample-efficient algorithms for RFE in tabular MDPs have been developed in [24, 37, 32]. It should be remarked that RFE requires more samples than common RL in tabular MDPs. [63] proposes a sample efficient algorithm for RFE in linear MDPs. [62] improves the algorithm of [63] and, interestingly, demonstrates that RFE is no harder than RL in Linear MDPs.

Online Apprenticeship Learning (AL). The first works that provide a theoretical analysis of the AL setting when the transition model is unknown are [3, 60]. Recently, [53] formulates the online AL problem, which closely resembles the online IRL problem. The main difference is that in online AL the ultimate goal is to imitate the expert, while in IRL is to recover a reward function. [66] improves the results in [53] by combining an RFE algorithm with an efficient algorithm for the estimation of the visitation distribution of the deterministic expert's policy in tabular MDPs, presented in [47]. We mention also [46, 59] for the sample complexity of estimating the expert's policy in problems with linear function approximation. In the context of Imitation Learning from Observation alone (ILfO) [34], the work [57] proposes a probably efficient algorithm for large-scale MDPs with unknown transition model. [35] provides an efficient AL algorithm based on GAIL [18] in Linear Kernel Episodic MDPs [69] with unknown transition model.

Others. We mention work [27], which considers a classification approach for IRL. However, this is fairly different from our IRL problem formulation in Section 4.

B Additional Results and Proofs for Section 3

In this section, we provide additional results beyond those presented in Section 3, and then we report the missing proofs. Specifically, in Appendix B.1, we provide two numerical examples that explain Proposition 3.1, in Appendix B.2 we show that some additional regularity assumptions beyond the Linear MDP cannot remove the dependence on the cardinality of the state space in the sample complexity. In Appendix B.3, we report and describe the sample efficient algorithm mentioned in Section 3, while in Appendix B.4 we collect all the missing proofs of this section.

B.1 Some Examples for Proposition 3.1

The following examples aim to explain Proposition 3.1 in a simple manner.

Example B.1 (Non-degenerate feasible set) . Let M ' p S , A , H, d 0 q Y t π E u be an IRL instance such that S ' t s 1 , s 2 u , A ' t a 1 , a 2 u , H ' 1 , d 0 p s 1 q ' d 0 p s 2 q ' 1 { 2 , π E p s 1 q ' π E p s 2 q ' a 1 . Consider the feature mapping ϕ 1 s.t. ϕ 1 p s, a q ' 1 t a ' a 1 u for all s P S . Then, we have Φ π E ' t 1 u and Φ ' t 0 u . Clearly, these sets can be separated by any hyperplane w P R ą 0 , since 1 ¨ w ą 0 ¨ w , and so R p,π E ‰ t r u , with r h p s, a q ' 0 @p s, a, h q P S ˆ A ˆ J H K . Actually, R p,π E ' t r P R | D θ P p 0 , 1 s : r 1 p s, a q ' x ϕ p s, a q , θ y @p s, a q P S ˆ A u .

Example B.2 (Degenerate feasible set) . Consider the same IRL instance as in the previous example, but this time consider the feature mapping ϕ 2 s.t. ϕ 2 p s 1 , a q ' 1 t a ' a 1 u , and ϕ 2 p s 2 , a q ' 1 t a ' a 2 u . Then, we have Φ π E ' t 0 , 1 u and Φ ' t 0 , 1 u . Clearly, the two sets coincide, thus they cannot be separated, and R p,π E ' t r u , with r h p s, a q ' 0 @p s, a, h q P S ˆ A ˆ J H K .

B.2 Additional Regularity Assumptions of the State Space do not Make the Problem Learnable

In tabular MDPs with small state space S , collecting samples from every state s P S is feasible, and it is exactly what previous works do:

Under the assumption that π E is deterministic, [39, 38] collect one sample from every p s, h q P S ˆ H using a generative model, obtaining π E exactly.

In the offline setting, assuming that the occupancy measure of the expert's policy is at least d min in all reachable p s, a q P S ˆ A , then [31] learns the support of π E exactly w.h.p. using 9 1 { d min episodes.

J K · If π E is stochastic, under the assumption that all actions in the support of the expert's policy are played with probability at least π min (see Assumption D.1 of [38]), both [38, 68] are able to learn the support of π E exactly w.h.p. using 9 1 { π min samples in the online setting. 9

However, when S is large, even under the Linear MDP assumption, this is not possible. In Section 3, we have formalized this fact with the following proposition:

Theorem 3.2 (Statistical Inefficiency) . Let M Yt π E u be a Linear IRL instance with finite state space S and deterministic expert's policy, and let ϵ, δ P p 0 , 1 q . If an algorithm A is p ϵ, δ q -PAC, then τ E ' Ω p S q , where S : ' | S | is the cardinality of the state space.

Theorem 3.2 tells us that the Linear MDP assumption is too weak for the feasible set to be learnable using the PAC framework of Definition 3.2 with a number of samples independent of the cardinality of the state space. Therefore, we can try to introduce an additional assumption on the structure of the IRL problem M Yt π E u and see whether it helps in alleviating the issue. Let us consider the following first assumption.

Assumption B.1. We assume a Lipschitz continuity property between features and states:

for some L ą 0 and some distance } ¨ ´ ¨ } in S .

The intuition is that, based on the fact that in Linear MDPs the Q -function of any policy π is linear in the feature mapping Q π h p¨ , ¨q ' p ϕ p¨ , ¨q , w π h q for some parameter vector w π h P R d (see [23]), then if we are able to ϵ -cover the state space S , we can approximate the Q -function Q π h p s, ¨q in any s P S with the Q -function Q π h p s 1 , ¨q of the closest point s 1 int the covering, so that | Q π h p s, a q ´ Q π h p s 1 , a q| ' |p ϕ p s, a q ´ ϕ p s 1 , a qq ⊺ w π h | ď } ϕ p s, a q ´ ϕ p s 1 , a q} 2 } w π h } 2 ď Lϵ } w π h } 2 . However, this assumption is not sufficient.

Proposition B.1. Under the setting of Proposition 3.2, even under Assumption B.1, then an algorithm is p ϵ, δ q -PAC only if τ E ' Ω p S q .

Assumption B.1 fails because it does not provide any information about how the knowledge of the expert's policy at a state can be 'transferred' to other states, and thus we still need to sample almost all the states of S p,π E to get an acceptable feasible set.

We devise another assumption to attempt to fix this issue.

Assumption B.2. We assume the following Lipschitz continuity property:

for some L ą 0 and some distance } ¨ ´ ¨ } in S .

This assumption says that states that are close to each other cannot have the features corresponding to the expert's action too far away from each other. From a high-level point of view, it says that the features are 'somehow' regular with π E , so that when the expert lies in s 1 which is really close to s , then she plays an action which has the same 'effect' (i.e., same transition model and same reward, due to the Linear MDP assumption) as the expert's action in s .

Assumption B.2 is not comparable with Assumption B.1 since, on the one hand, it does not hold for all actions in A , but only for those corresponding to π E , but, on the other hand, provides information on how to transfer knowledge about π E to neighbor states.

Let ∆ 1 : ' min s P S ,a,a 1 P A : ϕ p s,a q‰ ϕ p s,a 1 q } ϕ p s, a q ´ ϕ p s, a 1 q} 2 , i.e., the smallest non-zero distance between the features of different actions. Clearly, when S is finite, since in Linear MDPs also A : ' | A | is finite, then ∆ 1 is finite too. So we can define a new quantity ∆ to be any number 0 ă ∆ ă ∆ 1 .

Proposition B.2. Under the setting of Proposition 3.2, under Assumption B.2, then a number of samples τ E ' | N p ∆ 2 L ; S , } ¨ }q| is sufficient to recover π E exactly in any p s, h q P S , where | N p ∆ 2 L ; S , } ¨ }q| is the ∆ {p 2 L q -covering number of space S w.r.t. distance } ¨ } .

Intuitively, by constructing a covering with a sufficiently small radius in the state space S , then we are able to retrieve the exact expert's action in the neighborood of each state of the covering. Doing so, we are able to construct ϵ -correct estimates of the feasible set. Of course, this is possible as long as ∆ 1 is not too small, and L is not too large. When S is infinitely large or continuous, it might be possible to construct feature mappings in which ∆ 1 Ñ 0 , and so the approach would still require too many samples.

However, even for cases with finite and not too small ∆ 1 , the result in Proposition B.2 is not satisfactory, because it just allows to retrieve π E under a stronger assumption than Linear MDPs, but not to perform an interesting learning process. We observe that the feasible set is an 'unstable' concept, in the sense that, based on Proposition 3.1, changing the expert action in a single state might reduce the feasible set from a continuum of rewards to a singleton, or vice versa.

Remark B.1. If we want to be able to recover the exact feasible set efficiently, we need to recover the exact expert's policy almost everywhere.

B.3 Algorithm

By exploiting an RFE algorithm as sub-routine like that of Algorithm 1 in [63] or Algorithm 1 in [62], we are able to construct estimates of the transition model p p , that can be used to compute an 'empirical' estimate of the feasible set p R « R p p,π E (since ϕ and π E are known). The algorithm is presented in Algorithm 1.

Algorithm 1: IRL for Linear MDPs (known expert's policy)

Data: failure probability δ ą 0 , error tolerance ϵ ą 0 , expert policy π E , all sets Z Ď S ˆ J H p,π E p,π E

8 Return p R

Simply put, Algorithm 1 uses the dataset collected by an RFE algorithm to compute a least-squares estimate of the transition model p p , and then it returns the feasible set defined according to it (recall that ϕ and π E are known). Notice that this algorithm cannot be implemented in practice due to various reasons, like the presence of the Dirac delta δ measure in the definition of some quantities (see Appendix B.4.3), and the fact that the feasible set is, potentially, a set containing infinite rewards. Nevertheless, Theorem 3.3 states that this algorithm is sample efficient. The proof of the theorem is provided in Appendix B.4.3.

It should be remarked that Algorithm 1 takes in input also the true support of the visit distribution of the expert policy S p,π E in case S is finite, and all the possible sets Z that agree with S p,π E a.e. based on the measure d p,π E in case S is infinite. Intuitively, this set ( S p,π E ) of p s, h q pairs represents

the domain in which π E is defined. Indeed, since the expert in the true problem p never visits pairs p s 1 , h 1 q R S p,π E , its expert policy might reasonably be non well-defined there. When S is infinite, we require all sets Z because otherwise we cannot know which are the sets S p,π E z Z with zero measure, i.e., in which the reward can induce an optimal action different from the expert's one, since the overall contribution to the expected return is zero.

The proof of Theorem 3.3 is obtained by using Algorithm 1 of [62] at Line 1 of Algorithm 1. In Appendix B.4.3, we demonstrate an upper bound also if we use Algorithm 1 in [63].

B.4 Missing Proofs

Before diving into the proofs, we recall some important properties of the feasible set and of the Linear MDPs that will be useful in the proofs. First, we provide an explicit form for the feasible set presented at Definition 3.1.

Lemma B.3 (Lemma E.1 in [31]) . In the setting of Definition 3.1, if S is finite, then the feasible set R p,π E satisfies:

Notice that we have extended Lemma E.1 in [31] to consider stochastic expert policies (the extension is trivial). We can easily extend it to problems with large/continuous S .

Lemma B.4 (Feasible Set Explicit) . In the setting of Definition 3.1, then the feasible set R p,π E satisfies:

Simply, Lemma B.4 improves on Lemma B.3 by allowing the reward to enforce the 'wrong' action (i.e., different from the expert's action) in a subset with zero measure based on the visitation distribution.

Proof. The proof is completely analogous to that of Lemma E.1 in [31]. We just need to observe that if set S has zero measure (and the set of rewards R contains bounded rewards), then it does not affect the expected return.

Another useful property that we need is that the Q -function is always linear in the feature map for any policy in Linear MDPs.

Proposition B.5 (Proposition 2.3 in [23]) . For a Linear MDP, for any policy π , there exist weights t w π h u h P J H K such that, for any p s, a, h q P S ˆ A ˆ J H K , we have Q π h p s, a q ' x ϕ p s, a q , w π h y .

Wecan combine the results of Lemma B.4 and Proposition B.5 to obtain the following characterization of the feasible set in Linear MDPs.

Lemma B.6. In the setting of Definition 3.1, the feasible set R p,π E satisfies:

where θ h : ' w h ´ ş S max a 1 P A x ϕ p s 1 , a 1 q , w h ` 1 y dµ h p s 1 q for all h P J H K , and A E h p s q : ' t a P A | π E h p a | s q ą 0 u .

Proof. From [45], we know that in any MDP there exists an optimal policy. Therefore, thanks to Proposition B.5, we know that the optimal Q -function Q ˚ is linear in the feature map too. So, there

exist parameters t w h u h such that, for any p s, a, h q P S ˆ A ˆ J H K , the optimal Q -function can be rewritten as Q ˚ h p s, a q ' x ϕ p s, a q , w h y . From the Bellman equation, we know that:

By rearranging this equation, and removing the dot product with ϕ p s, a q , we obtain that:

Now, this holds in any Linear MDP. If we desire to enforce the constraints in Lemma B.4, we simply have to impose the constraint on the optimal Q -function using parameters t w h u h outside some S . This concludes the proof.

It is useful to introduce the following definitions. First we define the set of parameters that induce a Q -function compatible with π E :

Next, we define the set of parameters of the reward function obtained by using Q -functions parametrized by w P W p,π E :

ˇ Irrespective of the transition model t µ h u h and the feature map ϕ , we see that it is always possible to construct a surjective map from Θ p,π E to W p,π E (the map in the definition of Θ p,π E ). Thanks to these definitions, the feasible set can be rewritten as:

We are now ready to provide the proofs of the various results of this section.

B.4.1 Proof of Proposition 3.1

Proposition 3.1. Let M be a Linear MDP without reward with a finite state space, and let ϕ be a feature mapping. Let t Φ π E h u h P J H K and t Φ h u h P J H K be the sets of expert's and non-expert's features, defined for every h P H as:

where A E h p s q : ' t a P A | π E h p¨| s q ą 0 u for every s P S . If for none of the H pairs of sets p Φ π E h , Φ h q there exists a separating hyperplane, then R p,π E ' t r u , with r h p s, a q ' 0 @p s, a, h q P S ˆ A ˆ J H K i.e., the feasible set with linear rewards in ϕ contains only the reward function that assigns zero reward everywhere.

Proof. From [8], we recall that two sets Y 1 , Y 2 are separated by a hyperplane H ' t x | a ⊺ x ' b u if each lies in a different closed halfspace associated with H , i.e., if either:

or:

By definition of W p,π E , for each stage h P J H K , we are looking for vectors w h P R d such that @p s, h q P S p,π E , it holds that:

In words, for each p s, h q P S p,π E , we are looking for non-affine separating hyperplanes between features of expert and non-expert actions. However, since the hyperplane parameter w h is common to all states s P S p,π E h , then it must separate expert from non-expert actions at all states. This is equivalent to finding the separating hyperplanes to the sets Φ π E h and Φ h which contain all the points. Clearly, when the separating hyperplanes do not exist at all h P J H K , then the condition in W p,π E is satisfied by the zero vector alone. As a consequence, set Θ p,π E contains only the zero vector, and so does R p,π E .

Remark B.2. By using the result of Lemma B.4, we can easily convert Proposition 3.1 into a more general result by considering the impossibility of separating any pair of sets constructed by varying at will some subsets with zero measure. We will not provide such result explicitly.

B.4.2 Proofs of Proposition 3.2 and Appendix B.2

In the PAC framework of Definition 3.2, we have not specified formally the inner distance d :

where:

where t θ h u h and t p θ h u h are the (unbounded) parameters of rewards r and p r . As explained in [31], such normalization term allows us to work with unbounded reward functions. In practice, we are relaxing the Linear MDP assumption presented in Section 2 about the boundedness of the parameters θ of the rewards to avoid the issue described in [38] and [31]. We still assume that the feature mapping is bounded. Observe that this relaxation does not affect the results we present, which would hold even if we considered bounded parameters θ . Indeed, as visible in the proofs, the instances do not need to be constructed with unbounded θ .

Theorem 3.2 (Statistical Inefficiency) . Let M Yt π E u be a Linear IRL instance with finite state space S and deterministic expert's policy, and let ϵ, δ P p 0 , 1 q . If an algorithm A is p ϵ, δ q -PAC, then τ E ' Ω p S q , where S : ' | S | is the cardinality of the state space.

Proof. We construct two problem instances that lie at a finite Hausdorff distance, and show that, with less than S calls to the sampling oracle, we are not able to discriminate between the two instances.

Let S be the finite state space with cardinality S , A ' t a 1 , a 2 u , H ' 1 , d 0 p s q ' 1 { S @ s P S , ϕ p s, a q ' 1 t a ' a 1 u , and consider two deterministic expert's policies π E 1 p s q ' a 1 @ s P S , and π E 2 p s q ' a 1 @ s P S zt s u , and π E 2 p s q ' a 2 , for a certain s P S . The set of parameters compatible with π E 1 is:

since Q π E 1 p s, a 1 q ě Q π E 1 p s, a 2 q ðñ r p s, a 1 q ě r p s, a 2 q ðñ ϕ p s, a 1 q θ ě ϕ p s, a 2 q θ ðñ 1 ¨ θ ě 0 ¨ θ . Observe that, for π E 2 , due to the presence of s , we have:

since s imposes θ ď 0 , and the other states impose θ ě 0 .

Therefore, the Hausdorff distance between the two problems is:

Obviously, we need a Ω p S q samples to spot, if it exists, state s , and thus distinguish between R π E 1 and R π E 2 .

Proposition B.1. Under the setting of Proposition 3.2, even under Assumption B.1, then an algorithm is p ϵ, δ q -PAC only if τ E ' Ω p S q .

Proof. The same proof of Proposition 3.2 works here.

In particular, we now show that Assumption B.1 does not help. The Hausdorff distance between the instances in the proof of Proposition 3.2 can be written as:

where s 1 is the state in the covering closest to state s ; while the first term can be bounded, the assumption does not help us with the second term.

Proposition B.2. Under the setting of Proposition 3.2, under Assumption B.2, then a number of samples τ E ' | N p ∆ 2 L ; S , } ¨ }q| is sufficient to recover π E exactly in any p s, h q P S , where | N p ∆ 2 L ; S , } ¨ }q| is the ∆ {p 2 L q -covering number of space S w.r.t. distance } ¨ } .

Proof. For any state s P S p,π E , by definition of covering N p ∆ 2 L ; S , } ¨ }q , there always exist another state s 1 P N p ∆ 2 L ; S , } ¨ }q such that } s 1 ´ s } ď ∆ 2 L . By Assumption B.2 we know that:

and since π E h p s 1 q and thus ϕ p s 1 , π E h p s 1 qq is known, then the fact that ∆ is finite guarantees us that π E h p s q is equal to the action a that minimizes the distance to ϕ p s 1 , π E h p s 1 qq . Notice that if, by contradiction, there were two actions a 1 , a 2 with } ϕ p s, a 1 q ´ ϕ p s 1 , π E h p s 1 qq} 2 ď ∆ 2 and } ϕ p s, a 2 q ´ ϕ p s 1 , π E h p s 1 qq} 2 ď ∆ 2 , then by triangle inequality and finiteness of ∆ , we would have:

which is clearly a contradiction.

B.4.3 Proof of Theorem 3.3

The proof is based on deriving an upper bound to the Hausdorff distance between the true feasible set and its estimate. To do so, first, using the notation of [23], let us define the following quantities:

where δ p¨ , x q is the Dirac measure, and p s k h , a k h q represents the state-action pair visited at stage h of exploration episode k P J τ K . In words, P denotes the true transition model, p P denotes the least squares estimate computed by Algorithm 1, and P represents a bridge between the two. As we will see, the core of the proof consists in upper bounding the term ˇ ˇ ` P h ´ p P ˘ V h ` 1 p s, a q ˇ ˇ at all h P J H K

and reachable p s, a q P S ˆ A , for all the bounded linear functions V in class V , defined as:

ˇ To achieve this goal, it will be useful to apply triangle inequality and to bound the following two terms separately:

Lemma B.7. For any value function V in the class V , for any p s, a, h q P S ˆ A ˆ J H K , it holds that:

Proof. We have:

r where at (1) we have defined vector r w h : ' ş S V h ` 1 p s 1 q dµ h p s 1 q , at (2) we have used the definition of Λ h , and at (3) we have recognized that ϕ p s k h , a k h q ⊺ r w h ' P h V h ` 1 p s k h , a k h q . By taking the absolute value, we can write:

where at (4) we have applied Cauchy-Schwarz's inequality, at (5) we have bounded the quadratic form with the 2-norm and the largest eigenvector of the matrix, i.e., } r w h } Λ ´ 1 h ' b r w ⊺ h Λ ´ 1 h r w h ď ? σ } r w h } 2 , where σ is the largest eigenvalue of matrix Λ ´ 1 h , and then we have upper bounded σ ď 1 , since 1 is the smallest eigenvalue of invertible matrix Λ h (see [23]); finally, at (6) we have used the fact that | V h ` 1 p¨q| ď H , and so that } r w h } 2 ' } ş S V h ` 1 p s 1 q dµ h p s 1 q} 2 ď H } µ h p S q} 2 ď H ? d .

The result follows by noticing that the quantity to bound cannot be larger than 2 H .

Lemma B.8. Let δ P p 0 , 1 q . For any value function V in the class V , for any p s, a, h q P S ˆ A ˆ J H K , with probability at least 1 ´ δ { 2 , it holds that:

for some constant c .

Proof. We can write:

where at (1) we have applied Cauchy-Schwarz's inequality, at (2) we have applied Lemma B.13, at (3) we have upper bounded N ϵ using Lemma B.12, at (4), similarly to [62], unlike [23], we see that no union bound is needed (because there is no dependence on Λ ), thus by choosing ϵ ' H ? d { τ , we get the passage. Passage (5) follows for some constant c .

The result follows by a union bound over h P J H K , and by noticing that the quantity to bound cannot be larger than 2 H .

We are now ready to upper bound the Hausdorff distance using the two lemmas just presented. Recall that we work with unbounded rewards (parameters θ ), and that the definition of inner distance d is provided in Equation (1).

Lemma B.9. With probability at least 1 ´ δ { 2 , the Hausdorff distance between the true feasible set R p,π E and its estimate p R returned by Algorithm 1 can be upper bounded by: H p R p,π E , p R q ď 4 J ˚ p u ; p q , where u h p s, a q : ' min t β } ϕ p s, a q} Λ ´ 1 h , H u for all p s, a, h q P S ˆ A ˆ J H K , and β : ' cH a d log p 1 ` τ q ` log p H { δ q for some absolute constant c ą 0 .

Proof. Let us begin to bound the first branch of the Hausdorff distance.

where at (1) we have simply applied the Bellman optimality equation twice w.r.t. the reward function, at (2) we have upper bounded the infimum over the second set of rewards p R with the specific choice of reward r r P p R provided by Lemma B.11, at (3) we use the property of r r described in Lemma B.11, at (4) we bring term 1 { M r, r r inside, and then we upper bound it by: 1 { M r, r r : ' 1 { max t ? d, max h } θ h } 2 , max h } r θ h } 2 u{ ? d ď 1 { max t 1 , max h } θ h } 2 { ? d u , i.e., by simply removing one of the terms inside the maximum operator at denominator. At (5) we define K : ' max t 1 , max h } θ h } 2 { ? d u , and, since the value function is linear in the reward, we apply K directly to the reward. At (6) we realize that the possible optimal value functions that can be constructed in p using rewards in R p,π E normalized by K are a subset of the value functions in class V , i.e., of all the possible optimal value functions with parameters } w h } 2 ď 2 H ? d . This is not trivial since we are working with unbounded rewards r , and thus their parameters t θ h u h can be any. The normalization by K permits this in the following manner. For any h P J H K , we have r h p¨ , ¨q{ K ' x ϕ p¨ , ¨q , θ h { K y ' x ϕ p¨ , ¨q , θ h { max t 1 , max h 1 } θ h 1 } 2 { ? d uy . Therefore, if max h 1 } θ h 1 } 2 ą ? d , then the normalization makes sure that max h 1 } θ h 1 } 2 ' ? d , while if max h 1 } θ h 1 } 2 ď ? d , then the normalization is by 1 and it has no effect. In this way, we see that value functions V ˚ h ` 1 p s, a ; p, r K q can be created by a simple r 1 with parameters t θ 1 h u h with 2-norms bounded by ? d . This guarantees that, since by hypothesis of Linear MDPs } ϕ p¨ , ¨q} 2 ď 1 , the value function never exceeds H , and that the norm of the Q -function parameters t w π h u h for any policy π can be bounded as: } w π h } 2 ď } θ h { K } 2 `} ş S V π h ` 1 p s 1 q dµ h p s 1 q} 2 ď ? d ` H } µ h p S q} 2 ď ? d ` H ? d ď 2 H ? d (similarly to Lemma B.1 of [23]). It should be remarked that class V is more general than the actual set of optimal value functions that can be obtained using r P R p,π E in p , since such rewards induce optimal value functions for which the optimal action in S p,π E is always the expert's action/s π E p s q .

Notice that the same derivation can be carried out also for the other branch of the Hausdorff distance, ending up with the same expression. Therefore, the last line is an upper bound to the Hausdorff distance:

where at (7) we have noticed that class V contains the cartesian product of H sets, one for each stage, and therefore the supremum can be brought inside the summation, at (8) we have applied triangle inequality, at (9) we have applied Lemma B.7 and Lemma B.8 and used some absolute constants c 1 , c 2 ą 0 , and also the fact that for any numbers x, y, w, z , we have min t x, y u ` min t w,z u ď min t x ` w,y ` z u .

To conclude the proof of the main theorem, we simply have to observe that any RFE algorithm provides a bound to J ˚ p u 1 ; p q for some u 1 similar to u . Depending on the RFE algorithm instantiated as sub-routine, the sample complexity of Algorithm 1 varies.

Theorem 3.3. Assume that π E (along with its support S p,π E ) is known. Then, for any ϵ, δ P p 0 , 1 q , Algorithm 1 is p ϵ, δ q -PAC for IRL with a number of episodes τ upper bounded by:

Proof. To get the result, we instantiate Algorithm 1 of [62] as RFE sub-routine. Simply, observe that [62] sets β 1 so that β 1 ě r β : ' c 1 H a d log p 1 ` dHτ q ` log p H { δ q ě β . By Lemma B.9, we know that:

for some absolute constant c 1 ą 0 . It should be remarked that the quantity in the last line is, modulo c 1 , the quantity that [62] bound in the proof of their Theorem 1 using their algorithm. Specifically, by taking:

and a union bound over the two events that hold w.p. 1 ´ δ { 2 , and re-setting ϵ Ð c 1 ϵ , we get the result.

Notice that if we run Algorithm 1 of [63] for exploration instead of Algorithm 1 of [62], we obtain:

Theorem B.10. If we use Algorithm 1 of [63] at Line 1 of Algorithm 1, then for any ϵ, δ P p 0 , 1 q , such algorithm is p ϵ, δ q -PAC for IRL with a number of episodes τ upper bounded by:

Proof. By Lemma B.9, we know that:

for β : ' cH a d log p 1 ` τ q ` log p H { δ q . Now, let us define, similarly to Appendix A of [63], the quantities u 1 h p s, a q : ' min t β 1 } ϕ p s, a q} Λ ´ 1 h , H u for all p s, a, h q P S ˆ A ˆ J H K , and β 1 : ' c 1 dH a log p dH { δ { ϵ q for some absolute constant c 1 ą 0 . In addition, set the number of exploration episodes τ to τ ' c 2 d 3 H 6 log p dHδ ´ 1 ϵ ´ 1 q{ ϵ 2 , and notice that, for appropriate choices of c 1 , c 2 , it holds that: β 1 ě c 1 dH a log p dHτ { δ q ě β : ' cH a d log p 1 ` τ q ` log p 1 { δ q . This entails that u 1 h p s, a q ě u h p s, a q at all s, a, h , and so:

where at (1) we have applied Lemma 3.2 of [63] (reported in Lemma B.14 for simplicity) with some new constant c 1 ą 0 , and at (2) we have simply replaced τ with its value defined in Algorithm 1 of [63].

The result follows by union bound between the two events that hold w.p. 1 ´ δ { 2 to get 1 ´ δ , and by noticing that c 2 is a constant, thus setting ϵ Ð c 2 ϵ provides the result.

Lemma B.11. Let R p,π E be the feasible set of policy π E w.r.t. transition models p , and let p R be its estimate constructed as in Algorithm 1 using the true π E , S p,π E (or sets Z ) and some p p . For any reward r P R p,π E , the reward r such that, for all p s, a, h q P S ˆ A ˆ H :

belongs to p R . Moreover, observe that: Q ˚ h p s, a ; p, r q ' Q ˚ h p s, a ; p p, p r q at all p s, a, h q P S ˆ A ˆ J H K . In addition, for any reward p r P p R , it is possible to construct a reward r in analogous manner so that r P R p,π E , and such that Q ˚ h p s, a ; p, r q ' Q ˚ h p s, a ; p p, p r q at all p s, a, h q P S ˆ A ˆ J H K .

Proof. First, we consider the case when S is finite. By rearranging the terms in the definition of p r , we see that, for all p s, a, h q P S ˆ A ˆ H :

which, by the Bellman optimality equation, entails that Q ˚ h p s, a ; p, r q ' Q ˚ h p s, a ; p p, p r q . We recall that R is defined as:

while thanks to Lemma B.4, the feasible set R p,π E can be written as:

It is clear that, if Q ˚ h p s, a ; p, r q ' Q ˚ h p s, a ; p p, p r q for all p s, a, h q P S ˆ A ˆ J H K , then since r P R p,π E we necessarily have p r P p R . The proof of the opposite case is completely analogous.

In the case with infinite S , notice that both the feasible set R p,π E in Lemma B.4 and the definition of p R in Algorithm 1 make use of the same sets Z . Thus, we simply make the choice of reward with same Z and proceed like in the finite case.

Lemma B.12 (Covering Number of Class V ) . Let V be defined as in Equation (2) , and define distance dist in V as dist p V, V 1 q : ' sup s P S | V p s q ´ V 1 p s q| . Then, the ϵ -covering number | N p ϵ ; V , dist q| of

set V with distance dist can be bounded as:

Proof. The proof follows that of Lemma D.6 of [23], but is simpler because of the different form of V .

For any V 1 , V 2 P V parametrized by w 1 , w 2 , we write:

If we construct an ϵ -cover of W : ' t w P R d | } w } 2 ď 2 H ? d u w.r.t. the 2-norm, we get a covering number bounded by | N p ϵ ; W , } ¨ } 2 q| ď p 1 ` 4 H ? d { ϵ q d . Clearly, this value upper bounds the covering number of class V and the result follows.

where at (1) we have used the common bound that the absolute difference of maxima is upper bounded by the maximum of the absolute difference of the two functions, at (2) we have used the fact that the feature map is always bounded by 1 in 2-norm, and at (3) we have recognized the dual norm of the 2-norm, i.e., itself.

Lemma B.13 (Lemma D.4 of [23]) . Let t s k u 8 k ' 1 be a stochastic process on state space S with corresponding filtration t F k u 8 k ' 0 . Let t ϕ k u 8 k ' 0 be an R d -valued stochastic process where ϕ k P F k ´ 1 , and } ϕ k } 2 ď 1 . Let Λ τ ' I ` ř τ k ' 1 ϕ k ϕ ⊺ k . Then, for any δ ą 0 , with probability at least 1 ´ δ , for all τ ě 0 , and any V P V so that sup s P S | V p s q| ď H , we have:

where N ϵ is the ϵ -covering number of V with respect to the distance dist p V, V 1 q : ' sup s P S | V p s q ´ V 1 p s q| .

LemmaB.14 (Lemma 3.2 of [63]) . With probability 1 ´ δ { 2 , for the function u 1 defined as u 1 h p s, a q : ' min ␣ β 1 } ϕ p s, a q} Λ ´ 1 h , H ( , with β 1 : ' c 1 dH a log p dHδ ´ 1 ϵ ´ 1 q , we have:

for some absolute constant c ą 0 .

C Additional Insights on Compatibility

In this appendix, we collect and describe additional insights to the notion of rewards compatibility introduced in Section 4. The appendix is organized in the following manner: Appendix C.1 provides a visual explanation to the notion of rewards compatibility, in Appendix C.2 we analyse a multiplicative alternative to the definition of rewards compatibility, and Appendix C.3 discusses the conditions under which a learned reward can be used for 'forward' RL, by comparing rewards with small (non)compatibility with rewards learned in previous works.

C.1 A Visual Explanation for Rewards Compatibility

In this appendix, we aim to provide a visual intuition to the notion of rewards compatibility. For this reason, the reader should keep in mind Figure 3.

d

p,π

1

d

p,π

2

. . .

d

p

,π

E

d

p

,π

d

p

,π

E

As explained in Section 4, even in the limit of infinite samples, i.e., even if we know M ' p S , A , H, d 0 , p q Y t π E u exactly, and even if we assume that the expert is exactly optimal, i.e., J ˚ p r E ; p q ´ J π E p r E ; p q ' 0 (where r E is the true reward optimized by the expert), then we still do not have idea of how other policies perform. Expert demonstrations only provide information about the performance of a single policy, π E , w.r.t. to the reference J ˚ p r E ; p q under the unknown r E , i.e., demonstrations say that π E in r E performs as good as J ˚ p r E ; p q . But what about other policies? Demonstrations provide no information.

To see this, consider Figure 3, in which each line exemplifies the visitation distribution induced by some policy π P Π , and the point in the middle represents the starting state s 0 ' d 0 . Intuitively, observing d p,π E along with knowing that J π E p r E ; p q is good (i.e., because of expert demonstrations), does not tell us anything about the distribution d p,π induced by some other policy π potentially arbitrarily different from d p,π E . Indeed, it might be the case that J π p r E ; p q is acceptable, or that it is as good as J π E p r E ; p q , or that it is very bad. We cannot know from demonstrations only.

For this reason, if we consider the set of rewards with 0-(non)compatibility, i.e., the feasible reward set, we notice that it contains the rewards r that make π optimal J π p r ; p q ' J ˚ p r ; p q , but also the rewards r 1 that make π nearly optimal J π p r 1 ; p q « J ˚ p r 1 ; p q , and also the rewards r 2 that make π a very bad-performing policy J π p r 2 ; p q ! J ˚ p r 2 ; p q . Indeed, as long as both r, r 1 , r 2 make the direction pointed by d p,π E in Figure 3 a good direction, then they are in accordance with the constraint imposed by the demonstrations. The additional Degrees of Freedom (DoF) provided by policies beyond π E (e.g., π, . . . ) permit the ill-posedness of IRL.

We said that expert demonstrations provide information just about the performance of a single policy, π E . However, to be precise, in the context of IRL, this is not correct. Indeed, differently from the mere learning from demonstrations setting, in which we just assume that π E is a very good-performing policy, in IRL we assume that the underlying problem is an MDP, i.e., that the expert agent is optimizing a reward function r E . 10 This additional structure (i.e., that the underlying environment is indeed an MDP), makes sure that the performances of various directions d p,π in Figure 3 are measured through a dot product with a fixed reward function r , i.e.:

For this reason, we have the guarantee that the directions in the red area surrounding d p,π E are almost as good as d p,π E . Indeed, for all policies π such that ř h P J H K } d p,π h ´ d p,π E h } 1 ď ϵ , i.e., for all policies

ϵ -close to π E in 1-norm, we can write:

However, it should be remarked that, since we aim to recover the rewards explaining the expert's preferences, then we are guaranteed that policies close in 1-norm perform similarly under any reward function (by definition of 1-norm), and so we do not risk to incur in the error of representing d p,π E and a direction d p,π inside the red area of Figure 3 with very different performances.

C.2 A Multiplicative Compatibility

In Section 4, we have defined an additive notion of (non)compatibility, based on the difference of performance between π E and π ˚ (the optimal policy). Here, we analyze a multiplicative notion of (non)compatibility, based on the ratio of the performances. 11

We make the following observation. Any reward r P R induces, in the considered environment p , an ordering in the space of policies Π , based on the performance J π p r ; p q of each policy π P Π . It is easy to notice that for any scaling and translation parameters α P R ą 0 , β P R , the reward constructed as r 1 p¨ , ¨q ' αr p¨ , ¨q ` β induces the same ordering as r in the space of policies. 12

For this reason, it seems desirable to use a notion of (non)compatibility such that rewards r and r 1 p¨ , ¨q ' αr p¨ , ¨q ` β for some α, β , suffer from the same (non)compatibility w.r.t. some expert policy π E . However, observe that, for the notion of compatibility C in Definition 4.1, we have that, for any r P R :

Simply put, for the additive notion of (non)compatibility C , the scale ( α ) of a reward matters, and rescaling the reward modifies the (non)compatibility.

To solve this issue, one might introduce a multiplicative notion of compatibility F (defined only for non-negative rewards and setting F p,π E p r q ' 0 when the denominator is 0):

Clearly, the larger F p,π E p r q , the closer is the performance of π E to the optimal performance. Observe that, for this definition,we have:

i.e., this definition does not care about the scaling α of the reward, but it is sensitive to the actual position β of that reward.

Therefore, both C and F suffer from some 'rescaling' issues. Is it possible to devise a notion of compatibility, i.e., a measure of suboptimality, for a policy, that is independent of both the scale α and position β ? Formally, we are looking for a function (notion of distance) f : R ˆ R Ñ R ě 0 such that, for any J 1 , J 2 P R :

for all α P R ą 0 , β P R . Unfortunately, this is not possible, since it is easy to show that all the functions f of this kind are of the following type:

for some reals K ` , K 0 , K ´ . In words, any function f that satisfies Equation (3) is able to express just an ordering between inputs J 1 and J 2 , but not an actual measure of sub-optimality/compatibility.

We conclude by stating that we prefer to use C instead of F for the following reasons:

First, most RL literature prefers the additive notion of suboptimality towards the multiplicative one.

The additive notion of suboptimality is simpler to analyze w.r.t. the multiplicative one.

C.3 When can a learned reward be used for 'forward' RL?

In this appendix, we exploit the intuition developed in Appendix C.1 to discuss under which conditions we can exploit demonstrations alone to recover a single reward that can be used for 'forward' RL , i.e., to recover a single reward r for which we have the guarantee that any ϵ -optimal policy π to r in the true environment p has similar performance in the same environment p under the true reward r E , that is, policy π is an f p ϵ q -optimal policy to r E in p , for some function f .

Applications of IRL range from Apprenticeship Learning (AL), to reward design, to interpretability of expert's preferences. Concerning AL, it is common to 'use' the reward r learned through IRL to optimize our learning agent. But what properties r should satisfy in order to obtain performance guarantees on our learning agent w.r.t. the true (unknown) r E ? We now list and analyze various plausible requirements.

First, we might ask that, being π E optimal w.r.t. r E , then π E P arg max π J π p r q , i.e., that the expert policy π E is optimal under the learned reward r . However, this requirement is not satisfactory for the following reason. Reward r might induce more than one optimal policy (e.g., it might induce both π, π E as optimal), and optimal policies other than π E (e.g., π ) are not guaranteed to perform well under r E (actually, π can be any policy in Π ). Clearly, this is not satisfactory. Observe that there are rewards in the feasible set R p,π E for which multiple policies are optimal (thus, not all the rewards in the feasible set are satisfactory).

We might additionally ask that π E is the unique optimal policy of reward r (similarly to what happens in entropy-regularized MDPs [70, 15]). However, this is not satisfactory for the following reason. In practice, it is really difficult (almost impossible) to compute the optimal policy of a given reward. Thus, what is usually done in RL, is to settle for an ϵ -optimal policy. Since any policy can be ϵ -optimal under reward r , then no guarantee we can have for such policy w.r.t. r E .

What if we ask that π E is at least ϵ -optimal under r (i.e., the requirement provided by ϵ -(non)compatible rewards)? Well, this is not satisfactory because optimal policies can be any, and because there might be other ϵ -optimal policies that can perform arbitrarily bad under r E .

All the three requirements described above on r do not provide guarantees that optimizing the considered reward r provides a policy with satisfactory performance w.r.t. the true r E . However, as mentioned in Section 4 and in Appendix C.1, expert demonstrations do not provide any information about the performance of policies other than π E under r E .

Remark C.1. If we want to be sure that an ϵ -optimal policy π for the learned reward r in p is if f p ϵ q -optimal for r E in p (for some function f ), then, clearly, we need that all the (at least) ϵ -optimal policies under the learned r have visitation distribution close to that of π E in 1-norm (see Appendix C.1).

We stress that many IRL algorithms for AL, like max-margin [2], learn a reward function just as a mere mathematical tool to compute a policy π which is close in 1-norm } d π ´ d π E } 1 to π E .

A remark about works on the feasible set. If we look at recent works about the feasible set [38, 31, 68], it might seem that these works are able to provide guarantees between r, r E under distance d all (see Section 3.1 of [68]), defined as:

If d all p r, r E q is small, then the performance of any policy in r , not just optimal policy or ϵ -optimal policy, is similar also under r E . In other words, if we use/optimize reward r , then we have the guarantee that the performance of the retrieved policy under r E is more or less the same as its performance in r . Therefore, clearly, rewards r with small distance to r E w.r.t. d all can be used for 'forward' RL . However, we have the following result:

Proposition C.1. Let M ' p S , A , H, d 0 , p q be a known MDP without reward, and let π E be a known expert's policy. Let r E the true unknown reward optimized by the expert to construct π E . Then, there does not exist a learning algorithm that receives in input the pair p M , π E q and outputs a single reward r such that d all p r, r E q ď ϵ w.p. 1 ´ δ .

Proof. The proof is trivial. Indeed, since the feasible set R p,π E contains an infinite amount of reward functions along with r E , and the learning algorithm cannot discriminate r E inside R p,π E , then the best it can do is to output an arbitrary reward function r P R p,π E . However, since R p,π E contains, for any reward r P R p,π E , at least another reward r 1 P R p,π E such that d all p r, r 1 q ' c is finite and equal to some positive constant c ą 0 , 13 then we can simply construct the problem instance with r E : ' r 1 to make the learning algorithm not able to output rewards that can be used for forward learning.

Nevertheless, [38, 31, 68] seem to provide sample efficient algorithms w.r.t. d all . 14 By looking at Proposition C.1, we realize that this is clearly a contradiction . What is the right interpretation?

The trick is that the algorithms proposed in works [68, 38, 31] are not able to output a single reward r which is close to r E w.r.t. d all , but, for any possible reward r E ' r E p V, A q parametrized 15 by some value and advantage functions V, A , they are able to output a reward r such that d all p r, r E p V, A qq is small. In other words, it is like if these works assume to know the V, A parametrization of the true reward r E . Simply put, these works are able to output a reward r that can be used for 'forward' RL just under such assumption. Otherwise those algorithms do not provide such guarantee.

Conclusions. To sum up, we conclude that, in general, an arbitrary reward function with small (non)compatibility can not be used for 'forward' learning (see Proposition C.1), because we cannot know given demonstrations alone whether the performances assigned by such reward to policies other than the expert policy are meaningful. In addition, for the same reason, we realize that also an arbitrary reward with zero (non)compatibility, i.e., an arbitrary reward in the feasible set, can not be used for 'forward' learning.

C.4 Comparing the (non)compatibility of various rewards

In Section 4, we said that rewards r with smaller values of C p,π E p r q are more compatible with π E in M ' p S , A , H, d 0 , p q . However, one might provide the following 'counter-example':

Example C.1 (Question by Reviewer KyLX) . Let r 1 , r 2 be two rewards such that r 2 h p s, a q ' 2 r 1 h p s, a q ě 0 for all p s, a, h q P S ˆ A ˆ J H K . Then, clearly, C p,π E p r 2 q ' 2 C p,π E p r 1 q . Therefore, based on Section 4, we say that reward r 1 is more compatible than r 2 w.r.t. π E in M . However, since r 2 is just r 1 re-scaled by a constant, the two MDPs M Yt r 1 u and M Yt r 2 u should be 'equivalent', thus, r 1 and r 2 should be, intuitively, equally compatible with π E .

However, Example C.1 misleads the correct interpretation of the notion of reward function in MDPs, and in particular about the scale of the rewards. Let us explain better our point.

The MDP is a model, i.e., a simplified representation of reality, which is commonly applied to 2 different kinds of real-world scenarios: p i q problems in which the agent (learner in RL or expert in IRL) actually receives some kind of scalar feedback from the environment, which can be modelled as a reward function; p ii q problems in which the agent does not receive a feedback from the environment,

but its objective, i.e., its structure of preferences among state-action trajectories (which trajectories are better than others), satisfies some axioms that permit to represent it through a scalar reward [52, 9] (this is referred to as the Reward Hypothesis in literature [58]).

There is an enormous difference between scenario p i q and scenario p ii q . In p i q the notion of ϵ -optimal policy is well-defined for any fixed ϵ ą 0 , because the reward function is given and, thus, fixed. Instead, in p ii q , the notion of reward function is a mere mathematical artifact used to represent preferences among trajectories, whose existence is guaranteed by a set of assumptions/axioms [52, 9, 58]. As Example C.1 shows, positive affine transformations of the reward do not affect the structure of preferences represented (see [52] or Section 16.2 of [50] or [30]). Therefore, in p ii q , the notion of ϵ -optimal policy is not well-defined, because rescaling a reward function r to kr changes the suboptimality of some policy π from ϵ to kϵ . In other words, for fixed ϵ ą 0 , any policy can be made ϵ -optimal by simply rescaling a reward r to kr for some small enough k ą 0 .

In IRL, this issue is even more influential because, although we are in setting p i q , we have no idea on the scale of the true reward function. For this reason, our solution is to attach to any reward r a notion of compatibility C p r q which implicitly contains information about the scale of the reward r . Compatibilities of different rewards (e.g., r 1 and r 2 in Example C.1) cannot be compared unless the rewards have the same scale (e.g., r 1 and r 2 have different scales, thus their compatibilities shall not be compared).

It should be observed that in Appendix C.2 we discuss a notion of compatibility independent of the scale of the reward. However, we show that it suffers from major drawbacks that make the notion of compatibility introduced in the main paper (Definition 4.1) more suitable for the IRL problem.

In conclusion, to settle Example C.1, rewards r 1 and r 2 should not have the same compatibility, because they have different scales, and the notion of compatibility (i.e., suboptimality) is strictly connected to the scale of the reward. To carry out a fair comparison of compatibilities, one should rescale the compatibility of each reward based on the scale of the reward.

D Missing Proofs and Additional Results for Section 5

This appendix is organized as follows. First, we report the full pseudo-code of CATY-IRL . Then, we provide the proof of Theorem 5.1 in Appendix D.2.

D.1 Algorithm

In this section, we provide the extended version of CATY-IRL containing the explicit conditions under which we shall instantiate one BPI/RFE algorithm instead of another.

Algorithm 2: CATY-IRL - exploration

Algorithm 3: CATY-IRL - classification

D.2 Proof of Theorem 5.1

Notice that, according to Definition 4.3, an algorithm is p ϵ, δ q -PAC for IRL if it computes an estimate ϵ -close to the true (non)compatibility w.h.p.. Such definition does not depend on the specific strategy adopted by the algorithm to actually classify the input reward using the computed estimate of (non)compatibility.

Before diving into the proof of Theorem 5.1, we make the following considerations.

In the common tabular MDPs setting without additional structure, we know that the expected utility J π p r ; p q of policy π under reward r in environment with dynamics p can computed as:

where d p,π h is the occupancy measure of policy π in p . It should be remarked that both r h and d p,π h have SA components for all h P J H K .

In tabular MDPs with linear reward functions and in Linear MDPs, the reward function is linear in some feature map ϕ , i.e.:

where } ϕ p s, a q} 2 ď 1 for all p s, a q P S ˆ A and max h } θ h } 2 ď ? d . Using this decomposition, we can rewrite the expected utility J π p r ; p q as:

where we have defined the feature expectations t ψ p,π h u h P J H K as ψ p,π h : ' E p s,a q' d p,π h ϕ p s, a q . Observe that vector ψ p,π h has d components instead of the SA components of each d p,π h vector.

Since in our setting the IRL algorithm receives in input the reward function (or its parameter θ P R d ), to estimate the expected utility J π p r ; p q we must estimate the visit distributions t d p,π h u h or the feature expectations t ψ p,π h u h . However, because of the different dimensionalities of such quantities ( SA versus d ), the estimates might require different amounts of samples.

Theorem 5.1 (Sample Complexity of CATY-IRL ) . Let ϵ, δ P p 0 , 1 q . Then CATY-IRL is p ϵ, δ q -PAC for IRL with a sample complexity upper bounded by:

Tabular MDPs: τ E ď r O ´ H 3 SA ϵ 2 log 1 δ ¯ , τ ď r O ´ H 3 SA ϵ 2 ´ N ` log 1 δ ¯¯ , Tabular MDPs with linear rewards: τ E ď r O ´ H 3 d ϵ 2 log 1 δ ¯ , τ ď r O ´ H 3 SA ϵ 2 ´ N ` log 1 δ ¯¯ , Linear MDPs: τ E ď r O ´ H 3 d ϵ 2 log 1 δ ¯ , τ ď r O ´ H 5 d ϵ 2 ´ d ` log 1 δ ¯¯ , where N ' 0 if | R | ' Θ p 1 q , and N ' S otherwise.

Proof. To prove the theorem, we aim to find a bound to the number of samples τ E such that the estimate p J E p r q « J π E p r ; p q is ϵ { 2 -correct with probability at least 1 ´ δ { 2 . Next, similarly, we aim to bound τ so that p J ˚ p r q « J ˚ p r ; p q is ϵ { 2 -correct with probability at least 1 ´ δ { 2 . Then, the conclusion follows after performing a union bound and observing that, for any r P R :

Estimating p J E p r q « J π E p r ; p q To estimate J π E p r ; p q , CATY-IRL simply computes the empirical estimate of t d p,π E h u in case of tabular MDPs, and the empirical estimate of t ψ p,π E h u in case of tabular MDPs with linear rewards and Linear MDPs. Notice that by empirical estimates we mean:

and:

Concerning the estimate of the visit distribution p d E , we can use the result of Lemma 6 in [53] (we are working with bounded rewards), to obtain that:

Solving w.r.t. τ E we get the bound on τ E .

In a completely analogous manner, we can bound the feature expectations as:

Again, solving w.r.t. τ E we get the bound on τ E .

r RF-Express [37], and τ ď r O ´ H 5 d ϵ 2 ` d ` log 1 δ ˘ ¯ for the algorithm in [62] (we omit linear terms in 1 { ϵ ), they compute an estimate p « p of the true transition model such that:

Clearly, if such property holds, then by computing the performance of the policy p π outputted by the RFE algorithm we are able to obtain an ϵ { 2 -correct estimate of J ˚ p r ; p q . 16

Concerning the case in which | R | is a finite small constant, for tabular and tabular with linear rewards MDPs, we can simply use algorithm BPI-UCBVI of [37] as sub-routine, and run it as many times as there are rewards in R . When | R | is a small constant, we can proceed with a union bound over R :

This allows us to formally distinguish between small and large | R | based on the following inequality:

E Missing Proofs and Additional Results for Section 6.1

This appendix is organized as follows. First, in Appendix E.1, we introduce two problems that share similarities with RFE and IRL, and we characterize the main differences among them. In addition, we enunciate a lower bound to the sample complexity that is common to some of these 4 problems. Next, in Appendix E.2, we provide the missing proofs.

E.1 Four Problems

The 4 problems that we consider here are Reward-Free Exploration (RFE), Inverse Reinforcement Learning (IRL), Matching Performance (MP), and Imitation Learning from Demonstrations alone (ILfO). MP represents a novel generalization of RFE, while ILfO, introduced in [34], represents an exemplification of MP. Before enunciating the minimax lower bound, it is important to formally define each of these problems, as well as what we mean by learning in each problem.

E.1.1 Definition of the Problems

In all the 4 problems, the learner is placed into an unknown MDP without reward M ' p S , A , H, d 0 , p q , i.e., an environment whose dynamics p d 0 , p q is unknown to the learner. For simplicity, w.l.o.g., we assume that there is a single initial state s 0 : ' d 0 . In each problem, the learner can explore the environment at will to collect samples about the dynamics p , whose knowledge improves the performance of the agent at solving the task. However, at exploration phase, the learner does not

know which is the specific task it has to solve. It just knows that the specific task belongs to a given set of tasks T (e.g., set of reward functions). The agent can use the knowledge of T to engage in a more efficient task-driven exploration. For any ϵ, δ P p 0 , 1 q , the goal of the agent is to being able to ouputting, for any task in T a quantity o (e.g., a policy) that solves that specific task in an ϵ -correct manner with probability at least 1 ´ δ . The ultimate goal of exploration is to collect the least number of samples that permits p ϵ, δ q -correctness for all the tasks in T .

Now, let us see what the quantities T and o represent in each of the 4 problems. In Table 1, we provide a sum up of the various definitions.

Reward-Free Exploration (RFE). In RFE, the learner receives a set of reward functions T ' R Ď R in input, and the goal is to exploit the information about p collected at exploration phase to output, for any reward r P R , an ϵ -optimal policy o ' p π r w.p. 1 ´ δ . When T ' t r u is a singleton, the RFE problem is commonly termed the BPI problem. In symbols, any RFE algorithm must guarantee that:

where p π r is the estimate of the algorithm for reward r . Inverse Reinforcement Learning (IRL). In IRL, the learner receives in input an occupancy measure 17 t d p,π E h u h P J H K and a set of reward functions R Ď R : T ' p d p,π E , R q , but it does not know which specific reward it will have to classify. Under the assumption that the occupancy measure d p,π E is known, 18 the problem reduces to exploiting the information about p collected at exploration phase to output, for any reward r P R , an ϵ -correct estimate o ' p J p r q of the optimal utility J ˚ p r q w.p. 1 ´ δ . In symbols, under these conditions, any IRL algorithm must guarantee that:

Matching Performance (MP). In MP, the learner receives in input a set of reward functions R Ď R and a measure of performance for each of them J : R Ñ R : T ' p J, R q . For any r P R , the utility J p r q represents a performance measure for which we aim to find the policy that achieves closest performance. Thus, in MP, the goal is to exploit the information about p collected at exploration phase to output, for any reward r P R , a policy o ' p π r such that, if we denote the policy with performance closest to J p r q by π r P arg min π | J π p r q ´ J p r q| , then the utility of policy p π r is ϵ -close to the utility of policy π r w.p. 1 ´ δ . In symbols, any MP algorithm must guarantee that:

where p J p r q is the estimate of the algorithm for reward r .

where π r P arg min π | J π p r q ´ J p r q| , and p π r is the estimate of the algorithm for reward r . Imitation Learning from Demonstrations alone (ILfO). In ILfO, the learner receives in input a set of state-only reward functions R Ă R and a state-only occupancy measure t d h u h P J H K : T ' p d, R q . Under the assumption that d does not leak any information about the true transition model p , the goal is to exploit the information about p collected at exploration phase to output, for any reward r P R , a policy o ' p π r such that, if we denote the policy with performance closest to J p r q : ' ř h P J H K x r h , d h y by π r P arg min π | J π p r q ´ J p r q| , then the utility of policy p π r is ϵ -close to the utility of policy π r w.p. 1 ´ δ . Simply put, ILfO, as defined in this manner, exemplifies the MP setting by providing a functional form to J : R Ñ R as an inner product between a certain state-only occupancy measure and the input reward. It should be remarked that the assumption made for ILfO is mild, because it is

satisfied by the setting in which the expert and the learner have the same state space but different action spaces (or different dynamics). Indeed, in such case, the visit distribution d of the expert would not leak any information about p . In symbols, any ILfO algorithm must guarantee that:

where π r P arg min π | J π p r q ´ J p r q| and J p r q : ' ř h P J H K x r h , d h y , and p π r is the estimate of the algorithm for reward r .

E.1.2 Lower Bound

We now present a minimax lower bound rate that is common to RFE, IRL, and MP. We report here the lower bounds presented in Section 6.1.

Theorem 6.1 (IRL Classification - Lower Bound) . Let A be an p ϵ, δ q -PAC algorithm for the IRL classification in tabular MDPs. Let τ be the number of exploration episodes. Then, there exists an IRL classification instance such that:

Proof. The proof is similar to that of [38]. We split the proof in two parts, by considering two classes of difficult problem instances in Lemma E.2 and Lemma E.3. Next, we combine the two bounds through max t a, b u ě p a ` b q{ 2 for all a, b ě 0 . For the proof, we will assume that the expert visit distribution is known. The obtained bound represents a lower bound to the more general setting in which it is unknown.

Theorem E.1 (RFE - Refined Lower Bound) . Let A be an p ϵ, δ q -PAC algorithm for RFE in tabular MDPs. Let τ be the number of exploration episodes. Then, there exists an RFE instance such that:

Proof. The proof of this result is analogous to that of Theorem 6.1, and it employs Lemma E.2 and Lemma E.3.

Some observations are in order. First, since MP is a more general setting than RFE, then this lower bound is a lower bound for MP too. However, this is not guaranteed for ILfO. We observe that, while for RFE and IRL the bound is tight, for MP we cannot say so because we do not have the upper bound. Notice that, in case the expert state-only distribution d was unknown at exploration phase, and revealed afterwards, then the lower bound of Theorem 6.1 holds for ILfO too, because we might a posteriori reveal the state-only distribution d of the optimal policy, and thus, in such manner, ILfO would be reduced to RFE.

E.2 Missing proofs

Lemma E.2. Let IRL and RFE be the learning problems defined as in Appendix E.1. Then, for each problem, any p ϵ, δ q -PAC algorithm must collect at least the following number of exploration episodes:

Proof. Observe that the proof for RFE is present in [12]. Thus, we have to prove just the result for IRL. For doing so, we will use both the results of [12] and [38]. Notice that for the sake of this proof we consider R ' t r u , that will reduce our problem to simple RL as, in order to compute the function C p,π E p r q , we just need to compute J ˚ p r ; p q , being J π E p r ; p q known from the availability of d p,π E and r .

s

w

s

root

s

E

action =

π

E

p

s

w

q

1

s

1

s

2

s

3

s

4

action

‰

a

w

action =

a

w

s

g

s

b

r

h

p

s

g

, a

q '

1

t

h

ě

H

`

d

`

1

u

r

h

p

s

b

, a

q '

0

1

2

1

2

1

2

`

ϵ

1

1

2

´

ϵ

1

1

1

Instances Description The hard instances considered are exactly the same as [12], and are reported in Figure 4 for simplicity. The only difference is the presence of state s E , to which the expert's policy π E brings, which is absorbing. Such state is needed to make the knowledge of the expert's visit distribution d p,π E useless at inferring information about the transition model in other parts of the state-action space. Based on [12], we describe such hard instances. Similarly to [12], we assume that S ě 7 , A ě 2 , and there exists an integer d such that S ' 4 `p A d ´ 1 q{p A ´ 1 q , and we assume that H ě 3 d . Note that [12] show how to relax the assumption on the existence of d .

There are the initial state s w , from which the agent starts, and states s g , s b , respectively, the 'good' and 'bad' states which are absorbing. Moreover, there is state s E , which is reached by the expert, and is absorbing. The remaining S ´ 4 states are arranged in a full A -ary tree of depth d ´ 1 with root s root. We denote by H ď H ´ d a certain integer parameter, and by L : ' t s 1 , s 2 , . . . , s L u the set of leaves of the tree. We define I : ' t 1 ` d, . . . , H ` d u ˆ L ˆ A . For any ı P I , we define and MDP M ı as follows. In any state of the tree, i.e., in states S zt s w , s g , s b , s E u , the transitions are deterministic, and the a -th action of a state brings to the a -th child of that node.

The transitions from s w are given by

In other words, action a w allows the agent to remain in the initial state s w up to stage H . After stage H , the agent is forced to leave s w and to traverse the tree down to the leaves. Action a E ' π E 1 p s w q is the only action that brings to state s E , which is absorbing. The transitions from any leaf s i P L are given, as in [12], by:

where ∆ p h ˚ ,ℓ ˚ ,a ˚ q p h, s i , a q : ' 1 tp h, s i , a q ' p h ˚ , s ℓ ˚ , a ˚ qu ¨ ϵ 1 , for some ϵ 1 P r 0 , 1 { 2 s . For this reason, there exists a (single) leaf ℓ ˚ where the agent can choose an action a ˚ at stage h ˚ to increase its probability of arriving to the good state s g , which provides higher reward. We define states s g and s b to be absorbing, i.e., they satisfy p h p s b | s b , a q : ' p h p s g | s g , a q : ' 1 for any action a . The reward function is state-only and is defined as

so that even though the agent decides to stay at s w until stage H , it does not lose any reward. Observe that state s E does not provide any reward, so that to estimate the (non)compatibility, any algorithm must provide a good estimate of the optimal performance.

Finally, we define a reference MDP M 0 which is an MDP of the above type but for which ∆ 0 p h, s i , a q : ' 0 for all p h, s i , a q . For certain ϵ 1 and H to choose, we define the class M to be the set M : ' t M 0 u Y t M ι u ι P I .

Distance between problems We will prove the lower bound for instance M 0 . Observe that, in M 0 , the optimal utility is:

because there is no triple with additional bias towards s g . Instead, for any other M ı P M , the optimal utility is:

Therefore, if we choose ϵ 1 : ' 2 ϵ {p H ´ H ´ d q , we have that, for any ı P I :

p Identifying the underlying problem Following [38], let us consider a generic p ϵ, δ q -correct algorithm A that outputs the estimated optimal utility J . Then, for all ı P I , we have:

ˇ ˇ J ˚ 0 ´ J ˚ ı ˇ ˇ ' 2 ϵ. Thus, in particular, for any estimate p J P R we necessarily have | J ˚ 0 ´ p J | ď ϵ ùñ | J ˚ ı ´ p J | ą ϵ , and vice versa, i.e., we cannot provide an estimate J that is ϵ -close to both J ˚ 0 and J ˚ ı .

For every ı P I , we define the identification function Ψ ı as the index of the problem 'recognized' by algorithm A . In symbols:

In words, given estimate p J returned by algorithm A , the identification function Ψ ı returns the problem between M 0 and M ı whose optimal utility is closest to the estimate p J . For what we have seen in the previous paragraph, problems M 0 and M ı lie at a distance of at least 2 ϵ for all ı P I . Therefore, for ȷ P t 0 , ı u , we have the following inclusion of events:

s

w

s

root

s

E

. . .

. . .

s

1

s

S

s

1

1

s

1

2

s

1

S

. . .

action=

a

w

action=

π

E

p

s

w

q

action

‰

a

w

a

j

w.p.

1

`

ϵ

1

v

p

s S

,a j ,h

q

1

S

a

j

w.p.

1

`

ϵ

1

v

p

s S

,a j ,h

q

2

S

a

j

w.p.

1

`

ϵ

1

v

p

s S

,a j ,h

q

S

S

a

j

w.p.

1

`

ϵ

1

v

p

s

1

,a j ,h

q

1

S

a

j

w.p.

1

`

ϵ

1

v

p

s

1

,a j ,h

q

2

S

a

j

w.p.

1

`

ϵ

1

v

p

s

1

,a j ,h

q

S

S

Thanks to this fact, we can continue lower bounding the probability as:

where at (1) we have lower bounded the maximum with the average, i.e., max t a, b u ě p a ` b q{ 2 for all a, b ě 0 , and at (2) we have applied the Bretagnolle-Huber's inequality [38].

KL-divergence computation The proof can be concluded by upper bounding the KL divergence KL p P M 0 , A , P M ı , A q as in the proof of Theorem 7 in [12], and then summing over all the Θ p SAH q instances to retrieve the result.

Lemma E.3. Let IRL and RFE be the learning problems defined as in Appendix E.1. For each problem, if the set of reward functions R in input is R ' R , then any p ϵ, δ q -PAC algorithm must collect at least the following number of exploration episodes:

Proof. Instances description The hard instances that we use for the proof of this lemma are obtained by combining the hard instances in Lemma E.2 (i.e., the hard instances of [12]), with those in [38]. Specifically, this construction is based on the intuition described in [21] that, if we want to increase the sample complexity, we have to learn transitions also to Θ p S q states, and not just from Θ p S q states. Observe the presence of state s E (only for IRL), which plays the same role as in the proof of Lemma E.2. Any action in such state receives always reward ´ 1 , thus it is meaningless for the estimate of the (non)compatibility, which reduces to the estimation of the optimal performance. In this manner, the expert distribution d p,π E does not provide additional information about the transition model of

other portion of the state-action space. Therefore, in the following, we will present the lower bound construction as if such state did not exist.

The hard instances are reported in Figure 5. Notice that they are exactly the same instances as those presented in the proof of Lemma E.2, with the difference that, from the S leaves (differently from earlier, we now denote the number of leaves through S instead of L ), we do not reach just two states s g , s b , but we reach Θ p S q absorbing states, i.e., s 1 1 , s 1 2 , . . . , s 1 S . The transitions from the leaves to such states is the same as in [38], and we report a description below.

Let us introduce the set I : ' t s 1 , . . . , s S u ˆ A ˆt 1 ` d, . . . , H ` d u . Let ı : ' p s 1 , a 1 , 1 ` d q P I be a specific triple of set I , and denote I : ' I zt ı u . Let us also introduce set V : ' t v P t´ 1 , 1 u S : ř S j ' 1 v j ' 0 u . Thanks to Lemma E.6 of [38] (that we report in Lemma E.4 for simplicity), we know that there exists a subset V Ď V (of transition models) with cardinality at least 2 S { 5 such that, for every pair v, w P V with v ‰ w , we have that } v ´ w } 1 ě S { 16 . In other words, we know that there exists a S { 16 -packing of V with cardinality at least 2 S { 5 .

Following [38], we denote by v ' p v ı q ı P I P V I the generic vector of V I . Now, for any v P V I , for any triple ȷ P I , and for some parameter ϵ 1 P r 0 , 1 { 2 s to choose, we construct problem instance M v ,ȷ as follows.

First of all, we define the transition model at triple ı as:

where observe that we use notation ı ' p s ı , a ı , h ı q P I to denote triples in I . Instead, for the generic triple ı P I (including triple ȷ ), the probability distribution of the next state is given by:

where v ı i represents the i -th component of the ı -th vector in v . In words, the i -th component of vector v ı P V creates a bias of ϵ 1 { S towards the next state s 1 i for all i P J S K . Since v ı P V , then p h ı p¨| s ı , a ı q P ∆ J S K for all ı P I .

We consider non-stationary reward functions. Specifically, all the rewards r P R that we consider assign reward 1 to both triples ı and ȷ , i.e., r h ı p s ı , a ı q ' 1 and r h ȷ p s ȷ , a ȷ q ' 1 . Next, for any other triple p s, a, h q P S ˆ A ˆ J H K with state different from s 1 1 , s 1 2 , . . . , s 1 S , we assign reward 0. For states s 1 1 , s 1 2 , . . . , s 1 S , we consider state-only rewards whose value is always 0 in stages r 1 , H ` d s , and whose value is stationary and arbitrary afterwards. Intuitively, as in [12], forcing the reward to be 0 up h ' H ` d guarantees that we cannot obtain a higher expected return J by reaching the leaves states earlier (i.e., by exiting from s w before H ).

Given the definition above, we construct the class of instances M : ' t M v ,ı : ı P I , v P V I u . Moreover, we will use the notation M v ı Ð w,ȷ to denote the instance in which we replace the ı component of v , i.e., v ı , with w P V and M v ı Ð 0 ,ȷ the instance in which we replace the ı component of v , i.e., v ı , with the zero vector. Since we will always use this notation when substituting triple ȷ , i.e., we always use this notation in situations as M v ȷ Ð w,ȷ , then we omit the second parameter, and write just M v ı Ð w : ' M v ı Ð w,ȷ .

Distance between problems Consider an arbitrary problem instance M v ,ı P M , for certain ı P I and v P V I . Let r P R be an arbitrary reward function that satisfies the constraints described earlier. Let π ı P Π be the deterministic policy that brings to triple ı . Then, its expected return is:

where r i : ' r H ` d ` 1 p s 1 i q for all i P J S K . Let policy π ı P Π be the deterministic policy that brings to triple ı . Then, its expected return is:

Finally, let policy π ȷ P Π be the deterministic policy that brings to any other triple ȷ P I zt ı u . Then, its expected return is:

It should be remarked that p v, r q ' ř i P J S K v i r i P r´ S, S s for any r P R and v P V , therefore, as long as:

then any policy π ȷ is cannot be ϵ -optimal in problem M v ,ı , in which, thus, the optimal policy shall be searched for between π ı and π ı .

Now, consider an arbitrary pair v, w P V such that v ‰ w , and an arbitrary triple ı P I and vector P V I . We now compare problem instances M v ı Ð v and M v ı Ð w . Among all possible reward functions that satisfy the definition provided in the construction of the hard instances, we find reward r 1 such that, in every component i P J S K , satisfies:

.

For what we have seen before about class V , we know that } v ´ w } 1 ' ř i P J S K | v i ´ w i | ě S { 16 , thus, since v, w P V , i.e., their components belong to t´ 1 , ` 1 u , we know that there are at least S { 32 components of v, w that differ from each other. By using reward r 1 , we have that:

As a consequence, the expected returns of policies π ı and π ı in problems M v ı Ð v and M v ı Ð w are:

from which we infer that:

Now, let us choose ϵ 1 ą 64 ϵ {p H ´ H ´ d q . To satisfy also the constraint in Equation (5), we can roughly assume ϵ ă 1 { 256 and set ϵ 1 ' 65 ϵ {p H ´ H ´ d q . Thanks to this choice, observe that:

In words, policy π ı is optimal in problem M v ı Ð v , and policy π ı is worse than 2 ϵ -suboptimal in such problem. In addition, observe that policy π ı is optimal in problem M v ı Ð w , and policy π ı is worse than 2 ϵ -suboptimal in such problem. We stress that any stochastic policy in-between π ı and π ı cannot be ϵ -optimal for both problems.

To sum up, for the choice of ϵ 1 made earlier, for arbitrary pairs of problems M v ı Ð v and M v ı Ð w , we have seen that there exist rewards in R for which a policy ϵ -optimal for problem M ı,v is not ϵ -optimal for problem M ı,w , and vice versa.

Identifying the underlying problem: RFE. We consider first RFE, and then IRL.

Let us consider an p ϵ, δ q -correct algorithm A for RFE, that outputs, for any reward function r P R , a policy p π r . For simplicity, we consider as output of Algorithm A a function p π : R Ñ Π , that takes in input a reward and outputs a policy.

For any ı P I and v P V I , we can lower bound the error probability as:

where at (1) we have lower bounded by replacing all possible RFE problem instances with problem instances in M , and at (2) we have lower bounded by replacing all instances in M with just instances t M v ı Ð w : w P V u for the fixed triple ı and vector v .

For every ı P I and v P V I , we define the identification function Ψ ı, v as the index of the problem w P V 'recognized' by algorithm A . In symbols:

In words, given estimate p π : R Ñ Π returned by algorithm A , the identification function Ψ ı, v returns the problem in t M v ı Ð w : w P V u whose solution π : R Ñ Π is closest to the estimate p π . For what we have seen in the previous paragraph, for any v, w P V with v ‰ w , for any fixed ı P I and v P V I , there exists a reward function r 1 P R such that no policy can have expected utility ϵ -close to the optimal expected utility of both problems M v ı Ð v and M v ı Ð w . Therefore, for w P V , we have the following inclusion of events:

We can continue to lower bound the probability as:

where at (3) we have lower bounded the maximum over V with the average, and at (4) we have applied, similary to [38], the Fano's inequality, reported in Theorem E.5 for simplicity.

Identifying the underlying problem: IRL. For IRL, it is possible to carry out a similar derivation. However, we remark that, now, the error is measured based on the expected utilities, and not on the policies.

Let us consider an p ϵ, δ q -correct algorithm A for IRL, that outputs, for any reward function r P R , a utility p J r . For simplicity, we consider as output of Algorithm A a function p J : R Ñ R , that takes in input a reward and outputs a utility.

For any ı P I and v P V I , we can lower bound the error probability as:

For any ı P I and v P V I , we define an identification function Ψ ı, v as:

and by a reasoning analogous to that for RFE, we can continue to lower bounding as:

which represents the same lower bound obtained also for RFE.

KL-divergence computation The following derivation is analogous to that of [38]. To bound the KL-divergence term, for any ı P I , we can write:

where at (1) we have applied Lemma E.7, and at (2) we have applied Lemma E.6 (having observed that the transition models differ in ı and defined N τ h ı p s ı , a ı q ' ř τ t ' 1 1 tp s t , a t , h t q ' p s ı , a ı , h ı qu ). Plugging into Equation (6), we get:

Notice that, since | V | ' Θ p e S q and ϵ 1 ' Θ p ϵ { H q , then this bound is in the order of Ω p H 2 S ϵ 2 q . To get the additional Ω p SAH q dependence, we can make the same observation as in [38], i.e., that ince the derivation is carried out for every ı P I and v P V I , we can perform the summation over ı and the average over v . By noticing that we get a guarantee on a mean under the uniform distribution of the instances of the sample complexity, we realize that there must exist one v hard P V for which it holds the desired Ω ´ H 3 S 2 A ϵ 2 ¯ dependency.

E.2.1 Technical Tools

We report here some results from other works. The notation adopted is the same as the original works.

Lemma E.4 (Lemma E.6 of [38]) . Let V ' t v P t´ 1 , 1 u D : ř D j ' 1 v j ' 0 u . Then, the D 16 -packing number of V w.r.t. the metric d p v, v 1 q ' ř D j ' 1 | v j ´ v 1 j | is lower bounded by 2 D 5 .

Theorem E.5. ( Theorem E.2 of [38] ) Let P 0 , P 1 , . . . , P M be probability measures on the same measurable space p Ω , F q , and let A 1 , . . . , A M P F be a partition of Ω . Then,

where A c ' Ω z A is the complement of A .

Lemma E.6 (Lemma E.4 of [38]) . Let ϵ P r 0 , 1 { 2 s and v P t´ ϵ, ϵ u D such that ř d i ' 1 v i ' 0 . Consider the two categorical distributions P ' ` 1 D , 1 D , . . . , 1 D ˘ and P ' ` 1 ` v 1 D , 1 ` v 2 D , . . . , 1 ` v D D ˘ . Then, it holds that:

Lemma E.7 (Lemma 5 of [12]) . Let M and M 1 be two MDPs that are identical except for their transition probabilities, denoted by p h and p 1 h , respectively. Assume that we have @p sa q , p h p¨| s, a q ! p 1 h p¨| s, a q . Then, for any stopping time τ with respect to p F t H q t ě 1 that satisfies P M τ ă 8 ' 1 , where N τ h,s,a : ' ř τ t ' 1 1 tp S t h , A t h q ' p s, a qu and I τ H : Ω Ñ Ť t ě 1 I t H : ω ÞÑ I τ p ω q H p ω q is the random vector representing the history up to episode τ .

F A Use Case for Objective-Free Exploration (OFE)

Consider the following setting. You are given a certain MDP without reward M ' p S , A , H, d 0 , p q , in which you do not know neither d 0 nor p . Your job is to explore the environment to collect samples that allow you to construct estimates p d 0 « d 0 and p p « p , that will be subsequently used to perform a task in a given class F in an p ϵ, δ q -correct manner. Of course the number of samples should be as small as possible. How do you explore? It depends on which problems are contained in class F .

A use case for OFE is the following.

Example F.1. Assume that we are given a single fixed environment (for instance, a warehouse), in which there are many tasks to do (e.g., labelling objects, putting stuff on the shelves, bringing products from one side to the other), and assume (it is reasonable) that it is desirable to have one robot for each task. To teach these robots how to behave, we decide to use RL. Since all the robots work in the same environment (warehouse), then the (unknown) transition model is the same. For this reason, an efficient exploration (potentially through RFE) is meaningful. However, we realize that some tasks are difficult to design (i.e., the rewards of such tasks). For these tasks, we prefer to use a human expert to exhibit demonstrations, and then use ReL (in particular, IRL), to learn the reward, that will be subsequently used for AL. To perform IRL nicely, the samples collected at the beginning shall be used. To sum up, we might be interested in performing multiple RL and IRL tasks in the same unknown MDP, and, for efficiency reasons, our exploration of the environment has to be performed only once (before) being given the tasks to solve.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The Reviewer can find the list of contributions in Section 1, and the related content in the rest of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The Reviewer can find a 'Limitations' paragraph in Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The Reviewer can find the full set of complete proofs in the supplemental material.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper does not harm human subjects or participants, and there are no data-related concerns. In addition, we believe that there are no noteworthy potential harmful consequences of our research.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: This paper represents foundational research, and it is not tied to particular applications. In addition, we believe that there is no direct path to any negative applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies IRL in linear MDPs. The authors first demonstrate that the feasible reward set cannot be efficiently learned in large state and action spaces. To address this challenge, they propose a new IRL framework called reward compatibility, where the goal is to learn a classifier that determines whether the expert is approximately optimal for a given reward. Using CATY-IRL, they also introduce a new sample-efficient algorithm for this task, which first explores the MDP via reward-free exploration (RFE) and then evaluates whether a given reward is compatible with the expert's demonstrations. Furthermore, for the tabular setting, they provide a tight minimax lower bound and show that a similar bound also improves on existing lower bounds for RFE. Lastly, the authors propose a novel problem setting of objective-free exploration, which generalizes RFE to arbitrary tasks of interest.\n\nSTRENGTHS:\nThe paper provides a good and original contribution to the challenging problem of IRL in large state-action spaces. In my opinion, its strengths are:\n1. Technical quality: The authors use elegant notation and carefully introduce all symbols. Moreover, clear proofs for all results are provided in the appendix.\n2. Sample complexity bounds: The sample complexity of CATY-IRL is thoroughly analyzed for tabular MDPs, tabular MDPs with linear rewards, and linear MDPs. Additionally, a tight minimax lower bound for the tabular setting is given. This lower bound is particularly appreciated, as lower bounds are still rare in the IRL literature.\n3. Originality: After proving that identifying the feasible reward set is intractable, the authors introduce the novel problem setting of checking for reward compatibility and provide a sample efficient algorithm for it.\n\nWEAKNESSES:\n1. Motivation: While original, the motivation for learning a reward compatibility classifier is not entirely clear to me. I would expect the authors to better explain its usefulness and potential applications.\n2. Writing: The paper is not easy to understand on the first read. There are many propositions and theorems, but more emphasis on motivation and intuition building would be beneficial. For instance, a simple example for Proposition 3.1 could help build geometric intuition.\n3. Related work: The authors introduce reward non-compatibility as a novel metric for IRL. However, minimizing the suboptimality of the expert is a core idea behind many imitation learning and IRL algorithms, such as GAIL and others. Of course, in the unregularized setting minimizing the suboptimality would lead to trivial solutions. However, I would expect a discussion about similarities and differences to min-max IL/IRL in the main part of the paper.\n4. Computational complexity: The authors claim that their algorithm is also computationally efficient, but I couldn't find a discussion about this.\n5. This is a minor point, but in line 103 you use but don't introduce the notation $Y^X$ for the set of functions mapping from $X$ to $Y$. Moreover, slightly inconsistently the set of functions from $X$ to $\\Delta^Y$ is denoted as $\\Delta_X^Y$. Why don't you change the notation to $\\Delta_Y^X$?\n\nQUESTIONS:\nHow would you apply your algorithm to a real world IRL problem? What is the computational complexity of the classification step?\n\nLIMITATIONS:\nI think the practical limitations should be discussed more extensively.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper finds that the feasible reward set cannot be efficiently learned even under linear MDPs. Therefore, the paper proposes a new notion called \"reward compatibility\" that generalizes the notion of \"feasible set\" and thereby casts IRL as a classification problem. The paper proposes an algorithm to solve this new classification problem and theoretically show that the sample complexity of the proposed algorithms is independent of state cardinality for linear MDPs.\n\nSTRENGTHS:\n1. This paper proposes a new notion called \"reward compatibility\" and novelly formulates IRL as a classification problem based on the notion.\n\n2. This paper is theoretically solid, which is also the biggest strength of the paper.\n\nWEAKNESSES:\n1. The paper title is a little bit exaggerating. The title is \"scale IRL to large state spaces\", while what the paper does is to scale one specific kind of IRL, i.e., learning the feasible reward set, to large state spaces. In fact, many other kinds of IRL algorithms can work quite efficiently in continuous state spaces without the linear MDP assumption. For example, maximum likelihood IRL [1] is sample efficient where only one sample is needed for each reward update. I think what the authors mean here is that learning the feasible reward set is difficult in large state space, however, learning feasible reward set is only a kind of IRL, but does not represent all IRL methods. I suggest that the authors change the general terminology \"IRL\" to some more specific terminologies, given that some other kinds of IRL methods can already scale to large state spaces.\n\n2. The paper uses the terminology \"online IRL\" to represent IRL that needs to interact with the environment. The paper also adds a footnote to explain this notion. I know that this is to contrast offline IRL. However, it is still confusing because online IRL is already defined in literature [2,3,4], i.e., the IRL setting where the demonstrated trajectories are revealed sequentially. Therefore, I highly suggest the authors to use another terminology to avoid confusion.\n\n3. In example 4.1, the authors mention that the reward function with smaller $\\bar{C}(r)$ is more compatible. This can be questionable. For example, suppose $r_2=2 r_1$, then $\\bar{C}(r_2)>\\bar{C}(r_1)$ if they are both positive. However, can we say that $r_1$ is more compatible than $r_2$? The MDPs are equivalent if we multiply the reward by a constant. Then intuitively, $r_1$ and $r_2$ should be same compatible right?\n\n4. Lack of empirical evaluation.\n\n[1] Maximum-likelihood inverse reinforcement learning with finite-time guarantees\n\n[2] First-person activity forecasting with online inverse reinforcement learning\n\n[3] Online inverse reinforcement learning under occlusion\n\n[4] Learning multi-agent behaviors from distributed and streaming demonstrations\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nThe limitation is discussed in Section 7.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEven under the strong assumption implicit in Linear MDPs, the learning of the set of rewards that make the expert’s policy optimal doesn’t scale well. This improves somewhat, if additionally a notion of compatibility of rewards is introduced, because in this way and under these conditions the IRL problem can be seen as a classification task. In this context a number of theoretical results are possible minimax optimality of a thus formulated IRL algorithm, complexity bounds and some contribution to the complexity theory of reward-free exploration.\n\nSTRENGTHS:\nOutsourcing some of related work to the appendix seems a good idea here, as the proper introduction is still readable and interesting. However, it may have been an alternative option to move the “original contributions” to the appendix as these are already mentioned in the abstract and in --- the paper. The paper aims at much, and although it delivers, but could be more focused and concise.\n\nWEAKNESSES:\n“How to” is not really addressed, “How to Scale Inverse RL to Large State Spaces?” should be “How does Inverse RL Scale to Large State Spaces?”\n\nThe separation of exploration and classification phases may not appear to be a problem at this level of abstraction, but practically this can be a forbidding features of an algorithm.\n\nThe approach uses a restrictive problem setting, while it could be preferable to attempt to discover any compositional structure or any latent low-dimensional manifolds, as would be the present in any practical problem if the problem can be treated at all at larger scales.\n\nNo simulation included, although it should be easy provide some illustration, in particular for any worst-case results.\n\nObjective-free exploration need more study to receive the attention that it is claimed to deserve, but the current definition probably needs to be more precise.\n\nHere (as in most contributions to this conference) the use of display equations is dearly missed. Couldn’t the amount of ink be used to measure the length of the papers?\n\nThe use of color in the manuscript could be more systematic, if it is encouraged at all.\n\nThere is too much material in the paper. It may be tempting to publish a systematic study at a conference to reach some visibility, but it creates an imbalance among the contribution and may bias future submission.\n\nQUESTIONS:\nWould objective-free exploration be independent on the linear MDP assumption? Is it needed at all in the present paper?\n\nCan you compare to other IRL algorithms?\n\nLIMITATIONS:\nPractical application is limited, but this is outside the scope of this very impressive paper.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper shows that finding the feasible reward set in IRL needs to sample $\Omega(S)$ number of samples, even when the MDP possesses a linear structure. To enable more efficient scaling with $S$, the authors propose another task in IRL called rewards compatibility: deciding whether $\pi^{E}$ is $\epsilon$-optimal under a given reward $r$. They further use reward-free algorithms to solve such tasks and propose a matching lower bound. As a byproduct, the lower bound also improves the existing lower bounds in tabular reward-free RL.

STRENGTHS:
The theoretical analysis is solid. In particular, the lower bounds for feasible reward set learning and reward compatibility are novel and significant as they quantify the hardness of these two tasks in IRL.

WEAKNESSES:
1. The reward compatibility framework is not that interesting in my opinion because it requires you to input a reward function, which indeed makes IRL a standard RL problem given that reward. More specifically, the reward compatibility framework is just policy optimization and evaluation of $\pi^E$ under a given reward $r$, which has been studied sufficiently before.
2. The algorithms that the authors propose are also just the existing algorithms in standard RL, so there is no novelty in the algorithm design.

QUESTIONS:
The lower bound in Theorem 3.2 characterizes the difficulty of identifying the exact feasible reward set. However, in many cases we just want to learn a feasible reward with some desirable properties instead of the whole set. For this setting will the lower bound still hold?

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16