Title: Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Inverse reinforcement learning

Transferability

Identifiability

Robustness

Alignment
Abstract: Inverse reinforcement learning (IRL) aims to infer a reward from expert demonstrations, motivated by the idea that the reward, rather than the policy, is the most succinct and transferable description of a task [Ng et al., 2000]. However, the reward corresponding to an optimal policy is not unique, making it unclear if an IRL-learned reward is transferable to new transition laws in the sense that its optimal policy aligns with the optimal policy corresponding to the expert's true reward. Past work has addressed this problem only under the assumption of full access to the expert's policy, guaranteeing transferability when learning from two experts with the same reward but different transition laws that satisfy a specific rank condition [Rolland et al., 2022]. In this work, we show that the conditions developed under full access to the expert's policy cannot guarantee transferability in the more practical scenario where we have access only to demonstrations of the expert. Instead of a binary rank condition, we propose principal angles as a more refined measure of similarity and dissimilarity between transition laws. Based on this, we then establish two key results: 1) a sufficient condition for transferability to any transition laws when learning from at least two experts with sufficiently different transition laws, and 2) a sufficient condition for transferability to local changes in the transition law when learning from a single expert. Furthermore, we also provide a probably approximately correct (PAC) algorithm and an end-to-end analysis for learning transferable rewards from demonstrations of multiple experts.

Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning

Abstract

Inverse reinforcement learning (IRL) aims to infer a reward from expert demonstrations, motivated by the idea that the reward, rather than the policy, is the most succinct and transferable description of a task [Ng et al., 2000]. However, the reward corresponding to an optimal policy is not unique, making it unclear if an IRL-learned reward is transferable to new transition laws in the sense that its optimal policy aligns with the optimal policy corresponding to the expert's true reward. Past work has addressed this problem only under the assumption of full access to the expert's policy, guaranteeing transferability when learning from two experts with the same reward but different transition laws that satisfy a specific rank condition [Rolland et al., 2022]. In this work, we show that the conditions developed under full access to the expert's policy cannot guarantee transferability in the more practical scenario where we have access only to demonstrations of the expert. Instead of a binary rank condition, we propose principal angles as a more refined measure of similarity and dissimilarity between transition laws. Based on this, we then establish two key results: 1) a sufficient condition for transferability to any transition laws when learning from at least two experts with sufficiently different transition laws, and 2) a sufficient condition for transferability to local changes in the transition law when learning from a single expert. Furthermore, we also provide a probably approximately correct (PAC) algorithm and an end-to-end analysis for learning transferable rewards from demonstrations of multiple experts.

1 Introduction

Reinforcement learning (RL) has achieved remarkable success in various domains such as robotics [Hwangbo et al., 2019], autonomous driving [Lu et al., 2023], or fine-tuning of large language models [Stiennon et al., 2020]. Despite these advances, a key challenge lies in designing appropriate reward functions that reflect the desired outcomes and align with human values. Misaligned rewards can lead to suboptimal behaviors [Ngo et al., 2022], undermining the potential benefits of RL in practical scenarios. Inverse reinforcement learning (IRL), also known as inverse optimal control [Kalman, 1964] or structural estimation [Rust, 1994], addresses this problem by inferring a reward from demonstrations of an expert acting optimally in a Markov decision process (MDP).

Compared to behavioral cloning [Pomerleau, 1988], which directly fits a policy to the expert's demonstrations, IRL is believed to provide a more transferable description of the expert's task [Ng et al., 2000], as recovering the expert's underlying reward would enable us to train a policy in a new environment with different dynamics. However, it is also known that the reward corresponding to some optimal policy is not unique [Ng et al., 1999], making it difficult to recover the expert's true underlying reward. This raises the question: Is a reward recovered via IRL transferable to a new

environment in the sense that its optimal policy aligns with the expert's true reward? For example, in autonomous driving, could we effectively reuse a reward learned from demonstrations of one car in a given city to train or fine-tune a policy for another car in another city?

Ensuring transferability is challenging, as neither the optimal policy corresponding to a reward nor the reward corresponding to an optimal policy is unique. This leads to trivial solutions to the IRL problem, such as constant rewards that make all policies optimal. Common approaches to address this challenge include characterizing the entire set of rewards for which the expert is optimal [Metelli et al., 2021], or assuming the expert is optimal with respect to an entropy regularized RL problem [Ziebart, 2010], leading to many popular IRL and imitation learning algorithms [Ho and Ermon, 2016, Fu et al., 2017, Garg et al., 2021]. Entropy regularization results in a unique and more uniform optimal policy, serving as a model for the expert's bounded rationality [Ortega et al., 2015].

In the entropy-regularized setting, several recent works study the set of rewards for which a given expert policy is optimal. In particular, Cao et al. [2021], Skalse et al. [2023] show that under entropy regularization, the expert's reward can be identified up to so-called potential shaping transformations [Ng et al., 1999]. The authors of [Schlaginhaufen and Kamgarpour, 2023] extend this result to more general steep regularization. Furthermore, they show that to guarantee transferability to any transition law, the expert's reward needs to be identified up to a constant. The latter can be achieved either by restricting the reward class, e.g., to state-only rewards [Amin et al., 2017], or by learning from multiple experts with the same reward but different transition laws, given that a specific rank condition is satisfied [Cao et al., 2021, Rolland et al., 2022]. However, the above results cannot be applied directly in practice, as they rely on having full access to the experts' policies, whereas in practice, we typically only have a finite set of demonstrations available.

Contributions We consider the framework of regularized IRL [Jeon et al., 2021] and address the transferability of rewards recovered from a finite set of expert demonstrations.

We define a novel notion of transferability (Definition 3.1), to address the practical limitation of not having perfect access to the experts' policies. Furthermore, we show that when learning from finite data, the conditions developed under full access to the experts' policies are not sufficient to guarantee transferability (Example 3.3).

Instead of a binary rank condition, we propose to use principal angles to characterize the similarity and dissimilarity between transition laws (Definition 3.8). Based on these principal angles, we then establish two key transferability results: 1) a guarantee for transferability to any transition laws when learning from at least two experts with sufficiently different transition laws (Theorem 3.10), and 2) a guarantee for transferability to local changes in the transition law when learning from a single expert (Theorem 3.11).

Assuming oracle access to a probably approximately correct (PAC) algorithm for the forward RL problem, we provide a PAC algorithm for the IRL problem, which in O ( K 2 / ˆ ε 2 ) steps recovers a reward for which, with high probability, all K experts are ˆ ε -optimal (Theorem 4.1). Together with our results on transferability, this establishes end-to-end guarantees for learning transferable rewards from a finite set of expert demonstrations.

We experimentally validate our results in a gridworld environment (Section 5). 1

2 Background

Notation Given x and y in some Euclidean vector space V , we denote the p -norm by ∥ x ∥ p , the orthogonal projection onto a closed convex set X ⊂ V by Π X ( x ) = arg min y ∈X ∥ x -y ∥ 2 , and the standard dot product by ⟨ x, y ⟩ . For a linear operator A , we denote its image and rank by im A and rank A , respectively. Given two sets X and Y , we denote X + Y for their Minkowski sum and Y X for the set of all functions mapping from X to Y . Additionally, we denote ∆ X for the probability simplex over X and ✶ for the indicator function. The interior int X , the relative interior relint X , the relative boundary relbd X , and the convex hull conv X of some set X are defined in Appendix A, along with an overview of all other notations.

Regularized MDPs We consider a regularized MDP [Geist et al., 2019] defined by a tuple ( S , A , P, ν 0 , r, γ, h ) . Here, S and A represent finite state and action spaces with |S| , |A| > 1 , ν 0 ∈ ∆ S the initial state distribution, P ∈ ∆ S×A S the transition law, r ∈ R S×A the reward, and γ ∈ (0 , 1) the discount factor. Furthermore, h : X → R is a strictly convex regularizer that is defined on a closed convex set X ⊆ R A with relint ∆ A ⊆ int X . Starting from some initial state s 0 ∼ ν 0 the agent can at each step in time t , choose an action a t ∈ A , will arrive in state s t +1 ∼ P ( ·| s t , a t ) , and receives reward r ( s t , a t ) . The goal is to find a Markov policy π ∈ ∆ S A maximizing the regularized objective E π [ ∑ ∞ t =0 γ t [ r ( s t , a t ) -h ( π ( ·| s t ))]] . Following the classical linear programming approach to MDPs [Puterman, 2014], this can be cast equivalently as the convex optimization problem

where M denotes the set of occupancy measures, µ π ( s, a ) := (1 -γ ) E π [ ∑ ∞ t =0 γ t ✶ ( s t = s, a t = a )] , and we have ¯ h ( µ ) := E ( s,a ) ∼ µ [ h ( π µ ( ·| s ))] , with π µ being the policy corresponding to µ (see Appendix A). The set of occupancy measures is characterized by the Bellman flow constraints

where E : R S → R S×A and P : R S → R S×A are the linear operators mapping v ∈ R S to ( Ev )( s, a ) = v ( s ) and ( Pv )( s, a ) = ∑ s ′ P ( s ′ | s, a ) v ( s ′ ) , respectively.

Due to the strict convexity of h , the regularized MDP problem has a unique optimal policy [Geist et al., 2019], hence guaranteeing the uniqueness of the optimal occupancy measure in (O-RL). In addition, we assume that the gradients of h become unbounded towards the relative boundary of the simplex as detailed in Assumption 2.1 below.

Assumption 2.1 (Steep regularization) . Suppose that h : X → R is differentiable in int X and that lim l →∞ ∥∇ h ( p l ) ∥ 2 = ∞ if ( p l ) l ∈ N is a sequence in int X converging to a point p ∈ relbd ∆ A .

Assumption 2.1 ensures that the optimal policy is non-vanishing, and together with Assumption 2.2 below, we also have that the optimal occupancy measure is non-vanishing.

One way to guarantee Assumption 2.2 is to impose a lower bound on the initial state distribution ν 0 . In the following, it will be convenient to denote the optimal solution to (O-RL) for the reward r as

and the suboptimality of some occupancy measure µ for the reward r as

That is, µ = RL ( r ) if and only if SubOpt ( r, µ ) = 0 .

Remark 2.3 . As we aim to analyze the transferability of rewards to new transition laws P ∈ ∆ S×A S , it will often be useful to explicitly specify the dependency on P . We do so by adding a subscript e.g. we write M P , RL P , and SubOpt P . However, for better readability, we drop these subscripts whenever there is no potential for confusion.

Inverse reinforcement learning Given a dataset of trajectories sampled from an expert µ E that is optimal for some reward r E , the goal in IRL is to recover a reward ˆ r ∈ R , within a predefined reward class R⊆ R S×A , such that the expert is optimal for ˆ r . That is, ideally, we aim to find a reward in the feasible reward set

However, since we don't have direct access to the expert's policy but only to a finite set of demonstrations, the best we can hope for is an algorithm that with high probability outputs a reward ˆ r ∈ R such that SubOpt (ˆ r, µ E ) is small - i.e. an algorithm that is PAC [Syed and Schapire, 2007].

Reward equivalence The reward corresponding to an optimal occupancy is not unique. For example, all rewards in the affine subspace r + U , where U := im( E -γP ) is the subspace of so-called potential shaping transformations, correspond to the same optimal occupancy measure [Ng et al., 1999]. From a geometric perspective, the subspace U = im ( E -γP ) lies perpendicular to the set of occupancy measures M . Therefore, adding an element of U to the reward leaves the performance difference between any two occupancy measures invariant. Hence, it is often convenient to consider these rewards as equivalent [Kim et al., 2021] and to measure distances between rewards in the resulting quotient space. Given a linear subspace V ⊂ R S×A , the quotient space R S×A / V is the set of all equivalence classes [ r ] V := { r ′ ∈ R S×A : r ′ -r ∈ V } , which is itself a vector space with addition and multiplication operation defined by [ r ] V +[ r ′ ] V = [ r + r ′ ] V and c [ r ] V = [ cr ] V for c ∈ R . Intuitively, R S×A / V is the vector space obtained by collapsing V to zero, or in other words, it is isomorphic to the orthogonal complement of V . We endow R S×A / V with the quotient norm ∥ [ r ] V ∥ 2 := min v ∈V ∥ r + v ∥ 2 = ∥ Π V ⊥ r ∥ 2 and we say that r and r ′ are close in R S×A / V if ∥ [ r ] V -[ r ′ ] V ∥ 2 is small. Moreover, the expert's reward is said to be identifiable up to some equivalence class [ · ] V if IRL ( µ E ) ⊆ [ r E ] V . In this paper, we will consider the equivalence relations induced by constant shifts, i.e., V = 1 := { r ∈ R S×A : r ( s, a ) = c ∈ R } , and by potential shaping transformations, i.e., V = U . Note that since 1 is a subspace of U and U is |S| -dimensional, [ r ] 1 is a strict subset of [ r ] U whenever |S| > 1 .

3 Transferability

In this section, we present our main results on transferability in IRL. To this end, we first introduce the problem of learning ε -transferable rewards from multiple experts acting in different environments.

3.1 Problem formulation

Let R⊆ R S×A be a compact reward class, and suppose we are given access to K expert data sets,

consisting of trajectories sampled independently from the experts µ E P 0 , . . . , µ E P K -1 . Each expert is optimal for the same unrevealed reward r E ∈ R , but under different transition laws, P 0 , . . . , P K -1 . Our goal is to recover a reward ˆ r ∈ R that is transferable across a set of transition laws P ⊆ ∆ S×A S . Specifically, the optimal occupancy measure corresponding to ˆ r should remain approximately optimal for r E under every transition law in P . This yields the following definition of ε -transferability.

Definition 3.1 ( ε -transferability) . Fix some ε > 0 . We say that ˆ r is ε -transferable to some set of transition laws P ⊆ ∆ S×A S if SubOpt P ( r E , RL P (ˆ r )) ≤ ε for all P ∈ P . We say that ˆ r is exactly transferable to P if it is ε -transferable to P with ε = 0 .

The error margin of ε is crucial, as exact transferability is unrealistic when learning from finite expert data. Moreover, note that Definition 3.1 is a definition of uniform transferability, as it requires ˆ r to be ε -transferable to any P ∈ P with the same fixed ε . In the following, we will analyze the transferability of a reward ˆ r for which all experts are ˆ ε -optimal for some ˆ ε > 0 . That is,

In particular, we aim to establish appropriate conditions for choosing ˆ ε so as to guarantee ε -transferability to some set of transition laws P . In Section 4, we will then provide an IRL algorithm that, with high probability, outputs a reward ˆ r such that (3) holds.

Remark 3.2 . As discussed in Appendix J, the assumption of perfect expert optimality with respect to r E can be relaxed to allow for a misspecification error. All our results remain applicable in this setting but include an additional error term due to the experts' suboptimality.

3.2 Related work

Most previous work has focused on reward identifiability. For a single expert, Cao et al. [2021], Skalse et al. [2023], Schlaginhaufen and Kamgarpour [2023] show that under Assumption 2.1 (steepness)

the feasible reward set (2) can be expressed as

In other words, steepness ensures that the expert's reward is identifiable up to potential shaping. To identify the reward up to a constant, we can either restrict the reward class, e.g. to state-only rewards as explored by Amin et al. [2017], or learn from multiple experts [Cao et al., 2021, Rolland et al., 2022]. In particular, when we are given access to two experts, µ E P 0 and µ E P 1 , we can identify the experts' reward up to the intersection

That is, for the unrestricted reward class, R = R S×A , the reward is identifiable up to a constant if and only if U P 0 ∩ U P 1 = 1 . Or equivalently, if and only if the rank condition

is satisfied [Rolland et al., 2022]. Moreover, Schlaginhaufen and Kamgarpour [2023] show that identifying the expert's reward up to a constant is a necessary and sufficient condition for exact transferability to any full-dimensional set P ⊆ ∆ S×A S (a set P whose interior, with respect to the subspace topology on ∆ S×A S [Bourbaki, 1966], is non-empty).

Limitations The above results assume perfect access to the expert's policy, which isn't realistic. In practice, we can only learn a reward for which the experts are approximately optimal. In Example 3.3 below, we show that under approximate optimality of the experts, the learned reward can perform very poorly in a new environment, even if the rank condition in Equation (5) is satisfied.

Example 3.3. We consider a two-state, two-action MDP with S = A = { 0 , 1 } , uniform initial state distribution, discount rate γ = 0 . 9 , and Shannon entropy regularization h = -H (see Appendix C). Suppose the expert reward is r E ( s, a ) = ✶ { s = 1 } and consider the transition laws, P 0 and P 1 , defined by P 0 (0 | s, a ) = 0 . 75 and P 1 (0 | s, a ) = 0 . 25 + β · ✶ { s = 0 , a = 0 } for some β ∈ [0 , 0 . 75] . Also, consider the two experts µ E P 0 = RL P 0 ( r E ) and µ E P 1 = RL P 1 ( r E ) , and suppose we recovered the reward ˆ r ( s, a ) = -r E . Then, as detailed in Appendix E, the following holds: 1) We have SubOpt P 0 (ˆ r, µ E P 0 ) = 0 and SubOpt P 1 (ˆ r, µ E P 1 ) = O ( β ) . That is, for small β , the reward ˆ r is a good solution to the IRL problem, as both experts are approximately optimal under ˆ r . 2) The rank condition (5) between P 0 and P 1 is satisfied for any β > 0 . 3) For a new transition law P defined by P (0 | s, a ) = ✶ { s = 1 , a = 0 } , we have SubOpt P ( r E , RL P (ˆ r )) ≈ 4 . 81 , i.e. RL P (ˆ r ) performs poorly under the experts' reward.

3.3 Theoretical insights

To establish a sufficient condition for ε -transferability, our goal is to bound the suboptimality of an optimal occupancy measure, RL ( r ) , for some reward r ′ , in terms of reward distances measured in the quotient space R S×A / U . To this end, we first establish the relationship between the suboptimality in Equation (1) and the Bregman divergence corresponding to the occupancy measure regularization.

Bregman divergences The Bregman divergence [Teboulle, 1992] associated to ¯ h is defined as

Proposition 3.4. Under Assumptions 2.1 and 2.2, we have SubOpt ( r ′ , µ ) = D ¯ h ( µ, RL ( r ′ )) for any µ ∈ M .

Proposition 3.4 above demonstrates that the suboptimality of an occupancy measure µ for the reward r ′ coincides with the Bregman divergence between µ and the optimal occupancy measure under r ′ . This generalizes [Mei et al., 2020, Lemma 26] from entropy regularization to any steeply regularized MDP. The proof is presented in Appendix D.6.

Reward approximation Next, we show that under strong convexity and local Lipschitz gradients, the Bregman divergence between two optimal occupancy measures is bounded in terms of reward distances in R S×A / U .

Assumption 3.5 (Regularity) . Suppose the following holds:

The regularizer ¯ h is η -strongly convex over the set of occupancy measures M . That is, we have

The gradient ∇ ¯ h is locally Lipschitz continuous over relint M . That is, for any closed convex subset K ⊂ relint M there exists L K > 0 such that

We will show later that Assumption 3.5 is met for Shannon and Tsallis entropy regularization (see Proposition D.9). Under the above assumption, the following lemma establishes the desired upper and lower bound on the Bregman divergence between two optimal occupancy measures with respect to reward distances measured in R S×A / U .

Lemma 3.6. Suppose Assumptions 2.1,2.2, and 3.5 hold, and let r, r ′ ∈ R . Then, we have

for some problem-dependent constant σ R > 0 .

Remark 3.7 . The proof of Lemma 3.6 hinges on the duality between equivalence classes of rewards and optimal occupancy measures (see Appendix B). The main idea is to leverage duality of Bregman divergences, and a dual smoothness and strong convexity result in Proposition D.7. A key challenge arises because, by Assumption 2.1, the regularizer cannot be globally smooth. This results in a problem-dependent dual strong convexity constant σ R [Goebel and Rockafellar, 2008]. In Proposition D.9, we will provide a lower bound on σ R for the specific choices of Shannon and Tsallis entropy regularization. For more details, we refer to the full proof in Appendix D.6.

The above lemma has two key implications: First, the lower bound in (6) implies that if we recover a reward ˆ r for which all experts are approximately optimal, then the distance between ˆ r and r E can be bounded in the quotient spaces R S×A / U P k . Second, the upper bound shows that to control the performance of RL P (ˆ r ) in a new environment P , we need to tightly bound the distance between ˆ r and r E in R S×A / U P . As distances in R S×A / U P are bounded by distances in R S×A / 1 , this can be achieved by bounding the distance between ˆ r and r E in R S×A / 1 . However, revisiting Example 3.3 in light of Lemma 3.6 shows that even though ˆ r and r E are close in R S×A / U P k , this does not guarantee their proximity in R S×A / 1 and R S×A / U P .

Example 3.3 (continued) . Recall the definition U P k = im( E -γP k ) . Given that in Example 3.3 we have SubOpt P 0 (ˆ r, µ E P 0 ) = 0 and SubOpt P 1 (ˆ r, µ E P 1 ) = O ( β ) , Lemma 3.6 ensures that ˆ r and r E coincide in R S×A / U P 0 , and for small β , they are close in R S×A / U P 1 . However, as illustrated in Figure1(a) this doesn't ensure that ˆ r and r E are close in R S×A / 1 and R S×A / U P . In particular, it can be computed that ∥ ∥ [ˆ r ] U P -[ r E ] U P ∥ ∥ 2 ≈ 1 . 51 , which by Lemma 3.6 explains the poor transferability to P .

3.4 Sufficient conditions for transferability

With Lemma 3.6 in place, we are set to present our results on ε -transferability. Example 3.3 indicates that a sufficient condition for learning transferable rewards from experts ( K = 2 ) should not rely solely on the binary rank condition (5), which only checks if U P 0 ∩ U P 1 = 1 . Instead, we should consider the relative orientation between U P 0 and U P 1 . To formalize this, we need to introduce the concept of principal angles between linear subspaces, as outlined in Definition 3.8 below.

Definition 3.8 (Principal angles [Galántai, 2013]) . Let V , W ⊆ R n be two subspaces of dimension m ≤ n . The principal angles 0 ≤ θ 1 ( V , W ) ≤ . . . ≤ θ m ( V , W ) =: θ max ( V , W ) ≤ π/ 2 between V and W are defined recursively via

where v j , w j are the maximizers corresponding to the angle θ j . For two transition laws P, P ′ , we define θ i ( P, P ′ ) := θ i ( U P , U P ′ ) and refer to θ i ( P, P ′ ) as the i -th principal angles between P and P ′ .

r E

+

U

P

1

r E

ˆ

r

r E

+

U

P

0

r E

+

U

P

R

S×A

/

1

( a ) Rewards in Example 3.3.

r E

θ

∆

R

S×A

/

1

¯

ε r E

+

U

P

1

r E

+

U

P

0

Principal angles are the natural generalization of angles between two lines or planes to higher dimensional subspaces. For principal angles between transition laws, we have the following proposition.

Proposition 3.9. Let P, P ′ ∈ ∆ S×A S and H γ = 1 / (1 -γ ) . Then, we have θ 1 ( P, P ′ ) = 0 and sin ( θ max ( P, P ′ )) ≤ γH γ √ |S| / |A| ∥ P -P ′ ∥ , where ∥·∥ denotes the spectral norm.

The proof can be found in Appendix D.7. The above result shows that while the first principal angle between two transition laws is always zero, all principal angles are small if the transition laws are close to one another. In Example 3.3, we have sin( θ 2 ( P 0 , P 1 )) = O ( β ) , indicating that the second and in this case maximal principal angle is small when β is small (see Appendix E). The following result shows that when learning from two experts, the transferability error is directly controlled by the second principal angle between the experts' transition laws.

Theorem 3.10. Let K = 2 , θ 2 ( P 0 , P 1 ) > 0 , and suppose that Assumptions 2.1,2.2, and 3.5 hold. If SubOpt P k (ˆ r, µ E P k ) ≤ ˆ ε for k = 0 , 1 , then ˆ r is ε -transferable to P = ∆ S×A S with

Sketch of proof. The main idea of the proof is illustrated in Figure 1(b). First, it follows from Lemma 3.6 that ˆ r and r E are ¯ ε = √ 2ˆ ε/σ R -close in R S×A / U P k for k = 0 , 1 , respectively. From Figure 1(b) we see - using basic trigonometry - that this implies that ˆ r and r E are at least ∆ = ¯ ε/ sin( θ/ 2) -close in R S×A / 1 . As shown in the full proof in Appendix F, the relevant angle, θ , is the second principal angle θ 2 ( P 0 , P 1 ) . The result then follows from the upper bound in Lemma 3.6.

Some observations are in order. First, the above theorem shows that the larger the second principal angle between the two experts' transition laws, the better the recovered reward transfers to a new environment. Second, observe that θ 2 ( P 0 , P 1 ) > 0 is equivalent to the rank condition (5), as the second principal angle between two subspaces is non-zero if and only if their intersection is at most one-dimensional. Therefore, for exact transferability, Theorem 3.10 requires the rank condition (5) to be satisfied and ˆ ε = 0 , recovering the results by Cao et al. [2021], Rolland et al. [2022], Schlaginhaufen and Kamgarpour [2023]. But in contrast to past results, Theorem 3.10 applies to more realistic scenarios, where ˆ ε is merely small, not zero. Finally, we note that Theorem 3.10 can be trivially generalized to K ≥ 2 experts by replacing θ 2 ( P 0 , P 1 ) with the maximum of θ 2 ( P k , P l ) over 0 ≤ k ≤ l ≤ K -1 . However, such bounds may be loose for K > 2 , potentially leaving considerable room for improvement in this setting.

Local transferability When learning a reward ˆ r from a single expert ( K = 1 ), Schlaginhaufen and Kamgarpour [2023] show that, without reducing the dimension of the reward class, ˆ r cannot be exactly transferable to any neighborhood of the expert's transition law P 0 . However, Theorem 3.11 below shows that by allowing for an ε of error, we can guarantee transferability to a neighborhood of P 0 .

Theorem 3.11. Let K = 1 , D := max r,r ′ ∈R ∥ r -r ′ ∥ 2 , and suppose that Assumptions 2.1,2.2, and 3.5 hold. If SubOpt P 0 (ˆ r, µ E ) ≤ ˆ ε , then ˆ r is ε P -transferable to P ∈ ∆ S×A S with

The above theorem (which is proven in Appendix G) shows that the reward learned from a single expert transfers to transition laws that are sufficiently close to the expert's, where the closeness is measured in terms of the maximal principal angle. In other words, while a large second principal angle between two experts' transition laws, as per Theorem 3.10, ensures that the reward recovered from these two experts is transferable to arbitrary transition laws, a small largest principal angle between two transition laws ensures that a reward recovered in one environment can be successfully transferred to the other environment.

Remark 3.12 . As discussed in Appendix H, we can compute the principal angles using a singular value decomposition. Moreover, given estimates ˆ P 0 , ˆ P 1 of the transition laws P 0 , P 1 , the error in the estimate of sin θ i ( P 0 , P 1 ) scales with O (max {|| P 0 -ˆ P 0 || , || P 1 -ˆ P 1 ||} ) .

Regularizers To provide more insights about Theorems 3.10 and 3.11, we provide explicit values for the primal and dual strong convexity constants, η and σ R , respectively. To this end, we focus on the Shannon entropy regularization h ( p ) = -τ H ( p ) and the Tsallis-1/2 entropy regularization h ( p ) = -τ H 1 / 2 ( p ) as defined in Appendix C. While the Shannon entropy regularization is commonly used in IRL [Ziebart, 2010, Ho and Ermon, 2016], the Tsallis-1/2 entropy is more often adopted in the multi-armed bandit literature Zimmert and Seldin [2021]. Both regularizations satisfy Assumption 2.1 as well as Assumption 3.5 with the constants detailed in Proposition D.9 in the appendix. In general, the Tsallis entropy leads to a slightly smaller strong convexity constant η , but avoids an exponential dependence on the effective horizon H γ = 1 / (1 -γ ) in σ R . Below, we summarize the implications of Proposition D.9 for ε -transferability of a reward ˆ r recovered from two experts.

Corollary 3.13. Suppose the conditions in Theorem 3.10 hold. Furthermore, let H γ := 1 / (1 -γ ) , R := max r ∈R ∥ r ∥ ∞ , D = max r,r ′ ∈R ∥ r -r ′ ∥ 2 , and τ < D . Then, for the Shannon entropy ˆ r is ε -transferable to P = ∆ S×A S with

and for the Tsallis entropy with

We observe that transferability generally becomes more challenging with decreasing regularization parameter τ , i.e. if the expert's policy becomes more deterministic. Furthermore, we see that it is easier to recover a transferable reward in a Tsallis entropy-regularized MDP. Corollary 3.13 also shows that the constant between ε and ˆ ε tends to be large, meaning that we need to recover a reward for which the experts are ˆ ε -optimal with a very small ˆ ε to guarantee ε -transferability for a reasonable ε . However, it's important to note that our results provide sufficient conditions for the worst case, and it remains for future work to determine under what conditions these constants can be improved.

Remark 3.14 . Our results in this section, especially Proposition 3.4 and Lemma 3.6, are critically relying on the steepness of the regularization (Assumption 2.1), which is essential to ensure that the expert's reward can be identified up to the equivalence class of potential shaping transformations. Although we can still upper bound the suboptimality SubOpt ( r E , RL (ˆ r )) in terms of the distance between ˆ r to r E in R S×A / U without this assumption (see Proposition D.10), we no longer have a lower bound as in Lemma 3.6, which is essential for establishing closeness of ˆ r and r E in R S×A / U . Hence, we expect it to be difficult to obtain guarantees similar to those in Theorem 3.10 and 3.11 for the unregularized setting, without either reducing the dimension of the reward class [Amin et al., 2017] or making specific assumptions about the feasible reward sets [Metelli et al., 2021, Assumption 4.1].

4 Algorithm

To provide end-to-end guarantees for recovering transferable rewards from a finite set of expert demonstrations, we analyze the convergence and sample complexity (in terms of expert demonstra-

tions) of an algorithm for recovering a reward for which all K experts are approximately optimal. To this end, we focus on the reward class R = { r ∈ R S×A : ∥ r ∥ 1 ≤ 1 } . Furthermore, we assume oracle access to a ( ε, δ ) -PAC algorithm for the forward problem (O-RL). That is, a polynomial-time algorithm, A ε,δ , that outputs a policy π = A ε,δ ( r ) such that with probability at least 1 -δ it holds that SubOpt ( r, µ π ) ≤ ε (see e.g. [Lan, 2023] for a specific example). The key idea of our meta-algorithm is to learn a reward minimizing the sum of the suboptimalities of the K experts µ E P 0 , . . . , µ E P K -1 . This leads us to the following multi-expert IRL problem

where ˆ µ D E k ( s, a ) := (1 -γ ) /N E ∑ N E -1 i =0 ∑ H E -1 t =0 γ t ✶ { s k,i t = s, a k,i t = a } is an empirical expert occupancy measure. To solve Problem (O-IRL), we propose the projected gradient descent scheme as detailed in Algorithm 1 below, where rollout P k ( π, N, H ) samples N independent trajectories of length H from policy π . Using a stochastic online gradient descent analysis, Theorem 4.1 shows that any PAC algorithm for the forward problem yields a PAC algorithm for the inverse problem.

Theorem 4.1. Suppose that N E = Ω ( K log( |S||A| / ˆ δ ) / ˆ ε 2 ) and H E = Ω ( log( K/ ˆ ε ) / log(1 /γ ) ) . Running Algorithm 1 for T = Ω ( K 2 / ˆ ε 2 ) iterations with step-size α = 1 / ( K √ T ) , where δ opt = O ( ˆ δ ˆ ε 2 /K 3 ) , ε opt = O (ˆ ε/K ) , N = Ω ( K log( K |S||A| / ( ˆ δ ˆ ε )) / ˆ ε 2 ) , and H = H E , it holds with probability at least 1 -ˆ δ that SubOpt P k (ˆ r, µ E P k ) ≤ ˆ ε, for k = 0 , . . . , K -1 .

The above result generalizes [Syed and Schapire, 2007, Theorem 2] by considering multiple experts and by proving convergence in terms of the expert suboptimality. We refer to Appendix I for the proof and the precise constants. Theorem 4.1 shows that with Ω( K/ ˆ ε 2 ) demonstrations of each expert, we recover in Ω( K 2 / ˆ ε 2 ) steps of Algorithm 1 a reward ˆ r for which all experts are ˆ ε -optimal. Together with Theorem 3.10 and 3.11, this provides a bound on the sample and time complexity of recovering in ε -transferable rewards in regularized IRL.

5 Experiments

To validate our results experimentally, we adopt a stochastic variant of the WindyGridworld environment [Sutton and Barto, 2018]. In this environment, the agent moves to the intended grid cell with a probability of (1 -β ) and is pushed one step further in the direction of the wind with a probability of β . Using Algorithm 1, we recover a reward ˆ r from demonstrations of two experts, both exposed to the same wind strength β but different wind directions - North and East. The experiments are repeated for a varying number of expert demonstrations N E ∈ { 10 3 , 10 4 , 10 5 , 10 6 } and wind strengths β ∈ { 0 . 01 , 0 . 1 , 0 . 5 , 1 . 0 } . We then test the transferability to two different environments: one with South wind, P South , and a zero-wind environment with cyclically shifted actions, P Shifted . Figure 2(a) shows that the second principal angle between the two experts' transition laws P 0 and

(

a

)

(

b

)

(

c

)

(

d

)

0.25

β=0.01

20

β =0.1

6

9

0.20

reward distance

β=0.5

transferability

transferability

15

β =1.0

4

4

#

10

0.10

2

2

0.05

5

0.00+

0

0

0.0

0.5

1.0

103

104

105

106

103

104

105

106

103

104

105

106

β

NE

NE

NE

P 1 increases with increasing wind strength. Moreover, Figure 2(b)-(d) show that both the closeness between ˆ r and r E in R S×A / 1 and the transferability to P South and P Shifted improve with a larger second principal angle, as expected from Theorem 3.10. For a more detailed discussion of the experiments we refer to Appendix K.

6 Conclusion

Summary In this paper, we investigated the transferability of rewards in regularized IRL. We showed that the conditions established under full access to the experts' policies do not guarantee transferability when learning a reward from a finite set of expert demonstrations. To address this issue, we proposed using principal angles as a more refined measure of the similarity and dissimilarity of transition laws. Assuming a strongly convex and locally smooth regularization, we then showed that if we recover a reward for which at least two experts are nearly optimal, and their environments are sufficiently different in terms of the second principal angle between their transition laws, then the recovered reward is universally transferable. Furthermore, we showed that if two environments are sufficiently similar in terms of the maximal principal angle between their transition laws, rewards learned in one environment can be effectively transferred to the other environment. Additionally, we provided explicit constants for the Shannon and Tsallis entropy, as well as a PAC algorithm for recovering a reward for which all experts are approximately optimal. As a result, we established end-to-end guarantees for learning transferable rewards in regularized IRL. Additionally, we experimentally validated our results through gridworld experiments.

Limitations and future work Our results provide only sufficient conditions for transferability. It would be valuable to investigate necessary conditions to check whether our bounds are tight. Furthermore, extending our analysis to lower-dimensional reward classes could reduce the complexity of learning transferable rewards. Although our paper focuses on discrete state and action spaces, an exciting avenue for future research would be to extend our results to continuous state and action spaces, which are more commonly encountered in practice. We expect that our proof methods can be generalized to this setting, but the analysis will be more intricate due to the infinite-dimensional reward and occupancy measure spaces. Finally, as our work is mainly theoretical, experimental validation on real-world applications could provide valuable insight into the practical aspects and challenges of transferability.

Acknowledgments Andreas Schlaginhaufen is funded by a PhD fellowship from the Swiss Data Science Center.

References

Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , 2004.

Amin, N. Jiang, and S. Singh. Repeated inverse reinforcement learning. Advances in neural information processing systems , 2017.

Beck. First-order methods in optimization . SIAM, 2017.

Bourbaki. Elements of mathematics: General topology . Hermann, 1966.

P. Boyd and L. Vandenberghe. Convex optimization . Cambridge university press, 2004.

Cao, S. Cohen, and L. Szpruch. Identifiability in inverse reinforcement learning. Advances in Neural Information Processing Systems , 34, 2021.

M. Cover. Elements of information theory . John Wiley & Sons, 1999.

Dann, C.-Y. Wei, and J. Zimmert. Best of both worlds policy optimization. arXiv preprint arXiv:2302.09408 , 2023.

Drmac. On principal angles between subspaces of euclidean space. SIAM Journal on Matrix Analysis and Applications , 2000.

Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248 , 2017.

Galántai. Projectors and projection methods . Springer Science & Business Media, 2013.

Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems , 2021.

Geist, B. Scherrer, and O. Pietquin. A theory of regularized markov decision processes. In International Conference on Machine Learning . PMLR, 2019.

Goebel and R. T. Rockafellar. Local strong convexity and local lipschitz continuity of the gradient of convex functions. Journal of Convex Analysis , 2008.

Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems , 2016.

Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association , 1963.

Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics , 2019.

Jeon, C.-Y. Su, P. Barde, T. Doan, D. Nowrouzezahrai, and J. Pineau. Regularized inverse reinforcement learning. In International Conference on Learning Representations , 2021.

Ji-Guang. Perturbation of angles between linear subspaces. Journal of Computational Mathematics , 1987.

E. Kalman. When Is a Linear Control System Optimal? Journal of Basic Engineering , 1964.

Kim, S. Garg, K. Shiragur, and S. Ermon. Reward identification in inverse reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning , pages 5496-5505, 2021.

V. Knyazev and M. E. Argentati. Principal angles between subspaces in an a-based scalar product: algorithms and perturbation estimates. SIAM Journal on Scientific Computing , 2002.

Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical programming , 2023.

Lu, J. Fu, G. Tucker, X. Pan, E. Bronstein, R. Roelofs, B. Sapp, B. White, A. Faust, S. Whiteson, et al. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2023.

Mei, C. Xiao, C. Szepesvari, and D. Schuurmans. On the global convergence rates of softmax policy gradient methods. In International conference on machine learning . PMLR, 2020.

M. Metelli, G. Ramponi, A. Concetti, and M. Restelli. Provably efficient learning of transferable rewards. In International Conference on Machine Learning . PMLR, 2021.

Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml , 1999.

Y. Ng, S. Russell, et al. Algorithms for inverse reinforcement learning. In Icml , 2000.

Ngo, L. Chan, and S. Mindermann. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626 , 2022.

A. Ortega, D. A. Braun, J. Dyer, K.-E. Kim, and N. Tishby. Information-theoretic bounded rationality. arXiv preprint arXiv:1512.06789 , 2015.

Ouhamma and M. Kamgarpour. Learning nash equilibria in zero-sum markov games: A single time-scale algorithm under weak reachability. arXiv preprint arXiv:2312.08008 , 2023.

Penrose. A generalized inverse for matrices. In Mathematical proceedings of the Cambridge philosophical society . Cambridge University Press, 1955.

A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems , 1988.

L. Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley &Sons, 2014.

T. Rockafellar. Convex Analysis . Princeton University Press, 1970.

T. Rockafellar and R. J.-B. Wets. Variational analysis . Springer Science & Business Media, 2009.

Rolland, L. Viano, N. Schürhoff, B. Nikolov, and V. Cevher. Identifiability and generalizability from multiple experts in inverse reinforcement learning. Advances in Neural Information Processing Systems , 2022.

Rust. Structural estimation of markov decision processes. Handbook of econometrics , pages 3081-3143, 1994.

Schlaginhaufen and M. Kamgarpour. Identifiability and generalizability in constrained inverse reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning . PMLR, 2023.

M. V. Skalse, M. Farrugia-Roberts, S. Russell, A. Abate, and A. Gleave. Invariance in policy optimisation and partial identifiability in reward learning. In International Conference on Machine Learning . PMLR, 2023.

Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 2020.

S. Sutton and A. G. Barto. Reinforcement learning: An introduction . MIT press, 2018.

Syed and R. E. Schapire. A game-theoretic approach to apprenticeship learning. Advances in neural information processing systems , 2007.

Teboulle. Entropic proximal mappings with applications to nonlinear programming. Mathematics of Operations Research , 1992.

D. Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy . Carnegie Mellon University, 2010.

Zimmert and Y. Seldin. Tsallis-inf: An optimal algorithm for stochastic and adversarial bandits. The Journal of Machine Learning Research , 2021.

Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03) , 2003.

Appendix

Table of Contents

A Notations

Overview Here, we provide an overview of some of the most important notations. However, every notation is defined when it is introduced as well.

Additional definitions In the following, we briefly recall some additional definitions. To this end, we denote B ( x, r ) := { x ∈ R n : ∥ x ∥ 2 < r } for an open ball of radius r with center x .

Definition A.1 (Interior) . The interior of a set X ⊆ R n is defined as

Definition A.2 (Affine hull) . The affine hull of a set X ⊆ R n is defined as

Definition A.3 (Relative interior) . The relative interior of a set X ⊆ R n is defined as

Definition A.4 (Relative boundary) . The relative boundary of a closed set X ⊆ R n is defined as relbd X := X \ relint X .

Definition A.5 (Convex hull) . The convex hull of a set X ⊆ R n is defined as

B Conjugate duality in regularized IRL

In this section, we first recall some background from convex analysis and then briefly discuss the duality between reward equivalence classes and optimal occupancy measures.

̸

Definitions We recall a few definitions related to convex functions. In convex analysis it is standard to consider extended real value functions f : R n → R := [ -∞ , ∞ ] , where convex functions defined on some subset X ⊂ R n are extended over the entire space by setting their value to + ∞ outside of their domain. The effective domain is defined as dom f := { x : f ( x ) < ∞} , and a convex function f is said to be proper if f > -∞ and dom f = ∅ . Furthermore, f is referred to as closed if its epigraph { ( x, y ) : x ∈ dom f, y ≥ f ( x ) } is a closed set. 2 In particular, f is closed if it is continuous on dom f and dom f is a closed set [Boyd and Vandenberghe, 2004]. Lastly, we recall two key concepts in convex analysis - the subdifferential and the convex conjugate of some convex function.

Definition B.1 (Subdifferential) . A subgradient of f : R n → R at some point x ∈ R n is a vector g ∈ R n such that f ( x ′ ) ≥ f ( x ) + g ⊤ ( x -x ) for all x ′ ∈ X . The subdifferential ∂f ( x ) at x ∈ X is the set of all subgradients at x , where ∂f ( x ) is defined to be empty if x / ∈ dom f .

Definition B.2 (Convex Conjugate) . The convex conjugate of f : R n → R is the function f ∗ : R n → R defined as

Key results Next, we list two key results from convex analysis.

Theorem B.3 ([Rockafellar, 1970]) . A function f : R n → R is differentiable at some point x ∈ dom f if and only if ∂f ( x ) is singleton. In this case we have ∂f ( x ) = {∇ f ( x ) } .

Theorem B.4 ([Rockafellar, 1970]) . For any proper convex function f : R n → R it holds

If additionally f is closed, then

Duality in IRL Let f : R S×A → R be given by f := ¯ h + δ M , where δ M is a characteristic function defined as δ M ( µ ) = 0 if µ ∈ M and δ M ( µ ) = ∞ , otherwise. Since f is closed proper convex, Theorem B.3 and B.4 imply that for a strictly convex ¯ h we have

Additionally, under Assumption 2.1 and Slater's condition, which is ensured by Assumption 2.2, we have ∂f ( µ ) = ∇ ¯ h ( µ ) + U [Schlaginhaufen and Kamgarpour, 2023]. Hence, µ is optimal for r if and only if r ∈ [ ∇ ¯ h ( µ )] U . Therefore, there is a one-to-one mapping between elements of the quotient space R S×A / U , i.e. reward equivalence classes, and corresponding optimal occupancy measures in M . This mapping is given by

and its inverse by

C Regularizers

We dedicate this section to discuss optimal policies in regularized MDPs and to recall their explicit form for Shannon and Tsallis entropy regularization.

Optimal policies Throughout the appendix, it will convenient to use the notation π s := π ( ·| s ) . Given some proper closed strongly convex policy regularizer h , it can be shown [Geist et al., 2019] that the optimal policy, π ∗ , satisfies

where q ∗ s ∈ R A is the optimal q -function defined via

Next, we discuss the explicit form of π ∗ s = ∇ h ∗ ( q ∗ s ) for the specific cases of Shannon and Tsallis1 / 2 entropy regularization.

Shannon entropy For some τ > 0 , we define the Shannon entropy regularizer as h := -τ H , where

is the Shannon entropy satisfying 0 ≤ H ≤ log |A| . It can be shown that h is τ -strongly convex with respect to ∥·∥ 1 [Cover, 1999], and the optimal policy satisfies [Geist et al., 2019]

Tsallis entropy For some parameter α ∈ R , the Tsallis entropy, H α , is defined as

In the limit α → 1 , the Tsallis entropy equals the Shannon entropy defined above. However, in this paper, we use Tsallis entropy to refer to the choice α = 1 / 2 , which is often adopted as regularization in multi-armed bandit and, more recently, policy optimization algorithms [Zimmert and Seldin, 2021,

Dann et al., 2023]. That is, we consider h ( π s ) = -τ H 1 / 2 ( π s ) = -2 τ ( ∑ a √ π s ( a ) -1 ) for some

τ > 0 . We have 0 ≤ -h ≤ 2 τ ( √ |A| 1 ) and it can be shown that the optimal policy satisfies

where x s is a normalization parameter such that ∑ a π ∗ ( a | s ) = 1 [Zimmert and Seldin, 2021]. Furthermore, since h has diagonal Hessian ∇ 2 h ( π s ) a,a = τ/ (2 π s ( a ) 3 / 2 ) , it is τ/ 2 -strongly convex with respect to ∥·∥ 2 and hence also τ/ (2 |A| ) -strongly convex with respect to ∥·∥ 1 .

D Technical Lemmas

In this section, we show several new technical results that are required for the proofs of our main theorems.

D.1 Lipschitz continuity from policies to occupancy measures

Proposition D.1. Let Assumption 2.2 hold. For any µ 1 , µ 2 ∈ M , we have

Proof. To show the first inequality, we decompose

where we used the triangle and Hölder's inequality. From the Bellman flow constraints,

it follows that

where we again used the triangle inequality. Hence, we have

To show the second inequality, we use the reverse triangle inequality

where in the last step we used again that ∥ ν 1 -ν 2 ∥ 1 ≤ γ ∥ µ 1 -µ 2 ∥ 1 . By rearranging terms we arrive at the desired inequality

D.2 Strong convexity

Next, we show that strong convexity of the policy regularizer translates into strong convexity in the occupancy measure.

Proposition D.2 (Strong convexity) . Let Assumption 2.2 hold and suppose that h is η h -strongly convex with respect to the ∥·∥ 1 norm. Then, ¯ h is η h ν min /H 2 γ -strongly convex with respect to ∥·∥ 1 .

Proof. We need to show that for α ∈ (0 , 1) , ¯ α = 1 -α and any two µ 1 , µ 2 ∈ M it holds that

for η = η h ν min /H 2 γ . To this end, we start similarly as in the proof of strict convexity by Schlaginhaufen and Kamgarpour [2023], but use ν ( s ) ≥ ν min and strong convexity of h .

From here on, we use that

where we used Proposition D.1 in the last step. Plugging the inequality (8) back into (7) concludes the proof.

D.3 Lipschitz gradients

In this section, we show how we can get bounds on the Lipschitz constant L K . To this end, we first need to lower bound the optimal policies.

Policy lower bounds The following proposition establishes a lower bound for optimal policies with Shannon and Tsallis entropy regularization.

Proposition D.3. Let H γ = 1 / (1 -γ ) and r max := ∥ r ∥ ∞ . Then, we have the following lower bounds:

Proof. Recall the formula for the optimal policies in Appendix C.

Part a): Since, -r max H γ ≤ q ∗ ( s, a ) ≤ ( r max + τ log |A| ) H γ , it holds that

Part b): The proof of b) is similar to [Ouhamma and Kamgarpour, 2023, Lemma 8]. However, our settings are slightly different. Recall that

By Ouhamma and Kamgarpour [2023, Lemma 10] we have τ ≤ x s -∥ q ∗ s ∥ ∞ ≤ τ √ |A| . Furthermore, it holds that -r max H γ ≤ q ∗ ( s, a ) ≤ ( r max +2 τ √ |A| ) H γ . Hence, we have

which yields the desired lower bound.

We also highlight the following result, which shows that if the policy π µ is lower bounded on some set K ⊂ M , then it is also lower bounded on its convex hull conv K .

Proposition D.4. Suppose µ = αµ 1 +(1 -α ) µ 2 with α ∈ (0 , 1) and µ 1 , µ 2 ∈ M . Then,

where β s ∈ (0 , 1) .

Proof. The proof follows immediately from the proof of Proposition D.2.

Hessian upper bounds In the following, we establish upper bounds for the Hessians of the occupancy measure regularizations, ¯ h , resulting from Shannon and Tsallis entropy regularization of the policy. In particular, we aim to upper-bound the maximum norm of the Hessian in terms of the smallest entry of the policy.

Proposition D.5. Let Assumption 2.2 hold. Consider µ ∈ M and let π min = min s,a π µ ( a | s ) > 0 . Then, the Hessian of ¯ h is upper bounded as follows:

Here, ∥·∥ ∞ denotes the maximum norm ∥ A ∥ ∞ = max ij | A ij | .

Proof. As shown by Schlaginhaufen and Kamgarpour [2023, Proposition B.2], the gradient of ¯ h satisfies

Moreover, we have

Using the above two formulas, we can calculate the Hessians explicitly.

Part a): For the Shannon entropy it holds by (9) that ∇ ¯ h ( µ )( s, a ) = τ log π µ ( a | s ) . Hence,

and

Part b): For the Tsallis entropy it holds by (9) that

Therefore, the second derivative is bounded as follows

Now, since ∑ a √ π µ ( a | s ) ≤ √ A ≤ 1 / √ π min, we have ( A ) + ( B ) ≤ 1 / √ π min +1 /π 3 / 2 min ≤ 2 /π 3 / 2 min , which yields the desired result

Lipschitz gradients Next, we provide explicit Lipschitz constants L K for ∇ ¯ h corresponding to Shannon and Tsallis entropy regularization.

Proposition D.6 (Lipschitz gradients) . Consider some closed convex set K ⊂ M and suppose π min = min µ ∈K min s,a π µ ( a | s ) > 0 . Then, the gradient of ¯ h is Lipschitz continuous over K i.e.

where the respective Lipschitz constants are as follows:

Proof. Defining h = µ 2 -µ 1 , Lipschitz continuity follows from

Here, we used in ( i ) and ( v ) that ∥ x ∥ 1 ≤ √ n ∥ x ∥ 2 ≤ n ∥ x ∥ ∞ for x ∈ R n , and in ( ii ) we applied the fundamental theorem of calculus. Moreover, ( iii ) follows from ∣ ∣ ∫ f ∣ ∣ ≤ ∫ | f | , and ( iv ) from Hölder's inequality. Now, by convexity µ 1 , µ 2 ∈ K implies that µ 1 + th ∈ K for t ∈ [0 , 1] . Hence, plugging in the upper bounds from Proposition D.5 concludes the proof.

D.4 Dual smoothness and strong convexity

Next, we show that the convex conjugate, f ∗ , of the extended real value function f := ¯ h + δ M (see Appendix B) is - if understood as a mapping from R S×A / U to R - both smooth and strongly convex on R with respect to the quotient norm. While it is well-known that global smoothness and strong convexity are dual properties [Rockafellar and Wets, 2009, Proposition 12.60], the key challenge for proving dual strong convexity is that ¯ h has only locally Lipschitz gradients (see Proposition D.6). Proposition D.7 below shows that η -strong convexity of ¯ h implies dual 1 /η -smoothness and locally Lipschitz gradients imply dual σ R -strong convexity on R for some σ R > 0 . Moreover, we provide explicit lower bounds on σ R for Shannon and Tsallis entropy entropy regularization.

Proposition D.7. Let f ∗ be the convex conjugate of f := ¯ h + δ M . Then, the following holds:

Suppose that ¯ h is η -strongly convex over M , that is for all µ, µ ′ ∈ M it holds that

then we have for all r, r ′ ∈ R S×A that

Suppose that for any closed convex subset K ⊂ relint M , there is some L K > 0 such that for all µ, µ ′ ∈ K it holds that

then we have for all r, r ′ ∈ R that

for some σ R > 0 .

Let H γ := 1 / (1 -γ ) , R := max r ∈R ∥ r ∥ ∞ , D = max r,r ′ ∈R ∥ r -r ′ ∥ 2 , and suppose that τ < D , then for the Shannon entropy the inequality (11) holds with

and for the Tsallis entropy with

Proof. Part a): Since f is η -strongly convex with respect to ∥·∥ 2 , the convex conjugate f ∗ is 1 /η -smooth with respect to the dual norm in R S×A / U [Rockafellar and Wets, 2009, Proposition 12.60], which is equivalent to (10).

Part b): The show b), we closely follow [Goebel and Rockafellar, 2008, Theorem 4.1], but we need to account for the quotient spaces. We define the sets K = ∇ f ∗ ( R ) = RL ( R ) and K ϵ = conv( K ) + ϵ ( B ∩ aff( M )) , where B ⊂ R S×A denotes the closed unit ball with respect to ∥·∥ 2 and ϵ > 0 is chosen such that K ϵ ⊂ relint M . Moreover, we let L be the Lipschitz constant of ∇ ¯ h over

K ϵ . Now, consider r ∈ R and µ = ∇ f ∗ ( r ) . Then, for any r ′ ∈ R , we have

Here, ( i ) follows from K ϵ ⊂ R S×A , ( ii ) from the fact that Lipschitz gradients imply that

for any g ∈ ∂f ( µ ) [Beck, 2017, Lemma 5.7] and r ∈ ∂f ( µ ) (see Theorem B.4). Moreover, ( iii ) and ( v ) follow from rearranging terms, and ( iv ) from f ∗ ( r ) = ⟨ r, µ ⟩ -f ( µ ) . Now, consider any α > 0 such that σ R = 2( α -Lα 2 / 2) > 0 and αD ≤ ϵ . Setting ¯ µ ∈ K ϵ in the above supremum to (¯ µ -µ ) = α Π U ⊥ ( r ′ -r ) yields the desired result

Note that we indeed have ¯ µ ∈ K ϵ as µ ∈ K and ∥ µ -¯ µ ∥ 2 ≤ α ∥ r -r ′ ∥ 2 ≤ αD ≤ ϵ .

Part c): To get an explicit constant for σ R , we need to appropriately choose ϵ and calculate the corresponding Lipschitz constant. To this end, we first recall that according to Proposition D.3 and D.4 policies corresponding to occupancy measures in conv K are, for Shannon and Tsallis entropy, lower bounded by

respectively. Furthermore, for any µ ∈ K ϵ , we have by Proposition D.1 and equivalence of norms

Hence, by setting ϵ = ν min π min / (4 √ |S||A| ) , we have π µ ( a | s ) ≥ π ′ min = π min / 2 for any µ ∈ K ϵ . As for the Lipschitz constant over K ϵ , we have by Proposition D.6

for the Shannon and Tsallis entropy, respectively. Now, we need to ensure that α > 0 such that σ R = 2( α -Lα 2 / 2) > 0 and α ≤ ϵ/D . To that end, we set for both regularizations α = τ/ ( LD ) , which in light of (13) ensures that

for |S| , |A| ≥ 2 . Moreover, we get the dual strong convexity constant

which is larger than τ/LD for τ < D . We can therefore choose σ R = τ/LD as a dual strong convexity constant. Plugging in the Lipschitz constants for the two regularizations yields

for the Shannon entropy, and

for the Tsallis entropy.

Remark D.8 (Large τ regime) . Note that if τ ≥ 2 D/ √ |S||A| , we can set σ R = α = 1 /L , while still satisfying the condition (14). This leads to the strong convexity constants

D.5 Regularity constants

In the following Proposition, we summarize the regularity constants for Shannon and Tsallis entropy regularization. We highlight that these constants are lower bounds for η and σ R .

Proposition D.9. Let H γ := 1 / (1 -γ ) , R := max r ∈R ∥ r ∥ ∞ , and D = max r,r ′ ∈R ∥ r -r ′ ∥ 2 . Suppose that τ < D , then for the Shannon entropy, Assumption 3.5 holds with

and for the Tsallis entropy with

Proof. The derivation for η is given in Proposition D.2 and for σ R in Proposition D.7 above.

D.6 Suboptimality bounds

Proposition 3.4. Under Assumptions 2.1 and 2.2, we have SubOpt ( r ′ , µ ) = D ¯ h ( µ, RL ( r ′ )) for any µ ∈ M .

Proof. Let µ = RL ( r ) . We have

where the second equality holds, as by (4) we have r -∇ ¯ h ( µ ) ∈ U , and µ -µ ′ ∈ U ⊥ .

Lemma 3.6. Suppose Assumptions 2.1,2.2, and 3.5 hold, and let r, r ′ ∈ R . Then, we have

for some problem-dependent constant σ R > 0 .

Proof. Let f := ¯ h + δ M and µ = RL ( r ) , µ ′ = RL ( r ′ ) . We then have

Here, ( i ) follows from the definition of f and r ′ ∈ [ ∇ ¯ h ( µ ′ )] U , in ( ii ) we use that f ( µ ) = ⟨ r, µ ⟩ -f ∗ ( r ) and f ( µ ′ ) = ⟨ r ′ , µ ′ ⟩ -f ∗ ( r ′ ) , and ( iii ) follows from rearranging terms and µ = ∇ f ∗ ( r ) . The result then follows from dual strong convexity and smoothness as established in Proposition D.7.

Note that without steep regularization it is impossible to lower bound the suboptimality in terms of reward distances in R S×A / U (Proposition 3.4 doesn't hold). However, we still have the following upper bound.

Proposition D.10. Consider an arbitrary regularization and let µ ∈ RL ( r ) , µ ′ ∈ RL ( r ′ ) . Then,

Proof. Let r ′′ := argmin ˜ r ∈ [ r ′ ] U ∥ ˜ r -r ∥ ∞ , then the following holds

Here, ( i ) follows from the triangle inequality and optimality of µ ′ , ( ii ) from | max f -max g | ≤ max | f -g | and simplifying, ( iii ) from Hölder's inequality, and ( iv ) from the definition of r ′′ and the quotient norm.

D.7 Perturbation bounds

Next, we provide a bound quantifying the change in the quotient norm when changing the generating subspace.

Proposition D.11. Consider x, y ∈ R n and two subspaces V , W ⊂ R n of dimension m<n . Then,

where ∥ Π W -Π V ∥ = sin ( θ max ( V , W )) .

Proof. The result follows from the triangle inequality and the definition of the spectral norm:

Furthermore, for a proof of ∥ Π W -Π V ∥ = sin ( θ max ( V , W )) we refer to [Drmac, 2000].

The following proposition shows that the maximal principal angle between two transition laws can be upper bounded by the spectral norm difference of the transition laws.

Proposition 3.9. Let P, P ′ ∈ ∆ S×A S and H γ = 1 / (1 -γ ) . Then, we have θ 1 ( P, P ′ ) = 0 and sin ( θ max ( P, P ′ )) ≤ γH γ √ |S| / |A| ∥ P -P ′ ∥ , where ∥·∥ denotes the spectral norm.

Before proceeding with the proof of Proposition 3.9, we need the following technical result. Proposition D.12. For any P ∈ ∆ S S×A the smallest singular value of E -γP satisfies

̸

Proof. The main idea of the proof is to use that σ min ( A ) = min x =0 ∥ Ax ∥ 2 / ∥ x ∥ 2 for any matrix A ∈ R n × m . We first lower bound σ min ( I -γP a ) = σ min ( I -γ ( P a ) ⊤ ) , where P a denotes the state transition matrix under action a . Let x ∈ R S , then we have

where the third inequality follows from

∥ ∥ Hence, σ min ( I -γP a ) ≥ (1 -γ ) / √ |S| . Therefore, we have for any x ∈ R S that ∥ ( E -γP ) x ∥ 2 2 = ∑ a ∥ ( I -γP a ) x ∥ 2 2 ≥ |A| / |S| (1 -γ ) 2 ∥ x ∥ 2 2 , which yields the desired result.

We are now ready to prove Proposition 3.9.

Proof of Proposition 3.9. The first principal angle θ 1 ( P, P ′ ) = 0 is zero as we always have 1 ⊆ U . The bound on the maximal angle follows from a well-known perturbation result for orthogonal projections. Namely, if A,B ∈ R n × m are matrices of the same rank and Π A , Π B denote the orthogonal projections onto their column span, then we have [Ji-Guang, 1987]

where A † denotes the Moore-Penrose inverse [Penrose, 1955]. Recall that U P = im( E -γP ) , sin ( θ max ( P, P ′ )) = ∥ ∥ Π U P -Π U P ′ ∥ ∥ , and by Proposition D.12

Therefore, we get

E Proof of claim in Example 3.3

We recall Example 3.3 from the main paper.

Example 3.3. We consider a two-state, two-action MDP with S = A = { 0 , 1 } , uniform initial state distribution, discount rate γ = 0 . 9 , and Shannon entropy regularization h = -H (see Appendix C). Suppose the expert reward is r E ( s, a ) = ✶ { s = 1 } and consider the transition laws, P 0 and P 1 , defined by P 0 (0 | s, a ) = 0 . 75 and P 1 (0 | s, a ) = 0 . 25 + β · ✶ { s = 0 , a = 0 } for some β ∈ [0 , 0 . 75] . Also, consider the two experts µ E P 0 = RL P 0 ( r E ) and µ E P 1 = RL P 1 ( r E ) , and suppose we recovered the reward ˆ r ( s, a ) = -r E . Then, the following holds: 1) We have SubOpt P 0 (ˆ r, µ E P 0 ) = 0 and SubOpt P 1 (ˆ r, µ E P 1 ) = O ( β ) . That is, for small β , the reward ˆ r is a good solution to the IRL problem, as both experts are approximately optimal under ˆ r . 2) The rank condition (5) between P 0 and P 1 is

satisfied for any β > 0 . 3) For a new transition law P defined by P (0 | s, a ) = ✶ { s = 1 , a = 0 } , we have SubOpt P ( r E , RL P (ˆ r )) ≈ 4 . 81 , i.e. RL P (ˆ r ) performs poorly under the experts' reward.

In the following we prove the claims 1. and 2., while 3. is computed via regularized dynamic programming [Geist et al., 2019]. 3 Furthermore, we illustrate the occupancy measure spaces corresponding to P 0 and P 1 for different β in Figure 3.

0.4

0.2

0

-0.2

-0.6

-0.4

-0.4

-0.2

0

v_2

0.4

0.2

0.2

0.4

0.6

-0.4

-0.6

Mp1 small β

Mpi large β

Consider the transition law P ′ defined by P ′ (0 | s, a ) = 0 . 25 . First, we observe that while ∥ ∥ P 0 -P ′ ∥ ∥ is large, the potential shaping spaces U P 0 and U P ′ coincide. To see this note that we have P ′ ( ·| s, a ) = P 0 ( ·| s, a ) + ∆ , where ∆ = [ -0 . 5 , 0 . 5] ⊤ . Hence, we have for any x ∈ R 2 that

where 1 n denotes the all-one vector in R n . Therefore, span( E -γP ′ ) = span( E -γP 0 ) . Moreover, we have

In light of Propositions D.10, D.11, and 3.9, this implies that

We need to show that P 0 and P 1 are satisfying the rank condition

By the same reasoning as above, we can equivalently show the rank condition for the transition laws P 0 ′ , P 1 ′ defined by P 0 ′ (0 | s, a ) = 1 and P 1 ′ (0 | s, a ) = β · ✶ { s = 0 , a = 0 } . To this end, we choose the matrix representation

where I ∈ R |S|×|S| is the identity matrix and P a 0 , P a 1 ∈ R |S|×|S| are the state transition matrices corresponding to the actions 0 , 1 , respectively. Let C = [ E -γP 0 ′ , E -γP 1 ′ ] . We have

and

It's straightforward to see that the vector [1 1 -1 -1] ⊤ lies in the kernel of C , but there is a 3 × 3 submatrix with non-zero determinant:

In other words, we have rank C = 3 for any β > 0 .

F Proof of Theorem 3.10

Theorem 3.10. Let K = 2 , θ 2 ( P 0 , P 1 ) > 0 , and suppose that Assumptions 2.1,2.2, and 3.5 hold. If SubOpt P k (ˆ r, µ E P k ) ≤ ˆ ε for k = 0 , 1 , then ˆ r is ε -transferable to P = ∆ S×A S with

The proof of Theorem 3.10 hinges on Lemma 3.6 and the following reward approximation result.

Proof of Lemma F.1. Throughout this proof, we will use the short-hand notation U k := U P k for k = 0 , 1 . Recall that since 1 ⊆ U 0 ∩ U 1 , we have θ 1 ( U 0 , U 1 ) = 0 and by assumption we also have θ 2 ( U 0 , U 1 ) > 0 , which implies that U 0 ∩ U 1 = 1 . Furthermore, since for k = 0 , 1 we can rewrite R S×A as the orthogonal sum R S×A = U k ∩ 1 ⊥ ⊕U ⊥ k ⊕ 1 , we can uniquely decompose r E -ˆ r into r E -ˆ r = x k + y k + z , where x k ∈ U k ∩ 1 ⊥ , y k ∈ U ⊥ k , z ∈ 1 , for k = 0 , 1 . Then, it holds that x 0 + y 0 = x 1 + y 1 . Since ∥ ∥ [ r E ] P k -[ˆ r ] P k ∥ ∥ P k , 2 = ∥ y k ∥ 2 , the Assumption of Lemma F.1 implies

that ∥ y k ∥ 2 ≤ ¯ ε . For the 2 -distance between the equivalence classes [ r E ] 1 and [ˆ r ] 1 the Pythagorean theorem implies that

where the upper bound follows from x 0 + y 0 = x 1 + y 1 and ∥ y k ∥ 2 ≤ ¯ ε . Next, we want to show that the maximum on the right-hand side of (27) is achieved for β 0 = β 1 = ¯ ε . To see this, note that taking inner products between u 0 and u 1 , respectively, and the equation α 0 u 0 + β 0 v 0 = α 1 u 1 + β 1 v 1 , we arrive at

which is for any choice of β k , u k , v k , k = 0 , 1 an invertible linear system of equations for α 0 , α 1 with the solutions

where ⟨ u 0 , u 1 ⟩ < 1 , due to U 0 ∩ U 1 ∩ 1 ⊥ = 0 . As the sign of ⟨ u 0 , u 1 ⟩⟨ u 1 , v 0 ⟩ and ⟨ u 0 , v 1 ⟩ can be chosen arbitrarily by an appropriate choice of v 0 , v 1 , the objective in the right-hand-side of (27) is increasing in β 0 , β 1 and hence the maximum is achieved for β 0 = β 1 = ¯ ε and α := α 0 = α 1 = ¯ ε ⟨ u 0 ,v 1 ⟩ 1 -⟨ u 0 ,u 1 ⟩ . Therefore, it holds that

Here, we took the maximum over u 1 , v 1 in ( i ) , we used that max v ∈V , ∥ v ∥ 2 =1 ⟨ v, u ⟩ = ∥ Π V u ∥ 2 in ( ii ) , and ( iii ) follows from the Pythagorean theorem. Furthermore, ( iv ) follows from u 0 ∈ 1 ⊥ and ( v ) from simplifying. In ( vi ) we then again use max v ∈V , ∥ v ∥ 2 =1 ⟨ v, u ⟩ = ∥ Π V u ∥ 2 , the definition of the second principal angle (Definition 3.8), and the fact that the first principal vectors lie in 1 . Lastly, ( vii ) follows from simplifying and ( viii ) from sin( x/ 2) 2 = (1 -cos x ) / 2 .

Proof of Theorem 3.10. As mentioned in the proof sketch in the main paper, it follows from the lower bound in Lemma 3.6 that ∥ ∥ [ r E ] U P k -[ˆ r ] U P k ∥ ∥ 2 ≤ √ 2ˆ ε/σ R . In light of Lemma F.1, this implies that for any P ∈ ∆ S S×A we have

Hence, applying the upper bound in Lemma 3.6 yields

G Proof of Theorem 3.11

Theorem 3.11. Let K = 1 , D := max r,r ′ ∈R ∥ r -r ′ ∥ 2 , and suppose that Assumptions 2.1,2.2, and 3.5 hold. If SubOpt P 0 (ˆ r, µ E ) ≤ ˆ ε , then ˆ r is ε P -transferable to P ∈ ∆ S×A S with

Proof. Similar to Theorem 3.10, it follows from Lemma 3.6 that ∥ ∥ [ r E ] U P 0 -[ˆ r ] U P 0 ∥ ∥ 2 ≤ √ 2ˆ ε/σ R . By Proposition D.11, we then have that

Hence, applying Lemma 3.6 again yields

H Estimating principal angles

Consider two full rank matrices A,B ∈ R n × m and let the columns of U A , U B ∈ R n × m form an orthonormal basis of V = im A and W = im B , respectively. Then, as discussed by [Ji-Guang, 1987] we have

where 1 ≥ σ 1 ≥ . . . ≥ σ m ≥ 0 denote the singular values of U ⊤ A U B sorted in decreasing order. Hence, given the transition matrices P 0 , P 1 , we can compute the principle angles θ i ( P 0 , P 1 ) by first computing orthonormal bases for the column spans of E -γP i , i = 1 , 2 , and then computing the singular values as described above.

Now, suppose that ˆ P 0 , ˆ P 1 are empirical estimates of P 0 , P 1 , then we have by [Ji-Guang, 1987, Theorem 3.1] the following perturbation result

where the last inequality follows from Propositions D.11 and 3.9. Hence, we can estimate sin θ i ( P 0 , P 1 ) up to an error of O ( max {∥ ∥ ∥ P 0 -ˆ P 0 ∥ ∥ ∥ , ∥ ∥ ∥ P 1 -ˆ P 1 ∥ ∥ ∥ }) .

I Proof of Theorem 4.1

Theorem 4.1. Suppose that N E = Ω ( K log( |S||A| / ˆ δ ) / ˆ ε 2 ) and H E = Ω ( log( K/ ˆ ε ) / log(1 /γ ) ) . Running Algorithm 1 for T = Ω ( K 2 / ˆ ε 2 ) iterations with step-size α = 1 / ( K √ T ) , where δ opt = O ( ˆ δ ˆ ε 2 /K 3 ) , ε opt = O (ˆ ε/K ) , N = Ω ( K log( K |S||A| / ( ˆ δ ˆ ε )) / ˆ ε 2 ) , and H = H E , it holds with probability at least 1 -ˆ δ that SubOpt P k (ˆ r, µ E P k ) ≤ ˆ ε, for k = 0 , . . . , K -1 .

The proof of Theorem 4.1 is inspired by [Syed and Schapire, 2007, Theorem 2]. However, in contrast to Syed and Schapire [2007], we consider the regularized problem with multiple experts, we use the suboptimality as the convergence metric, and we use a projected gradient descent update (instead of multiplicative weights). The proof hinges on Hoeffding's inequality and a regret bound for online gradient descent, which are provided in Theorem I.1 and I.2 below.

Theorem I.1 (Hoeffding's inequality [Hoeffding, 1963]) . Let X 0 , . . . , X M -1 be independent random variables with X l ∈ [ a, b ] and let S M := X 0 + . . . + X M -1 . Then,

Theorem I.2 (Online gradient descent [Zinkevich, 2003]) . Consider some bounded closed convex set X ⊂ R n with D := max x,x ′ ∈X ∥ x -x ′ ∥ 2 . Moreover, let Π X : R n → X be the orthogonal projection onto X . For any sequence of convex differentiable functions f 0 , . . . , f T -1 : X → R satisfying max x ∈X ∥∇ f t ( x ) ∥ 2 ≤ G , the online projected gradient descent update

with step-size α = D/ ( G √ T ) satisfies

Proof of Theorem 4.1. The proof is in three steps. First, we use Hoeffding's inequality to prove concentration of the empirical occupancy measures around the true occupancy measures. Then, we use the union bound to upper bound the probability that any of our bounds fails to hold. Finally, we prove the convergence rate of Algorithm 1 using the regret bound in Theorem I.2.

Step 1: Let D = { ( s 0 , a 0 , . . . , s H -1 , a H -1 ) } N -1 i =0 be sampled from some policy π µ and recall that the corresponding empirical occupancy measure is defined as

It will be convenient to define the truncated occupancy measure

For K data sets D 1 , . . . , D K sampled from π µ k we then have

where ( i ) follows from Hölder's inequality, ( ii ) from our definition of R as the 1-norm ball, and ( iii ) from the triangle inequality. Since ∥ µ -µ H ∥ ∞ ≤ γ H , we have I 1 ≤ γ H K . Moreover, applying Hoeffding's inequality to the M = KN independent random variables

with X i ∈ [0 , 1 /N ] , we arrive at

Hence, applying the union bound over all |S||A| components of the occupancy measure yields

Therefore, to ensure that with probability at least 1 -δ stat it holds that

it suffices to choose

This concentration result applies to both empirical occupancy measures generated from the expert data sets D E k , as well as the data sets D k,t generated by Algorithm 1.

Step 2: When analyzing Algorithm 1 there are three sources of stochasticity. The first two are due to the randomness in the data sets D E k and D k,t , and the third is due to the randomness in the forward RL algorithm, A ε opt ,δ opt P k , that upon a query with the reward r t outputs a policy π k,t such that with probability at least 1 -δ opt it holds SubOpt P k ( r t , µ π k,t ) ≤ ε opt . Let's denote the event that max r ∈R ∑ K -1 k =0 ⟨ r, µ E P k -ˆ µ D E k ⟩ > ε stat , E by E stat , E , the event that max r ∈R ∑ K -1 k =0 ⟨ r, µ π k,t -ˆ µ D k,t ⟩ > ε stat by E stat ,t , and the event that SubOpt P k ( r t , µ π k,t ) > ε opt by E opt ,k,t . Moreover, let us assume that E stat , E happens with probability at most δ stat , E , E stat ,t happens with probability at most δ stat, and E opt ,k,t happens with probability at most δ opt. By union bound, the probability of the event

that none of the above events happens is lower bounded by

Hence, to ensure that F happens with probability at least 1 -ˆ δ , it suffices to choose

Step 3: Note that we can bound ∥ g t ∥ 2 ≤ ∥ g t ∥ 1 ≤ ∑ K -1 k =0 ∥ ∥ ∥ ˆ µ D E k ∥ ∥ ∥ 1 + ∥ ˆ µ k,t ∥ 1 ≤ 2 K =: G and the diameter of R is D = 2 . Hence, given that event F happens, we can bound the suboptimalities of the

K experts under the reward, ˆ r , recovered by Algorithm 1 with stepsize α = D/ ( G √ T ) as follows

Here, the inequalities ( i ) , ( iv ) , and ( vi ) follow from the concentration bound established in step 1. Moreover, inequality ( ii ) holds since ˆ r ↦→ max µ ∈M P k ⟨ ˆ r, µ -µ E P k ⟩-¯ h ( µ )+ ¯ h ( µ E P k ) is the pointwise maximum of affine functions and therefore convex. Furthermore, ( iii ) follows from ε opt-optimality of µ k,t , ( v ) from Theorem I.2, and ( vii ) from concavity of the mapping µ k,t ↦→⟨ r, µ k,t -µ E P k ⟩-¯ h ( µ k,t ) . Finally, ( viii ) holds because all experts are optimal for the reward r E . In conclusion, to ensure that with probability at least 1 -ˆ δ it holds that SubOpt P k (ˆ r, µ E P k ) ≤ ∑ K -1 k =0 SubOpt P k (ˆ r, µ E P k ) ≤ ˆ ε it suffices to choose T = 256 K 2 ˆ ε 2 , α = ˆ ε 16 K 2 , N = 128 K log ( 6 |S||A| / ˆ δ ) ˆ ε 2 , H = H t = log(16 K/ ˆ ε ) log(1 /γ ) , N t = 128 K log ( 1536 K 2 |S||A| / ( ˆ δ ˆ ε 2 ) ) ˆ ε 2 , δ opt = ˆ δ ˆ ε 2 768 K 3 , ε opt = ˆ ε 4 K .

J Suboptimal experts

In our problem formulation, we assumed that the K experts are optimal with respect to r E , i.e. µ E P k = RL P k ( r E ) for k = 0 , . . . , K -1 . This assumption can be weakened by requiring that

where ε mis is some misspecification error. The transferability results in Theorem 3.10 and 3.11 still apply whenever we recover a reward ˆ r such that SubOpt P k (ˆ r, RL P k ( r E )) ≤ ˆ ε . Moreover, with a straightforward modification of the proof of Theorem 4.1, it follows that with high probability Algorithm 1 recovers a reward ˆ r such that SubOpt P k (ˆ r, RL P k ( r E )) ≤ ˆ ε + 2 Kε mis . Hence, our end-to-end transferability guarantees apply with ˆ ε ← ˆ ε +2 Kε mis. However, ε mis cannot be further reduced by collecting more samples from the expert or MDP.

K Experimental details

Setup To validate our results experimentally, we are using a stochastic adaption of the WindyGridworld environment [Sutton and Barto, 2018]. 4 In particular, we consider a 6x6 grid with 4 actions (Up, Down, Left, Right), a wind direction (North, East, South, West), and a wind strength β ∈ [0 , 1] . When the agent takes an action, with probability (1 -β ) , it moves to the intended grid cell, and with probability β , the wind pushes the agent one step further in the direction of the wind. This means that the transition law is a convex combination of two laws: (1 -β ) P Gridworld + βP Wind , where P Gridworld and P Wind represent the transition laws for a deterministic Gridworld and a deterministic WindyGridworld . For our experiments, we then consider the pairs of expert transition laws P 0 β = (1 -β ) P Gridworld + βP North and P 1 β = (1 -β ) P Gridworld + βP East with β in { 0 . 01 , 0 . 1 , 0 . 5 , 1 . 0 } . As shown in Figure 4(a), the second principal angle between P 0 β and P 1 β , calculated using a singular value decomposition [Knyazev and Argentati, 2002], increases as the wind strength β increases.

Inverse reinforcement learning We observed that under a small second principal angle, the recovered reward heavily depends on both the expert reward and the reward initialization. Hence, we sample 10 independent expert rewards, each generated by first sampling a random set of 10 state-action pairs and then randomly assigning a reward of ± 1 . Using Shannon entropy regularization with τ = 0 . 3 , we then use soft policy iteration to get expert policies for each combination of expert reward and wind strength β . For each of these expert policies, we then generate expert data sets with N E ∈ { 10 3 , 10 4 , 10 5 , 10 6 } trajectories of length H = 100 . Next, we run Algorithm 1, with soft policy iteration as a subroutine, for 30 ′ 000 iterations, where rewards are initialized by sampling from a standard normal distribution. As a reward class, we choose the ∥·∥ 1 -ball with radius 10 3 (essentially unbounded), as a stepsize α = 0 . 05 for the first 15 ′ 000 iterations and α = 0 . 005 for the second half. Moreover, we sample N = 100 new trajectories of horizon H = 100 at each gradient step. Figure 4(b) illustrates the distances between the recovered ˆ r and the experts' reward r E , measured in R S×A / 1 . It is evident that the recovered reward gets closer to the experts' reward as the number of expert demonstrations increases. Moreover, we observe that the recovered reward is closer to the experts' reward when the second principal angle between the experts is larger, as expected from Lemma F.1.

Transferability We evaluate the transferability of the obtained reward by considering two new environments. First, a south wind setting P South with wind strength β = 1 , and second, a deterministic gridworld P Shifted , with cyclically shifted actions, i.e., Right → Down, Up → Right, Left → Up, Down → Left. In Figure 4(c) and (d), we illustrate the transferability in terms of SubOpt P South ( r E , RL P South (ˆ r )) and SubOpt P Shifted ( r E , RL P Shifted (ˆ r )) , respectively. We observe that for both environments the transferability improves with a larger second principal angle, thus confirming our theoretical result in Theorem 3.10. The effect is even more pronounced for the shifted environment. While confirming our results, the experiments also reveal a high sample complexity in terms of expert demonstrations. This is to be expected, as IRL aims to match the expert's empirical occupancy measure, leading to overfitting when there are not enough demonstrations [Ho and Ermon, 2016]. This issue can be mitigated by reducing the dimension of the reward class (see e.g. [Abbeel and Ng, 2004]).

(

a

)

(

b

)

(

c

)

(

d

)

0.25

β=0.01

20

β =0.1

6

9

0.20

reward distance

β=0.5

transferability

transferability

15

β =1.0

4

4

#

10

0.10

2

2

0.05

5

0.00+

0

0

0.0

0.5

1.0

103

104

105

106

103

104

105

106

103

104

105

106

β

NE

NE

NE

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: As mentioned in the abstract and introduction, our main contributions are sufficient conditions for learning transferable rewards, as given in Theorem 3.10 and 3.11. Furthermore, we also provide an algorithm and experiments in Section 4 and 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in the Limitations and future work paragraph in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes, all assumptions are stated in the main paper, and proofs are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All details about our experiments are provided in Appendix K. Furthermore, we also provide the code attached as a .zip file.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same

dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All code (for experiments and computations in Example 3.3) is attached as a .zip file. Furthermore, we provide a README.md with instructions how to run the code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, we discuss the chosen stepsizes and parameters in Appendix K.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We repeated our experiments over 10 random seeds and provide quantile plots. Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, the hardware details are specified as a footnote in Appendix K.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [NA]

Justification: This work is purely theoretical and did not cause any harm to society.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work is purely theoretical.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We don't work with any potentially harmful models or datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The experiments are based on our own codebase.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide our code as a .zip file. We will make it openly available on github later.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This research does not involve any human experts but only synthetically generated ones.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No humans are involved in this research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyses the problem of the transferability of the reward functions learned from expert demonstrations in regularized MDPs. After having formalized the problem, the authors introduce some assumptions that permit to connect the distance between equivalence classes of reward functions to the suboptimality of the expert. Next, they use the notion of principal angles to relate the distance between transition models with the \"amount of information\" provided by demonstrations in such settings. Finally, authors propose an algorithm with theoretical guarantees for solving the problem, which they validate empirically.\n\nSTRENGTHS:\n- The paper is written in a clear manner.\n- The goal is to bring theoretical results (i.e., about known expert's policy) to the real world, by analysing the setting with demonstrations.\n- The transferability issue is one of the peculiar advantages of IRL over, for instance, BC, thus it is interesting.\n- The contributions are based on technically solid theoretical results (e.g., authors provide sample complexity analysis of the algorithm as well as convergence analysis).\n\nWEAKNESSES:\n- The framework of regularized MDPs makes things much easier w.r.t. the setting of common MDPs.\n- There are various assumptions along the way (which seem necessary).\n- Regularized MDPs do not represent a realistic model for expert behavior.\n\nQUESTIONS:\nquestions:\n- Assumption 2.2 is rather strong, because you assume that the transition model is such that, for any arbitrary policy, each state is visited with a fixed non-zero probability. Do you agree?\n- As you comment in Remark 3.12, it seems clear that all the results depend a lot in how different the considered regularized MDP differs from a common MDP. Since common MDPs are more realistic models to model experts behavior, how limited do you think that your results are when applied to practical applications?\n\nminors:\n- the references should be made aligned with each other, and not like now where some have DOI code, others do not, some have volume, others number of pages, etc...\n- the checklist shall be put at the end of the appendix\n\ntypos:\n- line 130 the operation of difference between equivalence classes is not defined\n- line 140 it is not clear what is the difference between $N^E$ (which is not defined) and $K$\n- line 245 the simplex symbol has $\\mathcal{S}$ and $\\mathcal{S}\\times\\mathcal{A}$ reversed.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper establishes bounds on the reward transferability for inverse reinforcement learning in discrete MDPs with known transition matrix. It is well known that the reward function learned in a single environment is in general not transferable as the reward shaping potential can not be identified and therefore, the recovered reward function may perform poorly under different dynamics. Prior work showed conditions for recovering the true reward function (up to a constant) when observing multiple experts acting in MDPs with different dynamics. Probably most relevant, Rolland et al. (2022) showed that reward identifiability is ensured under some rank conditions on a matrix constructed based on the transition matrixes. However, their analysis assumes that the IRL problem can be accurately solved in the different environments.\n\n* The submission extends these previous results by allowing for an error in the recovery of the reward function. Namely, the paper derives a bound based on the principal angles between the subspaces of potential shaping transformations corresponding to the different transformation matrices of the experts. That is, the second largest principle angle between the two observed MDPs, in combination with the optimality gap of the recovered reward function, is used to bound the optimality gap of the optimal policy (wrt the recovered reward function) under any other transition matrix.\n\n* Similarly, the paper uses the principle angle to establish a bound on the optimality gap on a given environment, when the reward was recovered in a different environment.\n\n* The theoretical results are accompanied with empirical results on a gridworld environment that show the relation between the principle angle and the error of the recovered reward function. The results use projected gradient descent to optimize the reward function based on demonstration under different system dynamics.\n\nSTRENGTHS:\n**Originality**\n\nUsing prinicpal angles for bounding reward transferability seems to be a novel and useful concept. The previous rank conditions only provided a binary criterium usable for perfect conditions, whereas the principal angles enable us to relate approximation errors to the transferability of the recovered reward.\n\n**Significance**\n\nThe theoretical results could be a step towards better understanding reward transferability also for more realistic settings (e.g. non-finite MDPs, unknown transition matrices). Reward transferability is an important open and critical question in inverse reinforcement learning. Indeed, arguably there is little additional value in a non-transferable reward function compared to an imitation learning policy.   \n\n**Clarity**\n\nOverall, I find the paper well-written and easy to follow. The paper gives a good overview of the work and the theoretical results and assumptions are clear. I also like the geometrical proof Fig 1b a lot. In general the paper avoids unnecessary complications.  \n\n**Quality**\n\nThe assumptions (e.g. steepness & convexity of the regularizer and Lipschitz continuity of its gradient, full support for every state-occupancy measure) are reasonable.\n\nThe claims are well-supported and seem correct (I did not check all derivations in detail).\n\nWEAKNESSES:\n**Significance**\n\nThe paper builds on the concept of principle angles between different subspaces, which can be only computed in simple settings (finite MDPs with known transition functions). While I can imagine that concepts could be transferred to continuous MDPs and to settings where approximations based on transition samples are necessary, it is not fully clear if and how such extension could be accomplished.\n\n**Clarity** \n\nWhile overall the paper is clear, I think that the paper should provide some intution about its most relevant concepts. Namely, I am missing an intuition about the subspaces of potential shaping transformations $\\text{Im}(E - \\gamma T^\\top)$, which in turn would be useful for getting an intuition about the  principal angles of interest. \n\n**Quality**\n\nThe paper mainly considers the setting when learning with demonstrations from K=2 different transition dynamics. The paper proposes a trivial extension of the bound for K>2, that merely uses the tightest bound among all combinations of two different dynamics. However, I think it can be safely assumed that much better transferability can be guaranteed when learning in K>2 enviroments, as these could complement each other.\n\nThe bounds are quite loose as evident from Corollary 3.11 in the case of Shannon- and Tsallis-entropy regularization. In particular the term $\\exp(H_\\gamma)$ should easily explode for the Shannon-Entropy, and similarly the term $H_\\gamma^\\top$ for the Tsallis-entropy, where the term $H_\\gamma$ can easily be in the range of hundreds or thousands for reasonable discount factors. However, I give credit for the fact that the paper mentions this limitation, and at least the large bounds can be somewhat compensated by very small approximation errors (also recovering previous results for $\\hat{\\epsilon}=0$).\n\nQUESTIONS:\n- Line 133 states that the paper considers equivalence relations induced by constant shifts and the \"aforementioned potential shaping transformations\". Do these include all general potential shaping transformations?\n\n- Could we use the principle angle to design experiments in order to learn transferable reward functions?\n\n- Any ideas on how the angles could transfer to continuous MDPs?\n\n- How could we estimate them from transition samples?\n\n- Could the angles be generalized for K>2?\n\nLIMITATIONS:\nLimitations and potential negative societal impact have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the logical framework regarding rewards identification (up to potential shaping transformations, which can be achieved under entropy regularization) and rewards transferability (up to a constant, more difficult to achieve). The paper considers a practical scenario where only expert demonstrations are accessible, not the full expert policy, and analyzes $\\epsilon$-transferability using the principal angles technique. Additionally, when only expert demonstrations are available, they analyze two cases: for two experts, transferring to an arbitrary transition law requires the two laws (from two experts) to be sufficiently different; for a single expert, transferring from the original to the target environment requires the transition laws to be closely related. Finally, they present a probably approximately correct (PAC) algorithm and an end-to-end analysis for learning transferable rewards from multiple expert demonstrations.\n\nSTRENGTHS:\n1. I have particularly appreciated the logical framework regarding rewards identification (up to potential shaping transformations, which can be achieved under entropy regularization) and rewards transferability (up to a constant). I also value the introduction of two methods for attaining rewards transferability: first, by restricting the reward class, such as to state-only rewards, and second, by learning from multiple experts who share the same reward but have different transition laws, provided a specific rank condition is met.\n\n2. The topic selection is meaningful, exemplifying that under the approximate optimality of experts, the learned reward can perform poorly in a new environment, even if the rank condition in Equation (5) is satisfied. \n\n3. They consider a practical scenario where only expert demonstrations are accessible, not the full expert policy, and analyze $\\epsilon$-transferability using the principal angles technique. This result encompasses the scenario where the full expert policy is accessible, recovering a reward r^ for which all experts are exactly optimal, with ϵ being 0 for all experts.\n\n4. The comparison between single-expert transfer and multi-expert transfer is insightful.\n\nWEAKNESSES:\n1. This work is mainly theoretical. I think experimental validation on real-world applications could provide valuable insight into the practical aspects and challenges of transferability (if possible).\n\n2. I think the example used to describe the rank condition's insufficiency for reward transfer when $\\epsilon$ is not exactly 0 (for all experts) is not effective. As $\\beta$ tends to 0, the two transition laws become completely different, which is too extreme, despite being easy to follow and aligned with the main theorem. Maybe a toy demo with illustrations is better.\n\n3. When $\\epsilon$ is exactly 0 (recover a reward $\\hat{r}$ for which all experts are exactly optimal), the work is already established, making the theoretical contribution slightly inferior. I believe a detailed discussion of this work's theoretical innovations and how it differs from the case when the expert policy can be achieved is necessary (from a theory aspect).\n\nI will be inclined to increase my score if I appreciate your response.\n\nQUESTIONS:\n1. As the authors mentioned in Section 2, the reward in $\\mathcal{U}$ and $\\mathcal{V}$ is defined on $\\mathcal{S} \\times \\mathcal{A}$. According to Definition 3.7 (lines 237-239), how are the elements in $\\mathcal{U}_{P}, \\mathcal{U}_{P'} \\in \\mathbb{R}^{\\mathcal{S} \\times \\mathcal{A}}$ processed for inner product and modulus operations (to be a constant)? \n\n2. $\\|[r]_{\\mathcal{V}}-[r']_{\\mathcal{V}}\\|_{2}$ is small. How to define 'minus' and 'small' needs clarification, as I believe this involves the difference between two sets or two equivalence classes. A detailed explanation would be better.\n\n3. The description of the lower and upper bounds in (6) is unsatisfactory. For the recovered reward, we need to consider both the lower bound (not optimal) and the upper bound (approximately optimal). Therefore, both the lower and upper bounds should be included. \n\n4. I think the implications of Theorem 3.10 should be introduced separately according to single expert and multi experts due to the different meanings of transition law differences. For multiple-experts, the difference in transition laws pertains to the source environments. For single experts, it pertains to the difference between the source environment and the target environment.\n\nLIMITATIONS:\nThe authors have discussed the limitations and future work in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the context of inverse reinforcement learning (IRL) -- i.e. the task of recovering a reward based on demonstrations from experts acting (approximately) optimally with respect to that reward -- this paper studies the question of transferability to new environment dynamics: if we learn a reward function from demonstrations on environments with dynamics P1 and P2, when can we expect for this reward to lead to the optimal policy on a third environment with dynamics P3? The authors show that previous results guaranteeing transferability do not generalize to cases where we are able to learn the reward only approximately, which would be the case in most practical settings. They then provide two theoretical results giving sufficient conditions for transferability.\n\nSTRENGTHS:\n- The broader question the paper addresses (transferability in IRL) is a key question for the practical usefulness of IRL. IRL papers often claim that the recovered reward is a more generalizable representation of the goal than a policy (that could be recovered e.g. using behavioural cloning, which is usually cheaper). This paper makes a step toward putting such claims on firmer theoretical basis.\n- The paper clearly shows important limits of prior results in the area.\n- The authors prove novel theoretical results using an original method.\n- Once one grasps the necessary formalisms, the rest of the paper has a clear and easy-to-follow structure.\n- The writing is largely free of typos and other mistakes.\n\nWEAKNESSES:\nOnly major objection is the lack of pedagogical effort: I think the paper is not optimized to be read by a broader NeurIPS audience. I think it would be easy to read to someone who is (1) an expert on IRL (already a somewhat niche area) and at the same time (2) an expert on the kind of theoretical analysis used in the paper, which leaves only handful of people (I think an audience not big enough to even hold a workshop). Even someone from category (1) with a degree in math and some familiarity with similar theoretical results, I would really appreciate if authors put much more effort into explaining what are the intuitions behind the different abstract concepts that they use. E.g. I think most people active in IRL and RL have much stronger existing intuitions in thinking about policies, rather than occupancy measures, so a slight build-up toward defining stuff in terms of measures would help them. Similarly when introducing a \"strictly convex regularizer\" it may be useful to give an intuition what is the purpose of that regularizer .Often adding a single sentence in places like this would make a huge difference to many readers. Also defining the transition operator as $(Pv)(s,a) = \\sum_{s'} P(s'|s,a)v(s')$ is about as unintuitive as it gets for an otherwise so intuitive concept.\n\nQUESTIONS:\nI would appreciate more discussion on how you would expect similar results to generalize to continuous state and action spaces - you title the paper \"Towards the Transferability ...\" so it would be good to have more discussion of whether and how this simple finite linear case is a useful step toward establishing more practically useful results. This is key for assessing the importance of this paper.\n\nLIMITATIONS:\nThe results in the paper are limited to only finite state and action spaces and furthermore to recovering reward from an expert whose policy is corresponding to a particular kind of regularization. It is unclear whether this kind of regularization anyhow corresponds e.g. to how a human would demonstrate a practical task to a robot (I would say it doesn't).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}