Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch

Abstract:
We study the inverse reinforcement learning (IRL) problem under a transition dynamics mismatch between the expert and the learner. Specifically, we consider the Maximum Causal Entropy (MCE) IRL learner model and provide a tight upper bound on the learner’s performance degradation based on the `1-distance between the transition dynamics of the expert and the learner. Leveraging insights from the Robust RL literature, we propose a robust MCE IRL algorithm, which is a principled approach to help with this mismatch. Finally, we empirically demonstrate the stable performance of our algorithm compared to the standard MCE IRL algorithm under transition dynamics mismatches in both finite and continuous MDP problems.

heading: 1 Introduction
text: Recent advances in Reinforcement Learning (RL) [1, 2, 3, 4] have demonstrated impressive performance in games [5, 6], continuous control [7], and robotics [8]. Despite these successes, a broader application of RL in real-world domains is hindered by the difficulty of designing a proper reward function. Inverse Reinforcement Learning (IRL) addresses this issue by inferring a reward function from a given set of demonstrations of the desired behavior [9, 10]. IRL has been extensively studied, and many algorithms have already been proposed [11, 12, 13, 14, 15, 16].
Almost all IRL algorithms assume that the expert demonstrations are collected from the same environment as the one in which the IRL agent is trained. However, this assumption rarely holds in real world because of many possible factors identified by [17]. For example, consider an autonomous car that should learn by observing expert demonstrations performed on another car with possibly different technical characteristics. There is often a mismatch between the learner and the expert’s transition dynamics, resulting in poor performance that are critical in healthcare [18] or autonomous driving [19]. Indeed, the performance degradation of an IRL agent due to transition dynamics mismatch has been noted empirically [20, 21, 22, 23], but without theoretical guidance.
To this end, our work first provides a theoretical study on the effect of such mismatch in the context of the infinite horizon Maximum Causal Entropy (MCE) IRL framework [24, 25, 26]. Specifically, we bound the potential decrease in the IRL learner’s performance as a function of the `1-distance between the expert and the learner’s transition dynamics. We then propose a robust variant of the MCE IRL algorithm to effectively recover a reward function under transition dynamics mismatch, mitigating degradation. There is precedence to our robust IRL approach, such as [27] that employs ∗Correspondence to: Parameswaran Kamalaruban <kparameswaran@turing.ac.uk>
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
an adversarial training method to learn a robust policy against adversarial changes in the learner’s environment. The novel idea of our work is to incorporate this method within our IRL context, by viewing the expert’s transition dynamics as a perturbed version of the learner’s one.
Our robust MCE IRL algorithm leverages techniques from the robust RL literature [28, 29, 30, 27]. A few recent works [20, 21, 31] attempt to infer the expert’s transition dynamics from the demonstration set or via additional information, and then apply the standard IRL method to recover the reward function based on the learned dynamics. Still, the transition dynamics can be estimated only up to a certain accuracy, i.e., a mismatch between the learner’s belief and the dynamics of the expert’s environment remains. Our robust IRL approach can be incorporated into this research vein to further improve the IRL agent’s performance.
To our knowledge, this is the first work that rigorously reconciles model-mismatch in IRL with only one shot access to the expert environment. We highlight the following contributions:
1. We provide a tight upper bound for the suboptimality of an IRL learner that receives expert demonstrations from an MDP with different transition dynamics compared to a learner that receives demonstrations from an MDP with the same transition dynamics (Section 3.1).
2. We find suitable conditions under which a solution exists to the MCE IRL optimization problem with model mismatch (Section 3.2).
3. We propose a robust variant of the MCE IRL algorithm to learn a policy from expert demonstrations under transition dynamics mismatch (Section 4).
4. We demonstrate our method’s robust performance compared to the standard MCE IRL in a broad set of experiments under both linear and non-linear reward settings (Section 5).
5. We extend our robust IRL method to the high dimensional continuous MDP setting with appropriate practical relaxations, and empirically demonstrate its effectiveness (Section 6).

heading: 2 Problem Setup
text: This section formalizes the IRL problem with an emphasis on the learner and expert environments. We use bold notation to represent vectors. A glossary of notation is given in Appendix C.

heading: 2.1 Environment and Reward
text: We formally represent the environment by a Markov decision process (MDP) Mθ := {S,A, T, γ, P0, Rθ}, parameterized by θ ∈ Rd. The state and action spaces are denoted as S and A, respectively. We assume that |S| , |A| <∞. T : S ×S ×A → [0, 1] represents the transition dynamics, i.e., T (s′|s, a) is the probability of transitioning to state s′ by taking action a from state s. The discount factor is given by γ ∈ (0, 1), and P0 is the initial state distribution. We consider a linear reward function Rθ : S → R of the form Rθ(s) = 〈θ,φ(s)〉, where θ ∈ Rd is the reward parameter, and φ : S → Rd is a feature map. We use a one-hot feature map φ : S → {0, 1}|S|, where the sth element of φ (s) is 1 and 0 elsewhere. Our results can be extended to any general feature map (see empirical evidence in Fig. 6), but we use this particular choice as a running example for concreteness.
We focus on the state-only reward function since the state-action reward function is not that useful in the robustness context. Indeed, as [22] pointed out, the actions to achieve a specific goal under different transition dynamics will not necessarily be the same and, consequently, should not be imitated. Analogously, in the IRL context, the reward for taking a particular action should not be recovered since the quality of that action depends on the transition dynamics. We denote an MDP without a reward function by M = Mθ\Rθ = {S,A, T, γ, P0}.

heading: 2.2 Policy and Performance
text: A policy π : S → ∆A is a mapping from a state to a probability distribution over actions. The set of all valid stochastic policies is denoted by Π := {π : ∑ a π(a|s) = 1,∀s ∈ S;π(a|s) ≥ 0,∀(s, a) ∈ S ×A}. We are interested in two different performance measures of any policy π acting in the MDP Mθ: (i) the expected discounted return V πMθ := E [ ∑∞ t=0 γ
tRθ (st) | π,M ], and (ii) its entropy regularized variant V π,softMθ := E [ ∑∞ t=0 γ t {Rθ (st)− log π (at|st)} | π,M ]. The state occupancy measure of a policy π in the
MDPM is defined as ρπM (s) := (1− γ) ∑∞ t=0 γ
tP [st = s | π,M ], where P [st = s | π,M ] denotes the probability of visiting the state s after t steps by following the policy π in M . Note that ρπM (s) does not depend on the reward function. Let ρπM ∈ R|S| be a vector whose sth element is ρπM (s). For the one-hot feature map φ, we have that V πMθ = 1 1−γ ∑ s ρ π M (s)Rθ(s) = 1 1−γ 〈θ,ρ π M 〉. A policy π is optimal for the MDP Mθ if π ∈ arg maxπ′ V π ′ Mθ , in which case we denote it by π∗Mθ . Similarly, the soft-optimal policy (always unique [32]) in Mθ is defined as πsoftMθ := arg maxπ′ V π′,soft Mθ
(see Appendix D for a parametric form of this policy).
2.3 Learner and Expert
Our setting has two entities: a learner implementing the MCE IRL algorithm, and an expert. We consider two MDPs, MLθ = { S,A, TL, γ, P0, Rθ } and
MEθ = { S,A, TE , γ, P0, Rθ } , that differ only in the transition dynamics. The true reward parameter θ = θ∗ is known only to the expert. The expert provides demonstrations to the learner: (i) by following policy π∗
ME θ∗
in ME
when there is a transition dynamics mismatch between the learner and the expert, or (ii) by following policy π∗
ML θ∗
in ML otherwise. The
learner always operates in the MDP ML and is not aware of the true reward parameter and of the expert dynamics TE2, i.e., it only has access to MLθ∗\Rθ∗ . It learns a reward parameter θ and the corresponding soft-optimal policy πsoft
MLθ , based on the state occupancy measure ρ received from the
expert. Here, ρ is either ρ π∗ ME θ∗
ME or ρ
π∗ ML θ∗ ML depending on the case. Our results can be extended to the
stochastic estimate of ρ using concentration inequalities [11].
Our learner model builds on the MCE IRL [24, 25, 26] framework that matches the expert’s state occupancy measure ρ. In particular, the learner policy is obtained by maximizing its causal entropy while matching the expert’s state occupancy:
max π∈Π
E [ ∞∑ t=0 −γt log π(at|st) ∣∣∣∣ π,ML ] subject to ρπML = ρ. (1)
Note that this optimization problem only requires access to MLθ \Rθ. The constraint in (1) follows from our choice of the one-hot feature map. We denote the optimal solution of the above problem by
πsoft MLθ
with a corresponding reward parameter: (i) θ = θE , when we use ρ π∗ ME θ∗
ME as ρ, or (ii) θ = θL,
when we use ρ π∗ ML θ∗
ML as ρ. Here, the parameters θE and θL are obtained by solving the corresponding
dual problems of (1). Finally, we are interested in the performance of the learner policy πsoft MLθ in the MDP MLθ∗ . Our problem setup is illustrated in Figure 1.

heading: 3 MCE IRL under Transition Dynamics Mismatch
text: This section analyses the MCE IRL learner’s suboptimality when there is a transition dynamics mismatch between the expert and the learner, as opposed to an ideal learner without this mismatch. The proofs of the theoretical statements of this section can be found in Appendix E.

heading: 3.1 Upper bound on the Performance Gap
text: First, we introduce an auxiliary lemma to be used later in our analysis. We define the distance between the two transition dynamics T and T ′, and the distance between the two poli-
2The setting with TE known to the learner has been studied under the name of imitation learning across embodiments [33].
cies π and π′ as follows, respectively: ddyn (T, T ′) := maxs,a ‖T (· | s, a)− T ′ (· | s, a)‖1, and dpol (π, π
′) := maxs ‖π(·|s)− π′(·|s)‖1. Consider the two MDPs Mθ = {S,A, T, γ, P0, Rθ} and M ′θ = {S,A, T ′, γ, P0, Rθ}. We assume that the reward function is bounded, i.e., Rθ (s) ∈ [ Rminθ , R max θ ] ,∀s ∈ S. Also, we define the following two constants: κθ :=√
γ ·max { Rmaxθ + log |A| ,− log |A| −Rminθ } and |Rθ|max := max {∣∣Rminθ ∣∣ , |Rmaxθ |}. Lemma 1. Let π := πsoftMθ and π ′ := πsoftM ′θ be the soft optimal policies for the MDPs Mθ and M ′θ respectively. Then, the distance between π and π ′ is bounded as follows: dpol (π′, π) ≤
2 min
{ κθ √ ddyn(T ′,T )
(1−γ) , κ2θddyn(T ′,T) (1−γ)2
} .
The above result is obtained by bounding the KL divergence between the two soft optimal policies, and involves a non-standard derivation compared to the well-established performance difference theorems in the literature (see Appendix E.1). The lemma above bounds the maximum total variation distance between two soft optimal policies obtained by optimizing the same reward under different transition dynamics. It serves as a prerequisite result for our later theorems (Theorem 1 for soft optimal experts and Theorem 6). In addition, it may be a result of independent interest for entropy regularized MDP.
Now, we turn to our objective. Let π1 := πsoftMLθL be the policy returned by the MCE IRL algorithm when there is no transition dynamics mismatch. Similarly, let π2 := πsoftMLθE be the policy returned by the MCE IRL algorithm when there is a mismatch. Note that π1 and π2 are the corresponding solutions to the optimization problem (1), when ρ ← ρ π∗ ML θ∗
ML and ρ ← ρ
π∗ ME θ∗ ME , respectively. The
following theorem bounds the performance degradation of the policy π2 compared to the policy π1 in the MDP MLθ∗ , where the learner operates on: Theorem 1. The performance gap between the policies π1 and π2 on the MDP MLθ∗ is bounded as follows: ∣∣∣V π1ML θ∗ − V π2 ML θ∗
∣∣∣ ≤ γ·|Rθ∗ |max(1−γ)2 · ddyn (TL, TE). The above result is obtained from the optimality conditions of the problem (1), and using Theorem 7 from [34]. In Section 4.4, we show that the above bound is indeed tight. When the expert policy is soft-optimal, we can use Lemma 1 and Simulation Lemma [35, 36] to obtain an upper bound on the performance gap (see Appendix E.2). For an application of Theorem 1, consider an IRL learner that first learns a simulator of the expert environment, and then matches the expert behavior in the simulator. In this case, our upper bound provides an estimate (sufficient condition) of the accuracy required for the simulator.

heading: 3.2 Existence of Solution under Mismatch
text: The proof of the existence of a unique solution to the optimization problem (1), presented in [37], relies on the fact that both expert and learner environments are the same. This assumption implies that the expert policy is in the feasible set that is consequently non-empty. Theorem 2 presented in this section poses a condition under which we can ensure that the feasible set is non-empty when the expert and learner environments are not the same.
Given ML and ρ, we define the following quantities useful for stating our theorem. We define, for each state s ∈ S , the probability flow matrix F (s) ∈ R|S|×|A| as follows: [F (s)]i,j := ρ(s)TLsi,s,aj , where TLsi,s,aj := T
L(si|s, aj) for i = 1, . . . , |S| and j = 1, . . . , |A|. Let B(s) ∈ R|S|×|A| be a row matrix that contains only ones in row s and zero elsewhere. Then, we define the matrix T ∈ R2|S|×|S||A| by stacking the probability flow and the row matrices as follows:
T :=
[ F (s1) F (s2) . . . F (s|S|)
B(s1) B(s2) . . . B(s|S|)
] . In addition, we define the vector v ∈ R2|S| as fol-
lows: vi = ρ(si)− (1− γ)P0(si) if i ≤ |S|, and 1 otherwise. Theorem 2. The feasible set of the optimization problem (1) is non-empty iff the rank of the matrix T is equal to the rank of the augmented matrix (T |v).
The proof of the above theorem leverages the fact that the Bellman flow constraints [15] must hold for any policy in an MDP. This requirement leads to the formulation of a linear system whose solutions
set corresponds to the feasible set of (1). The Rouché-Capelli theorem [38][Theorem 2.38] states that the solutions set is non-empty if and only if the condition in Theorem 2 holds. We note that the construction of the matrix T does not assume any restriction on the MDP structure since it leverages only on the Bellman flow constraints. Theorem 2 allows us to develop a robust MCE IRL scheme in Section 4 by ensuring the absence of duality gap. To this end, the following corollary provides a simple sufficient condition for the existence of a solution under transition dynamics mismatch.
Corollary 1. Let |A| > 1. Then, a sufficient condition for the non-emptiness of the feasible set of the optimization problem (1) is given by T being full rank.

heading: 3.3 Reward Transfer under Mismatch
text: Consider a classM of MDPs such that it contains both the learner and the expert environments, i.e., ML,ME ∈M (see Figure 2). We are given the expert’s state occupancy measure ρ = ρ π∗ ME θ∗
ME ;
but the expert’s policy π∗ ME θ∗ and the MDP ME are unknown. Further, we assume that every MDP M ∈M satisfies the condition in Theorem 2. We aim to find a policy πL that performs well in the MDP MLθ∗ , i.e., V πL
ML θ∗
is high. To this end, we
can choose any MDP M train ∈M, and solve the MCE IRL problem (1) with the constraint given
by ρ = ρπMtrain . Then, we always obtain a reward parameter θ train s.t. ρ = ρ
πsoft Mtrain θtrain Mtrain , since M train
satisfies the condition in Theorem 2. We can use this reward parameter θtrain to learn a good policy πL in the MDP ML
θtrain , i.e., πL := π∗
ML θtrain or πL := πsoft ML θtrain . Using Lemma 1, we obtain a bound
on the performance gap between πL and π1 := πsoftMLθL (see Theorem 6 in Appendix E.4).
However, there are two problems with this approach: (i) it requires access to multiple environments M train, and (ii) unless M train happened to be closer to the expert’s MDP ME , we cannot recover the true intention of the expert. Since the MDP ME is unknown, one cannot compare the different reward parameters θtrain’s obtained with different MDPs M train’s. Thus, with θtrain, it is impossible to ensure that the performance of πL is high in the MDP MLθ∗ . Instead, we try to learn a robust policy πL over the class M, while aligning with the expert’s
occupancy measure ρ, and acting only in ML. By doing this, we ensure that πL performs reasonably well on any MDP Mθ∗ ∈M including MLθ∗ . We further build upon this idea in the next section.

heading: 4 Robust MCE IRL via Two-Player Markov Game

heading: 4.1 Robust MCE IRL Formulation
text: This section focuses on recovering a learner policy via MCE IRL framework in a robust manner, under transition dynamics mismatch, i.e., ρ = ρ πsoft ME θ∗
ME in Eq. (1). In particular, our learner policy matches
the expert state occupancy measure ρ under the most adversarial transition dynamics belonging to a set described as follows for a given α > 0: T L,α := { αTL + (1− α)T̄ ,∀T̄ ∈ ∆S|S,A } , where ∆S|S,A is the set of all the possible transition dynamics T : S × S × A → [0, 1]. Note that the set T L,α is equivalent to the (s, a)-rectangular uncertainty set [28] centered around TL, i.e., T L,α = { T : ddyn ( T, TL ) ≤ 2(1− α) } . We need this set T L,α for establishing the equivalence between robust MDP and action-robust MDP formulations. The action-robust MDP formulation allows us to learn a robust policy while accessing only the MDP ML.
We define a class of MDPs as follows: ML,α := {{ S,A, TL,α, γ, P0 } ,∀TL,α ∈ T L,α } . Then, based on the discussions in Section 3.3, we propose the following robust MCE IRL problem:
max πpl∈Π min M∈ML,α
E [ ∞∑ t=0 −γt log πpl(at|st) ∣∣∣∣ πpl,M ] subject to ρπ pl M = ρ (2)
The corresponding dual problem is given by:
min θ max πpl∈Π min M∈ML,α
E [ ∞∑ t=0 −γt log πpl(at|st) ∣∣∣∣ πpl,M ] + θ> ( ρπ pl M − ρ )
(3)
In the dual problem, for any θ, we attempt to learn a robust policy over the classML,α with respect to the entropy regularized reward function. The parameter θ plays the role of aligning the learner’s policy with the expert’s occupancy measure via constraint satisfaction.

heading: 4.2 Existence of Solution
text: We start by formulating the IRL problem for any MDP ML,α ∈ ML,α, with transition dynamics TL,α = αTL + (1− α)T̄ ∈ T L,α, as follows:
max πpl∈Π
E [ ∞∑ t=0 −γt log πpl(at|st) ∣∣∣∣ πpl,ML,α ] subject to ρπ pl ML,α = ρ (4)
By introducing the Lagrangian vector θ ∈ R|S|, we get:
max πpl∈Π
E [ ∞∑ t=0 −γt log πpl(at|st) ∣∣∣∣ πpl,ML,α ] + θ> ( ρπ pl ML,α − ρ )
(5)
For any fixed θ, the problem (5) is feasible since Π is a closed and bounded set. We define U(θ) as the value of the program (5) for a given θ. By weak duality, U(θ) provides an upper bound on the optimization problem (4). Consequently, we introduce the dual problem aiming to find the value of θ corresponding to the lowest upper bound, which can be written as
min θ U(θ) := max πpl∈Π E [ ∞∑ t=0 −γt log πpl(at|st) ∣∣∣∣ πpl,ML,α ] + θ> ( ρπ pl ML,α − ρ ) . (6)
Given θ, we define πpl,∗ := πsoft ML,αθ . Due to [32][Theorem 1], for any fixed ML,αθ , the policy π pl,∗ exists and it is unique. We can compute the gradient3 ∇θU = ρπ pl,∗
ML,α − ρ, and update the parameter via gradient descent: θ ← θ −∇θU . Note that, if the condition in Theorem 2 holds, the feasible set of (4) is non-empty. Then, according to [37][Lemma 2], there is no duality gap between the programs (4) and (6). Based on these observations, we argue that the program (2) is well-posed and admits a unique solution.

heading: 4.3 Solution via Markov Game
text: In the following, we outline a method (see Algorithm 1) to solve the robust MCE IRL dual problem (3). To this end, for any given θ, we need to solve the inner max-min problem of (3). First, we express the entropy term E [∑∞ t=0−γt log πpl(at|st) ∣∣πpl,M] as follows:∑ s∈S ρπ pl M (s) ∑ a∈A { −πpl(a|s) log πpl(a|s) } = ∑ s∈S ρπ pl M (s)H πpl (A | S = s) = ( Hπ pl )> ρπ pl M ,
whereHπ pl ∈ R|S| a vector whose sth element is the entropy of the player policy given the state s. Since the quantityHπ pl
+ θ depends only on the states, to solve the dual problem, we can utilize the equivalence between the robust MDP [28, 29] formulation and the action-robust MDP [30, 27, 40] formulation shown in [27]. We can interpret the minimization over the environment class as the
3In Appendix F.2, we proved that this is indeed the gradient update under the transition dynamics mismatch.
Algorithm 1 Robust MCE IRL via Markov Game Input: opponent strength 1− α Initialize: player policy πpl, opponent policy πop, and parameter θ while not converged do
compute ραπ pl+(1−α)πop
ML by dynamic programming [37][Section V.C]. update θ with Adam [39] using the gradient ( ρ απpl+(1−α)πop ML − ρ ) .
use Algorithm 2 with R = Rθ to update πpl and πop s.t. they solve the problem (9). end while Output: player policy πpl
minimization over a set of opponent policies that with probability 1− α take control of the agent and perform the worst possible move from the current agent state. Indeed, interpreting ( Hπ pl + θ )> ρπ pl
M
as an entropy regularized value function, i.e., θ as a reward parameter, we can write:
max πpl∈Π min M∈ML,α
( Hπ pl + θ )> ρπ pl
M = max πpl∈Π min T̄
E [ G ∣∣ πpl, P0, αTL + (1− α)T̄ ] (7)
≤ max πpl∈Π min πop∈Π
E [ G ∣∣ απpl + (1− α)πop,ML] , (8)
where G := ∑∞ t=0 γ t { Rθ(st) +H πpl (A | S = st) }
. The above inequality holds due to the derivation in section 3.1 of [27]. Further details are in Appendix F.1.
Finally, we can formulate the problem (8) as a two-player zero-sum Markov game [41] with transition dynamics given by T two,L,α(s′|s, apl, aop) = αTL(s′|s, apl) + (1− α)TL(s′|s, aop), where apl is an action chosen according to the player policy and aop according to the opponent policy. Note that the opponent is restricted to take the worst possible action from the state of the player, i.e., there is no additional state variable for the opponent. As a result, we reach a two-player Markov game with a regularization term for the player as follows:
arg max πpl∈Π min πop∈Π
E [ G ∣∣ πpl, πop,M two,L,α] , (9)
where M two,L,α = { S,A,A, T two,L,α, γ, P0, Rθ } is the two-player MDP associated with the above game. The repetition of the action space A denotes the fact that player and adversary share the same action space. Inspired from [42], we propose a dynamic programming approach to find the player and opponent policies (see Algorithm 2 in Appendix F.3).

heading: 4.4 Performance Gap of Robust MCE IRL
text: Let πpl be the policy returned by our Algorithm 1 when there is a transition dynamics mismatch. Recall that π1 := πsoftMLθL is the policy recovered without this mismatch. Then, we obtain the following upper-bound4 for the performance gap of our algorithm via the triangle inequality: Theorem 3. The performance gap between the policies π1 and πpl on the MDP MLθ∗ is bounded as follows: ∣∣∣V π1ML θ∗ − V πpl ML θ∗ ∣∣∣ ≤ |Rθ∗ |max(1−γ)2 · {γ · ddyn (TL, TE)+ 2 · (1− α)}.
However, we now provide a constructive example, in which, by choosing the appropriate value for α, the performance gap of our Algorithm 1 vanishes. In contrast, the performance gap of the standard MCE IRL is proportional to the mismatch. Note that our Algorithm 1 with α = 1 corresponds to the standard MCE-IRL algorithm.
4This bound is worst than the one given in Theorem 1. When the condition in Theorem 2 does not hold, the robust MCE IRL achieves a tighter bound than the MCE IRL for a proper choice of α (see Appendix F.5).
Consider a reference MDPM ( ) = { S,A, T ( ), γ, P0 } with variable (see Figure 3). The state space is S = {s0, s1, s1}, where s1 and s2 are absorbing states. The action space is A = {a1, a2} and the initial state distribution is P0 (s0) = 1. The transition dynamics is defined as: T ( )(s1|s0, a1) = 1− , T ( )(s2|s0, a1) = , T ( )(s1|s0, a2) = 0, and T ( )(s2|s0, a2) = 1. The true reward function is given by: Rθ∗ (s0) = 0, Rθ∗ (s1) = 1, and Rθ∗ (s2) = −1. We define the learner and the expert environment as: ML := M (0) and ML := M ( E). Note that the distance between the two transition dynamics is ddyn ( TL, TE ) = 2 E . Let πpl and π2 := πsoftMLθE be the policies returned by Algorithm 1 and the MCE IRL algorithm, under the above mismatch. Recall that π1 is the policy recovered by the MCE IRL algorithm without this mismatch. Then, the following holds:
Theorem 4. For this example, the performance gap of Algorithm 1 vanishes by choosing α = 1− ddyn(T
L,TE) 2 , i.e., ∣∣∣V π1ML θ∗ − V πpl ML θ∗ ∣∣∣ = 0. Whereas, the performance gap of the standard MCE IRL is given by: ∣∣∣V π1ML θ∗ − V π2 ML θ∗
∣∣∣ = γ1−γ · ddyn(TL, TE).

heading: 5 Experiments
text: This section demonstrates the superior performance of our Algorithm 1 compared to the standard MCE IRL algorithm, when there is a transition dynamics mismatch between the expert and the learner. All the missing figures and hyper-parameter details are reported in Appendix G.
Setup. Let M refθ∗ = ( S,A, T ref , γ, P0, Rθ∗ ) be a reference MDP. Given a learner noise L ∈ [0, 1],
we introduce a learner MDP without reward function as ML, L = ( S,A, TL, L , γ, P0 ) , where TL, L ∈ ∆S|S,A is defined as TL, L := (1 − L)T ref + LT̄ with T̄ ∈ ∆S|S,A. Similarly, given an expert noise E ∈ [0, 1], we define an expert MDP ME, Eθ∗ = ( S,A, TE, E , γ, P0, Rθ∗ ) , where TE, E ∈ ∆S|S,A is defined as TE, E := (1 − E)T ref + E T̄ with T̄ ∈ ∆S|S,A. Note that a pair ( E , L) corresponds to an IRL problem under dynamics mismatch, where the expert acts in the MDP ME, E θ∗ and the learner in ML, L . In our experiments, we set T ref to be deterministic, and T̄ to be
uniform. Then, one can easily show that ddyn ( TL, L , TE, E ) = 2 ( 1− 1|S| ) | L − E |. The learned policies are evaluated in the MDP ML, L θ∗ , i.e., ML, L endowed with the true reward function Rθ∗ .
Baselines. We are not aware of any comparable prior IRL work that exactly matches our setting: (i) only one shot access to the expert environment, and (ii) do not explicitly model the expert environment. Note that Algorithm 2 in [33] requires online access to TE (or the expert environment) to empirically estimate the gradient for every (time step) adversarial expert policy π̌∗, whereas we do not access the expert environment after obtaining a batch of demonstrations, i.e., ρ. Thus, for each pair ( E , L), we compare the performance of the following: (i) our robust MCE IRL algorithm with different values of α ∈ {0.8, 0.85, 0.9, 0.95}, (ii) the standard MCE IRL algorithm, and (iii) the ideal baseline that utilizes the knowledge of the true reward function, i.e, π∗
M L, L θ∗
.
Environments. We consider four GRIDWORLD environments and an OBJECTWORLD [43] environment. All of them are N × N grid, where a cell represents a state. There are four actions per state, corresponding to steps in one of the four cardinal directions; T ref is defined accordingly. GRIDWORLD environments are endowed with a linear reward function Rθ∗(s) = 〈θ∗,φ(s)〉, where φ is a one-hot feature map. The entries θ∗s of the parameter θ
∗ for each state s ∈ S are shown in Figures 4a, 10e, 10i, and 10m. OBJECTWORLD is endowed with a non-linear reward function, determined by the distance of the agent to the objects that are randomly placed in the environment. Each object has an outer and an inner color; however, only the former plays a role in determining the reward while the latter serves as a distractor. The reward is −2 in positions within three cells to an outer blue object (black areas of Figure 4e), 0 if they are also within two cells from an outer green object (white areas), and −1 otherwise (gray areas). We shift the rewards originally proposed by [43] to non-positive values, and we randomly placed the goal state in a white area. We also modify the reward features by augmenting them with binary features indicating whether the goal state has been reached. These changes simplify the application of the MCE IRL algorithm in the infinite horizon setting. For this non-linear reward setting, we used the deep MCE IRL algorithm from [44], where the reward function is parameterized by a neural network.
Results. In Figure 4, we have presented the results for two of the environments, and the complete results can be found in Figure 10. Also, in Figure 4, we have reported the results of our algorithm with the best performing value of α; and the performance of our algorithm with different values of α are presented in Figure 11. In all the plots, every point in the x-axis corresponds to a pair ( E , L). For example, consider Figure 4b, for a fixed learner environment ML, L with L = 0, and different expert environments ME, E by varying E along the x-axis. Note that, in this figure, the distance ddyn ( TL, L , TE, E ) ∝ | L − E | increases along the x-axis. For each pair ( E , L), in the y-axis, we present the performance of the learned polices in the MDPML, L θ∗
, i.e., V π M L, L θ∗ . In alignment with our
theory, the performance of the standard MCE IRL algorithm degrades along the x-axis. Whereas, our Algorithm 1 resulted in robust performance (even closer to the ideal baseline) across different levels of mismatch. These results confirm the efficacy of our method under mismatch. However, one has to carefully choose the value of 1−α (s.t. TE, E ∈ T L,α): (i) underestimating it would lead to a linear decay in the performance, similar to the MCE IRL, (ii) overestimating it would also slightly hinder the performance, and (iii) given a rough estimate T̂E of the expert dynamics, choosing 1−α ≈ ddyn(T L,T̂E) 2 would lead to better performance in practice. The potential drop in the performance of our Robust MCE IRL method under the low expert noise regime (see Figures 4c, 4d, and 4h) can be related to the overly conservative nature of robust training. See Appendix G.3 for more discussion on the choice of 1− α. In addition, we have tested our method on a setting with low-dimensional feature mapping φ, where we observed significant improvement over the standard MCE IRL (see Appendix G.2).

heading: 6 Extension to Continuous MDP Setting
text: In this section, we extend our ideas to the continuous MDP setting, i.e., the environments with continuous state and action spaces. In particular, we implement a robust variant of the Relative Entropy IRL (RE IRL) [15] algorithm (see Algorithm 3 in Appendix H). We cannot use the dynamic programming approach to find the player and opponent policies in the continuous MDP setting. Therefore, we solve the two-player Markov game in a model-free manner using the policy gradient methods (see Algorithm 4 in Appendix H).
We evaluate the performance of our Robust RE IRL method on a continuous gridworld environment that we called GAUSSIANGRID. The details of the environment and the experimental setup are given in Appendix H. The results are reported in Figure 5, where we notice that our Robust RE IRL method outperforms standard RE IRL.

heading: 7 Related Work
text: In the context of forward RL, there are works that build on the robust MDP framework [28, 29, 45], for example, [46, 47, 48]. However, our work is closer to the line of work that leverages on the equivalence between action-robust and robust MDPs [49, 50, 30, 27, 40]. To our knowledge, this is the first work to adapt the robust RL methods in the IRL context. Other works study the IRL problem under a mismatch between the learner and the expert’s worldviews [51, 52]. However, these works do not consider the dynamics mismatch.
Generative Adversarial Imitation Learning (GAIL) [53] and its variants are IRL methods that use a GAN-based reward to align the distribution of the state-action pairs between the expert and the learner. When there is a transition dynamics mismatch, the expert’s actions are not quite useful for imitation. [54, 55] have considered state only distribution matching when the expert actions are not observable. Building on these works, [22, 23] have studied the imitation learning problem under transition dynamics mismatch. These works propose model-alignment based imitation learning algorithms in the high dimensional settings to address the dynamics mismatch. Finally, our work has the following important differences with AIRL [56]. In AIRL, the learner has access to the expert environment during the training phase, i.e., there is no transition dynamics mismatch during the training phase but only at test time. In contrast, we consider a different setting where the learner can not access the expert environment during the training phase. In addition, AIRL requires input demonstrations containing both states and actions, while our algorithm requires state-only demonstrations.

heading: 8 Conclusions
text: In this work, we theoretically analyze the MCE IRL algorithm under the transition dynamics mismatch: (i) we derive necessary and sufficient conditions for the existence of solution, and (ii) we provide a tight upper bound on the performance degradation. We propose a robust MCE IRL algorithm and empirically demonstrate its significant improvement over the standard MCE IRL under dynamics mismatch. Even though our Algorithm 1 is not essentially different from the standard robust RL methods, it poses additional theoretical challenges in the IRL context compared to the RL setup. In particular, we have proved: (i) the existence of solution for the robust MCE IRL formulation, and (ii) the performance gap improvement of our algorithm compared to the non-robust MCE IRL in a constructive example. We present empirical results for the settings not covered by our theory: MDPs with non-linear reward function and continuous state and action spaces.
Code Repository
https://github.com/lviano/RobustMCE_IRL/tree/master/robustIRLcode
Acknowledgments and Disclosure of Funding
This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement n° 725594 - time-data). Research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF-19-1-0404, by the Department of the Navy, Office of Naval Research (ONR) under a grant number N62909-17-1-2111 and by Hasler Foundation Program: Cyber Human Systems (project number 16066). This work has been supported by 2021 gift from the Schindler Group for research excellence in reinforcement learning. This work has received financial support from the Enterprise for Society Center (E4S).
Parameswaran Kamalaruban acknowledges support from The Alan Turing Institute. He carried out part of this work while at LIONS, EPFL.
Adrian Weller acknowledges support from a Turing AI Fellowship under grant EP/V025379/1, The Alan Turing Institute, and the Leverhulme Trust via CFI.

Review summaries:
This paper studies the performance degradation of a Maximum Causal Entropy learner under a transition dynamics mismatch between the expert and the learner in the imitation learning setting. The authors show that the performance of the learner is bounded by the $L_1$ distance between the transition dynamics of the expert and the learner.    Using Two-Player Markov games, the paper proposes an algorithm to robustly obtain a policy under transition dynamic mismatch and shows how it performance against MCE. 

They address the problem where the imitator should imitate an expert, but the expert's environment (MDP M^E) is unknown, and there is no access to it; instead, the imitator can just train in some M^L that may have a different dynamics (but nonetheless, we want to optimize the original experts reward -- independently of the dynamics).  They first derive some bounds on performance loss, given a bound on the deviation between the dynamics (sec3), and if a solution of the occupancy matching nonetheless exists. While one could simply train the imitator in M^L or some M^train to match the experts occupancy (under M^E), based on sec3, they propose a more systematic approach of optimizing over some worst case over possible environments (adversarial-Markov-game) to find a good imitator policy in spite of perturbation of dynamics (sec4).  Generally they assume finite state/action spaces.