Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch
Authors: Luca Viano
Yu-Ting Huang
Parameswaran Kamalaruban
Adrian Weller
Volkan Cevher
Venue: NeurIPS 2021 Poster (2021)
Keywords: Inverse Reinforcement Learning
Imitation Learning
Robust Reinforcement Learning

Abstract: We study the inverse reinforcement learning (IRL) problem under a transition dynamics mismatch between the expert and the learner. Specifically, we consider the Maximum Causal Entropy (MCE) IRL learner model and provide a tight upper bound on the learner's performance degradation based on the $\ell_1$-distance between the transition dynamics of the expert and the learner. Leveraging insights from the Robust RL literature, we propose a robust MCE IRL algorithm, which is a principled approach to help with this mismatch. Finally, we empirically demonstrate the stable performance of our algorithm compared to the standard MCE IRL algorithm under transition dynamics mismatches in both finite and continuous MDP problems.

Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch
Luca Viano LIONS, EPFL
Yu-Ting Huang EPFL
Parameswaran Kamalaruban' The Alan Turing Institute
Adrian Weller
Volkan Cevher LIONS, EPFL
University of Cambridge & The Alan Turing Institute
Abstract
We study the inverse reinforcement learning (IRL.) problem under transition dynamics mismatch between the expert and the learner. Specifically, we consider the Maximum Causal Entropy (MCE) IRL. learner model and provide a tight upper bound on the learner's performance degradation based the fi-distance between transition dynamics the expert and learner. Leveraging insights from the Robust RL literature, we propose robust MCE IRL algorithm, which is a principled approach to help with this mismatch. Finally, we empirically demonstrate the stable performance of our algorithm compared to the standard MCE IRL algorithm under transition dynamics mismatches both finite and continuous MDP problems.
Introduction
Recent advances in Reinforcement Learning (RL) 3 have demonstrated impressive perfor- mance in games 5 continuous control (7]. and robotics ( Despite these successes, a broader application of RL in real-world domains is hindered by the difficulty of designing proper reward function. Inverse Reinforcement Learning (IRL. addresses this issue by inferring reward function J0] IRL has been extensively studied,
from a given of demonstrations of the desired behavior 99
and many algorithms have already been proposed [I]
12
13 14
15 [6].
collected from the same environment as the one which the IRL agent is trained. However, this assumption rarely holds in real world because of many possible factors identified by [17]. For example, consider autonomous car that should learn by observing expert demonstrations performed on another car with possibly different technical characteristics. There is often mismatch between the learner and the expert's transition dynamics, resulting in poor performance that are critical in healthcare [18 or autonomous driving [19]. Indeed, the performance degradation of IRL agent due transition dynamics mismatch has been noted empirically (20 21 22 23) but without theoretical guidance.
Almost all IRL algorithms assume that the expert demonstrations
To this end, our work first provides theoretical study on the effect of such mismatch in the context of the infinite horizon Maximum Causal Entropy (MCE) IRL framework 24 25 26). Specifically, bound the potential decrease in the IRL learner's performance between the expert and the learner's ' transition dynamics. We then propose a robust variant of the MCE IRL algorithm to effectively recover reward function under transition dynamics mismatch. mitigating degradation. There is precedence to our robust IRI. approach, such as [27] that employs
function of the ε fi-distance
"Correspondence to: Parameswaran Kamalaruban ckparameswaraanduuring ac.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

an adversarial training method to learn a robust policy against adversarial changes in the learner's 's environment. The novel idea of our work is incorporate this method within our IRL context, by viewing the expert's s transition dynamics as perturbed version of the learner's one.
Our robust MCE IRL algorithm leverages techniques from the robust RL literature 28
29
30 27 A
few recent works [20] 22 [1] attempt to infer the expert's 's transition dynamics from the demonstration set via additional information, and then apply the standard IRL method recover the reward function based on the learned dynamics. Still, the transition dynamics can be estimated only to a certain accuracy, i.e., mismatch between the learner's belief and the dynamics of the expert's environment remains. Our robust IRL approach can be incorporated into this research vein to further improve the IRL agent's performance.
To our knowledge, this is the first work that rigorously reconciles model-mismatch in IRL with only one shot access the expert environment. We highlight the following contributions:
1. We provide a tight upper bound for the suboptimality of an IRL learner that receives expert demonstrations from an MDP with different transition dynamics compared to learner that receives demonstrations from an MDP with the same transition dynamics (Section 3.1).
2. We find suitable conditions under which a solution exists to the MCE IRL optimization problem with model mismatch (Section 3.2)
3. We propose a robust variant of the MCE IRL. algorithm to learn a policy from expert demonstrations under transition dynamics mismatch (Section I
4. We demonstrate our method's s robust performance compared to the standard MCE IRL in broad set of experiments under both linear and non-linear reward settings (Section 5
5. We extend our robust IRL method to the high dimensional continuous MDP setting with appropriate practical relaxations, and empirically demonstrate its effectiveness (Section 6
2 Problem Setup
This section formalizes the IRL problem with an emphasis on the learner and expert environments. We use bold notation represent vectors. A glossary of notation is given in Appendix C
2.1 Environment and Reward
We formally represent the environment by a Markov decision process (MDP) Mo {S, A, T, Po. Re). parameterized by 0 E Rd. The state and action spaces are denoted as S and A. respectively. We assume that SS ||| < co. T x S x A - [0, 1] represents the transition dynamics, i.c., T(s'|s, |s, a) is the probability of transitioning state s' by taking action from state The discount factor is given by 7 E (0, 1). and Po is the initial state distribution. We consider a linear reward function Rg S + of the form Rs) = (0, cfs)), where 0 E Idd the reward parameter, and Φ: S + Rd is feature map. We use a one-hot feature map Φ S - {0, 1| where the th element of Φ (s) is and elsewhere. Our results can be extended to any general feature map (see empirical evidence in Fig. 6). but we use this particular choice as a running example for concreteness.
We focus on the state-only reward function since the state-action reward function is not that useful in the robustness context. Indeed, as (22 pointed out, the actions to achieve a specific goal under different transition dynamics will not necessarily be the same and, consequently, should not be imitated. Analogously, in the IRL context, the reward for taking particular action should not be recovered since the quality of that action depends on the transition dynamics. We denote an MDP without a reward function by M = Me Ri = A,T. Po).
2.2 Policy and Performance
A policy π S tion over actions. The set of all valid stochastic policies is denoted by II == (iT E. <[als)= = 1, Vs S; mals) 2 0, (ss E S x A). We interested in two different performance measures of any policy v acting in the MDP Mo: (i) the expected discounted return ViT = E E 7 Ro (8t) n, MJ. and (ii) its entropy regularized variant Me E 1 7 {Re (st) - log 7 (ools:)) M.. The state occupancy measure of policy in the
A is mapping from state to probability distribu-
saft ==
2

---

= MDP M is defined as (T) (=) :- (1-7) Ei=o? >' the probability of visiting the state s after t steps by following the policy π in M. Note that PM does not depend on the reward function. Let Pîs E RR|
M, where P [s+ = s π. M. denotes
be a vector whose 8th element is P (s). For
- (0, PM optimal for the MDP Mg if arg maxx Va in which case we denote it by ग TM. Similarly, the seft-optimal policy (always unique (32) in Mo is defined x = :- arg max,' V soft (see Me
the one-hot feature map Φ. we have that VM I-y
policy i
1-7
Appendix D for a parametric form of this policy).
2.3 Learner and Expert
Our setting has two entities: a learner implementing the MCE IRL algorithm, and an expert. We consider two MDPs, M
Expert
Learner
MM. Pas-
Mg- Flg.
Mg. Vag
{S,A, ,, Ro) and M {S,A,T Po. Rø). that differ only in the transition dynamies. The true reward parameter = 0* is known only to the expert. The expert provides demonstrations to the learner: (i) by following policy "isg. in M when there is a transition dynamics mismatch between the learner and the expert, or (ii) by following policy TMg. in ML otherwise. The
O2 Mr
MB Fiag.
Figure I: An illustration of the IRL problem under transition dynamics mismatch: See Section 2.
learner always operates the MDP ML and is not aware of the true reward parameter and of the expert dynamics 7 4 i.e., it only has access to My- \Rg.. It learns reward parameter 0 and the based on the state occupancy measure p received from the corresponding soft-optimal policy ı "ag.
"isg.
expert. Here, P is either pyE or P depending on the case. Our results can be extended to the stochastic estimate of p using concentration incqualities
Our learner model builds on the MCE IRL. 24 25 26 framework that matches the expert's state occupancy measure p. In particular, the learner policy is obtained by maximizing its causal entropy while matching the expert's 's state occupancy:
mAx WEIl
E
- log x (aglst))
n, Mr.
subject to
P7Ac p.
(1)
Note that this optimization problem only requires access to Mf Rg. The constraint in ( follows from our choice of the one-hot feature map. We denote the optimal solution of the above problem by
THE
soft with corresponding reward parameter: (i) 0 θg. when use ParE g as p. or (ii) 0 OL. Mg
Fiel when we use Part as p. Here, the parameters OE and O, are obtained by solving the corresponding dual problems of (T). Finally, we are interested in the performance the learner policy л soft in the "M..
MDP My. Our problem setup is illustrated in Figure
3
MCE IRL under Transition Dynamics Mismatch
This section analyses the MCE IRL learner's suboptimality when there is transition dynamics mismatch between the expert and the learner, as opposed to an idcal learner without this mismatch. The proofs of the theoretical statements of this section can be found in Appendix E
3.1 Upper bound on the Performance Gap
First, we introduce an auxiliary lemma to be used later in our analysis. We define the dis- tance between the two transition dynamics T and T', and the distance between the two poli-
'The setting with 1 known to the learner has been studied under the name of imitation learning across embodiments (33].
3

---

cies and n' as follows, respectively: dayn (T,T) == max,,a (- s.a) s.a) - s, and
dpol (T, X² == max, [[|(s x' (-1s)]l Consider the two MDPs Mn = (S.A.T, Po. Ro}
and Ma = AS 7. Po. Ro
We assume that the reward function is bounded, i.e.,
Ro (8) E Rggin Rgax vs E S. Also, we define the following two constants: K60
V max (Rg Rgpax log |A|, - log |A]
Rg Rmin) } and [Rg]
10 := max { (
Lemma 1. Let = x
soft and nO soft be the soft optimal policies for the MDPs Mo and Mo
M'
M/ respectively. Then, the distance between π and n' is bounded as follows: dpol (T's π) S (hayl("" .T) Kig sgdayn (abgn((''77)) (T' (1-9))
2 min
(1-7)2
The above result is obtained by bounding the KL divergence between the two soft optimal policies, and involves non standard derivation compared to the well-established performance difference theorems in the literature (see Appendix I..). The lemma above bounds the maximum total variation distance between two soft optimal policies obtained by optimizing the same reward under different transition dynamics. It serves as prerequisite result for our later theorems (Theorem for soft optimal experts and Theorem 6). In addition, it may be result of independent interest for entropy regularized MDP.
soft be the policy returned by the MCE IRL algorithm "Ma, when there is no transition dynamics mismatch. Similarly, let æ2 == π soft be the policy returned "
Now, we turn to our objective. Let T ==
by the MCE IRL algorithm when there is mismatch. Note that Ti and 72 are the corresponding "mg-
solutions to the optimization problem ( when p + PM and p + pM respectively. The following theorem bounds the performance degradation of the policy л2 compared to the policy ι in the MDP My. where the learner operates on:
Theorem 1. The performance gap between the policies Tı and T2 on the MDP My. is bounded as (TL,TE)
follows:
VTRa ML
I 7--Ra-1 pmax
(1-7)² ddayu
The above result is obtained from the optimality conditions of the problem ( and using Theorem 7 from (34). In Section 4.4] we show that the above bound is indeed tight. When the expert policy is soft-optimal, we can use Lemma [ and Simulation Lemma [35 36 to obtain upper bound on the performance gap (see Appendix F.22). For an application of Theorem I. consider an IRL learner that first learns simulator of the expert environment, and then matches the expert behavior in the simulator. In this case, our upper bound provides an estimate (sufficient condition) of the accuracy required for the simulator.
3.2 Existence of Solution under Mismatch
The proof of the existence of a unique solution to the optimization problem (1. presented [77 relies on the fact that both expert and learner environments are the same. This assumption implies that the expert policy is in the feasible set that is consequently non-empty. Theorem presented in this section poses condition under which we can ensure that the feasible set is non-empty when the expert and learner environments are not the same.
Given M and p. we define the following quantities useful for stating our theorem. We define, for each state s E S, the probability flow matrix F()) E R[S]x|A| as follows: [F())la, == p(8)TT s1 where TL. Mi,,,yy := (s, 0,) for i 1,.... |S] and j 1...., IAA. Let B(s) SIxÁA be row matrix that contains only ones in row s and zero elsewhere. Then, we define the matrix T R21S|I|81IA| by stacking the probability flow and the row matrices follows:
F((s) F(22)
F(sis)) In addition, define the vector v R2IS fol- B(ss))
T
B(ss) B(s2)
lows: U; = p(ss) - (1- if S S), and otherwise.
Theorem 2. The feasible set of the optimization problem ① is non-empty iff the rank of the matrix
T is equal to the rank of the augmented matrix (T
The proof of the above theorem leverages the fact that the Bellman flow constraints 15 must hold for any policy in MDP. This requirement leads to the formulation of linear system whose solutions

---

set corresponds to the feasible set of (I. The Bouché-Capelly theorem [Thorrem 2.38] states that the solutions set non-empty if and only if the condition in Theorem holds.. We note that the construction of the matrix T does not assume any restriction on the MDP structure since it leverages only on the Bellman flow constraints. Theorem alllow us to develop a robust MCE IRL scheme in Section 4 by ensuring the absence of duality gap. To this end, the following corollary provides a simple sufficient condition for the existence of a solution under transition dynamics mismatch.
Corollary 1. Let |A| > 1. Then, a sufficient condition for the non-emptiness of the feasible set of the optimization problem 1 is given by T being full rank.
3.3 Reward Transfer under Mismatch
Consider a class M of MDPs such that it contains both the learner and the expert environments,
* Mr
M E M (see Figure 2. We are given the expert's state occupancy measure p = M but the expert's policy "Agg and the MDP M are unknown. Further, we assume that every MDP M E M satisfies the condition in Theorem 2
i.e, M
We aim to find a policy x- that performs well in the MDP My., i.e., Mr..
is high. To this end, we
can choose any MDP M ttrin M, and solve the MCE IRL problem [I with the constraint given
T
Jofe
by P Then, we always obtain a reward parameter @train s.t. p= P= Maraann since Mtaain satisfies the condition in Theorem 2 We can use this reward parameter @train
garain
to learn good policy we obtain a bound
or in the MDP Mt. girnius i.e., π = "is or π == "ML Using Lemma
gtrain
gerain
on the performance gap between n² and I == x Mb.
sofs
(see Theorem in Appendix E.4
However, there two problems with this ap- proach: (i) it requires access to multiple envi- ronments Mirain and (ii) unless Miaai hap- pened to be closer to the expert's MDP M we cannot recover the true intention of the expert. Since the MDP M E is unknown, one cannot compare the different reward parameters @trains obtained with different MDPs M s. Thus, with gtrain performance of T² is high in the MDP M Instead, we try to learn robust policy n² over the class M, while aligning with the expert's occupancy measure p. and acting only in M By doing this, ensure that π performs reasonably
M
matching MDP
(for training)
M
Expert MDP
Learner MDP 4. M (for testing)
M + Class of MDPs
impossible to ensure that the
Figure 2: Illustrative example of learning a pol- icy I act in one MDP M given the expert occupancy measure p.
well on any MDP Mg. E M including M.. We further build upon this idea in the next section.
4 Robust MCE IRL via Two-Player Markov Game
4.1 Robust MCE IRL Formulation
This section focuses on recovering a learner policy via MCE IRL framework in robust manner, under
T
transition dynamics mismatch, i.e., p= Park in Eq. (1. In particular, our learner policy matches the expert state occupancy measure p under the most adversarial transition dynamics belonging to a set described as follows for given > 0: TL.a == {aT4 + a)T, vT e Asts,A) where S A. is the set of all the possible transition dynamics T S
[0, 1]. Note that the set TL.a is equivalent to the (s, recetanuula uncertainty set (28 centered around T', i.e., TL.a = {T: dayn (T, T4) < 2(1 - } We need this set 72.a for establishing the equivalence between robust MDP and action-robust MDP formulations. The action-robust MDP formulation allows us to learn robust policy while accessing only the MDP -
A
5

---

We define a class of MDPs as follows: M =
{ 77, Po
VT Tl.a). Then,
based on the discussions in Section 3.3
we propose the following robust MCE IRL problem:
max min gPl M MEM
IE
log x ( (aelse)
x M
subject
mrl PM
p
The corresponding dual problem is given by:
min max min EPET MEM
Σ
y log T (aels))
TPl, M
+ +
PMM
(P)
(3)
In the dual problem, for any 0, we attempt to learn robust policy over the class M with respect the entropy regularized reward function. The parameter plays the role of aligning the learner's policy with the expert's 's occupancy measure via constraint satisfaction.
4.2 Existence of Solution
We start by formulating the IRL problem for any MDP ML. ML.a, with transition dynamics TL.a aT + (1 a)T's E TL.a as follows:
80
max gal EII
7 log n (aslss)
TPl Mhha
subject to Pirha =
(4)
By introducing the Lagrangian vector O IRSS), we get:
max
E
y log T (arlst) TPl Miaa
+ T (priL.
(5)
Ell
1=0
For any fixed 0, the problem (5) is feasible since Il is a closed and bounded set. We define (0) as the value of the program (5) for given O. By weak duality, (0)) provides upper bound on the optimization problem ( Consequently, we introduce the dual problem aiming to find the value of corresponding to the lowest upper bound, which can be written as
min U (0)
max E
>² log Te
xPI aP),MLn
+ T
(pkiL..
(6)
xP'ell
1=
Given O, we define pll,* =
π soft a.
Due to
32 Theorem 1], for any fixed My.a, the policy pll,
exists and it unique. We can compute the gradien VaU = Pir - p. and update the parameter
via gradient descent: 0 0 VoU. Note that, if the condition in Theorem 2 holds, the feasible set of (4) is non-empty. Then, according to [Lemma 2), there is no duality gap between the programs (4) and (6). Based on these observations, argue that the program is well well-posed and admits unique solution.
Solution via Markov Game
In the following, outline method (see [lgorithm[] to solve the robust MCE IRL dual problem 3 To this end, for any given 0, we need to solve the inner max-min problem of 3 First, we express the entropy term E E 7 log nPl (ae)s)) PM M)
follows:
Σ SES
(1 log (als Pis
E (s) SES
(ASS-)
(H**))
PM
a
where R
ER vector whose th element the entropy of the player policy given the state s. Since the quantity = + A depends only on the states, to solve the dual problem, we can utilize the
equivalence between the robust MDP 28 29] formulation and the action-robust MDP (30 40)
formulation shown in [27] We can interpret the minimization over the environment class as the
In Appendix
E2
proved that this is indeed the gradient update under the transition dynamics mismatch.

---

Algorithm Robust MCE IRL via Markov Game
Input:
opponent strength 1 a
Initialize: player policy xP), opponent policy = and parameter while not converged do
an a.* =1-a)()))=" by dynamic programming [37 SSettin V.C..
compute PM
update A with Adam 39 using the gradient (PMr (22-4 - p)
use Algorithm 2 with R = Rg to update n and TP s.t. they solve the problem
end while Output: player policy nPl
minimization over a set of opponent policies that with probability 1 take control of the agent and
perform the worst possible move from the current agent state. Indeed, interpreting HHah + 0
PM
entropy regularized value function, i.e., 0 as reward parameter,
write:
max min (HT +0) Pis MEMC.
max min E GG' TP! Po, aT (1--a)1
(7)
sPEll
max min E [G ax + (1 -1-00 (1-a) " MM]. xp #"Elll
(8)
where G =i=EE=o 7¹ Ro(si) -1 (A S = se) ) The above inequality holds due to the deriva-
tion section 3.1 of 277. Further details are Appendix F.T
Finally, we can formulate the problem as a two-player zero-sum Markov game 41 with transition dynamics given by T'tw,,,,ass' apl, aPP) = aT (s'sss aPI) + (1 as)"(s')ss aP), where aPl is an action chosen according to the player policy and a according to the opponent policy. Note that the opponent is restricted take the
possible action from the state of the player, i.e., there is no additional state variable for the opponent. As a result, we reach a two-player Markov game with regularization term for the player as follows:
arg max min E [G] Tll n Ltwo zllen *"elll
Lta. where L. = A, 7, Po, Ro } is the two-player MDP associated with the above game. The repetition of the action space A denotes the fact that player and adversary share the same action space. Inspired from 42. propose dynamic programming approach find the player and opponent policies (see Algorithm 2 in Appendix F.3)
4.4 Performance Gap of Robust MCE IRL
Let Pl be the policy returned by Algorithm I when there is transition dynamics mismatch.
Recall that TT = π soft is the policy recovered without this mismatch. Then, we obtain the following aff,
upper-bound for the performance gap of algorithm via the triangle inequality:
Theorem 3. The performance gap between the policies ii and nPl on the MDP M is bounded as
follows:
VTL MM.
V
Mr.
<
[Ra= Imnx
(1-))²
{r daya (T TE) + aa)
However, we now provide constructive ex- ample, which, by choosing the appropriate value for the performance gap of our Algo- rithm I vanishes.
a1, =6
S2
SO
S
contrast, the performance gap of the standard MCE IRL is proportional to the mismatch. Note that our Algorithm with corresponds to the standard MCE-IRL algorithm.
az, P
a1 p
Figure 3: Constructive example study the per-
formance gap of Algorithm [ and the MCE IRL.
'This bound is worst than the one given in Theorem When the condition in Theorem 2 does not hold, the robust MCE IRL achieves tighter bound than the MCE IRL for proper choice of (see Appendix F.5

---

Consider reference MDP M = [S.A, y. Po} with variable e (see Figure 3 The state space is S = (8o. 81, )1 where 81 and #2 are absorbing states. The action space is A [11, (2) and the initial state distribution is Po (so) = 1. The transition dynamics is defined as: (s) (51 |s0, @1) = 1 -,, ((() (#2| a1) ) = 6, (+)) (81 80: 02) = O, and ( 02) = 1. The true reward function is given by: Ro- (80) = 0. Rg- (81)
1, and Rg- (82) -1. We define the learner and the expert
environment as: M
M and M M(+E), Note that the distance between the two transition xofft be the policies returned by Algorithm | dynamics is dayn (T4: (2)) = 2eg. Let nPl and π2 =- MMg
and the MCE IRL algorithm, under the above mismatch. Recall that тı is the policy recovered by the MCE IRL algorithm without this mismatch. Then, the following holds:
Theorem 4. For this example, the performance gap of Algorithm
? vanishes by choosing = M = O. Whereus, the performance gap of the standard MCE IRL
1 (oy 2
i.e. VT - Vasi
2
Ma-
is given by: VM V
1- ,
5
Experiments
This section demonstrates the superior performance of our Algorithm I compared to the standard MCE IRL algorithm, when there is a transition dynamics mismatch between the expert and the learner. All the missing figures and hyper- details are reported in Appendix G
Setup. Let Mg = (S, AT ,%, Ro, be a reference MDP. Given a learner noise é,. [0, 1], we introduce learner MDP without reward function as _L.e TL..1 A defined as TLe. == (1 - e + T with T E ASIS.A. Similarly, given an expert noise c 1], we define expert MDP M (S,A, ,, Po. Ro where TE E Asis.A A is defined as TE == (1 "g) + EgT with T E ASis.A- Note that a pair (CE, (L) corresponds to an IRL problem under dynamics mismatch, where the expert acts in the MDP MESE and the learner in M our experiments, we set T be deterministic, and T to be uniform. Then, one can easily show that dayu (TI..ss, "II..E,,EEEE TT = - |
AS,,, Po), where
lel Egl. The learned policies are evaluated in the MDP bi i.e., M h, endowed with the true reward function Rø-
Baselines. We are not aware of any comparable prior IRL work that exactly matches our setting: (i) only one shot access to the expert environment, and (ii) do not explicitly model the expert environment. Note that Algorithm in [33 requires online access to T (or the expert environment) to empirically estimate the gradient for every (time step) adversarial expert policy Æ", whereas we do not access the expert environment after obtaining batch of demonstrations, i.e., p. Thus, for each pair (r.,, we compare the performance of the following: (i) our robust MCE IRI. algorithm with different values of a E {0.8, 0.85, 0.9, 0.95). (ii) the standard MCE IRL, algorithm, and (iii) the ideal baseline that utilizes the knowledge of the true reward function, i.e, n "M
Environments. We consider four GRIDWORLD environments and OBJECTWORLD vironment. All of them are N x grid, where a cell represents a state. There are four actions per state, corresponding to steps in one of the four cardinal directions; Tee is defined accordingly. GRRDWORLD environments are endowed with a linear reward function Rg-(s) = (0* Φ(*)), where & is one-hot feature map. The entries 0* of the parameter 0* for each state
43 en-
S are shown in Figures 4a 10e TOi and T0m] OBJECT ORLD is endowed with a non-linear reward function, determined by the distance of the agent to the objects that are randomly placed in the environment. Each object has an outer and an inner color; however, only the former plays a role in determining the reward while the latter serves as a distractor. The reward is 2 in positions within three cells to an outer blue object (black areas of Figure 4e). O if they are also within two cells from an outer green object (white areas), and 1 otherwise (gray areas). We shift the rewards originally proposed by [43] to non-positive values, and we randomly placed the goal state in a white area. We also modify the reward features by augmenting them with binary features indicating whether the goal state has been reached. These changes simplify the application of the MCE IRL algorithm in the infinite horizon setting. For this non-linear reward setting, we used the deep MCE IRL. algorithm from 44 where the reward function is parameterized by neural network.
8

---

-150
-175
Totel Ratur
-200
Total Return
-200
Total Return
-200
-225
-1
-250
-250
-250
-300
Robusl MCE
Robust MCE: 0.85
-275
Robust MCE 0.9
expert
300
expert
expert
-100
-300
0.1
Noise
Noise
Noise
(a) GRIDWORL.D-
(b) GRW
(c) GRW 0.05
(d) GRW 0.1
Tata Return
Total Resurn
-6
Total
MCE
MGE expert
MGE
-7
-7
MCE
expert
expert
0.0
0.1 Expert Noise
0.2
0.1 Expert Noise
0.2
0.0
0.1
Expert Noise
(e) OBJECTWOORE WORLD
(f) OBW EL
OBW 0.05
(h) OBW 0.1
Figure 4: Comparison of the performance Algorithml 1 aggainst the baselines, under different levels of mismatch: (Cg, <t) E {0.0, 0.05, 0.1, 0.15, 0.2) ) x (0.0, 0.05, 0.13. Each plot corresponds to a fixed leaner environment L, with E,. {0.0, 0.05, 0.1). The values of a used for Algorithm | are reported in the legend. The vertical line indicates the position of the learner environment in the x-axis. We abbreviated the environment names GRW, and OBW. Note that our Robust MCE IRL outperforms standard MCE IRL when the expert noise increases along the x-axis. At the same time, Robust MCE IRL might perform slightly worse the low expert noise regime. This observation aligns with the overly conservative nature of robust training methods.
Results. In Figure 4 we have presented the results for two of the environments, and the complete results can be found in Figure I0, Also, Figure we have reported the results of our algorithm with the best performing value of o; and the performance of our algorithm with different values of are presented in Figure II In all the plots, every point the x-axis corresponds For example, consider Figure 4b for fixed learner environment M with 6 O, and different expert environments E by varying c along the x-axis. Note that, in this figure, the distance dayn (TLee TB. Icz cgl increases along the x-axis. For each pair (CE, CL), in the y-axis, we alignment with our
pair (Eg, (Eg,,r, Er.).
present the performance of the learned polices in the MDP M..'L, i.e., V Mr
theory, the performance of the standard MCE IRL algorithm degrades along the x-axis. Whereas, our Algorithm resulted in robust performance (even closer the ideal baseline) across different levels of mismatch. These results confirm the efficacy of our method under mismatch. However, one has to care- fully choose the value of (s.t. T F E TT.,0:: (i) underestimating it would lead a linear decay in the performance, similar to the MCE IRL, (ii) overestimating it would also slightly hinder the per-
formance, and (iii) given rough estimate 7E of the expert dynamics, choosing 1 a dayn T*) would lead to better performance in practice. The potential drop in the performance of our Robust MCE IRL. method under the low expert noise regime (see Figures 4c] 4d and 4hj can be related the overly conservative nature of robust training. See Appendix G.3 for more discussion on the choice of a. addition, we have tested our method on setting with low-dimensional feature mapping Φ, where we observed significant improvement over the standard MCE IRL (see Appendix G.2
6 Extension to Continuous MDP Setting
In this section, we extend our ideas to the continuous MDP setting, i.e., the environments with continuous state and action spaces. In particular, we implement robust variant of the Relative Entropy IRL (RE IRL) [15] algorithm (see Algorithm Appendix HH. We cannot use the dynamic programming approach to find the player and opponent policies in the continuous MDP setting. Therefore, we solve the two-player Markov game model-free manner using the policy gradient methods (see Algorithm 4 in Appendix H

---

10
Total Cont
RE
Total Cost
Total Cast
RE
10a
Robust
00,85
10a
Robust RE: 0.85
Robust RE 009
expert
expert
expert
0.0
0.2
0.15 Nosse
0.05
0.15
Noise
Noise
(a) GAUSSIANGRID
(b) M F. with er.
(c) MA with EL = 0.05
(d) M M with ez 0.1
Figure 5: Comparison of the performance our Robust RE IRL (Algorithm 3) against the standard
RE IRL, under different levels of mismatch: (EE,EL) EL) E 0..0,0.05, 0.1, 0.15..1 0.2] x {0.0, 0.05, 0.1}. Each plot corresponds
fixed leaner environment MZ, with e, (0.0, 0.05, 0.1}. The values of a used for Algorithm 3 are reported in the legend. The vertical line indicates the position of the learner environment in the x-axis. The results are averaged across seeds.
We evaluate the performance of our Robust RE IRL method on continuous gridworld environment that we called GAUSSIANGRID. The details of the environment and the experimental setup are given in Appendix H The results are reported in Figure 5 where we notice that our Robust RE IRL method outperforms standard RE IRL.
7 Related Work
In the context of forward RL, there are works that build on the robust MDP framework (28 29 45), for example, 46
However, our work is closer to the line of work that leverages on the equivalence between action-robust and robust MDPs 49 50 30 27 40) To our knowledge, this is the first work to adapt the robust RL methods in the IRL context. Other works study the IRL problem
under mismatch between the learner and the expert's worldviews [51, 52]. However, these works do not consider the dynamics mismatch.
Generative Adversarial Imitation Learning (GAIL) 53 and its variants are IRL. methods that use a GAN-based reward to align the distribution of the state-action pairs between the expert and the learner. When there is transition dynamics mismatch, the expert's actions are not quite useful for imitation. [54 (5) have considered state only distribution matching when the expert actions are observable. Building on these works, [22] (23 have studied the imitation learning problem under transition dynamics mismatch. These works propose model-alignment based imitation learning algorithms in the high dimensional settings to address the dynamics mismatch. Finally, our work has the following important differences with AIRL [56]. In AIRL, the learner has access to the expert environment during the training phase. i.c.. there is no transition dynamics mismatch during the training phase but only test time. In contrast, we consider a different setting where the learner can not access the expert environment during the training phase. In addition, AIRL requires input demonstrations containing both states and actions, while our algorithm requires state-only demonstrations.
8 Conclusions
In this work, we theoretically analyze the MCE IRL algorithm under the transition dynamics mismatch: (i) we derive necessary and sufficient conditions for the existence of solution, and (ii) we provide a tight upper bound on the performance degradation. We propose robust MCE IRL algorithm and empirically demonstrate its significant improvement over the standard MCE IRL under dynamics mismatch. Even though our Algorithm i is not essentially different from the standard robust RL methods, it poses additional theoretical challenges the IRL context compared to the RL setup. In particular, we have proved: (i) the existence of solution for the robust MCE IRL. formulation, and (ii) the performance gap improvement of our algorithm compared to the non-robust MCE IRL in a constructive example. We present empirical results for the settings not covered by our theory: MDPs with non-linear reward function and continuous state and action spaces.
Code Repository
10

---

Acknowledgments and Disclosure of Funding
This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement n" 725594 time-data). Research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF-19-1-0A0 by the Department of the Navy, Office of Naval Research (ONR) under grant number 1122909-171-221 number 16066). This work has been supported by 2021 gift from the Schindler Group for research excellence in reinforcement learning. This work has received financial support from the Enterprise for Society Center (E4S).
and by Hasler Foundation Program: Cyber Human Systems (project
Parameswaran Kamalaruban acknowledges support from The Alan Turing Institute. He carried out part of this work while at LIONS, EPFL.
Adrian Weller acknowledges support from Turing Al Fellowship under grant EP/V025379/1, The Alan Turing Institute, and the Leverhulme Trust via CFI.
References
[1] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NeurIPS), 2000.
[2] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proc. Inil Conf. on Machine Learning (ICML), 2014.
[3] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proc. Intl Conf. on Machine Learning (ICML), 2015.
[4] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:100700337, 2017.
[5] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.
[6] David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. Nature, 2017.
[7] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proc. Intl Conf. on Learning Representations (ICLR), 2016.
[8] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbecl. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 2016.
[9] Stuart Russell. Learning agents for uncertain environments. Proc. Conf. on Learning Theory (COLT), 1998.
[10] Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Proc. Intl Conf. on Machine Learning (ICML), 2000.
[11] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proc. Intl Conf. on Machine Learning (ICML), 2004.
[12] Nathan Ratliff, J Andrew Bagnell, and Martin Zinkevich. Maximum margin planning. In Proc. Intl Conf. on Machine Learning (ICML), 2006.
[13] Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind Dey. Maximum entropy inverse reinforcement learning, In Proc. AAAI Conference Artificial Intelligence, 2008.

---

[14] Umar Syed and Robert E Schapire. A game-theoretic approach to apprenticeship learning. In Advances in Neural Information Processing Systems (NeurIPS), 2008.
[15] Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In Proc. Intl Conf. on Artificial Intelligence and Statistics (AISTATS), 2011.
[16] T Osa, J Pajarinen, G Neumann, JA Bagnell, P Abbeel, and J Peters. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 2018.
[17] Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforce- ment learning. arXiv preprint arXiv: 1ekiv:1004 wee...944..9990 12901,, 2019.
[18] Chao Yu, Jiming Liu, and Shamim Nemati. Reinforcement learning in healthcare: survey. arXiv preprint arXiv: 1908.00908 2019.
[19] Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey, arXiv preprint arXiv:2002.00444 2020.
[20] Sid Reddy, Anca Dragan, and Sergey Levine. Where do you think you're re going?: Inferring beliefs about dynamics from behavior. In Advances in Neural Information Processing Systems (NeurIPS), 2018
[21] Ze Gong and Yu Zhang. What it you really want of me? generalized reward learning with biased beliefs about domain dynamics. In Proc. AAAI Conference on Artificial Intelligence, 2020.
[22] Tanmay Gangwani and Jian Peng. State-only imitation with transition dynamics mismatch. In Proc. Intl Conf. on Learning Representations (ICLR), 2020.
[23] Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment-basee imitation learning. In Proc. Inil Conf. on Learning Representations (ICLR), 2020.
[24] Brian Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. PhD thesis, Carnegie Mellon University, 2010.
[25] Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. The principle of maximum causal entropy for estimating interacting processes. IEEE Transactions Information Theory, 2013.
[26] Zhengyuan Zhou, Michael Bloem, and Nicholas Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. IEEE Transactions on Automatic Control, 2017.
[27] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In Proc. Intl Conf. on Machine Learning (ICML), 2019.
[28] Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 2005.
[29] Arna Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 2005.
[30] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In Proc. Intl Conf. on Machine Learning (ICMI), 2017.
[31] Michael Herman, Tobias Gindele, Jörg Wagner, Felix Schmitt, and Wolfram Burgard. Inverse reinforcement learning with simultancous estimation of rewards and dynamics. In Proc. Intl Conf. on Artificial Intelligence and Statistics (AISTATS), 2016.
[32] Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In Proc. Intl Conf. on Machine Learning (ICML), 2019.
[33] Xiangli Chen, Mathew Monfort, Brian D Ziebart, and Peter Carr. Adversarial inverse optimal control for general imitation learning losses and embodiment transfer. In Proc. Conf. on Uncertainty in Artificial Intelligence (UAI), 2016.
12

---

[34] Amy Zhang, Shagun Sochani, Khimya Khetarpal, and Joelle Pineau. Multi-task reinforcement learning as a hidden-parameen block mdp. arXiv preprint arXiv:2007.07206, 2020.
[35] Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynominal time. In Proc. Inil Conf. on Machine Learning (ICML), 1998.
[36] Eyal Even-Dar and Yishay Mansour. Approximate equivalence of Markov decision processes. In Learning Theory and Kernel Machines. 2003.
[37] Michael Bloem and Nicholas Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. In IEEE Conference on Decision and Control, 2014.
[38] LRR Shafarevich, A.O. Remizov, D.P. Kramer, and L. Nekludova. Linear Algebra and Geometry. Springer Berlin Heidelberg, 2014.
[39] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. Intl Conf. on Learning Representations (ICLR), 2015.
[40] Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, and Volkan Cevher. Robust reinforcement learning via adversarial training with langevin dynamics. In Advances in Neural Information Processing Systems (NeurlPS). 2020.
[41] Michael L Littman. Markov games as framework for multi-agent reinforcement learning. In Machine learning proceedings. 1994.
[42] Jordi Grau-Moya, Felix Leibfried, and Haitham Bou-Ammar. Balancing two-player stochastic games with soft q-learning. In Proc. Intl Joint Conf. on Artificial Intelligence (IJCAI), 2018.
[43] Sergey Levine, Zoran Popovic, and Vladlen Koltun. Nonlinear inverse reinforcement learning with gaussian processes. In Advances in Neural Information Processing Systems (NeurIPS), 2011.
[44] Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforcement learning, arXiv preprint arXiv: arXiv:/507.004 1557000888, 2015.
[45] Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathematics of Operations Research, 2013.
[46] Shirli Di-Castro Shashua and Shie Mannor. Deep robust kalman filter. arXiv preprint MrXiv:1703.02333 2017.
[47] Xue Bin Peng. Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In IEEE international conference on robotics and automation (ICRA), 2018
[48] Daniel J Mankowitz, Nir Levine, Rae Jeong, Abbas Abdolmaleki, Jost Tobias Springenherg, Yuanyuan Shi, Jackie Kay, Todd Hester, Timothy Mann, and Martin Riedmiller. Robust reinforcement learning for continuous control with model misspecification. In Proc. Intl Conf. on Learning Represemtations (ICLR), 2020.
[49] Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 2005.
[50] John C Doyle, Bruce A Francis, and Allen R Tannenbaum. Feedback control theory. Courier Corporation, 2013.
[51] Luis Haug, Sebastian Tschiatschek, and Adish Singla. Teaching inverse reinforcement learners via features and demonstrations. In Advances in Neural Information Processing Systems (NeurIPS), 2018
[52] Sebastian Tschiatschek, Ahana Ghosh, Luis Haug, Rati Devidze, and Adish Singla. Learner- aware teaching: Inverse reinforcement learning with preferences and constraints. In Advances Neural Information Processing Systems (NeurIPS), 2019.
[53] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
13

---

[54] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observa- tion, arXiv preprint wwXiv:1877061133 2018.
[55] Wen Sun, Anirudh Vemula, Byron Boots, and J Andrew Bagnell. Provably efficient imitation learning from observation alone. In Proc. Inil Conf. Machine Learning (ICML), 2019.
[56] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In Proc. Intl Conf. on Learning Representations (ICLR), 2018.
[57] Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun, Reinforcement Learning: Theory and Algorithms, 2019.
[58] Abdeslam Boularias and Brahim Chaib-Draa. Bootstrapping apprenticeship learning. In Advances in Neural Information Processing Systems (NeurlPS). 2010.
[59] Wen Sun, Geoffrey J Gordon, Byron Boots, and Andrew Bagnell. Dual policy iteration. In Advances Neural Information Processing Systems (NeurIPS), 2018.
[60] Parameswaran Kamalaruban, Rati Devidze, Volkan Cevher, and Adish Singla. Interactive teaching algorithms for inverse reinforcement learning. In Proc. Intl Joint Conf. Artificial Intelligence (IJCAI), 2019.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the pa- per's contributions and scope? [Yes] The paper is organized exactly according to the contributions listed at the end of the introduction section.
(b) Did you describe the limitations of your work? [Yes] Yes, highlight the fact that the worst case guarantees of the robust MCE IRL can be worst than the standard MCE IRL, and describe how a good choice for a is crucial for better performance.
(c) Did you discuss any potential negative societal impacts your work? [Yes] This work presents theoretical investigation of the imitation learning under transition dynamics mismatch (see Section S). As such in the present form there are no direct negative societal impacts of our work. However, in future, when the proposed methods are applied on real-world systems, the practitioner has to be careful with the choice of a. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We confirm that our paper conforms with the ethics review guidelines.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] For example, see Theorem 2
(b) Did you include complete proofs of all theoretical results? [Yes] Complete proofs of all the theoretical results can be found in the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URL)? [Yes] The code and instructions are included in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We have provided all the training and hyperparameters details in the Experiments section, and in the Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] This can be seen in all the Figures.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We used an internal cluster with CPU nodes for the experiments; but we do not have an estimate of the total amount of compute.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "j_p9cD2M1aU", "reviewer": "Reviewer_Zhym", "summary": "This paper studies the performance degradation of a Maximum Causal Entropy learner under a transition dynamics mismatch between the expert and the learner in the imitation learning setting. The authors show that the performance of the learner is bounded by the $L_1$ distance between the transition dynamics of the expert and the learner.  \n\nUsing Two-Player Markov games, the paper proposes an algorithm to robustly obtain a policy under transition dynamic mismatch and shows how it performance against MCE. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is trying to address an important problem in imitation learning, that expert demonstrations are obtained under different system dynamics. I think this is an interesting direction and I enjoyed reading this paper. There are substantial theoretical results in the paper, however, the significance of the paper is lowered because of the following issues: \n\n1. There are other papers that consider transition mismatch or \"Embodiment Transfer\". Mainly: \"Adversarial Inverse Optimal Control for\nGeneral Imitation Learning Losses and Embodiment Transfer\" by Chen et al considers settings where transition matrices are different and also the state-action space can potentially differ between the learner and the expert's demonstrations. Chen et al also consider a two player game to find a policy, however, their approach seems quite different. These methods should be compared in both literature background and experiment section. It is true that the setting is slightly different here and with Chen et al, however, experiment can show how having access to expert transition matrix can affect the performance of produced policy of each method. Moreover, comparing Robust MCE with other robust methods will benefit this work as demonstrations from transition dynamic mismatch can be seen as noisy demonstrations.   \n\n2. The theoretical results in this paper is limited to the choice of the IRL algorithm. Is there any insight that this result could be generalized to to any method? \n\n\n## Originality\n\nThe theoretical analysis on transition dynamic mismatch is novel and I think that the new insights might be valuable for some researchers.\n\n\n## Quality\nMethodology and claims seem valid and sound and derivations seem correct, but I did not check them thoroughly. \n\n\n## Clarity\n\nThis work is generally well-written.\n\nSome minor comments \nThere are typos in the paper. However, I saw \"returned returned\" has occurred in multiple location. lines 132, 134, 252. \nAlso, the choice of occupancy measure representation makes it extremely hard to read. For example lines 174 and 200. \n\n\n## Significance\n\nThe idea of bounding the performance of IRL agent with the distance between the expert and learner's transition matrix is interesting in my opinion and could be beneficial for other researchers. \n\n## Update after reading author response\n\nThank you for your response. I will keep my original score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact was sufficiently addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "OROtIzxFMMA", "reviewer": "Reviewer_z8oS", "summary": "This paper considers the problem of robust inverse reinforcement learning when the teacher and learner have different transition dynamics. The paper provides theoretical results that bound the performance gap due to the dynamics mismatch and provides empirical results justifying their proposed approach to address this dynamics mismatch.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well organized. The problem setting is interesting and seems relatively novel. \n\nHow does this relate to Fu et al. \"Learning robust rewards with adversarial inverse reinforcement learning.\" 2017. Here the goal is also to learn a reward function that is not tied to the dynamics such that it can transfer to very different dynamics for the learner. \n\nProviding theoretical bounds on performance gaps due to transition differences seems like a nice contribution. \n\nI was confused about the family of MDPs in 3.3 and 4.1. Where does this family (class) of MDPs come from?\n\n---\nPost Author Response\n\nThank you for your response and clarifications. I think this is a good paper and have updated my recommendation to an accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "KbPZSVvFl7W", "reviewer": "Reviewer_sUbA", "summary": "They address the problem where the imitator should imitate an expert, but the expert's environment (MDP M^E) is unknown, and there is no access to it; instead, the imitator can just train in some M^L that may have a different dynamics (but nonetheless, we want to optimize the original experts reward -- independently of the dynamics).\n\nThey first derive some bounds on performance loss, given a bound on the deviation between the dynamics (sec3), and if a solution of the occupancy matching nonetheless exists. While one could simply train the imitator in M^L or some M^train to match the experts occupancy (under M^E), based on sec3, they propose a more systematic approach of optimizing over some worst case over possible environments (adversarial-Markov-game) to find a good imitator policy in spite of perturbation of dynamics (sec4).\n\nGenerally they assume finite state/action spaces.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this is an interesting paper with interesting approach ideas and theoretical results. Deviations in dynamics is a relevant topic. Overall the writing is OK.\n\nOn the conceptual level, and in terms of motivating the precise problem/approach formulation, some things were not so clear for me, and potentially could be improved:\n- I appreciate that sec4.1 is more systemmatic compared to the \"naive\" way in 3.3. But wouldn't there be a fundamentally more systematic way in abstaining from perfectly matching rho? One could imagine starting from the very basic problem formulation, i.e., there is a dynamics mismatch, and I feel this problem fromulation (together with, say, some adversarial uncertainty formulation) would give the most systematic formulation/approach, no?\n- Why does the adversarial in 4.1 minimize entropy? I guess there is some rationale behind it, but one would rather expect the adversarial to make the imitator as different as possible from the expert (or the reward estimate as different as possible, respectively).\n- Why is M^E unknown (l188)? At least we have the experts samples inforamtion about this environment, right?\n\nWhat also remained a bit unclear is the precise contribution compared to previous work on dynamics deviation: The authors do comment on this (l130-1, etc.), and I believe it is substantial, but I didn't fully get how precisely the difference is to previous work on performance difference analysis. Or is it new to build on the existing work in RL to apply this to IRL? The related work section is also rather short on this. Also the relation to AIRL (which also studies dynamics perturbation) should be discussed (or did I miss it?).\n\nQuality:\n- I just took a brief look at the proofs -- that of thm1, to be specific. It seems correct, and the authors seem to be knowing what they are doing, but I feel things could be made a bit more specific/careful. E.g.: The 1/(1-gamma) may be missing in l666c. And it would be helpful if Theorem 7 of [33] would be restated, especially because in [33] it seems to have a slightly different form than needed here.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "s.a.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "2_9tNhkr8s", "reviewer": "Reviewer_d6zC", "summary": "This study focuses on inverse RL under transition model mismatch between the expert and the learner. A robust version of maximum causal entropy (MCE) inverse RL is proposed and tested numerically. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality and significance*\n\nI am not an IRL expert (only a learner), so I cannot fully assess the novelty of this work in the IRL literature. Yet, from a robust RL perspective, the connection between robust MDP and IRL is new, especially the usage of action robustness for solving the problem.\nThis work is well motivated and addresses an important problem of expert-learner model mismatch.  \n\n*Clarity*\n- The problem description is well conducted and rigorously solved. Still, notations may be simplified, in particular, all of the indices in l. 99, 102, 103, and notably l. 179, should be condensed to facilitate the reading. \n- There is sometimes a lack of self-containment, like in l. 162-169 where a reference with a short description is missing: what are Bellman flow constraints? What is the Rouche-Capelli theorem? Ref? What is the Simulation lemma mentioned in l.142?\n\n*Minor comments/typos*\n- l. 26 \"mistmatches\" --> \"mismatch\"\n- Remove \"the\" - l. 20, 21, 30, 82, 103 (the concentration inequalities), 112 (the MDP), 138 (the policies; the MDP), 342\n- l. 41 \"to improve ... further\" --> \"to further improve\"\n- l. 43 \"(Appendix B)\": What is this reference doing here?\n- l. 60 \"denoted as\" --> \"denoted by\"\n- l. 72 \"goodness\" --> \"quality\"/ \"efficiency\"\n- l. 78 \"total expected return\" --> \"expected discounted return\"\n- l. 82: Why capital $S_t$? Inconsistent with previous notation\n- l. 75: The policy definition is not incorrect but not standard. Rather define it as $\\pi: \\mathcal{S}\\to\\Delta_{\\mathcal{A}}$ or $[0,1]^{\\mathcal{\\A}}$, to be consistent with the following remark on valid policies. \n- l. 86 \"and we denote an optimal policy by\" --> \"in which case we denote it by\"\n- l. 116 \"as compared to\" --> \"as opposed to\"\n- l. 132, 134: \"returned returned\"\n- l. 141 \"in [33]\" --> \"from [33]\"\n- l. 145 \"provide\" --> \"provides\"\n- Fig 4: the caption should provide a take-home message. What do we learn from that figure?\n- l. 340 - remove \"There are\"\n\n*Quality*\n1. l. 31-34 \"There is precedence to our robust IRL approach\": What is the difference with this work then? What is missing there that is covered here?\n2. l. 101, 111, 113: Why is it written $\\theta^*$ there, given that we are in the learner's MDP and it does not know the true parameter? This is quite confusing. Also, the parameter subscript sometimes appears sometimes not.\n3. Besides being a prerequisite result, what does Lemma 1 tells us? How is it useful? \n4. In Fig. 4, the performance of robust MCE is lower than MCE in most of the plots, when the expert noise is low. This is not mentioned in the paper. In fact, a major drawback of robustness is its conservativeness: although more stable, robust policies can be overly conservative, especially when there is a low mismatch. This seems to appear also in robust MCE IRL, most notably in fig (h).\n5. Experiments in continuous domains are announced in the abstract but missing in the text body. I would have liked it to appear (by condensing the theoretical part to satisfy space constraints). Alternatively, I would not mention it in the abstract, but rather as a side note in the experiments section.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See item 4. above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the inverse reinforcement learning setting with a mismatch between the estimated transition dynamics and the true dynamics (due to the environment being only known from state-only demonstrations and not possible to directly query). This is an important practical problem for IRL methods that the authors address rigorously by bounding the performance degradation created by the mismatch and constructing a game-theoretic algorithm for producing a robust policy in this setting. The paper's experiments show the improvement compared to obliviousness of the dynamics mismatch. The reviewers raised a number of clarification questions for the authors, that the authors did a great job addressing in their rebuttal---and I expect those response clarifications to improve the revision of the paper itself.  Given that all the final ratings at a minimum lean towards acceptance, I recommend the paper be accepted to the conference. 
Decision: Accept (Poster)