Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the performance degradation of a Maximum Causal Entropy learner under a transition dynamics mismatch between the expert and the learner in the imitation learning setting. The authors show that the performance of the learner is bounded by the $L_1$ distance between the transition dynamics of the expert and the learner.  

Using Two-Player Markov games, the paper proposes an algorithm to robustly obtain a policy under transition dynamic mismatch and shows how it performance against MCE. 
main_review: The paper is trying to address an important problem in imitation learning, that expert demonstrations are obtained under different system dynamics. I think this is an interesting direction and I enjoyed reading this paper. There are substantial theoretical results in the paper, however, the significance of the paper is lowered because of the following issues: 

1. There are other papers that consider transition mismatch or "Embodiment Transfer". Mainly: "Adversarial Inverse Optimal Control for
General Imitation Learning Losses and Embodiment Transfer" by Chen et al considers settings where transition matrices are different and also the state-action space can potentially differ between the learner and the expert's demonstrations. Chen et al also consider a two player game to find a policy, however, their approach seems quite different. These methods should be compared in both literature background and experiment section. It is true that the setting is slightly different here and with Chen et al, however, experiment can show how having access to expert transition matrix can affect the performance of produced policy of each method. Moreover, comparing Robust MCE with other robust methods will benefit this work as demonstrations from transition dynamic mismatch can be seen as noisy demonstrations.   

2. The theoretical results in this paper is limited to the choice of the IRL algorithm. Is there any insight that this result could be generalized to to any method? 


## Originality

The theoretical analysis on transition dynamic mismatch is novel and I think that the new insights might be valuable for some researchers.


## Quality
Methodology and claims seem valid and sound and derivations seem correct, but I did not check them thoroughly. 


## Clarity

This work is generally well-written.

Some minor comments 
There are typos in the paper. However, I saw "returned returned" has occurred in multiple location. lines 132, 134, 252. 
Also, the choice of occupancy measure representation makes it extremely hard to read. For example lines 174 and 200. 


## Significance

The idea of bounding the performance of IRL agent with the distance between the expert and learner's transition matrix is interesting in my opinion and could be beneficial for other researchers. 

## Update after reading author response

Thank you for your response. I will keep my original score.
limitations_and_societal_impact: Limitations and societal impact was sufficiently addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper considers the problem of robust inverse reinforcement learning when the teacher and learner have different transition dynamics. The paper provides theoretical results that bound the performance gap due to the dynamics mismatch and provides empirical results justifying their proposed approach to address this dynamics mismatch.
main_review: The paper is well organized. The problem setting is interesting and seems relatively novel. 

How does this relate to Fu et al. "Learning robust rewards with adversarial inverse reinforcement learning." 2017. Here the goal is also to learn a reward function that is not tied to the dynamics such that it can transfer to very different dynamics for the learner. 

Providing theoretical bounds on performance gaps due to transition differences seems like a nice contribution. 

I was confused about the family of MDPs in 3.3 and 4.1. Where does this family (class) of MDPs come from?

---
Post Author Response

Thank you for your response and clarifications. I think this is a good paper and have updated my recommendation to an accept.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This study focuses on inverse RL under transition model mismatch between the expert and the learner. A robust version of maximum causal entropy (MCE) inverse RL is proposed and tested numerically. 
main_review: *Originality and significance*

I am not an IRL expert (only a learner), so I cannot fully assess the novelty of this work in the IRL literature. Yet, from a robust RL perspective, the connection between robust MDP and IRL is new, especially the usage of action robustness for solving the problem.
This work is well motivated and addresses an important problem of expert-learner model mismatch.  

*Clarity*
- The problem description is well conducted and rigorously solved. Still, notations may be simplified, in particular, all of the indices in l. 99, 102, 103, and notably l. 179, should be condensed to facilitate the reading. 
- There is sometimes a lack of self-containment, like in l. 162-169 where a reference with a short description is missing: what are Bellman flow constraints? What is the Rouche-Capelli theorem? Ref? What is the Simulation lemma mentioned in l.142?

*Minor comments/typos*
- l. 26 "mistmatches" --> "mismatch"
- Remove "the" - l. 20, 21, 30, 82, 103 (the concentration inequalities), 112 (the MDP), 138 (the policies; the MDP), 342
- l. 41 "to improve ... further" --> "to further improve"
- l. 43 "(Appendix B)": What is this reference doing here?
- l. 60 "denoted as" --> "denoted by"
- l. 72 "goodness" --> "quality"/ "efficiency"
- l. 78 "total expected return" --> "expected discounted return"
- l. 82: Why capital $S_t$? Inconsistent with previous notation
- l. 75: The policy definition is not incorrect but not standard. Rather define it as $\pi: \mathcal{S}\to\Delta_{\mathcal{A}}$ or $[0,1]^{\mathcal{\A}}$, to be consistent with the following remark on valid policies. 
- l. 86 "and we denote an optimal policy by" --> "in which case we denote it by"
- l. 116 "as compared to" --> "as opposed to"
- l. 132, 134: "returned returned"
- l. 141 "in [33]" --> "from [33]"
- l. 145 "provide" --> "provides"
- Fig 4: the caption should provide a take-home message. What do we learn from that figure?
- l. 340 - remove "There are"

*Quality*
1. l. 31-34 "There is precedence to our robust IRL approach": What is the difference with this work then? What is missing there that is covered here?
2. l. 101, 111, 113: Why is it written $\theta^*$ there, given that we are in the learner's MDP and it does not know the true parameter? This is quite confusing. Also, the parameter subscript sometimes appears sometimes not.
3. Besides being a prerequisite result, what does Lemma 1 tells us? How is it useful? 
4. In Fig. 4, the performance of robust MCE is lower than MCE in most of the plots, when the expert noise is low. This is not mentioned in the paper. In fact, a major drawback of robustness is its conservativeness: although more stable, robust policies can be overly conservative, especially when there is a low mismatch. This seems to appear also in robust MCE IRL, most notably in fig (h).
5. Experiments in continuous domains are announced in the abstract but missing in the text body. I would have liked it to appear (by condensing the theoretical part to satisfy space constraints). Alternatively, I would not mention it in the abstract, but rather as a side note in the experiments section.  

limitations_and_societal_impact: See item 4. above
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: They address the problem where the imitator should imitate an expert, but the expert's environment (MDP M^E) is unknown, and there is no access to it; instead, the imitator can just train in some M^L that may have a different dynamics (but nonetheless, we want to optimize the original experts reward -- independently of the dynamics).

They first derive some bounds on performance loss, given a bound on the deviation between the dynamics (sec3), and if a solution of the occupancy matching nonetheless exists. While one could simply train the imitator in M^L or some M^train to match the experts occupancy (under M^E), based on sec3, they propose a more systematic approach of optimizing over some worst case over possible environments (adversarial-Markov-game) to find a good imitator policy in spite of perturbation of dynamics (sec4).

Generally they assume finite state/action spaces.
main_review: Overall this is an interesting paper with interesting approach ideas and theoretical results. Deviations in dynamics is a relevant topic. Overall the writing is OK.

On the conceptual level, and in terms of motivating the precise problem/approach formulation, some things were not so clear for me, and potentially could be improved:
- I appreciate that sec4.1 is more systemmatic compared to the "naive" way in 3.3. But wouldn't there be a fundamentally more systematic way in abstaining from perfectly matching rho? One could imagine starting from the very basic problem formulation, i.e., there is a dynamics mismatch, and I feel this problem fromulation (together with, say, some adversarial uncertainty formulation) would give the most systematic formulation/approach, no?
- Why does the adversarial in 4.1 minimize entropy? I guess there is some rationale behind it, but one would rather expect the adversarial to make the imitator as different as possible from the expert (or the reward estimate as different as possible, respectively).
- Why is M^E unknown (l188)? At least we have the experts samples inforamtion about this environment, right?

What also remained a bit unclear is the precise contribution compared to previous work on dynamics deviation: The authors do comment on this (l130-1, etc.), and I believe it is substantial, but I didn't fully get how precisely the difference is to previous work on performance difference analysis. Or is it new to build on the existing work in RL to apply this to IRL? The related work section is also rather short on this. Also the relation to AIRL (which also studies dynamics perturbation) should be discussed (or did I miss it?).

Quality:
- I just took a brief look at the proofs -- that of thm1, to be specific. It seems correct, and the authors seem to be knowing what they are doing, but I feel things could be made a bit more specific/careful. E.g.: The 1/(1-gamma) may be missing in l666c. And it would be helpful if Theorem 7 of [33] would be restated, especially because in [33] it seems to have a slightly different form than needed here.
limitations_and_societal_impact: s.a.
review_title: null

Meta-review: This paper considers the inverse reinforcement learning setting with a mismatch between the estimated transition dynamics and the true dynamics (due to the environment being only known from state-only demonstrations and not possible to directly query). This is an important practical problem for IRL methods that the authors address rigorously by bounding the performance degradation created by the mismatch and constructing a game-theoretic algorithm for producing a robust policy in this setting. The paper's experiments show the improvement compared to obliviousness of the dynamics mismatch. The reviewers raised a number of clarification questions for the authors, that the authors did a great job addressing in their rebuttal---and I expect those response clarifications to improve the revision of the paper itself.  Given that all the final ratings at a minimum lean towards acceptance, I recommend the paper be accepted to the conference. 
Decision: Accept (Poster)