Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch

1 Introduction . Recent advances in Reinforcement Learning ( RL ) [ 1 , 2 , 3 , 4 ] have demonstrated impressive performance in games [ 5 , 6 ] , continuous control [ 7 ] , and robotics [ 8 ] . Despite these successes , a broader application of RL in real-world domains is hindered by the difficulty of designing a proper reward function . Inverse Reinforcement Learning ( IRL ) addresses this issue by inferring a reward function from a given set of demonstrations of the desired behavior [ 9 , 10 ] . IRL has been extensively studied , and many algorithms have already been proposed [ 11 , 12 , 13 , 14 , 15 , 16 ] . Almost all IRL algorithms assume that the expert demonstrations are collected from the same environment as the one in which the IRL agent is trained . However , this assumption rarely holds in real world because of many possible factors identified by [ 17 ] . For example , consider an autonomous car that should learn by observing expert demonstrations performed on another car with possibly different technical characteristics . There is often a mismatch between the learner and the expert ’ s transition dynamics , resulting in poor performance that are critical in healthcare [ 18 ] or autonomous driving [ 19 ] . Indeed , the performance degradation of an IRL agent due to transition dynamics mismatch has been noted empirically [ 20 , 21 , 22 , 23 ] , but without theoretical guidance . To this end , our work first provides a theoretical study on the effect of such mismatch in the context of the infinite horizon Maximum Causal Entropy ( MCE ) IRL framework [ 24 , 25 , 26 ] . Specifically , we bound the potential decrease in the IRL learner ’ s performance as a function of the ` 1-distance between the expert and the learner ’ s transition dynamics . We then propose a robust variant of the MCE IRL algorithm to effectively recover a reward function under transition dynamics mismatch , mitigating degradation . There is precedence to our robust IRL approach , such as [ 27 ] that employs ∗Correspondence to : Parameswaran Kamalaruban < kparameswaran @ turing.ac.uk > 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . an adversarial training method to learn a robust policy against adversarial changes in the learner ’ s environment . The novel idea of our work is to incorporate this method within our IRL context , by viewing the expert ’ s transition dynamics as a perturbed version of the learner ’ s one . Our robust MCE IRL algorithm leverages techniques from the robust RL literature [ 28 , 29 , 30 , 27 ] . A few recent works [ 20 , 21 , 31 ] attempt to infer the expert ’ s transition dynamics from the demonstration set or via additional information , and then apply the standard IRL method to recover the reward function based on the learned dynamics . Still , the transition dynamics can be estimated only up to a certain accuracy , i.e. , a mismatch between the learner ’ s belief and the dynamics of the expert ’ s environment remains . Our robust IRL approach can be incorporated into this research vein to further improve the IRL agent ’ s performance . To our knowledge , this is the first work that rigorously reconciles model-mismatch in IRL with only one shot access to the expert environment . We highlight the following contributions : 1 . We provide a tight upper bound for the suboptimality of an IRL learner that receives expert demonstrations from an MDP with different transition dynamics compared to a learner that receives demonstrations from an MDP with the same transition dynamics ( Section 3.1 ) . 2 . We find suitable conditions under which a solution exists to the MCE IRL optimization problem with model mismatch ( Section 3.2 ) . 3 . We propose a robust variant of the MCE IRL algorithm to learn a policy from expert demonstrations under transition dynamics mismatch ( Section 4 ) . 4 . We demonstrate our method ’ s robust performance compared to the standard MCE IRL in a broad set of experiments under both linear and non-linear reward settings ( Section 5 ) . 5 . We extend our robust IRL method to the high dimensional continuous MDP setting with appropriate practical relaxations , and empirically demonstrate its effectiveness ( Section 6 ) . 2 Problem Setup . This section formalizes the IRL problem with an emphasis on the learner and expert environments . We use bold notation to represent vectors . A glossary of notation is given in Appendix C . 2.1 Environment and Reward . We formally represent the environment by a Markov decision process ( MDP ) Mθ : = { S , A , T , γ , P0 , Rθ } , parameterized by θ ∈ Rd . The state and action spaces are denoted as S and A , respectively . We assume that |S| , |A| < ∞ . T : S ×S ×A → [ 0 , 1 ] represents the transition dynamics , i.e. , T ( s′|s , a ) is the probability of transitioning to state s′ by taking action a from state s. The discount factor is given by γ ∈ ( 0 , 1 ) , and P0 is the initial state distribution . We consider a linear reward function Rθ : S → R of the form Rθ ( s ) = 〈θ , φ ( s ) 〉 , where θ ∈ Rd is the reward parameter , and φ : S → Rd is a feature map . We use a one-hot feature map φ : S → { 0 , 1 } |S| , where the sth element of φ ( s ) is 1 and 0 elsewhere . Our results can be extended to any general feature map ( see empirical evidence in Fig . 6 ) , but we use this particular choice as a running example for concreteness . We focus on the state-only reward function since the state-action reward function is not that useful in the robustness context . Indeed , as [ 22 ] pointed out , the actions to achieve a specific goal under different transition dynamics will not necessarily be the same and , consequently , should not be imitated . Analogously , in the IRL context , the reward for taking a particular action should not be recovered since the quality of that action depends on the transition dynamics . We denote an MDP without a reward function by M = Mθ\Rθ = { S , A , T , γ , P0 } . 2.2 Policy and Performance . A policy π : S → ∆A is a mapping from a state to a probability distribution over actions . The set of all valid stochastic policies is denoted by Π : = { π : ∑ a π ( a|s ) = 1 , ∀s ∈ S ; π ( a|s ) ≥ 0 , ∀ ( s , a ) ∈ S ×A } . We are interested in two different performance measures of any policy π acting in the MDP Mθ : ( i ) the expected discounted return V πMθ : = E [ ∑∞ t=0 γ tRθ ( st ) | π , M ] , and ( ii ) its entropy regularized variant V π , softMθ : = E [ ∑∞ t=0 γ t { Rθ ( st ) − log π ( at|st ) } | π , M ] . The state occupancy measure of a policy π in the MDPM is defined as ρπM ( s ) : = ( 1− γ ) ∑∞ t=0 γ tP [ st = s | π , M ] , where P [ st = s | π , M ] denotes the probability of visiting the state s after t steps by following the policy π in M . Note that ρπM ( s ) does not depend on the reward function . Let ρπM ∈ R|S| be a vector whose sth element is ρπM ( s ) . For the one-hot feature map φ , we have that V πMθ = 1 1−γ ∑ s ρ π M ( s ) Rθ ( s ) = 1 1−γ 〈θ , ρ π M 〉 . A policy π is optimal for the MDP Mθ if π ∈ arg maxπ′ V π ′ Mθ , in which case we denote it by π∗Mθ . Similarly , the soft-optimal policy ( always unique [ 32 ] ) in Mθ is defined as πsoftMθ : = arg maxπ′ V π′ , soft Mθ ( see Appendix D for a parametric form of this policy ) . 2.3 Learner and Expert Our setting has two entities : a learner implementing the MCE IRL algorithm , and an expert . We consider two MDPs , MLθ = { S , A , TL , γ , P0 , Rθ } and MEθ = { S , A , TE , γ , P0 , Rθ } , that differ only in the transition dynamics . The true reward parameter θ = θ∗ is known only to the expert . The expert provides demonstrations to the learner : ( i ) by following policy π∗ ME θ∗ in ME when there is a transition dynamics mismatch between the learner and the expert , or ( ii ) by following policy π∗ ML θ∗ in ML otherwise . The learner always operates in the MDP ML and is not aware of the true reward parameter and of the expert dynamics TE2 , i.e. , it only has access to MLθ∗\Rθ∗ . It learns a reward parameter θ and the corresponding soft-optimal policy πsoft MLθ , based on the state occupancy measure ρ received from the expert . Here , ρ is either ρ π∗ ME θ∗ ME or ρ π∗ ML θ∗ ML depending on the case . Our results can be extended to the stochastic estimate of ρ using concentration inequalities [ 11 ] . Our learner model builds on the MCE IRL [ 24 , 25 , 26 ] framework that matches the expert ’ s state occupancy measure ρ . In particular , the learner policy is obtained by maximizing its causal entropy while matching the expert ’ s state occupancy : max π∈Π E [ ∞∑ t=0 −γt log π ( at|st ) ∣∣∣∣ π , ML ] subject to ρπML = ρ . ( 1 ) Note that this optimization problem only requires access to MLθ \Rθ . The constraint in ( 1 ) follows from our choice of the one-hot feature map . We denote the optimal solution of the above problem by πsoft MLθ with a corresponding reward parameter : ( i ) θ = θE , when we use ρ π∗ ME θ∗ ME as ρ , or ( ii ) θ = θL , when we use ρ π∗ ML θ∗ ML as ρ . Here , the parameters θE and θL are obtained by solving the corresponding dual problems of ( 1 ) . Finally , we are interested in the performance of the learner policy πsoft MLθ in the MDP MLθ∗ . Our problem setup is illustrated in Figure 1 . 3 MCE IRL under Transition Dynamics Mismatch . This section analyses the MCE IRL learner ’ s suboptimality when there is a transition dynamics mismatch between the expert and the learner , as opposed to an ideal learner without this mismatch . The proofs of the theoretical statements of this section can be found in Appendix E .

Summary: This paper studies the performance degradation of a Maximum Causal Entropy learner under a transition dynamics mismatch between the expert and the learner in the imitation learning setting. The authors show that the performance of the learner is bounded by the $L_1$ distance between the transition dynamics of the expert and the learner.    Using Two-Player Markov games, the paper proposes an algorithm to robustly obtain a policy under transition dynamic mismatch and shows how it performance against MCE.