Imitation Learning from Observations under Transition Model Disparity

1 INTRODUCTION . Imitation Learning ( IL ) is a framework that trains an agent to perform desired skills by leveraging expert demonstrations of those skills . Compared to the standard Reinforcement Learning ( RL ) approach , IL offers the benefit of not requiring a reward function , that can be difficult to specify for complicated objectives . Recent IL methods that integrate efficiency with deep-RL and are performant in high-dimensional state-action spaces include behavioral-cloning-based algorithms ( Ross et al. , 2011 ; Brantley et al. , 2019 ) , and adversarial IL algorithms inspired by maximum entropy inverse-RL ( Ho & Ermon , 2016 ; Finn et al. , 2016 ) . Imitation Learning from Observations ( ILO ) refers to the setting where the expert demonstrations consist of only observations ( or states ) , while the expert actions are unavailable . ILO is beneficial when the measurement of the expert action is difficult , e.g. , in kinesthetic teaching in robotics or when learning with motion capture datasets . Adversarial IL methods frame the problem as the minimization of an f-divergence between the stateaction visitation distributions of the expert and the learner ( Ke et al. , 2019 ) . Since expert actions are absent in ILO , the analogous methodology here is to minimize the f-divergence between the statetransition distributions of the expert and the learner ( Arumugam et al. , 2020 ) . A state-transition distribution for a policy is the joint distribution over the current state and the next state , and is defined formally in Section 2 . Choosing the f-divergence to be the Jensen–Shannon ( JS ) divergence has enabled successful imitation using algorithms such as GAIL ( Ho & Ermon , 2016 ) and GAIfO ( Torabi et al. , 2018b ) , for IL and ILO , respectively . The transition dynamics model of an environment governs the distribution over the next state , given the current state and action . In this paper , we focus on ILO in the scenario of a transition dynamics mismatch between the expert and the learner environments . Such discrepancy could manifest in real-world applications of imitation learning when there are subtle differences in the physical attributes of the system used to collect the demonstrations and the system where the learner policy is run . Adversarial ILO methods such as GAIfO , that attempt to train the learner by matching its statetransition distribution with that of the expert , perform very well when the expert and learner operate in a shared environment , under the same dynamics . When the dynamics differ , however , matching the state-transition distributions becomes challenging since the state transitions provided in the ex- pert demonstrations could be infeasible under the dynamics in the learner ’ s environment . To form some intuition , consider ILO in a 2D grid with discrete states , where the path demonstrated by the expert follows the main diagonal from the bottom-left grid-cell to the top-right grid-cell . Suppose that the transition dynamics for the learner environment are such that only horizontal and vertical moves are permitted on the grid , i.e. , no diagonal motion for the learner . In this case , an optimal learner ’ s movement is still in the same general direction as the expert , and it covers all the expert states ( along with some adjacent states ) . However , matching the state-transition distributions is an ineffective strategy since the learner can not reproduce the ( one-step ) state transitions of the expert . To alleviate the challenges of ILO under dynamics mismatch , we propose an algorithm that trains an intermediary policy in the learner environment , and hope to use it as a surrogate expert for training the learner ( imitator ) . We refer to this policy as the advisor . For the advisor to be effective , the state transitions generated by it in the learner environment should be as close as possible to the state transitions in the expert dataset . We formalize this concept in terms of the cross-entropy distance between state-conditional next-state distributions of the expert and the advisor . To convert this into a practical and scalable algorithm for training the advisor , we incorporate ideas from distribution support estimation ( Wang et al. , 2019 ) . Simultaneous to the advisor training , the learner agent is updated to imitate the advisor . Crucially , the advisor operates in the same environment as the learner , making the distribution-matching IL objective amenable . We evaluate the efficacy of our ILO algorithm using five locomotion control tasks from OpenAI Gym where we introduce a mismatch between the dynamics of the expert and the learner by changing different configuration parameters . We demonstrate that our approach compares favorably to the baseline ILO algorithms in many of the considered scenarios . 2 PRELIMINARIES . We model the RL environment as a discounted , infinite horizon Markov Decision Process ( MDP ) . At every discrete timestep , the agent observes a state ( s ∈ S ) , generates an action ( a ∈ A ) from a stochastic policy π ( a|s ) , receives a scalar reward r ( s , a ) , and transitions to the next state ( s′ ) sampled from the transition dynamics model p ( s′|s , a ) . In the infinite horizon setting , the future rewards are discounted by a factor of γ ∈ [ 0 , 1 ) . Let dtπ ( s ) denote the distribution induced by π over the state-space at a particular timestep t. The stationary discounted state distribution of π is then defined as ρπ ( s ) = ( 1 − γ ) ∑∞ t=0 γ tdtπ ( s ) . The RL objective of maximizing the cumulative discounted sum of rewards can be framed as maxπ Eρπ ( s , a ) [ r ( s , a ) ] , where ρπ ( s , a ) = ρπ ( s ) π ( a|s ) is the state-action distribution ( also known as the occupancy measure ) . Lastly , we define the statetransition distribution for a policy as ρπ ( s , s′ ) = ∑ a p ( s ′|s , a ) ρπ ( s , a ) 1 . 2.1 GAIL AND GAIFO . Generative Adversarial Imitation Learning ( Ho & Ermon , 2016 ) is a widely popular model-free IL method that builds on the Maximum Causal Entropy Inverse-RL ( MaxEnt-IRL ) framework ( Ziebart , 2010 ) . MaxEnt-IRL models the expert behavior with a policy that maximizes its γ-discounted causal entropy , H ( π ) = Eπ [ − log π ( a|s ) ] , while satisfying a feature matching constraint . GAIL considers a regularized version of the dual to this primal problem . It shows that RL with the reward function recovered as the solution of the regularized dual is equivalent to directly learning a policy whose state-action distribution is similar to that of the expert . For a specific choice of the regularizer , this similarity is quantified by the JS divergence between the two state-action distributions , DJS [ ρπ ( s , a ) || ρπe ( s , a ) ] . Based on these ideas , GAIL seeks to learn a policy with the objective : min π max D Eρπ [ logD ( s , a ) ] + Eρπe [ log ( 1−D ( s , a ) ) ] − λH ( π ) where D : S×A → ( 0 , 1 ) is the discriminator that provides the rewards for training the learner policy π , and the inner maximization overD approximatesDJS ( · ) similar to GANs ( Goodfellow et al. , 2014 ) . To empirically estimate the expectation under ρπe ( s , a ) , state-action pairs are sampled from the available expert demonstrations . In the ILO setting , however , expert actions are not included in the demonstrations . To mitigate this challenge , Torabi et al . ( 2018b ) propose GAIfO , which adapts 1With slight abuse of notation , we use the symbol ρπ for state , state-action , and state-transition distributions of a policy π . We would provide context around their usage to avoid any confusion . to ILO by modifying the GAIL objective to match the state-transition distributions of the expert and the learner , i.e. , DJS [ ρπ ( s , s′ ) || ρπe ( s , s′ ) ] . Correspondingly , the discriminator in GAIfO is a function of the state transitions D ( s , s′ ) . 2.2 SUPPORT ESTIMATION VIA RED . We summarize a recently proposed method for estimating the support of a distribution in high dimensions ( RED ; Wang et al . ( 2019 ) ) , since it forms a core ingredient of our final algorithm . Let X denote a set and p be a probability distribution on X . Denote by supp ( p ) = { x ∈ X | p ( x ) 6= 0 } , the support of the distribution p. Given any x ∈ X , the task is to know if x ∈ supp ( p ) . Towards this goal , Wang et al . ( 2019 ) combine ideas from kernel-based support estimation ( De Vito et al. , 2014 ) and RND ( Burda et al. , 2018 ) , and consider the following objective : θ∗ = argmin θ Ex∼p ( x ) ‖fθ ( x ) − fθ̃ ( x ) ‖ 2 2 where fθ : X → RK is a trainable function parameterized by θ , while fθ̃ is a fixed function with randomly initialized parameters θ̃ . Define the score function as ( with constant positive scalar λ ) : rRED ( x ) = exp ( −λ‖fθ∗ ( x ) − fθ̃ ( x ) ‖ 2 2 ) Wang et al . ( 2019 ) conclude that the score rRED ( x ) is high if x ∈ supp ( p ) , and is low otherwise . With neural network function approximators , we thus obtain a smooth metric whose value decreases ( increases ) as we move farther from ( closer to ) the support of the distribution p . 3 METHODS . We begin by defining some notations for our setup . We denote the MDP in which the expert policy ( πe ) operates as the e-MDP , while the learner ( or imitator ) policy is run in the l-MDP . The two MDPs share all the attributes , except for the transition dynamics function , which we symbolize with pe ( s ′|s , a ) and pl ( s′|s , a ) , for the e-MDP and l-MDP , respectively . ρe ( s ) , ρe ( s , a ) , and ρe ( s , s′ ) are the state , state-action , and state-transition distribution for the expert policy . These distributions depend on the e-MDP dynamics pe ( s′|s , a ) . The corresponding distributions for the learner are denoted by ρπ ( · ) and they depend on the l-MDP dynamics pl ( s′|s , a ) . Adversarial ILO methods , such as GAIfO , that learn the imitator policy by matching the statetransition distributions of the expert and learner aim to solve the following primal problem : max π H ( π ) s.t . ρe ( s , s′ ) = ρπ ( s , s′ ) ∀ ( s , s′ ) ∈ S × S ( 1 ) If the expert and the learner operate under different transition dynamics , depending on the extent of the mismatch , it is possible that some ( or all ) of the one-step state transitions of the expert are infeasible under the dynamics function in l-MDP . Said differently , given a ( se , s′e ) pair sampled from the expert demonstrations , there could be no action in the l-MDP from the state se that results in s′e as the next state , as per the dynamics pl ( s′|s , a ) . This renders the state-transition matching objective hard to optimize in practice . For instance , in the practical implementation of GAIfO , the rewards for the imitator policy are computed from a discriminator that is trained with binary classification on ( s , s′ ) pairs from the expert and the learner . If the expert transitions can ’ t be generated in l-MDP by the learner , then a high-capacity discriminator could achieve perfect accuracy and thus fail to provide informative rewards for imitation . Consider a sequence of states { s1 , s2 , . . . } generated by the expert policy in the e-MDP , as shown in Figure 1a . Given an expert state si , it may not be possible to reach si+1 in a single timestep in the l-MDP . Instead , we would like to find an alternative state s̃i+1 that is reachable from si in one step ( i.e. , feasible under the dynamics pl ) and is close to the desired destination si+1 . For instance , in Figure 1a , starting from the expert state s1 , s̃2 is a more desirable next state compared to ŝ2 .

Summary: This paper tackles imitation learning from observations under model mismatch.  This is a realistic consideration when tackling more complex or real environments where methods such as IL (and its variants) may see dramatic improvements over more conventional RL-based approaches (to the point of being the only feasible approach).  Model mismatch makes existing ILO methods unstable (in theory, at the very least).  To ameliorate this, the paper introduces an _advisor_ policy, which learns from the expert demonstrations but under the constraints implicitly imposed by the capacity of the learner or the learners dynamics.  The advisor is then used to train the learner.  The method is empirically validated on some standard benchmarks and appears to perform reasonably well.