State-only Imitation with Transition Dynamics Mismatch

1 INTRODUCTION . In the Reinforcement Learning ( RL ) framework , the objective is to train policies that maximize a certain reward criterion . Deep-RL , which combines RL with the recent advances in the field of deeplearning , has produced algorithms demonstrating remarkable success in areas such as games ( Mnih et al. , 2015 ; Silver et al. , 2016 ) , continuous control ( Lillicrap et al. , 2015 ) , and robotics ( Levine et al. , 2016 ) , to name a few . However , the application of these algorithms beyond controlled simulation environments has been fairly modest ; one of the reasons being that manual specification of a good reward function is a hard problem . Imitation Learning ( IL ) algorithms ( Pomerleau , 1991 ; Ng et al. , 2000 ; Ziebart et al. , 2008 ; Ho & Ermon , 2016 ) address this issue by replacing reward functions with expert demonstrations , which are easier to collect in most scenarios . The conventional setting used in most of the IL literature is the availability of state-action trajectories from the expert , τ : = { s0 , a0 , . . . sT , aT } , collected in an environment modeled as a Markov decision process ( MDP ) with transition dynamics T exp . These dynamics govern the distribution over the next state , given the current state and action . The IL objective is to leverage τ to train an imitator policy in the same MDP as the expert . This is a severe requirement that impedes the wider applicability of IL algorithms . In many practical scenarios , the transition dynamics of the environment in which the imitator policy is learned ( henceforth denoted by T pol ) is different from the dynamics of the environment used to collect expert behavior , T exp . Consider self-driving cars as an example , where the goal is to learn autonomous navigation on a vehicle with slightly different gear-transmission characteristics than the vehicle used to obtain human driving demonstrations . We therefore strive 1Code for this paper is available at https : //github.com/tgangwani/RL-Indirect-imitation for an IL method that could train agents under a transition dynamics mismatch , T exp 6= T pol . We assume that other MDP attributes are the same for the expert and imitator environments . Beyond the dynamics equivalence , another assumption commonly used in IL literature is the availability of expert actions ( along with the states ) . A few recent works ( Torabi et al. , 2018a ; b ; Sun et al. , 2019 ) have proposed “ state-only ” IL algorithms , where expert demonstrations do not include the actions . This opens up the possibility of employing IL to situations such as kinesthetic teaching in robotics and learning from weak-supervision sources such as videos . Moreover , if T exp and T pol differ , then the expert actions , even if available , are not quite useful for imitation anyway , since the application of an expert action from any state leads to different next-state distributions for the expert and the imitator . Hence , our algorithm uses state-only expert demonstrations . We build on previous IL literature inspired by GAN-based adversarial learning - GAIL ( Ho & Ermon , 2016 ) and AIRL ( Fu et al. , 2017 ) . In both these methods , the objective is to minimize the distance between the visitation distributions ( ρ ) induced by the policy and expert , under some suitable metric d , such as Jensen-Shannon divergence . We classify GAIL and AIRL as direct imitation methods as they directly reduce d ( ρπ , ρ∗ ) . Different from these , we propose an indirect imitation approach which introduces another distribution ρ̃ as an intermediate or indirection step . In slight detail , starting with the Max-Entropy Inverse-RL objective ( Ziebart et al. , 2008 ) , we derive a lower bound which transforms the overall IL problem into two sub-parts which are solved iteratively : the first is to train a policy to imitate a distribution ρ̃ represented by a trajectory buffer , and the second is to move the buffer distribution closer to expert ’ s ( ρ∗ ) over the course of training . The first part , which is policy imitation by reducing d ( ρπ , ρ̃ ) is done with AIRL , while the second part , which is reducing d ( ρ̃ , ρ∗ ) , is achieved using a Wasserstein critic ( Arjovsky et al. , 2017 ) . We abbreviate our approach as I2L , for indirect-imitation-learning . We test the efficacy of our algorithm with continuous-control locomotion tasks from MuJoCo . Figure 1a depicts one example of the dynamics mismatch which we evaluate in our experiments . For the Ant agent , an expert walking policy π∗e is trained under the default dynamics provided in the OpenAI Gym , T exp = Earth . The dynamics under which to learn the imitator policy are curated by modifying the gravity parameter to half its default value ( i.e . 9.812 ) , T pol = PlanetX . Figure 1b plots the average episodic returns of π∗e in the original and modified environments , and proves that direct policy transfer is infeasible . For Figure 1c , we just assume access to state-only expert demonstrations from π∗e , and do IL with the GAIL algorithm . GAIL performs well if the imitator policy is learned in the same environment as the expert ( T exp = T pol = Earth ) , but does not succeed under mismatched transition dynamics , ( T exp = Earth , T pol = PlanetX ) . In our experiments section , we consider other sources of dynamics mismatch as well , such as agent-density and joint-friction . We show that I2L trains much better policies than baseline IL algorithms in these tasks , leading to successful transfer of expert skills to an imitator in an environment dissimilar to the expert . We start by reviewing the relevant background on Max-Entropy IRL , GAIL and AIRL , since these methods form an integral part of our overall algorithm . 2 BACKGROUND . An RL environment modeled as an MDP is characterized by the tuple ( S , A , R , T , γ ) , where S is the state-space , andA is the action-space . Given an action at ∈ A , the next state is governed by the transition dynamics st+1 ∼ T ( st+1|st , at ) , and reward is computed as rt = R ( rt|st , at ) . The RL objective is to maximie the expected discounted sum of rewards , η ( πθ ) = Ep0 , T , π [ ∑∞ t=0 γ tr ( st , at ) ] , where γ ∈ ( 0 , 1 ] is the discount factor , and p0 is the initial state distribution . We define the unnormalized γ-discounted state-visitation distribution for a policy π by ρπ ( s ) = ∑∞ t=0 γ tP ( st=s|π ) , where P ( st=s|π ) is the probability of being in state s at time t , when following policy π and starting state s0 ∼ p0 . The expected policy return η ( πθ ) can then be written as Eρπ ( s , a ) [ r ( s , a ) ] , where ρπ ( s , a ) = ρπ ( s ) π ( a|s ) is the state-action visitation distribution ( also referred to as the occupancy measure ) . For any policy π , there is a one-to-one correspondence between π and its occupancy measure ( Puterman , 1994 ) . 2.1 MAXIMUM ENTROPY IRL . Designing reward functions that adequately capture the task intentions is a laborious and error-prone procedure . An alternative is to train agents to solve a particular task by leveraging demonstrations of that task by experts . Inverse Reinforcement Learning ( IRL ) algorithms ( Ng et al. , 2000 ; Russell , 1998 ) aim to infer the reward function from expert demonstrations , and then use it for RL or planning . The IRL method , however , has an inherent ambiguity , since many expert policies could explain a set of provided demonstrations . To resolve this , Ziebart ( 2010 ) proposed the Maximum Causal Entropy ( MaxEnt ) IRL framework , where the objective is to learn a reward function such that the resulting policy matches the provided expert demonstrations in the expected feature counts f , while being as random as possible : max π H ( π ) s.t . Es , a∼π [ f ( s , a ) ] = f̂demo whereH ( π ) = Eπ [ − log π ( a|s ) ] is the γ-discounted causal entropy , and f̂demo denotes the empirical feature counts of the expert . This constrained optimization problem is solved by minimizing the Lagrangian dual , resulting in the maximum entropy policy : πθ ( a|s ) = exp ( Qsoftθ ( s , a ) − V softθ ( s ) ) , where θ is the Lagrangian multiplier on the feature matching constraint , and Qsoftθ , V soft θ are the soft value functions such that the following equations hold ( please see Theorem 6.8 in Ziebart ( 2010 ) ) : Qsoftθ ( s , a ) = θ T f ( s , a ) ︸ ︷︷ ︸ r ( s , a ) +Ep ( s′|s , a ) [ V softθ ( s′ ) ] , V softθ ( s ) = softmaxaQsoftθ ( s , a ) Inspired by the energy-based formulation of the maximum entropy policy described above , πθ ( a|s ) = exp ( Qsoftθ ( s , a ) − V softθ ( s ) ) , recent methods ( Finn et al. , 2016 ; Haarnoja et al. , 2017 ; Fu et al. , 2017 ) have proposed to model complex , multi-modal action distributions using energy-based policies , π ( a|s ) ∝ exp ( fω ( s , a ) ) , where fω ( s , a ) is represented by a universal function approximator , such as a deep neural network . We can then interpret the IRL problem as a maximum likelihood estimation problem : max ω Eτ∼demo [ log pω ( τ ) ] with , pω ( τ ) = p ( s0 ) ∏ t p ( st+1|st , at ) efω ( st , at ) Z ( ω ) ( 1 ) 2.2 ADVERSARIAL IRL . An important implication of casting IRL as maximum likelihood estimation is that it connects IRL to adversarial training . We now briefly discuss AIRL ( Fu et al. , 2017 ) since it forms a component of our proposed algorithm . AIRL builds on GAIL ( Ho & Ermon , 2016 ) , a well-known adversarial imitation learning algorithm . GAIL frames IL as an occupancy-measure matching ( or divergence minimization ) problem . Let ρπ ( s , a ) and ρE ( s , a ) represent the state-action visitation distributions of the policy and the expert , respectively . Minimizing the Jenson-Shanon divergence minπDJS [ ρπ ( s , a ) || ρE ( s , a ) ] recovers a policy with a similar trajectory distribution as the expert . GAIL iteratively trains a policy ( πθ ) and a discriminator ( Dω : S × A → ( 0 , 1 ) ) to optimize the min-max objective similar to GANs ( Goodfellow et al. , 2014 ) : min θ max ω E ( s , a ) ∼ρE [ logDω ( s , a ) ] + E ( s , a ) ∼πθ [ log ( 1−Dω ( s , a ) ) ] − λH ( πθ ) ( 2 ) GAIL attempts to learn a policy that behaves similar to the expert demonstrations , but it bypasses the process of recovering the expert reward function . Finn et al . ( 2016 ) showed that imposing a special structure on the discriminator makes the adversarial GAN training equivalent to optimizing the MLE objective ( Equation 1 ) . Furthermore , if trained to optimality , it is proved that the expert reward ( up to a constant ) can be recovered from the discriminator . They operate in a trajectorycentric formulation which can be inefficient for high dimensional state- and action-spaces . Fu et al . ( 2017 ) present AIRL which remedies this by proposing analogous changes to the discriminator , but operating on a single state-action pair : Dω ( s , a ) = efω ( s , a ) efω ( s , a ) + πθ ( a|s ) ( 3 ) Similar to GAIL , the discriminator is trained to maximize the objective in Equation 2 ; fω is learned , whereas the value of π ( a|s ) is “ filled in ” . The policy is optimized jointly using any RL algorithm with logDω − log ( 1 − Dω ) as rewards . When trained to optimality , exp ( fω ( s , a ) ) = π∗ ( a|s ) = exp ( A∗soft ( s , a ) /α ) ; hence fω recovers the soft advantage of the expert policy ( up to a constant ) .

Summary: The manuscript considers the problem of imitation learning when the system dynamics of the agent are different from the dynamics of the expert. The paper proposes Indirect Imitation Learning (I2L), which aims to perform imitation learning with respect to a trajectory buffer that contains some of the previous trajectories of the agent. The trajectory buffer has limited capacity and adds trajectories based on a priority-queue that prefers trajectories that have a similar state distribution to the expert. Similarity is hereby measured by the score of a WGAN-critic trained to approximate the W1-Wasserstein distance between the previous buffer and the expert distribution. By performing imitation learning with respect to a trajectory buffer, state-action trajectories of the agent's MDP are available, which enables I2L to apply AIRL (Fu et al. 2017). By using those trajectories for the transition buffer that have state-marginals close to the expert's trajectory, I2L produces similar behavior compared to the expert. I2L is compared to state-only GAIL, state-action-GAIL and AIRL on four MuJoCo tasks with modified dynamics compared to the expert policy. The experiments show that I2L may learn significantly better policies if the dynamics of agent and the expert do not match.