Adaptive Adversarial Imitation Learning

1 INTRODUCTION . Humans and animals can learn complex behaviors via imitation . Inspired by these learning mechanisms , Imitation Learning ( IL ) has long been a popular method for training autonomous agents from human-provided demonstrations . However , human and animal imitation differs markedly from commonly used approaches in machine learning . Firstly , humans and animals tend to imitate the goal of the task rather than the particular motions of the demonstrator ( Baker et al. , 2007 ) . Secondly , humans and animals can easily handle imitation scenarios where there is a shift in embodiment and dynamics between themselves and a demonstrator . The first feature of human IL can be represented within the framework of Inverse Reinforcement Learning ( IRL ) ( Ng et al. , 2000 ; Abbeel & Ng , 2004 ; Ziebart et al. , 2008 ) , which at a high level casts the problem of imitation as one of matching outcomes rather than actions . Recent work in adversarial imitation learning ( Ho & Ermon , 2016 ; Finn et al. , 2016 ) has accomplished this by using a discriminator to judge whether a given behavior is from an expert or imitator , and then a policy is trained using the discriminator expert likelihood as a reward . While successful in multiple problem domains , this approach makes it difficult to accommodate the second feature of human learning : imitation across shifts in embodiment and dynamics . This is because in the presence of such shifts , the discriminator may either simply use the embodiment or dynamics to infer whether it is evaluating expert behavior , and as a consequence fails to provide a meaningful reward signal . In this paper we are concerned with the problem of learning adaptive policies that can be transferred to environments with varying dynamics , by imitating a small number of expert demonstrations collected from a single source domain . This problem is important in robotic learning because it is better aligned with real world constraints : 1 ) reward functions are hard to obtain , 2 ) learned policies from one domain are hard to deploy to different domains due to varying source to target domain statistics , and 3 ) the target domain dynamics oftentimes changes while executing the learned policy . As such , this work assumes ground truth rewards are not available , and furthermore we assume that expert demonstrations come from only a single domain ( i.e . an instance of an environment where dynamics can not be exactly replicated by the policy at training time ) . To the best of our knowledge , this is the first work to tackle this challenging problem formulation . Our proposed method solves the above problem by building upon the GAIL ( Ho & Ermon , 2016 ; Finn et al. , 2016 ) framework , by firstly conditioning the policy on a learned dynamics embedding ( “ context variable ” in policy search literature ( Deisenroth et al. , 2013 ) ) . We propose two embedding approaches on which the policy is conditioned , namely , a direct supervised learning approach and a variational autoencoder ( VAE ) ( Kingma & Welling , 2013 ) based unsupervised approach . Secondly , to prevent the discriminator from inferring whether it is evaluating the expert behavior or imitator behavior purely through the dynamics , we propose using a Gradient Reversal Layer ( GRL ) to learn a dynamics-invariant discriminator . We demonstrate the effectiveness of the proposed algorithm on benchmark Mujoco simulated control tasks . The main contributions of our work include : 1 ) present a general and novel problem formulation that is well aligned with real world scenarios in comparison to recent literature 2 ) devise a conceptually simple architecture that is capable of learning an adaptive policy from a small number of expert demonstrations ( order of 10s ) collected from only one source environment , 3 ) design an adversarial loss for addressing the covariate shift issue in discriminator learning . 2 RELATED WORK . Historically , two main avenues have been heavily studied for imitation learning : 1 ) Behavioral Cloning ( BC ) and 2 ) Inverse Reinforcement Learning ( IRL ) . Though conceptually simple , BC suffers from compound errors caused by covariate shift , and subsequently , often requires a large quantity of demonstrations ( Pomerleau , 1989 ) , or access to the expert policy ( Ross et al. , 2011 ) in order to recover a stable policy . Recent advancements in imitation learning ( Ho & Ermon , 2016 ; Finn et al. , 2016 ) have adopted an adversarial formation that interleaves between 1 ) discriminating the generated policy against the expert demonstrations and 2 ) a policy improvement step where the policy aims to fool the learned discriminator . Dynamics randomization ( Tobin et al. , 2017 ; Sadeghi & Levine , 2016 ; Mandlekar et al. , 2017 ; Tan et al. , 2018 ; Pinto et al. , 2017 ; Peng et al. , 2018 ; Chebotar et al. , 2018 ; Rajeswaran et al. , 2016 ) has been one of the prevailing vehicles for addressing varying simulation to real-world domain statistics . This avenue of methods typically involves perturbing the environment dynamics ( often times adversarially ) in simulation in order to learn an adaptive policy that is robust enough to bridge the “ Reality Gap ” . While dynamics randomization has been explored for learning robust policies in an RL setting , it has a critical limitation in the imitation learning context : large domain shifts might result in directional differences in dynamics , therefore , the demonstrated actions might no longer be admissible for solving the task in the target domain . Our method ( Figure 2 ) also involves training in a variety of environments with different dynamics . However , we propose conditioning the policy on an explicitly learned dynamics embedding to enable adaptive policies based on online system ID . Yu et al . ( 2017 ) adopted a similar approach towards building adaptive policies . They learn an online system identification model and condition the policy on the predicted model parameters in an RL setting . In comparison to their work , we do not assume access to the ground truth reward signals or the ground truth physics parameters at evaluation time , which makes this work ’ s problem formulation a harder learning problem , but with greater potential for real-world applications . We will compare our method with Yu et al . ( 2017 ) in the experimental section . Third person imitation learning ( Stadie et al. , 2017 ) also employs a GRL ( Ganin & Lempitsky , 2014 ) under a GAIL-like formulation with the goal of learning expert behaviors in a new domain . In comparison , our method also enables learning adaptive policies by employing an online dynamics identification component , so that the policies can be transferred to a class of domains , as opposed to one domain . In addition , learned policies using our proposed method can handle online dynamics perturbations . Meta learning ( Finn et al. , 2017 ) has also been applied to address varying source to target domain dynamics ( Duan et al. , 2017 ; Nagabandi et al. , 2018 ) . The idea behind meta learning in the context of robotic learning is to learn a meta policy that is “ initialized ” for a variety of tasks in simulation , and then fine-tune the policy in the real-world setting given a specific goal . After the meta-learning phase , the agent requires significantly fewer environment interactions to obtain a policy that solves the task . In comparison to meta learning based approaches , fine-tuning on the test environment is not required in our method , with the caveat being that this is true only within the target domain where the dynamics posterior is effective . 2.1 BACKGROUND . In this section we will briefly review GAIL ( Ho & Ermon , 2016 ) . Inspired by GANs , the GAIL objective is defined as : min θ max ω EπE [ logDω ( s , a ) ] +Eπθ [ log ( 1−Dω ( s , a ) ) ] ( 1 ) Where πE denotes the expert policy that generated the demonstrations ; πθ is the policy to imitate the expert ; D is a discriminator that learns to distinguish between πθ and πE with generated stateaction pairs . In comparison to GAN optimization , the GAIL objective is rarely differentiable since differentiation through the environment step is often intractable . Optimization is instead achieved via RL-based policy gradient algorithms , e.g. , PPO ( Schulman et al. , 2017 ) or off policy methods , e.g. , TD3 ( Kostrikov et al. , 2018 ) . Without an explicit reward function , GAIL relies on reward signals provided by the learned discriminator , where a common reward formulation is rω ( s , a ) = − log ( 1−Dω ( s , a ) ) . 3 ADAPTIVE ADVERSARIAL IMITATION LEARNING ( ADAIL ) . 3.1 PROBLEM DEFINITION . Suppose we are given a classE of environments with different dynamics but similar goals , a domain generator g ( c ) which takes in a code c and generates an environment ec ∈ E , and a set of expert demonstrations { τexp } collected from one source environment eexp ∈ E. In adaptive imitation learning , one attempts to learn an adaptive policy πθ that can generalize across environments within E. We assume that the ground truth dynamics parameters c , which are used to generate the simulated environments , are given ( or manually sampled ) during the training phase . 3.2 ALGORITHM OVERVIEW . We allow the agent to interact with a class of similar simulated environments with varying dynamics parameters , which we call “ adaptive training ” . To be able to capture high-level goals from a small set of demonstrations , we adopt a approach similar to GAIL . To provide consistent feedback signals during training across environments with different dynamics , the discriminator should be dynamicsinvariant . We enable this desirable feature by learning a dynamics-invariant feature layer for the discriminator by 1 ) adding another head DR ( c|s , a ) to the discriminator to predict the dynamics parameters , and 2 ) inserting a GRL in-between DR and the dynamics-invariant feature layer . The discriminator design is illustrated in Figure 1 . In addition , to enable adaptive policies , we introduced a dynamics posterior that takes a roll-out trajectory and outputs an embedding , on which the policy is conditioned . Intuitively , explicit dynamics latent variable learning endows the agent with the ability to identify the system and act differently against changes in dynamics . Note that a policy can learn to infer dynamics implicitly , without the need for an external dynamics embedding . However , we find experimentally that policies conditioned explicitly on the environment parameters outperform those that do not . The overall architecture is illustrated in Figure 2 . We call the algorithm Adaptive Adversarial Imitation Learning ( ADAIL ) , with the following objective ( note that for brevity , we for now omit the GRL term discussed in Section 3.4 ) : min θ max ω , φ EπE [ logDω ( s , a ) ] +Eπθ ( ·|c ) [ log ( 1−Dω ( s , a ) ) ] +Eτ∼πθ ( ·|c ) [ logQφ ( c|τ ) ] ( 2 ) Where c is a learned latent dynamics representation that is associated with the rollout environment in each gradient step ; τ is a roll-out trajectory using πθ ( ·|c ) in the corresponding environment ; Q ( c|τ ) is a “ dynamics posterior ” for inferring the dynamics during test time ; The last term in the objective , Eτ∼πθ ( ·|c ) [ logQφ ( c|τ ) ] , is a general form of the expected log likelihood of c given τ . Note that , the posterior training is on-policy , meaning that the rollouts are collected online using the current Algorithm 1 ADAIL 1 : Inputs : 2 : An environment class E. 3 : Initial parameters of policy θ , discriminator ω , and posterior φ . 4 : A set of expert demonstrations { τexp } on one of the environment eexp ∈ E. An environment generator g ( c ) that takes a code and generates an environment ec ∈ E. A prior distribution of p ( c ) . 5 : for i = 1 , 2 , .. do 6 : Sample c ∼ p ( c ) and generate environment ec ∼ g ( c ) 7 : Sample trajectories τi ∼ πθ ( ·|c ) in ec and τei ∼ { τexp } 8 : Update the discriminator parameters ω with the gradients : Ê ( s , a ) ∼τi [ ∇w log ( Dw ( s , a ) ) ] + Ê ( s , a ) ∼τei [ ∇w log ( 1−Dw ( s , a ) ] 9 : Update the discriminator parameters ω again with the following loss , such that the gradients are reversed when back-prop through the dynamics-invariant layer : −Ê ( s , a ) ∼τi [ log ( D R ( c|s , a ) ) ] 10 : Update the posterior parameters φ with gradients Êτi [ ∇φ logQφ ( c|τi ) ) ] 11 : Update policy πθ ( ·|c ) using policy optimization method ( PPO ) with : Ê ( s , a ) ∼τi [ − log ( 1−Dω ( s , a ) ) ] 12 : Output : Learned policy πθ , and posterior Qφ . policy , thereby the last term of the objective is dependent on θ . One can employ various supervised and unsupervised methods towards optimizing this term . We will explore a few methods in the following subsections . The algorithm is outlined in Algorithm 1 .

Summary: This paper proposes an algorithm for imitation of expert demonstrations, in situations where the imitator is acting under a different environment (different dynamics, for instance) than the one used to collect expert demonstrations. The algorithm builds on GAIL with the following modifications – the discriminator is made dynamics-invariant by adding a domain-adversarial loss, and the policy is made to condition on a dynamics context. A separate dynamics posterior network is trained (either supervised or unsupervised) to predict this context at test-time.