Adaptive Adversarial Imitation Learning

1 INTRODUCTION . Humans and animals can learn complex behaviors via imitation . Inspired by these learning mechanisms , Imitation Learning ( IL ) has long been a popular method for training autonomous agents from human-provided demonstrations . However , human and animal imitation differs markedly from commonly used approaches in machine learning . Firstly , humans and animals tend to imitate the goal of the task rather than the particular motions of the demonstrator ( Baker et al. , 2007 ) . Secondly , humans and animals can easily handle imitation scenarios where there is a shift in embodiment and dynamics between themselves and a demonstrator . The first feature of human IL can be represented within the framework of Inverse Reinforcement Learning ( IRL ) ( Ng et al. , 2000 ; Abbeel & Ng , 2004 ; Ziebart et al. , 2008 ) , which at a high level casts the problem of imitation as one of matching outcomes rather than actions . Recent work in adversarial imitation learning ( Ho & Ermon , 2016 ; Finn et al. , 2016 ) has accomplished this by using a discriminator to judge whether a given behavior is from an expert or imitator , and then a policy is trained using the discriminator expert likelihood as a reward . While successful in multiple problem domains , this approach makes it difficult to accommodate the second feature of human learning : imitation across shifts in embodiment and dynamics . This is because in the presence of such shifts , the discriminator may either simply use the embodiment or dynamics to infer whether it is evaluating expert behavior , and as a consequence fails to provide a meaningful reward signal . In this paper we are concerned with the problem of learning adaptive policies that can be transferred to environments with varying dynamics , by imitating a small number of expert demonstrations collected from a single source domain . This problem is important in robotic learning because it is better aligned with real world constraints : 1 ) reward functions are hard to obtain , 2 ) learned policies from one domain are hard to deploy to different domains due to varying source to target domain statistics , and 3 ) the target domain dynamics oftentimes changes while executing the learned policy . As such , this work assumes ground truth rewards are not available , and furthermore we assume that expert demonstrations come from only a single domain ( i.e . an instance of an environment where dynamics can not be exactly replicated by the policy at training time ) . To the best of our knowledge , this is the first work to tackle this challenging problem formulation . Our proposed method solves the above problem by building upon the GAIL ( Ho & Ermon , 2016 ; Finn et al. , 2016 ) framework , by firstly conditioning the policy on a learned dynamics embedding ( “ context variable ” in policy search literature ( Deisenroth et al. , 2013 ) ) . We propose two embedding approaches on which the policy is conditioned , namely , a direct supervised learning approach and a variational autoencoder ( VAE ) ( Kingma & Welling , 2013 ) based unsupervised approach . Secondly , to prevent the discriminator from inferring whether it is evaluating the expert behavior or imitator behavior purely through the dynamics , we propose using a Gradient Reversal Layer ( GRL ) to learn a dynamics-invariant discriminator . We demonstrate the effectiveness of the proposed algorithm on benchmark Mujoco simulated control tasks . The main contributions of our work include : 1 ) present a general and novel problem formulation that is well aligned with real world scenarios in comparison to recent literature 2 ) devise a conceptually simple architecture that is capable of learning an adaptive policy from a small number of expert demonstrations ( order of 10s ) collected from only one source environment , 3 ) design an adversarial loss for addressing the covariate shift issue in discriminator learning . 2 RELATED WORK . Historically , two main avenues have been heavily studied for imitation learning : 1 ) Behavioral Cloning ( BC ) and 2 ) Inverse Reinforcement Learning ( IRL ) . Though conceptually simple , BC suffers from compound errors caused by covariate shift , and subsequently , often requires a large quantity of demonstrations ( Pomerleau , 1989 ) , or access to the expert policy ( Ross et al. , 2011 ) in order to recover a stable policy . Recent advancements in imitation learning ( Ho & Ermon , 2016 ; Finn et al. , 2016 ) have adopted an adversarial formation that interleaves between 1 ) discriminating the generated policy against the expert demonstrations and 2 ) a policy improvement step where the policy aims to fool the learned discriminator . Dynamics randomization ( Tobin et al. , 2017 ; Sadeghi & Levine , 2016 ; Mandlekar et al. , 2017 ; Tan et al. , 2018 ; Pinto et al. , 2017 ; Peng et al. , 2018 ; Chebotar et al. , 2018 ; Rajeswaran et al. , 2016 ) has been one of the prevailing vehicles for addressing varying simulation to real-world domain statistics . This avenue of methods typically involves perturbing the environment dynamics ( often times adversarially ) in simulation in order to learn an adaptive policy that is robust enough to bridge the “ Reality Gap ” . While dynamics randomization has been explored for learning robust policies in an RL setting , it has a critical limitation in the imitation learning context : large domain shifts might result in directional differences in dynamics , therefore , the demonstrated actions might no longer be admissible for solving the task in the target domain . Our method ( Figure 2 ) also involves training in a variety of environments with different dynamics . However , we propose conditioning the policy on an explicitly learned dynamics embedding to enable adaptive policies based on online system ID . Yu et al . ( 2017 ) adopted a similar approach towards building adaptive policies . They learn an online system identification model and condition the policy on the predicted model parameters in an RL setting . In comparison to their work , we do not assume access to the ground truth reward signals or the ground truth physics parameters at evaluation time , which makes this work ’ s problem formulation a harder learning problem , but with greater potential for real-world applications . We will compare our method with Yu et al . ( 2017 ) in the experimental section . Third person imitation learning ( Stadie et al. , 2017 ) also employs a GRL ( Ganin & Lempitsky , 2014 ) under a GAIL-like formulation with the goal of learning expert behaviors in a new domain . In comparison , our method also enables learning adaptive policies by employing an online dynamics identification component , so that the policies can be transferred to a class of domains , as opposed to one domain . In addition , learned policies using our proposed method can handle online dynamics perturbations . Meta learning ( Finn et al. , 2017 ) has also been applied to address varying source to target domain dynamics ( Duan et al. , 2017 ; Nagabandi et al. , 2018 ) . The idea behind meta learning in the context of robotic learning is to learn a meta policy that is “ initialized ” for a variety of tasks in simulation , and then fine-tune the policy in the real-world setting given a specific goal . After the meta-learning phase , the agent requires significantly fewer environment interactions to obtain a policy that solves the task . In comparison to meta learning based approaches , fine-tuning on the test environment is not required in our method , with the caveat being that this is true only within the target domain where the dynamics posterior is effective . 2.1 BACKGROUND . In this section we will briefly review GAIL ( Ho & Ermon , 2016 ) . Inspired by GANs , the GAIL objective is defined as : min θ max ω EπE [ logDω ( s , a ) ] +Eπθ [ log ( 1−Dω ( s , a ) ) ] ( 1 ) Where πE denotes the expert policy that generated the demonstrations ; πθ is the policy to imitate the expert ; D is a discriminator that learns to distinguish between πθ and πE with generated stateaction pairs . In comparison to GAN optimization , the GAIL objective is rarely differentiable since differentiation through the environment step is often intractable . Optimization is instead achieved via RL-based policy gradient algorithms , e.g. , PPO ( Schulman et al. , 2017 ) or off policy methods , e.g. , TD3 ( Kostrikov et al. , 2018 ) . Without an explicit reward function , GAIL relies on reward signals provided by the learned discriminator , where a common reward formulation is rω ( s , a ) = − log ( 1−Dω ( s , a ) ) . 3 ADAPTIVE ADVERSARIAL IMITATION LEARNING ( ADAIL ) . 3.1 PROBLEM DEFINITION . Suppose we are given a classE of environments with different dynamics but similar goals , a domain generator g ( c ) which takes in a code c and generates an environment ec ∈ E , and a set of expert demonstrations { τexp } collected from one source environment eexp ∈ E. In adaptive imitation learning , one attempts to learn an adaptive policy πθ that can generalize across environments within E. We assume that the ground truth dynamics parameters c , which are used to generate the simulated environments , are given ( or manually sampled ) during the training phase . 3.2 ALGORITHM OVERVIEW . We allow the agent to interact with a class of similar simulated environments with varying dynamics parameters , which we call “ adaptive training ” . To be able to capture high-level goals from a small set of demonstrations , we adopt a approach similar to GAIL . To provide consistent feedback signals during training across environments with different dynamics , the discriminator should be dynamicsinvariant . We enable this desirable feature by learning a dynamics-invariant feature layer for the discriminator by 1 ) adding another head DR ( c|s , a ) to the discriminator to predict the dynamics parameters , and 2 ) inserting a GRL in-between DR and the dynamics-invariant feature layer . The discriminator design is illustrated in Figure 1 . In addition , to enable adaptive policies , we introduced a dynamics posterior that takes a roll-out trajectory and outputs an embedding , on which the policy is conditioned . Intuitively , explicit dynamics latent variable learning endows the agent with the ability to identify the system and act differently against changes in dynamics . Note that a policy can learn to infer dynamics implicitly , without the need for an external dynamics embedding . However , we find experimentally that policies conditioned explicitly on the environment parameters outperform those that do not . The overall architecture is illustrated in Figure 2 . We call the algorithm Adaptive Adversarial Imitation Learning ( ADAIL ) , with the following objective ( note that for brevity , we for now omit the GRL term discussed in Section 3.4 ) : min θ max ω , φ EπE [ logDω ( s , a ) ] +Eπθ ( ·|c ) [ log ( 1−Dω ( s , a ) ) ] +Eτ∼πθ ( ·|c ) [ logQφ ( c|τ ) ] ( 2 ) Where c is a learned latent dynamics representation that is associated with the rollout environment in each gradient step ; τ is a roll-out trajectory using πθ ( ·|c ) in the corresponding environment ; Q ( c|τ ) is a “ dynamics posterior ” for inferring the dynamics during test time ; The last term in the objective , Eτ∼πθ ( ·|c ) [ logQφ ( c|τ ) ] , is a general form of the expected log likelihood of c given τ . Note that , the posterior training is on-policy , meaning that the rollouts are collected online using the current Algorithm 1 ADAIL 1 : Inputs : 2 : An environment class E. 3 : Initial parameters of policy θ , discriminator ω , and posterior φ . 4 : A set of expert demonstrations { τexp } on one of the environment eexp ∈ E. An environment generator g ( c ) that takes a code and generates an environment ec ∈ E. A prior distribution of p ( c ) . 5 : for i = 1 , 2 , .. do 6 : Sample c ∼ p ( c ) and generate environment ec ∼ g ( c ) 7 : Sample trajectories τi ∼ πθ ( ·|c ) in ec and τei ∼ { τexp } 8 : Update the discriminator parameters ω with the gradients : Ê ( s , a ) ∼τi [ ∇w log ( Dw ( s , a ) ) ] + Ê ( s , a ) ∼τei [ ∇w log ( 1−Dw ( s , a ) ] 9 : Update the discriminator parameters ω again with the following loss , such that the gradients are reversed when back-prop through the dynamics-invariant layer : −Ê ( s , a ) ∼τi [ log ( D R ( c|s , a ) ) ] 10 : Update the posterior parameters φ with gradients Êτi [ ∇φ logQφ ( c|τi ) ) ] 11 : Update policy πθ ( ·|c ) using policy optimization method ( PPO ) with : Ê ( s , a ) ∼τi [ − log ( 1−Dω ( s , a ) ) ] 12 : Output : Learned policy πθ , and posterior Qφ . policy , thereby the last term of the objective is dependent on θ . One can employ various supervised and unsupervised methods towards optimizing this term . We will explore a few methods in the following subsections . The algorithm is outlined in Algorithm 1 .

Summary: The submission considers the problem of imitation learning when the dynamics of the expert are not known to the agent and the dynamics of the agent may change frequently. It is however assumed that the agent has access to a parameterized simulator that can simulate the expert dynamics. The parameters for the simulator are not known but are assumed to be drawn from a known distribution.