Adaptive Adversarial Imitation Learning
Authors: Yiren Lu
Jonathan Tompson
Sergey Levine
Venue:  (2020)
Keywords: Imitation Learning
Reinforcement Learning

Abstract: We present the ADaptive Adversarial Imitation Learning (ADAIL) algorithm for learning adaptive policies that can be transferred between environments of varying dynamics, by imitating a small number of demonstrations collected from a single source domain. This problem is important in robotic learning because in real world scenarios 1) reward functions are hard to obtain, 2) learned policies from one domain are difficult to deploy in another due to varying source to target domain statistics, 3) collecting expert demonstrations in multiple environments where the dynamics are known and controlled is often infeasible. We address these constraints by building upon recent advances in adversarial imitation learning; we condition our policy on a learned dynamics embedding and we employ a domain-adversarial loss to learn a dynamics-invariant discriminator. The effectiveness of our method is demonstrated on simulated control tasks with varying environment dynamics and the learned adaptive agent outperforms several recent baselines.

Under review as a conference paper at ICLR 2020
ADAPTIVE ADVERSARIAL IMITATION LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
We present the ADaptive Adversarial Imitation Learning (ADAIL) algorithm for learning adaptive policies that can be transferred between environments of vary- ing dynamics, by imitating a small number of demonstrations collected from a single source domain. This is an important problem in robotic learning because in real world scenarios 1) reward functions are hard to obtain, 2) learned policies from one domain are difficult to deploy in another due to varying source to target domain statistics, 3) collecting expert demonstrations in multiple environments where the dynamics are known and controlled is often infeasible. We address these constraints by building upon recent advances in adversarial imitation learn- policy
ing; we condition a aadeiinad loss to learn a mnnaiccs-nnvarial discriminator. The effee- tiveness of our method is demonstrated simulated control tasks with varying environment dynamics and the learned adaptive agent outperforms several recent baselines.
learned dynamics embedding and we employ
INTRODUCTION
Humans and animals can learn complex behaviors via imitation. Inspired by these learning mecha- nisms, Imitation Learning (IL) has long been popular method for training autonomous agents from human-provided demonstrations. However, human and animal imitation differs markedly from com- monly used approaches in machine learning. Firstly, humans and animals tend imitate the goal of the task rather than the particular motions of the demonstrator (Baker et al. 2007). Secondly, humans and animals can easily handle imitation scenarios where there is shift in embodiment and dynamics between themselves and demonstrator: The first feature of human II. can be represented within the framework of Inverse Reinforcement Learning (IRI.) (Ng al. 2000 Abbeel & Ng. 2004 Ziebart et al.) 2008). which at a high level casts the problem of imitation as one of matching outcomes rather than actions. Recent work in adversarial imitation learning (Ho & Ermon 2016
Finn et al. 2016) has accomplished this by using discriminator to judge whether a given behavior is from an expert or imitator, and then a policy is trained using the discriminator expert likelihood as reward. While successful in multiple problem domains, this approach makes it difficult to accom- modate the second feature of human learning: imitation across shifts in embodiment and dynamics. This is because in the presence of such shifts, the discriminator may either simply use the embod- iment or dynamics to infer whether it is evaluating expert behavior, and as a consequence fails to provide a meaningful reward signal.
In this paper we are concerned with the problem of learning adaptive policies that can be transferred environments with varying dynamies, by imitating small number of expert demonstrations col- lected from single source domain. This problem is important in robotic learning because it is better aligned with real world constraints: reward functions are hard obtain, 2) learned policies from one domain are hard to deploy to different domains due to varying source to target domain statistics, and 3) the target domain dynamics oftentimes changes while executing the learned policy. As such, this work assumes ground truth rewards are not available, and furthermore we assume that expert demonstrations come from only a single domain (i.c. an instance of an environment where dynamics cannot be exactly replicated by the policy at training time). To the best of our knowledge, this is the first work to tackle this challenging problem formulation.
Our proposed method solves the above problem by building upon the GAIL (Ho & Ermon 2016 Finn et al.) (0166) framework, by firstly conditioning the policy on a learned dynamics embedding ("context v in policy search literature (Deisenroth et al. (013)). We propose two embedding

---

Under review as a conference paper at ICLR 2020
approaches which the policy is conditioned, namely. direct supervised learning approach and variational autoencoder (VAE) (Kingma & Welling 2013) based unsupervised approach. Secondly, to prevent the discriminator from inferring whether it is evaluating the expert behavior or imitator behavior purely through the dynamics, we propose using a Gradient Reversal Layer (GRL.) to learn Synamics-invariant discriminator. We demonstrate the effectiveness of the proposed algorithm on benchmark Mujoco simulated control tasks.
The main contributions of our work include:
1) present general and novel problem formulation that is well aligned with real world scenarios in comparison to recent literature 2) devise conceptually simple architecture that capable of learning an adaptive policy from a small number of expert demonstrations (order of 10s) collected from only one source environment, 3) design an adversarial loss for addressing the covariate shift issue discriminator learning.
2 RELATED WORK
Historically, two main avenues have been heavily studied for imitation learning: 1) Behavioral Cloning (BC) and 2) Inverse Reinforcement Learning (IRL). Though conceptually simple, BC suf- fers from compound errors caused by covariate shift, and subsequently, often requires large quan- tity of demonstrations (Pomerleau 1989), or access to the expert policy (Ross et al.) 2011) in order to recover stable policy.
Recent advancements in imitation learning (Ho & Ermon 2016 Finn et al. 2016) have adopted
an adversarial formation that interleaves between 1) discriminating the generated policy against the expert demonstrations and 2) a policy improvement step where the policy aims to fool the learned discriminator.
Dynamics randomization (Tobin et al.) 2017 Sadeghi & Levine 2016 Mandlekar et al. 2017 Tan
al.) 2018: Pinto al.) 2017; Peng et al. 2018; Chebotar al.) 2018: Rajeswaran al.) 2016)
has been one of the prevailing vehicles for addressing varying simulation to real-world domain statistics. This avenue of methods typically involves perturbing the environment dynamics (often times adversarially) in simulation in order to learn an adaptive policy that is robust enough to bridge the "Reality Gap""
While dynamics randonization has been explored for learning robust policies in RL setting, it has critical limitation the imitation learning context: large domain shifts might result in direc- tional differences in dynamics, therefore, the demonstrated actions might longer be admissible for solving the task in the target domain. Our method (Figure 2 also involves training in a vari- ety of environments with different dynamics. However, we propose conditioning the policy an explicitly learned dynamics embedding to enable adaptive policies based on online system ID.
Yu al. (017) adopted similar approach towards building adaptive policies. They learn an online system identification model and condition the policy on the predicted model parameters in an RL setting. In comparison their work, we do not assume access to the ground truth reward signals or the ground truth physics parameters at evaluation time, which makes this work's problem for- mulation harder learning problem. but with greater potential for real-world applications. We will compare our method with Yu et al. (2017) in the experimental section.
Third person imitation learning (Stadie et al., 2017) also employs GRL (Ganin & Lempitsky 2014) under GAlL-like formulation with the goal of learning expert behaviors in a new domain. In comparison, our method also enables learning adaptive policies by employing an online dynamics identification component, so that the policies can be transferred to class of domains, as opposed to one domain. addition, learned policies using our proposed method can handle online dynamics perturbations.
Meta learning (Finn et al., 2017) has also been applied to address varying source to target domain dynamics (Duan et al. 2017; Nagabandi al. 2018). The idea behind meta learning in the context of robotic learning is to learn a meta policy that is "initialized" for a variety of tasks in simulation, and then fine-tune the policy in the real-world setting given specific goal. After the meta-learning phase, the agent requires significantly fewer environment interactions to obtain a policy that solves the task. In comparison to meta learning based approaches, fine-tuning on the test environment is
2

---

Under review as a conference paper at ICLR 2020
not required in method, with the caveat being that this is true only within the target domain where the dynamics posterior is effective.
2.1 BACKGROUND
In this section we will briefly review GAIL (Ho Ermon 2016). Inspired by GANs, the GAIL objective is defined as:
min max E log (s. + E [log(1 (1
(1)
Where g denotes the expert policy that generated the demonstrations; Th is the policy to imitate the expert; D is discriminator that learns to distinguish between ITy and TE with generated state- action pairs. In comparison to GAN optimization, the GAIL objective is rarely differentiable since differentiation through the environment step is often intractable. Optimization is instead achieved via RL-based policy gradient algorithms, e.g., PPO (Schulman et al.) (017) or off policy methods, e.g., TD3 (Kostrikov al., 2018). Without an explicit reward function, GAIL relies on reward signals provided by the learned discriminator, where a common reward formulation is rw, -log(1 (w(s.))) (s.
ADAPTIVE ADVERSARIAL IMITATION LEARNING (ADAIL)
3.1 PROBLEM DEFINITION
Suppose we given class E of environments with different dynamics but similar goals, domain generator g(c) which takes in a code and generates an environment e, E E, and set of expert demonstrations [Teap] collected from one source environment Gezp E. In adaptive imitation learning, one attempts to learn an adaptive policy Ta that can generalize across environments within E. We assume that the ground truth dynamics parameters c, which are used to generate the simulated environments, are given (or manually sampled) during the training phase.
3.2 ALGORITHM OVERVIEW
We allow the agent to interact with a class of similar simulated environments with varying dynamics parameters, which we call "adaptive training". To be able capture high-level goaly from small set of demonstrations, we adopt approach similar to GAIL.. To provide consistent feedback signals during training across environments with different dynamics, the discriminator should be dynamics- invariant. We enable this desirable feature by learning dynamics-invarian feature layer for the discriminator by 1) adding another head [c(sss, a) the discriminator predict the dynamics parameters, and 2) inserting GRL in-between D and the dynamics-invariant feature layer. The discriminator design illustrated in Figure[] In addition, to enable adaptive policies, we introduced dynamics posterior that takes a roll-out trajectory and outputs embedding, on which the policy is conditioned. Intuitively, explicit dynamics latent variable learning endows the agent with the ability to identify the system and act differently against changes in dynamics. Note that policy can learn to infer dynamics implicitly, without the need for an external dynamics embedding. However, we find experimentally that policies conditioned explicitly the environment parameters outperform those that do not. The overall architecture is illustrated in Figure 2 We call the algorithm Adaptive Adversarial Imitation Learning (ADAIL), with the following objective (note that for brevity, we for now omit the GRL term discussed in Section 3.4):
min max Exg log (s, a)] + E (-1e) [lg([ - Ds + E [log Qo(clT)] 6..
(2)
Where is learned latent dynamics representation that is associated with the rollout environment in each gradient step: 5 is a roll-out trajectory using (-(-)c) in the corresponding environment; Q(c[r) is a "dynamics posterior" for inferring the dynamics during test time; The last term in the objective, E [log (c])), is a general form of the expected log likelihood of given т. Note that, the posterior training is on-policy, meaning that the rollouts collected online using the current
3

---

Under review as a conference paper at ICLR 2020
Algorithm ADAIL
1: Inputs:
2: An environment class E.
3: Initial parameters of policy 8. discriminator w. and posterior d.
set of expert demonstrations {Tezp) on one environment Gezp E environment generator
g(c) that takes code and generates an environment 6, E prior distribution of p(c).
5: for 1, 2, do
6: Sample p(c) and generate environment g(cc)
7: Sample trajectories T; (T(+)c) in c. and 7f [Trop)
8: Update the discriminator parameters with the gradients: Fi(aa))er VV log( (s, +
B(o.a)=rr log(1 (s. a)]
9:
Update the discriminator parameters again with the following loss, such that the gradients versed when pack-prop through the dynamics- layer: - ELs,a)rr, [log( (c(ss,))))
10: Update the posterior parameters & with gradients E., V log [c(c[r]]]
11: Update policy mo(+lc) using policy optimization method (PPO) with: E(aaa)rT I- log(1 - (s,
12:
Output: Learned policy Ta, and posterior Qo-
Dynamics
Closs tebel
not
(s. a) =
Dynamics Parameters
Gradeeis
Gradient Reveraal Layer
Figure 1:
Discriminator with Gradients Reversal Layer (GRL). The red layer the GRL which reverses the gradients during backprop. The yellow layer is dynamics-invariant layer that is shared with the classification task.
policy, thereby the last term of the objective is dependent on 0. One can employ various supervised and unsupervised methods towards optimizing this term. We will explore a few methods in the following subsections.
The algorithm outlined in Algorithm
3.3 ADAPTIVE TRAINING
Adaptive training is achieved through 1) allowing the agent interact with a class of similar simu- lated environments within class E, and 2) learning dynamics posterior for predicting the dynamics based on rollouts. The environment class E is defined as a set of parameterized environments with degrees of freedom, where is the total number of latent dynamics parameters that we can change. We assume that we have access environment generator g(c) that takes in sample of the
G lobel 01
6. di.))
Dynomics Posterior
Discriminate
Envinonment
Policry
GL a labal
Demonstmation
Figure 2: The ADAII. architecture. "Environment" is sampled from a population of environments with varying dynamics, "Demonstrations" are collected from one environment within the environ- ment distribution, "Posterior" is the dynamics predictor, Q(clr): Latent code "c" represents the ground truth or learned dynamics parameters; The policy input is extended to include the latent dynamics embedding

---

Under review as a conference paper at ICLR 2020
1s. 0.
Encoder (Posterior)
ple)
s.
Decodor Regressor)
q(c|s,a.5)
p(sts..c)
Figure 3: VAE-based unsupervised dynamics latent variable learning.
dynamics parameters с and generates an environment. At each time when an on-policy rollout is initiated, we re-sample the dynamics parameters e based on predefined prior distribution p(c).
3.4 LEARNING A DYNAMICS-INVARIANT DISCRIMINATOR
GAIL learns from the expert demonstrations by matching an implicit state-action occupancy mea- sure. However, this formulation might be prohlematic our training setting, where on-policy rollouts are collected from environments with varying dynamics. In non-source environments, the discriminator can no longer provide canonical feedback signals. This motivates us to learn a dynamics-invariant feature space, where, the behavior-oriented features preserved but dynamics- identifiable features are removed. We approach this problem by assuming that the behavior-oriema characteristics and cynamics--dentiffe learn a dynamics-invarian nique called Gradient Reversal Layer (GRL) (Ganin & Lempitsky (2014), which is widely used in image domain adaptation (Bousmalis et al. (016). The dynamics-
characteristics are loosely coupled and thereby representation for the discriminator. In particular, we employ a tech-
can
lynamics-invarriant invariant features layer is shared with the original discriminator classification head, illustrated in Figure
3.5 DIRECT SUPERVISED DYNAMICS LATENT VARIABLE LEARNING
Perhaps one of the best latent representations of the dynamics is the ground truth physics param- eterization (gravity, friction, limb length, etc). In this section we explore supervised learning for inferring dynamics. A neural network is employed to represent the dynamics posterior, which is learned via supervised learning by regressing to the ground truth physics parameters given a replay buffer of policy rollouts. We update the regression network using Huber loss to match environment dynamics labels. Details about the Huber loss can be found in appendix A.I During training, we condition the learned policy the ground truth physics parameters. During evaluation, the other hand, the policy is conditioned on the predicted physics parameters from the posterior.
We use (state, action, next state) the posterior's input, i.e., Qo(c)s, a, s'), and 3-layer fully- connected neural network to output the N-dimensional environment parameters. Note that one can recurrent neural network and longer rollout history for modeling complex dynamic structures. however we found that this was not necessary for the chosen evaluation environments.
3.6 VAE-BASED UNSUPERVISED DYNAMICS LATENT VARIABLE LEARNING
For many cases, the number of varying latent parameters of the environment is high, one might not know the set of latent parameters that will vary in a real world laboratory setting, or the latent parameters are oftentimes strongly correlated (e.g., gravity and friction) in terms of their effect on environment dynamics. In this case, predicting the exact latent parameterization is hard. The policy mainly concerned with the end effector of the latent parameters. This motivates us to use unsupervised tool to extract a latent dynamics embedding. In this section, we explore VAE-based unsupervised approach similar to conditional VAE (Sohn et al., 2015) with an additional contrastive regularization loss, for learning the dynamics without ground truth labels.
With the goal of capturing the underlying dynamics, we avoid directly reconstructing the (state, action, next state) tuple, (s, s'). Otherwise, the VAE would likely capture the latent structure of the state space. Instead, the decoder modified to take-in the state-action pair, (s, a), and a latent code, and outputs the next state, s'. The decoder now becomes forward dynamics predictive model. The unsupervised dynamics latent variable learning method is illustrated in Figure
The evidence lower bound (ELBO) used is:
5

---

Under review as a conference paper at ICLR 2020
Figure 4: Vary x-component of gravity in HalfCheetah environment. The red arrows in the picture show the gravity directions.
ELBO= (e (log Pols'|s, a, (L,
(3)
Where Qcc)s, s') is the dynamics posterior (encoder); P(*'|s, c) is forward dynamics pre- dictive model (decoder); P(c) is Gaussian prior over the latent code c. Similar to Davis et al. (2077) and Hsu & Kira (2015), avoid the encoder learning an identity mapping on s', we add the following KL-based contrastive regularization to the loss:
Leontrastive KL(Q& au (s),
min (onn(LL(Q 02 16)1 0s 16)), Dob
Where (sp, ao, s6) and (81, 01 $4) are sampled from the same roll-out trajectory; (s2, d2 5/2) and ($3, (3, s'y) are sampled from different roll-out trajectories. Do is constant. We use this regular- ization introduce additional supervision in order improve the robustness of the latent posterior.
The overall objective for the dynamics learner is
min ELBO + XL contrasttve
(4)
&.1"
where À is scalar to control the relative strength of the regularization term. The learned poste- rior (encoder) infers the latent dynamics, which is used for conditioning the policy. The modified algorithm can be found in the appendix (Algorithm) 2
EXPERIMENTS
4.1 ENVIRONMENTS
To evaluate the proposed algorithm consider 4 simulated environments: CartPole. Hopper, HalfCheetah and Ant. The chosen dynamics parameters are specified table 3 and an example of one such parameter (HalfCheetah gravity component a:) is shown in Figure
4 During training the
parameters are sampled uniformly from the chosen range. Source domain parameters are also given the table For each source domain, collect 16 expert demonstrations.
Gym CartPole-VO, VO: We vary the force magnitude in continuous range [-1, 1] in our training setting. Note that the force magnitude can take negative values, which flips the force direction.
3 Mujoco Environments: Hopper, HalfCheetah, and Ant: With these three environments, we vary 2d dynamics parameters: gravity x-component and friction.
Environment
Paramater
Parameter
Source
CartPole- VO
Fm
[111]]
Fmm 1.0
Hopper
Gæ
[-1.0,
1.0]
Fr
[1.5, 2.5]
Gæ 0.0, Fr Fr = 2.0
HalfCheetah
[-3.0, 3.0]
[0.0, 2.0
0.0, Fr
Ant
Gæ
[.5.0,
Fr
[0.0, 4.0]
Gz = Fr =
Table I: Environments. Fm Force magnitude; Gæ=Gravity x-component; Fr = Friction. For each environment, collect 16 expert demonstrations from the source domain.

---

Under review as a conference paper at ICLR 2020
4.2 ADAIL ON SIMULATED CONTROL TASKS
Is the dynamics posterior component effective under large dynamics shifts?
We first demonstrate the effectiveness of the dynamics posterior under large dynamics shifts on a toy Gym environment, Cartpole, by varying 1d force magnitude. As the direction of the force changes, blindly mimicking the demonstrations collected from the source domain (Fm 1.0) would not work on target domains with Fm < 0.0. This result is evident when comparing ADAIL to GAIL with dynamics randomization. As shown Figure 5a GAIL with Dynamics Randomization failed to generalize to Fm 0.0, whereas, ADAIL is able to achieve the same performance as Fm > 0.0. We also put a comparison with ADAIL-rand, where the policy is conditioned on uniformly random values of the dynamics parameters, which completely breaks the performance across the domains.
How does the GRL help improve the robustness of performance across domains?
To demonstrate the effectiveness of GRL in the adversarial imitation learning formulation, do a comparative study with and without GRL on GAIL with dynamics randomization the Hopper environment. The results are shown in Figure 5b
-98
PCo Scaett
AGM vaed
-95
-1.3
2002
0032
2002
2032
1002
1032
1203
105
Foice Hegriaue
1.0
1.6
1.3
77
3.0
7.4
1.0
2.0
77
(a)
(b)
Figure 5: (a): ADAIL on CartPole-VO Blue: PPO Expert; green: GAIL with Dynamics Ran- domization; red: ADAIL with latent parameters from the dynamics posterior; light blue: ADAIL with uniformly random latent parameters. (b): GAIL with Dynamics Randomization without (left, 2024.89 + 669.39) or with (right, 2453.63 63 = 430.51) GRL on Hopper
How does the overall algorithm work comparison with baseline methods?
We demonstrate the overall performance of ADAIL by applying it to three Mujoco control tasks: HalfChoetah, Ant and Hopper. For each of the Mujoco environments, we vary 2 continuous dynam- ies parameters and we compare the performance of ADAIL with few baseline methods, including 1) the PPO expert which was used to collect demonstrations; 2) the UP-true algorithm of Yf et al. (017)), which is essentially PPO policy conditioned ground truth physics parameters; and 3) GAIL with dynamics randomization, which is unmodified GAIL training on a variety of environ- ments with varying dynamics. The results of this experiment show in Figure 6
The experiments show that 1) as expected the PPO expert (Plot 6al has limited adapt- ability to unseen dynamics. 2) UP-true (Plot 6b) achieves similar performance across envi- romments. Note that since UP-true has access to the ground truth reward signals and the policy is conditioned on ground truth dynamics parameters, the Plot 6b shows an approximate expected up- bound for our proposed method since we do not assume access to reward signals during policy training, or to ground truth physics parameters at policy evaluation time. 3) GAIL with dynamics randomization (Plot ??) can generalize to some extent, but failed to achieve the demonstrated per- formance in the source environment (gravity x = 0.O, friction = 0.5) 4) Plots 91 show evaluation of the proposed method ADAIL with policy conditioned predicted physics parameters respectively: ADAIL matches the expert performance the source environment (gravity 0.0, friction 0.5) and generalizes unseen dynamics. In particular, when the environment dynamics favors the task, the adaptive agent was able to obtain even higher performance (around friction 1.2, gravity 2).
HalfCheetah
ground truth physics parameters and
Ant and Hopper. We again show favorable performance on both Ant and Hopper in Figure
How does the algorithm generalize to unseen environments?

---

Under review as a conference paper at ICLR 2020
wcb
HallCheetah
(a) PPO Expert (2991.23 + 2020.93)
(b) UP-true
(c) GAlL-rand
(d) ADAIL
(3441.76 + 1248.77)
(3182.72 1753388
(283. 20 + 1569.31)
(e) PPO Expert
(1) UP-true
(g) GAIL-rand
(h) ADAIL
Ant
(1972.06 2630.20)
(1521.14 1792.74)
(1579.36 + 2082.10
(2119.90 2531.03)
lus
Hopper
(i) PPO Expert (2196.88 955.15)
(j) Ut-true (2225.49 + 830.02)
(k) GAIL-rand (2024.89 89 + 669.39)
(1) ADAIL (2352.44 620.64)
Figure 6: Comparing ADAIL with baselines on Mujoco tasks. Each plot is heatmap that demon- strates the performance of an algorithm in environments with different dynamics. Each cell of the plot shows 10 episodes averaged cumulative rewards on particular 2D range of dynamics. Note that to aid visualization, we render plots for Ant log scale.
8

---

Under review as a conference paper at ICLR 2020
40
155
1000
(a) ADAIL-true (5x5)
(b) -Drll.-pred (Sx5)
(c) Posterior RMSE (5x5)
Figure 7: Generalization of policy held out parameters the HalfCheetah environment. The red rectangles in plots show the blackout regions not seen during policy training.
To understand how ADAIL generalizes environments sampled training time, do suite of studies in which the agent is only allowed to interact in a limited set of environments. Figure 7 shows the performance of ADAIL on different settings, where 5 region of enviromment parame- ters including the expert source environment are "blacked-out", This case is particularly challenging since the policy is not allowed access the domain from which the expert demonstrations were col- lected, and so our dynamics-invariann discriminator is essential. For additional held out experiments see Appendix A.5
The experiments show that, 1) without training on the source environment, ADAIL with the ground truth parameters tends to have performance drops on the blackout region but largely is able to gener- alize (Figure 7a): 2) the posterior's 's RMSE raises the blackout region (Figure 7c); 3) consequently ADAIL with the predicted dynamics parameters suffers from the posterior error on the blackout re- gion (Figure 7b).
How does unsupervised version of the algorithm perform?
VAE-ADAIL HalfCheetah. With the goal of understanding the characteristics of the learned dy- namics latent embedding through the unsupervised method and its impact on the overall algorithm, proof of concept apply VAE-ADAIL to HalfCheetah environment varying ID continuous dynamics, friction. The performance shown in Figure X
Mean Cum Rews
4000
2000
GAIL Dyanmics Randomization
1000
ADAIL w/ GRL
65
Friction
Figure 8: VAE-ADAIL performance on HalfCheetah
5 CONCLUSION
In this work we proposed the ADaptive Adversarial Imitation Learning (ADAIL) algorithm for learn- ing adaptive control policies from limited number of expert demonstrations, We demonstrated the effectiveness of ADAIL on two challenging MuloCo test suites and compared against recent SoTA. We showed that ADAIL extends the generalization capacities of policies to unseen environments, and we proposed variant our algorithm, VAE-ADAIL, that does not require environment dy- namics labels at training time. We will release the code aid in reproduction upon publication.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. ACM, 2004.
Chris L Baker, Joshua B Tenenbaum, and Rebecca R Saxe. Goal inference as inverse planning. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 29, 2007.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Advances in neural information processing systems, pp. 343-351, 2016.
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. arXiv preprint arXiv: / 8/0.05687, 2018.
Jason Davis, Brian Kulis, Prateek Jain, Suvrit Sra, and Inderjit S Dhillon. information-thcore metric learning. In Proceedings of the 24th international conference Machine learning, pp- 209-216. ACM, 2007.
Mare Peter Deisenroth, Gerhard Neumann, Jan Peters, et al. A survey policy search for robotics. Foundations and Trends() in Robotics, 2(1-2):1-142, 2013.
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-shot imitation learning. Advances in neural information processing systems, pp. 1087-1098, 2017.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pp. 49-58, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1136-1135. JMLR. org, 2017.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409.744: 7495, 2014.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565-4573, 2016.
Yen- Ken-Chang Hsu and Zsolt Kira. Neural network-based clustering using pairwise constraints. arXiv preprint arXiv:1511.06322 2015.
Diederik Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Ilya Kostrikov, Kumar Krishna Agrawal, Sergey Levine, and Jonathan Tompson. Addressing sample inefficiency and reward bias in inverse reinforcement learning. ICLR, 2018.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of physicall-pplauiie perturbations. In 2017 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pp. 3932-3939. IEEE, 2017.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt in dynamic. real-world environments through meta- reinforcement learning. arXiv preprint 1rXiv:1803.11344 2018.
Andrew Y Ng, Stuart J Russell, al. Algorithms for inverse reinforcement learning. In leml, volume 1, pp. 2, 2000.
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba. and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-8. IEEE, 2018.
10

---

Under review as a conference paper at ICLR 2020
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 2817-2826. JMLR. org. 2017.
Dean A Pomerleau. Alvinn: An autonomous land vehicle a neural network. In Advances in neural information processing systems, pp. 305-313, 1989.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv:1610.01283, 2016.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635, 2011.
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without single real image. arXiv preprint a6 1.04201, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, pp- 3491, 2015.
Bradly C Stadie, Pieter Abbecl, and Ilya Sutskever. Third Third-person imitation learning. arXiv preprint 17X3v:1003.07700 2017.
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv: 1804. 10332, 2018.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 20-30. IEEE, 2017.
Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. Preparing for the unknown: Learning universal policy with online system identification. arXiv preprint arXiv: 7002 2017.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, II. USA, 2008
A
APPENDIX
A.1 HUBER LOSS FOR DYNAMICS EMBEDDING Loss
We use the following loss function when training the dynamics embedding posterior:
| - Qo())) [8|c - Qa(()) 3,2 otherwise
for Ic lo (c- <
Lg(c, Qs(T))
(5)
Where controls the joint position between L.2 and L1 penalty in Huber loss.
Lemma 1. Minimizing the above Huber loss is equivalent to maximizing the log likelihood,
log T) assuming P(c/7) is distributed as a Gaussian distribution when |c c- Q ) < 8, and as Laplace distribution otherwise. See appendix A.2 for the proof.

---

Under review as a conference paper at ICLR 2020
A.2
LEMMA PROOF
Proof. For Qo(T)] < <,
(o- Q()))
log P(c/7) log
20
0n is positive constant
(6)
v /
log P(c|7) = log
(c -
Q(-)) Q(T))²
(7)
V
20
v log P(c/) - (log
(c ((T))
VTT0,
20
C.V. (c-(()))²
C1 is positive constant
(9)
=-CiVlae Qo(T))
(10)
Likewise, we can prove for Qo(T)) 2 8.
A.3 VAE-ADAIL ALGORITHM
Algorithm VAE-ADAIL
1: Inputs:
An environment class
3: Initial parameters of policy 0, discriminator w, and dynamics posterior Φ, 1/.
4: A set of expert demonstrations } on one of the environment Geap E E.
5: for 2, do
Sample environment e E.
Sample trajectories 7; ~ =o(:1Qo) and of ~ [Texp}
8: Update the discriminator parameters w with the gradients
((,,a),,, log[ a))] E( [V... log (m, a)]
9: Update the posterior parameters é, 1 with the objective described in Eq (3) & Update policy no <-(-)c using policy optimization method (TRPO/PPO) with:
10:
B log(1 (s(s,a))]
11: Output: Learned policy and posterior Q+-
A.4 HALFCHEETAH ADAIL PERFORMANCE COMPARISON
Figure 9: Comparing ADAIL with few basclines HalfCheetah. Each plot is heatmap that demonstrates the performance of an algorithm in environments with different dynamics. Each cell of the plot shows 10 episodes averaged cumulative rewards on particular 2D range of dynamics.
12

---

Under review as a conference paper at ICLR 2020
100
(a) PPO Expert
(b) GAIL (2189.72 76 + 2110.70)
(c) GAIL-rand (3182.72 + 1753.86)
State- only GAIL-rand
(2991.23- + (3)
(3301.20 .20 + 1350.29)
1500
100
000
534
100
000
500
034
reld
Vitann
(c) tue
(f) ADAIL-true
(g) ADAIL-pred (4283.20 + 1569.31)
(h) Posterior RMSE
(3441.76 1248.77)
(4419.75 75 1493.54)
(1.03 + 0.36)
A.5 HELD-OUT ENVIRONMENT EXPERIMENT
A.6 HYPERPARAMETERS
A. ADAIL
We used fully connected neural networks with 2 hidden layers for all three components of the system.
The Network hyperparameters for each of the test environments with 2D dynamics parameters are shown in the following table. For UP- True and GAIL-rand,
the same set of hyperparameters.
Environment
Policy
Discriminator
Posterior
Architecture
Learning rate
Architecture
Learning rate
Architecture
Learning rate
CartPole- VO
(s.a) 64 64
0,0005586
(s,a) 32 32
0,000167881
(s,a.s" 140-2
0.00532
Hopper
(s,a) - 64 64 (a)
0.000098646
(2.a)
0.0000261
(s,a.s" (s.a.5))-241-2336
0.00625
HalfCheetah
(s,a) 64 64 (a)
0.00005586
(s,a) 32 32-1
0.0000167881
(s,a,8 (s,a.5")-150-1500
0.003
Ant
(8,a) 64 64
0.000047
(s,a) 32 32
0.000037
(s.a.5)-72-177-22
0.002353
Table 2: ADAIL network architectures and learning rates on test environments
A.6.2 VAE-ADAIL
Here we show the network architectures and learning rates for VAE-ADAIL.
Encoder (Posterior)
Decoder
Policy
Discriminator
Architecture Learning rate
(s,a,s") 200 200 (c)
(s,a,c) 200 200 (8')
(s.a) 64 64 (a)
(s,a) 32 32 (1.c)
0.000094
0.000094
0.00005596
0.000046077
Table 3: VAE-ADAIL network architectures and learning rates
13


Reviews:
[{"review_id": "SkxqREopYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe submission considers the problem of imitation learning when the dynamics of the expert are not known to the agent and the dynamics of the agent may change frequently. It is however assumed that the agent has access to a parameterized simulator that can simulate the expert dynamics. The parameters for the simulator are not known but are assumed to be drawn from a known distribution.\nThe proposed method is based on GAIL but uses several modifications:\n- A contextual policy is trained that also takes the dynamics-parameters as additional input. At each iteration, a new environment is sampled for performing the policy-rollout.\n- A \"posterior\" prediction network is trained to maximize the likelihood of the parameters that were used for the different roll-outs. This network is used for the test case, where the true dynamics of the agent are not known.\n- The discriminator might use features of the state-action input that correlate with the corresponding dynamics. Classifying based on such features may be undesirable because the discriminator might no longer produce useful rewards. In order to address this problem, an additional head is added to the discriminator that outputs a prediction of the dynamic parameters. The prediction error is trained by backpropagation, however by flipping the sign of the gradient at the last shared layer, the features of the discriminator are optimized to be unsuited for predicting the dynamic parameters (the technique is known as Gradient Reversal Layer).\n- A VAE-based method for learning latent dynamic parameters is proposed, by training a conditional VAE to reconstruct the next state, where the current state and action are provided as context to the encoder and decoder.\n\nContribution:\nOne of the strong-points of the submission is the fact that it features several different, orthogonal contributions. I also think that the considered problem setting is relatively interesting. However, also when considering real applications such as robotics, I am not convinced that explicitly modeling the dynamic changes is necessary. Some existing imitation learning methods focus on a setting where the dynamics of the expert may differ from the agent, but the dynamics of agents do not change. This setting does not require dynamic-contextual policies and seems to be applicable to typical robot applications.\n\nSoundness:\nThe different components of the proposed methods seem reasonable to me. They do not come with (and arguably do not require) new derivations but seem rather like pragmatic solutions for the encountered problem.\nThe optimization problem (Eq.2) seems to be formulated slightly inaccurate, because the last term should in my opinion not depend on theta. If I understand correctly, the policy should not maximize the likelihood of the dynamics posterior.\nThe contrastive regularization loss needs to be better motivated. A high KL in the first term may not necessary be bad, for example, if the confidence in the prediction of (s_0, a_0, s'_0) is lower compared to (s_1, a_1, s'_1). If a similar regularizer has been used in prior work, such work should be referenced. Otherwise, it needs to be motivated.\n\nPresentation/Clarity:\nThe presentation of the work is arguably the main weakness of the paper.\nThe submission does not seem polished. It contains a large amount of typos. Figure 2 is confusing and adds little compared to the text description. Also the structure could be improved. For example, the submission introduces the posterior loss and outlines the algorithm before describing the individual components.\nThe paper uses some techniques such as conditional VAEs [1] or contextual policy search [2]\nare used but not described / referenced.\n\nEvaluation:\nI like that the different aspects of the proposed method are also evaluated individually. The ablations with respect to the adaptability and GRL are crucial. The evaluation of the performance could be improved. PPO and UP-True use the true reward function, so the only real competitor is a naive GAIL-baseline that uses randomized dynamics during training. I'm not aware of prior work that considers the exact same setting as the manuscript. However, one of the main arguments for inverse reinforcement learning is the claimed generalizability of a reward function as opposed to a policy. I see that learning a new policy after each change in the dynamics may be too costly in some settings. However, comparisons to methods such as AIRL that aim to learn reward functions that are robust to changes in the dynamics would be highly interesting.\n\n[1] Sohn Kihyuk, Honglak Lee, and Xinchen Yan. “Learning Structured Output Representation using Deep Conditional Generative Models.” Advances in Neural Information Processing Systems. 2015.\n[2] Marc Peter Deisenroth, Gerhard Neumann, and Jan Peters.  A survey on policy search forrobotics.Foundations and Trends in Robotics, 2(1–2):328–373, 2013.\n\nComparison: How about IRL, e.g. AIRL?\nWhat about state-only GAIL?\n\nTypos:\nEquations are not properly integrated into the sentences (missing punctuations)\n\"domains, as oppose to one domain.\"\n\"Inspired by to GANs\"\n\"and generates a environment\"\nFigure 5a (legend): \"dyanmics\"\n\"that can generalized across\"\nAlgorithmbox: \"A environment\", \"and Generate environment\"\n\"is achieved through 1) allowing\"\n\"the policy is mainly concerned with the end-effector of the latent parameters\"\n\n\nQuestion: \nWhat are the network architectures?\n\nAccording to line 10 of the algorithmbox only the current trajectory is used for updating the dynamics posterior. Why?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgSu9lhYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes an approach that combines domain adversarial neural network with generative adversarial imitation learning. In the setup, each environment is dependent on some latent context variable (e.g. physics parameters) through which latent variable dependent policy and latent variable independent discriminators are learned. \n\nI don't think the exact same idea has appeared in existing literature. The authors justifies its connections and differences between third person imitation learning, but it seems that the proposed method bear some similarities to the NeurIPS19 papers below. \n\nhttps://arxiv.org/pdf/1909.09314.pdf\nhttps://drive.google.com/file/d/1urPE7J5tT8dzoBHSFvZKwNLsQieU706o/view\n\nThe following papers also assumed that GAIL-like methods in a meta learning setup, where environments depend on context. Perhaps the difference here is that the discriminator is also trained with a gradient reversal layer, so it encourages the discriminator to not use redundant state information. Also in this paper the source domain only contains demos from one env, which might highlight the importance of the gradient reversal layer.\n\n\nQuestions:\n\nIn \"dynamics learning\", it seems that the inference network learns is mostly the context variable c, wonder if it is better to use terms like \"latent variable inference\" to avoid confusion.\n\nWhat does the standard deviation mean in Figure 6? It seems a lot of them are even larger than the mean.\n\nThere is little explanation to the VAE-ADAIL experiments -- is it safe to assume most of the experiments require certain knowledge of the latent variable in order to be successful? Maybe some of the arguments about VAE can go to the appendix.\n\nWhy would in some cases UP-True performs much worse than ADAIL? In Ant it is even worse than PPO expert.\n\nHow would you adapt to unseen environments in ADAIL-pred? I don't see explanations in the text about how this is done. Essentially, how are samples obtained from the environment in order to perform posterior inference?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJlqXqoptB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm for imitation of expert demonstrations, in situations where the imitator is acting under a different environment (different dynamics, for instance) than the one used to collect expert demonstrations. The algorithm builds on GAIL with the following modifications – the discriminator is made dynamics-invariant by adding a domain-adversarial loss, and the policy is made to condition on a dynamics context. A separate dynamics posterior network is trained (either supervised or unsupervised) to predict this context at test-time. \n\n\nI have the following concerns about the paper:\n\n1.\tLack of novelty: \n         a.\tLearning a dynamics-invariant discriminator with the gradient-reversal-layer was proposed in Stadie et. al (2017). How is this approach different? In particular, what is the new element in Figure 1. and complete Section 3.4?  \n         b.\tLearning a posterior network to predict context codes, and conditioning the policy on those was explored in papers such DIAYN and InfoGAIL. \n\n2.\tThe proposed algorithm is reliant on the possibility of being able to sample from a distribution of environments (with varying dynamics), and then collect many trajectories in that environment (Line 6-7 in Algorithm 1). This is a severe requirement for real-word scenarios, and somewhat antithetic to the robotics learning motivation given by the authors in the introduction. Moreover, Figure 7 seems to imply that the method doesn’t generalize well to unseen environments, if enough environments can’t be sampled during training time.\n\n\nMinor comment:\n\nFigure 8. Friction value should not go from -3 to 3. Also, this single result inspires no confidence in the benefit or general applicability of the VAE-based context prediction.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends adversarial imitation learning to an adaptive setting where environment dynamics change frequently. The authors propose a novel approach with pragmatic design choices to address the challenges that arise in this setting. Several questions and requests for clarification were addressed during the reviewing phase. The paper remains borderline after the rebuttal. Remaining concerns include the size of the algorithmic or conceptual contribution of the paper.
Decision: Reject