State-only Imitation with Transition Dynamics Mismatch
Authors: Tanmay Gangwani
Jian Peng
Venue:  (2020)
Keywords: Imitation learning
Reinforcement Learning
Inverse Reinforcement Learning

Abstract: Imitation Learning (IL) is a popular paradigm for training agents to achieve complicated goals by leveraging expert behavior, rather than dealing with the hardships of designing a correct reward function. With the environment modeled as a Markov Decision Process (MDP), most of the existing IL algorithms are contingent on the availability of expert demonstrations in the same MDP as the one in which a new imitator policy is to be learned. This is uncharacteristic of many real-life scenarios where discrepancies between the expert and the imitator MDPs are common, especially in the transition dynamics function. Furthermore, obtaining expert actions may be costly or infeasible, making the recent trend towards state-only IL (where expert demonstrations constitute only states or observations) ever so promising. Building on recent adversarial imitation approaches that are motivated by the idea of divergence minimization, we present a new state-only IL algorithm in this paper. It divides the overall optimization objective into two subproblems by introducing an indirection step and solves the subproblems iteratively. We show that our algorithm is particularly effective when there is a transition dynamics mismatch between the expert and imitator MDPs, while the baseline IL methods suffer from performance degradation. To analyze this, we construct several interesting MDPs by modifying the configuration parameters for the MuJoCo locomotion tasks from OpenAI Gym.

Published as a conference paper at ICLR 2020
STATE-ONLY IMITATION WITH TRANSITION DYNAM- ICS MISMATCH
Tanmay Gangwani
Jian Peng
Department of Computer Science University of Illinois, Krbana-CCampaann gangwan2e illinos innis s.. .ddu
Department of Computer Science University of Illinois, Urbana-Champaa j Lanpenge i1l ilinoiae 8. edu
ABSTRACT
Imitation Learning (IL) is a popular paradigm for training agents achieve com- plicated goals by leveraging expert behavior, rather than dealing with the hard- ships of designing correct reward function. With the environment modeled as a Markov Decision Process (MDP), most of the existing IL algorithms are con- tingent on the availability of expert demonstrations in the same MDP as the one in which a new imitator policy is to be learned. This is uncharacteristic of many real-life scenarios where discrepancies between the expert and the imitator MDPs are common, especially in the transition dynamics function. Furthermore, obtain- ing expert actions may be costly or infeasible, making the recent trend towards state-only IL (where expert demonstrations constitute only states observations) ever so promising. Building on recent adversarial imitation approaches that are motivated by the idea of divergence minimization, we present a new state-only IL algorithm in this paper. It divides the overall optimization objective into two subproblems by introducing an indirection step and solves the subproblems it- eratively. We show that our algorithm is particularly effective when there is a transition dynamics mismatch between the expert and imitator MDPs, while the baseline II. methods suffer from performance degradation. To analyze this, we construct several interesting MDPs by modifying the configuration parameters for the MuJoCo locomotion tasks from OpenAl Gym
INTRODUCTION
In the Reinforcement Learning (RL) framework, the objective is to train policies that maximize certain reward criterion. Deep-RL, which combines RL with the recent advances in the field of deep- learning, has produced algorithms demonstrating remarkable success in areas such as games (Mnih) et al. 2015 Silver et al., al., (0116). continuous control (Lillicrap el al. 2015). and robotics (Levine et al.) 2016 to name a few. However, the application of these algorithms beyond controlled simulation environments has been fairly modest; one of the reasons being that manual specification of good reward function is hard problem. Imitation Learning (II.) algorithms (Pomerleau 199T Ng al., 2000 Ziebart et al. [0088 Ho & Ermon 2016) address this issue by replacing reward functions with expert demonstrations, which are easier to collect in most scenarios.
The conventional setting used in most of the IL literature is the availability of state-action trajectories from the expert, T : {50, ag, 8T, aT), collected in an environment modeled as a Markov deci- sion process (MDP) with transition dynamics Texp, These dynamics govern the distribution over the next state, given the current state and action. The IL objective is to leverage T to train an imitator pol- icy in the same MDP the expert. This is a severe requirement that impedes the wider applicability of II. algorithms. In many practical scenarios, the transition dynamics of the environment in which the imitator policy is learned (henceforth denoted by TPol is different from the dynamics of the environment used to collect expert behavior, Tap Consider self-driving cars as an example, where the goal is to learn autonomous navigation on vehiele with slightly different tramsmisnnnnn characteristics than the vehicle used to obtain human driving demonstrations. We therefore strive
"Code for this paper available http:///giiii....mmmagga//////ir

---

Published as a conference paper at ICLR 2020
10 05K1
Art
R. executed
R2
yeur
2520
2010
5000
Gravity 9
Gravity g/2
Average Return
4000
2000
3000
1900
1000
2000
520
1000
Tinesiae Tinesaasemiiiio Onilions
(a)
(b)
(C)
Figure/ 1: (a) Different amount of gravitation pull is one example of transition dynamies mismatch between the expert and the imitator MDPs. (b) An expert policy n trained in 7 transfer poorly to an environment with dissimilar dynamics 7 (gravity 0.5x). (c) IL performance with GAIL degrades when T # T compared to the conventional IL setting of imitating in the same environment as the expert.
for an II. method that could train agents under a transition dynamics mismatch, Texp # TPol, We assume that other MDP attributes are the same for the expert and imitator environments.
Beyond the dynamics equivalence, another assumption commonly used in IL literature is the avail- ability of expert actions (along with the states). A few recent works (Torabi et al., 2018a 8a b Sun jet al. 2019) have proposed "state-only" IL algorithms, where expert demonstrations do not include the actions. This opens up the possibility of employing IL to situations such kinesthetic teaching in robotics and learning from weakk-supervision sources such as videos. Moreover, if TeXP and 7P differ, then the expert actions, even if available, are not quite useful for imitation anyway, since the application of an expert action from any state leads to different next-state distributions for the expert and the imitator. Hence, our algorithm uses state-only expert demonstrations.
We build on previous IL literature inspired by GAN-based adversarial learning GAIL (Ho & Er- mon 2016) and AIRL (Fu et al. 2017). In both these methods, the objective is to minimize the distance between the visitation distributions (p) induced by the policy and expert, under some suit- able metric d, such as Jensen- Jensen-Shannon divergence. We classify GAIL and AIRL as direct imitation methods as they directly reduce d(p=, Different from these, we propose an indirect imitation approach which introduces another distribution ý as an intermediate or indirection step. In slight detail, starting with the Max Max-Fntropy Inverse-RI. RL objective (Ziebart et al.] 2008). we derive lower bound which transforms the overall IL problem into two sub-parts which are solved iteratively: the first is to train policy to imitate distribution ō represented by a trajectory buffer, and the second is to move the buffer distribution closer to expert's ' (p") over the course of training. The first part, which is policy imitation by reducing d(pr. is done with AIRL, while the second part, which is reducing d(p, p²) is achieved using Wasserstein critic (Arjovsky et al. 2017). We abbreviate our approach I2I., for
We test the efficacy of our algorithm with continuous-control locomotion tasks from MuJoCo. Fig- ure [aa depicts one example of the dynamics mismatch which we evaluate in our experiments. For the Ant agent, an expert walking policy n is trained under the default dynamics provided in the OpenAl Gym, Tup Earth. The dynamics under which to learn the imitator policy are curated by modifying the gravity parameter to half its default value (i.e. 9,81), T PlanetX. Figure Ib plots the average episodic returns of म in the original and modified environments, and proves that direct policy transfer is infeasible. For Figure Tc we just assume access to state-only expert demon- strations from = and do II. with the GAII. algorithm. GAII, performs well the imitator policy is learned in the same environment as the expert (Texp- T Earth), but does not succeed un- der mismatched transition dynamics, (7exp Earth, T = PlanetX). our experiments section, consider other sources of dynamics mismatch as well, such as agent-density and joint-friction. We show that 12L trains much better policies than baseline IL algorithms in these tasks, leading to successful transfer of expert skills to an imitator in an environment dissimilar to the expert.
We start by reviewing the relevant background on Max- Max-ont IRL, GAIL and AIRL, since these methods form integral part of our overall algorithm.
2

---

Published as a conference paper at ICLR 2020
2 BACKGROUND
An RL environment modeled as an MDP is characterized by the tuple (S, R, T. 7), where is the state-space, and A is the action-space. Given action d; E A, the next state governed by the tran- sition dynamics $(+1 ~ [ az), and reward is computed as r; = R(r St, at). The RL ob- jective is to maximie the expected discounted sum of rewards, m)(T)) = To D y'r(s), at)]. where y E (0, is the discount factor, and Po is the initial state distribution. We define the unnor- malized 7-discounted state-visitation distribution for a policy π by P=(s) = Σx= P(s,=)))) where (() the probability of being in state at time f, when following policy and start- ing state Po- The expected policy return (TTT)) can then be written as Pp, (e..)) (r(s, a)}, where Pr (s. (() is the state-action visitation distribution (also referred to as the occupancy measure). For any policy TT, there is one-to-one correspondence between and its occupancy measure (Puterman 1994
2.1 MAXIMUM ENTROPY IRL
Designing reward functions that adequately capture the task intentions is laborious and error- proor-prone procedure. An alternative is to train agents solve particular task by leveraging demonstrations of that task by experts. Inverse Reinforcement Learning (IRL) algorithms (Ng et al.] 2000; Rus- sell 1998) aim to infer the reward function from expert demonstrations, and then use it for RL or planning. The IRL method, however, has an inherent ambiguity, since many expert policies could explain set of provided demonstrations. To resolve this, Ziebart (2010) proposed the Maximum Causal Entropy (MaxEnt) IRL. framework, where the objective is to learn a reward function such that the resulting policy matches the provided expert demonstrations in the expected feature counts f, while being as random as possible:
max H(T)) s.t. Eg, [f(s, faemo
where H(TT) [- log [[] is the d-diicuutted causal entropy, and fa denotes the empirical feature counts of the expert. This constrained optimization problem solved by minimizing the Lagrangian dual, resulting in the maximum entropy policy: (a(ass) exp(Qr a) Vioo((s)) where 0 is the Lagrangian multiplier on the feature matching constraint, and Q V are the soft value functions such that the following equations hold (please see Theorem 6.8 in Ziebart (2010):
Q (s, a) = 8T f(s,a) + + pppl''ss (V7 (s')]
(s) = softmax, 2 170 (s,
r(s,a)
Inspired by the energy-based formulation of the maximum entropy policy described above, To(als) exp(Qx (ep((Q)m (s. a)) & (s)). recent methods (Finn et al. 2016: Haarnoja et al. 2017 Fu ct al. 2017) have proposed to model complex, multi-modal action distributions using energy-basce based policies, (als)) x exp(fa (s, a)))) where fu(ss, a) is represented by a universal function approxima- tor, such as a deep neural network. We can then interpret the IRL. problem as a maximum likelihood estimation problem:
with. (T( p(50) II ((s11111. |s,, a (fu(aa))))))
max deoo Po Pa (T)
(1)
Z(w)
2.2 ADVERSARIAL IRL
An important implication of casting IRL as maximum likelihood estimation is that it connects IRL to adversarial training. We now briefly discuss AIRL (Fu et al. (017)) since it forms a compo- nent of our proposed algorithm. AIRL builds on GAIL (Ho & Ermon 2016), well-known ad- versarial imitation learning algorithm. GAIL. frames II. as an occupancy-measure matching (or divergence minimization) problem. Let p=(s,a) a) and pæ(s, represent the state-action visitation distributions of the policy and the expert, respectively. Minimizing the ming Dss(pa(s, a) PE a)] recovers a policy with similar trajectory distribution as the expert. GAIL iteratively trains a policy (#o) and discriminator (D., S x A + (0, 1)) to optimize the min-max objective similar GANs (Goodfellow et al. 2014)
divergence
min max E (s,))~ppp log (w, a)) + ((.)) log(1 Ds a)) XH(TT))
(2)
6
3

---

Published as a conference paper at ICLR 2020
GAIL attempts to learn policy that behaves similar the expert demonstrations, but it bypasses the process of recovering the expert reward function. Finn et al. (2016) showed that imposing a special structure on the discriminator makes the adversarial GAN training equivalent to optimizing the MLE objective (Equation [). Furthermore, trained to optimality, it is proved that the expert reward (up to constant) can be recovered from the discriminator. They operate in trajectory- centric formulation which can be inefficient for high dimensional state- and action-spaces, Fuu et al. (2017) present AIRL which remedies this by proposing analogous changes to the discriminator, but operating on a single state-action pair:
(s( efa(s.0) + no(als)
(3)
Similar to GAIL, the discriminator is trained to maximize the objective in Equation fw is learned,
whereas the value of ((]s)) is "filled in". The policy is optimized jointly using any RL. algorithm with log Du log(1 D as rewards. When trained optimality, exp(fu(s, a)) x= (a|ss) exp(A Aon (s,a)/oo a)/a;; hence fw recovers the soft advantage of the expert policy (up to constant).
2.3 STATE-ONLY IMITATION
State-only IL algorithms extend the scope of applicability of IL by relieving the need for expert ac- tions in the demonstrations. The original GAIL approach could be modified to work in the absence of actions. Specifically, Equation 2 could be altered to use a state-dependent discriminator (s), and state-visitation (instead of sitte-actionvviittaa gorithm, however, requires expert actions due to the special structure enforced on the discriminator (Equation 33. deeming it incompatible with state-only IL. This is because, even though fa could potentially be made a function of only the state 8, actions are still needed for the "filled in" To (als)) component. Inspired by GAIL., Torabi al.) (2018b) proposed GAIfO for state-only II. The moti- vation is to train the imitator to perform actions that have similar effects in the environment, rather than mimicking the expert action. Algorithmically, GAIL is modified make the discriminator function of state transitions Dus (s, s'), and include sate-transition distributions p(s,
distributions (g(s) and Pro (s). The AIRL al-
3 INDIRECT IMITATION LEARNING (I2L)
We now detail our 12L algorithm which alters the standard IL routine (used by GAIL, AIRL) by introducing an intermediate or indirection step, through new distribution represented by trajec- tory buffer. For this section, we ignore the properties of the transition dynamics for the expert and the imitator MDPs (TxX,, TPol); they can be the same or different, 12I. has no specific dependence on this. denotes trajectory, which is sequence of state-action pairs, (50, Ao...., ST, ar). We begin with the expert's 's (unknown) trajectory distribution, although our final algorithm works with state-only expert demonstrations.
Let the trajectory distribution induced by the expert be (r),, and its state-action visitation distribu- tion be p" (s. a). Using the parameterization from Equation I the likelihood objective maximize for reward learning in MaxEnt-IRI. can be written as (ignoring constants w.r.t w):
E Eyops(r) pw)) = (( [fa (s, a)] log ((w)
(4)
available, one could optimize for w by solving an equivalent adversarial min-max objective, as done in AIRL. To handle state-only IL, we proceed derive a lower bound to this objective and optimize that instead. Let there be a surrogate policy स with state-action distribution p(s,a), The following proposition provides a lower bound to the likelihood objective in Equation 4
As alluded to in Sections 2.2 2.3
if expert actions
Proposition. Under mild assumptions of Lipschitz continuity of the function fws we have that for two different state-action distributions p² and p.
E(s,a)<p* (s, a)] 2
LW1
where is the Lipschitz constant, and W (p", (p) is the I-Wasserstein (or Earth Mover's) distance between the state-action distributions.
4

---

Published as a conference paper at ICLR 2020
Proof. Let @ denote the concatenation of state and action. Under Lipschitz continuity assumption for fw (z), for any two imputs x ~ X and x' X', we have
fu(d') (a) V L||(2' =))
Let (xX,, be any joint distribution over the random variables representing the two inputs, such that the marginals are p (X) and p(X'). Taking expectation W:E on both sides, we get
Ra- [3)( - L. x)]l1
Since the above inequality holds for any μ., it also holds for μ² = arg min, Bu - x)111. which gives us the I-Wasserstein distance
Eg Ex-=plfu(<') - Ea-p- (fu(z)]] (W'
Rearranging terms,
Ex-p' [fw Bemplfu LWi(0".
We can therefore lower bound the likelihood objective (Equation as:
Br- [log (T)) 2 (Er-s(r)loog))) (T)] LW1(P" (p"
where ((T) is the trajectory distribution induced by the surrogate policy न. Since the original opti- mization (Equation [ is infeasible under the AIRL framework in the absence of expert actions, instead maximize the lower bound, which is to solve the surrogate problem:
max Eax B [log Pw (T)] LW1 .p) w.P
(5)
This objective can be intuitively understood as follows. Optimizing wirl w recovers the reward (or soft advantage) function of the surrogate policy म. in the same spirit as MaxEnt-IRL. Optimizing w.t. ý brings the state-action distribution of स close (in I-Wasserstein metric) to the expert's, ss along with a bias term that increases the log- liceeihooho of trajectories from under the current reward model w. We now detail the practical implementation of these optimizations.
Surrogate policy. We not use a separate explicit parameterization for æ. Instead, #t is implicitly represented by buffer B, with fixed capacity of k trajectories P In this way, at can be viewed as a mixture of deterministic policies, each representing delta distribution in trajectory space. B is akin to experience replay (Lin) 1992), in that it is filled with trajectories generated from the agent's 's interaction with the environment during the learning process. The crucial difference that inclusion to B is governed by a priority-basee protocol (explained below). Optimization w.E.f can now be done using adversarial training (AIRL), since the surrogate policy actions are available in B. Following Equation the objective for the discriminator is:
(x.00)
(a(a(s)
max E B( log (f-(),)) + (T(a(s) (e..a) llog efu(s,0) + тe
(6)
(7((a)))) (a.s)
where Ty is the learner (imitator) policy that is trained with log Du log(1 Dw) rewards.
Optimizing P. Since ý is characterized by the state-action tuples the buffer B, updating P amounts to refreshing the trajectories in B. For the sake of simplicity, distance objective and ignore the other bias term, when updating for in Equation Note that p", denote the state-action visitation distributions of the expert and the surrogate, respectively. Since we have state-only demonstrations from the expert (no expert actions), we minimize the Wasserstein distance between state visitations, rather than state-action visitations. Following the approach in WGANs (Arjovsky et al., 2017). we estimate W1 using the Sectttrovicc-Ruuiisttee duality, and train a critic network Is with Lipschitz continuity constraint,
only consider the Wasserstein
= sup Esnpr [x) - [xmplgo(s) lls.ll.g1
The empirical estimate of the first expectation term is done with the states in the provided expert demonstrations; for the second term, the states in B are used. With the trained critic gz. we obtain
kk 5 in all our experiments
5

---

Published as a conference paper at ICLR 2020
Envirvament
GAIL-S
[2C
Wilkirdd
3711
4107
g
Hopper
Density Friction
Grasily 92
Density 24
Frlction =
Half-Chheeuh
Figure Environments for training an imitator policy are obtained by Table 1: changing the default Gym configuration settings, one al a time.
Average episodic re-
turns when T T
Algorithm 1: Indirect Imitation Learning (12L)
Networks:
Policy (0), Discriminator (w), Wasserstein critic (e)
B +- empty buffer
T Taatess : (Sp, .1.....$8....................... ST} / State-only expert demonstration +/
for each iteration do
Run Te in environment and collect few trajectories 7
Update Wasserstein critic ē using B and siates /* Equation
+/
Obtain trajectory score [-| Exer (s(s) for each using e
Add r to B with the priority-based protocol, using the score as priority
Update the AIRL discriminator using 7 and B Equation
//
10
Update policy 0 with PPO using log D. - log(1 - Dw) as rewards
11 end
score for each trajectory generated by the agent. The score is calculated as + Σeer Po(s), where |T| is the length of the trajectory. Our buffer B is priority-queue structure of fixed number of trajectories, the priority value being the score of the trajectory. This way, over the course of training, is only updated with trajectories with higher scores, and by construction of the score function, these trajectories are closer to the expert's 's in terms of the Wasserstein metric. Further details on the update algorithm for the buffer and its alignment with the Wasserstein distance minimization objective are provided in Appendix 7.3
Algorithm. The major steps of the training procedure are outlined Algorithm The policy parameters (0) updated with the clipped-ratio version of PPO (Schulman al. 2017). State- value function baselines and GAE (Schulman et al. 2015) are used for reducing the variance of the estimated policy-gradiente
The priority butfer B uses the heap-queue algorithm (Appendix 773
The Lipschitz constant L in Equation unknown and task-dependent. If fu is fairly smooth, is pappee-tr and absorbed into the learning rate. Please
small constant that can be treated see Appendix 7.2 for details on the hyper-parameters.
4 RELATED WORK
There is an extensive amount of literature on IL with state-action expert demonstrations, and also on integrating IL and RL to bootstrap learning (Billard et al., 2008 Argall et al. (009).. Our work most closely related to state-only II. and adversarial Inverse- methods discussed in Section 2 Here, mention other related prior literature. BCO (Torabi et al. (0188)) is state-only IL approach that learns an inverse dynamics model p(a) s, s') by running a random exploration policy. The inverse model is then applied to infer actions from the state-only demonstrations, which in turn used for imitation via Behavioral Cloning, making the approach vulnerable to the well-known issue of compounding errors (Ross al. 2011). Kimura al. (2018) learn an internal model p(s' [s) on state-only demonstrations: the imitator policy is then trained with RL using rewards derived from the model. Imitation under domain shift has been considered in [Stadie al. (2017); Liu et al. (2018). These methods incorporate raw images observations and are designed to handle differences in context (such as viewpoints, visual appearance, object positions, surroundings) between the expert and the imitator environments. Gupta al. (2017) propose learning invariant feature mappings to transfer skills from an expert to an imitator with a different morphology. However, the reward function for such transfer is contingent our Algorithm[] I the adversarial training between the policy and buffer trajectories (AIRL, Line 9) bears resemblance the adversarial self-imitation approaches in (Guo al. 2018 Gangwani
the assumption of time-alignment in episodic tasks. In

---

Published as a conference paper at ICLR 2020
12L (state-only)
GAIL-S (state-only)
GAIL-SA (state action)
AIRL (state action)
Half Chectah (Gravily x0.5)
Wal Wd (Gravily x0.5)
Hupper (Gravily x0.5)
Ani (Gravily xD.5)
6000
3000
Average Return
40000
2000
4000
3000
2000
1500
2000
2000
1000
1000
1000
500
2.5 Timesteps (millions)
2.5 Timesteps (millions)
2.5
2.5 Timesteps (millions)
Timesteps (millions)
Figure 3: Training progress for I2L. and GAIL-S when the imitator and expert MDPs differ in the configuration of the gravity parameter. Gravity in 7 is 0.5x the gravity in 7TP.
(state-only)
GAIL-S (state-only)
(state action)
AIRL (state action)
Half-Cheetah (Density x2)
Walker2d (Density x2)
Hopper (Density x2)
Ant (Density x2)
4000
2560
4000
Average Return
2000
3000
2000
3000
1500
2000
1000
1000
2000
1000
500
1000
2.5 Timesteps (millions)
2.5
5
2.5
2.5 Timesteps (millions)
Timesteps (millions)
Timesteps (millions)
Figure 4: Training progress for 12L and GAIL-S when the imitator and expert MDPs differ in the configuration of the density parameter. Density of the bot in TPa is 2x the density in TT
et al. 2018). Those self-imitation methods are applicable for RL from sparse rewards, while our focus is IL from expert behavior, under transition dynamics mismatch.
5
EXPERIMENTS
In this section, we compare the performance of 12L to baseline methods for state-only IL from Sec- tion 2.3] namely GAIL with state-dependent discriminator, denoted by GAIL-S, and GAIfO Tooabi jet all.. 2018b). We do the evaluation by modifying the continuous-contro locomotion task from Mu- JoCo to introduce various types of transition dynamics mismatch between the expert and the imitator MDPs (Tep o T It should be noted that other aspects of the MDP (S, A. R, 7) are assumed be the same We, therefore, use dynamics and MDP interchangeably in this section. While the expert demonstrations are collected under the default configurations provided in OpenAI Gym, we construct the environments for the imitator by changing some parameters independently: a.) gravity in T is 0.5x the gravity in Tex,, b.) density of the bot in T is 2x the density in Texp, and c.) the friction coefficient on all the joints of the bot T is 3x the coefficient Texp, Figure 2 has visual. For all our experiments and tasks, assume single expert state-only demonstration of length 1000. We do not assume any access to the expert MDP beyond this.
Performance when Tep = T Table shows the average episodic returns for policy trained for 5M timesteps using GAIL-S and I2L in the standard IL setting. The policy learning curves are included in Appendix 7. All our experiments average 8 independent runs with random seeds. Both the algorithms work fairly well in this scenario, though I2L achieves higher scores in 3 out of benchmark when evaluate performance with transition dynamics
tasks. These numbers mismatch. The table also contains the expert demonstration score for each task.
Since state-only II. does not depend expert actions, A can also be made different hetween the MDPs without requiring any modifications the algorithm.
7

---

Published as a conference paper at ICLR 2020
I2L (state-only)
GAIL-S (state-only)
GAIL-SA (state + action)
AIRL (state action)
Holf-Cheetah (Friction x3)
Walker2d (Friction x3)
Hopper (Friction x3)
Ant (Friction x3)
6000
5000
3000
Average Return
1500
4000
4000
2000
3000
1000
2000
2000
1000
500
1060
2.5
2.5 Timesteps (millions)
2.5 Timestaps (millions)
2.5 Timesteps (millions)
Timesteps (millions)
Figure 5: Training progress for 12L and GAIL-S when the imitator and expert MDPs differ in the configuration of the friction parameter. The friction coefficient all the joints of the bot in 7 is 3x the coefficient in T
Performance when Tap * T Figures 3 4 and 5 plot the training progress (mean and standard
deviation) with GAIL-S and I2L under mismatched transition dynamics with low gravity, high den- sity and high friction settings, respectively, as described above. We observe that 12L achieves faster
res
No dynamics mismatch
Low gravity
HallCheetah Walker Hopper
Ant HallCheetah
Walker2d Hopper Ant
GAIRO
5082
3122
2121
3452
1518
2995
1683
594
2000
12L
5240
4107
2751
3320
4155
3547
2566
1617
2000
High density
High friction
2000
HalfCheetah
Walker2d Hopper
Ant
HalfCheetah
Walker2d Hopper
Ant
GAIIO 121.
-234
378
440
3667
2883
3858
876
380
3975
1988
1999
3319
5554
3825
2084
1145
Figure 6: Ablation capac- ity of butfer using low-gravity Half-Cheetab.
Table 2: Comparing performance 12L with GAlFO Torabi et al. 2018b a state-only IL. baseline.
than GAIL-S in most of the situations. GAIL-S degrades severely in some cases. For instance, for Half- Cheetah under high density, GAIL-S drops to 923 (compared to 5974 with no dynamics change, Table D). while 12L attains score of 3975 (compared to 5240 with no dynamics change). Similarly, with Hopper under high friction, GAIL-S score reduces to 810 (from 2130 with no dy- namics change), and the I2L score is 2084 (2751 1 with no dynamics change). The plots also indicate the final average performance achieved using the original GAIL (marked as GAIL-SA) and AIRL algorithms. Both of these methods require extra supervision in the form of expert actions. Even so, they generally perform worse than 12I., which can be attributed to the fact that the expert actions generated in e are not very useful when the dynamics shift to TPal
Comparison with GAIFO baseline. GAIFO Torabi et al. 2018b) is a recent state-only IL. method which we discuss in Section 2.3 contraco the performance of I2L with GAIfO for imitation tasks both with and without transition dynamics mismatch. We find GAIFO to be in the same ballpark as GAIL-S. It can learn good imitation policies if the dynamics are the same between the expert and the imitator, but loses performance with mismatched dynamics. Learning curves for GAIfO are included in Appendix 7.7 Furthermore, Appendix 7.6 we compare to BCO (Torabi et al. 2018 8a).
Ablation on buffer capacity. Algorithm uses priority-queut buffer B of fixed number of trajec- tories to represent the surrogate stte-actionn visitation p. All our experiments till this point fixed the buffer capacity to 5 trajectories. To gauge the sensitivity of our approach to the capacity [B], we ablate on and report the results in Figure ( The experiment Cheetah environment. We observe that the performance of I2L is fairly robust to [B]. Surprisingly, even a capacity of 1 trajectory works well, and having a large buffer (B)) = 50) also does not hurt performance much. The GAIL-S baseline the same task is included for comparison.
done with the low-gravity Half-
Empirical measurements of the lower-bound and Wasserstein approximations. Section 3 in- troduces a lower bound on the expected value of function (s, a) under the expert's state-action visitation. In Appendix 7.4, we analyze the quality of the lower bound by plotting the approximation- gap for the different و distributions obtained during training. We observe that the gap generally
8

---

Published as a conference paper at ICLR 2020
duces. Finally, in Appendix 7.3 plot the empirical estimate of the Wasserstein distance between the state- state-visitation visitations of the buffer distribution and the expert, and note that this value also typically decreases over the training iterations.
CONCLUSION
In this paper, presented 12L, indirect imitation- learning approach that utilizes state-only expert demonstrations collected the expert MDP, to train an imitator policy in an MDP with a dissimilar transition dynamics function. We derive a lower bound to the Max-Ent IRI objective that transforms it into two subproblems, We then provide a practical algorithm that trains policy to imitate the dis- tribution characterized by trajectory- -uffer using AIRL, whilst reducing the Wasserstein distance between the state- of the buffer and expert, over the course of training, Our experiments variety of MuJoCo-basee MDPs indicate that 12L is an effective mechanism for successful skill transfer from the expert to the imitator, especially under mismatched transition dynamics.
REFERENCES
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Breut Browning. A survey of robot learning from demonstration. Roboties and autonomous systems, 57(5):469 483, 2009.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.0787. 2017.
Wasserstein gan. arXiv preprint
Aude Billard, Sylvain Calinon, Ruediger Dillmann, and Stefan Schaal. Robot programming by demonstration. Springer handbook of robotics, pp. 1371-1394, 2008.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprini arXiv:1611.03852. 2016.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint 1710:11700.12465 2017.
Tanmay Gangwani, Qiang Liu, and Jian Peng. Learning self-imitating diverse policies. arXiv preprint arxiv:/805. 10309, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Yijie Guo, Junhyuk Oh, Satinder Singh, and Honglak Lee. Generative adversarial self-imitation learning. arXiv preprint 182.00180 2018.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces transfer skills with reinforcement learning. arXiv preprint arXiv: 2017.
Tuomas Haamoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference Machine Boarning-Volumm 70, pp. 1352-1361. JMLR. org, 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565-4573 2016.
Daiki Kimura, Subhajit Chaudhury, Ryuki Tachibana, and Sakyasingha Dasgupta. Internal model from observations for reward shaping. arXiv preprint arXiv:1806.01267, 2018.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-1375 2016.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint /509. 2015.

---

Published as a conference paper at ICLR 2020
Long-Ji Lin. Self-improving reactive agents based reinforcement learning, planning and teaching. Machine learning, 8(3-4):293-32) 321, 1992.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learn- ing imitate behaviors from raw video via context translation. In 2018 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 1118-1 125. IEEE, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 518(7540):529, 2015.
Andrew Y Ng. Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, pp- 663-670, 2000.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neu- ral Computation. 3(1):88-97, 97, 1991.
Martin L.. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., New York, NY, USA, 1st edition, 1994. ISBN 0471619779.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statistics, pp. 227-635,20 2011.
Stuart Russell. Learning agents for uncertain environments. In COLT, volume 98, pp. 101-103, 1998.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02435 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre,, George Van Den Driessche, Julian Schrittweseer, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. arXiv preprint 1703.01 2017.
Wen Sun, Anirudh Vemula, Byron Boots, and Andrew Bagnell. Provably efficient imitation learn- ing from observation alone. arXiv preprint arXiv:1905. 10948, 2019.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv: /a0i:v/800.0/55. 2018a.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.0611 2018b.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008
10

---

Published as a conference paper at ICLR 2020
7
APPENDIX
7.1
PERFORMANCE WHEN TEXP_ = -TOL
12L (state-onlyl
GAIL-S (state-only)
GAIL-SA (slate + action)
AIRL (slate action)
Half-Cheetah
Walker2d
Hopper
Ant
6000
5000
3000
4000
age Return
4000
3000
4000
3000
2000
2000
2000
2000
1000
1000
1000
2.5
2.5
2.5
Timesteps (millions)
Timesteps (millions)
Timesteps (millions)
Timesteps (millions)
Figure 7: Training progress for 12L. and GAIL-S when imitator and expert MDPs are the same.
7.2
PARAMEEES
Hyper-pane prree--paraeter
Value
Wasserstein critic é network
3 layers, 64 hidden, tanh
Discriminator network
layers, 64 hidden, tanh
Policy 0 network
3 layers, 64 hidden, tanh
Wasserstein critic optimizer, Ir, gradient-steps
RMS-Prop, Se-5, 20
Discriminator w optimizer, Ir, gradient-steps
Adam, 3e-4, 5
Policy algorithm, Ir
PPO (clipped ratio), le-4
Number of state-only expert demonstrations
(1000 states) 5 trajectorics
Butter B capacity
7, (GAE)
0.99, 0.95
7.3 FURTHER DETAILS ON BUFFER B AND THE UPDATE MECHANISM
The buffer B is priority-queue structure, with a fixed capacity K trajectories. Each trajectory is set of tuples - == {si, aa V Denote the trajectories by {TI,.... ). and let {s E 7 be the collection of states trajectory т. Buffer B characterizes the surrogate policy क defined in Section 3 The state-visitation distribution of मे can then be written as:
(())
ΣΣ 8(s)
KT
$ SET
where 5 denotes the delta measure. Following Equation 7 our objective for optimizing 7 is:
min (p (s), p(s)) = min
sup 1aalla
Eenp [g0())] Ex=p(go(s)]
The min-max objective is optimized using an iterative algorithm. The Wasserstein critic ( up- date is done with standard gradient descent using state samples from expert demonstrations and the buffer B. The update for is more challenging since is only available as an empirical measure (Equation 8). For the current iterate o, the objective for X then becomes:
max E (l = max
ΣΣ (x)
KT
1 BET
Section 3 defines the quantity Σ. E.Er, Io (s) as the score of the trajectory т₁. Therefore, the ob- jective in Equation is to update the buffer such that the average score of the K trajectories in increases.
K 5 in experiments
11

---

Published as a conference paper at ICLR 2020
Priority-queue with priority on Min-Heap. Let the current PQ be (fI..... JK ), sorted based on score such that score(T,) _ score(T,); Vi Let {T ........ be the new trajectories rolled out in the environment using the current learner policy To (Line 5 in Algorithm 1). For each of these, PQ updated using standard protocol:
Buffer B is implemented as priority-queu (PQ) based
Update scores of (TI...., TK] in PQ using the latest critic for each r do Calculate score(T;) using the latest critic ø score(T,)) > score(TI) then
// replace the lowest scoring buffer trajectory with the new trajectory // PQQllibrary call to maintain the heap-invariant, score(T)) S score(T)), vi vi<
heapify end end
It follows from the PQ-protocol that the average score of the K trajectories in the buffer B increases (or remains same), after the update, compared to the average score hefore. This aligns the update with the objective in Equation
7.4 EMPIRICAL CONVERGENCE OF LOWER BOUND
In main section, we derive the following lower bound which connects the expected value of function fw under the expert's state-action visitation to the expected value under another surrogate distribution, and the 1- I-Wasserstein distance between the distributions:
(x,a)~p=~ (s, a)] E (s, a)] (W'(p",
In this section, we provide empirical measurements on the gap between the original objective (LHS) and the lower bound (RHS). This gap depends the specific choice of the surrogate distribution is characterized by trajectories in the priority-queue buffer B, and is
p(s, a). In our algorithm, updated during the course of training based the protocol detailed in Appendix 7.3 Figure plots the estimated value of the lower bound for these different أ, and shows that the gap generally reduces over time. To get estimates LHS and RHS, we need the following:
p(s. a) We take snapshots of the buffer B at periodic intervals of the training to obtain the different ý distributions.
p" (s. This is the expert's state-action distribution. We train separate oracle experts in the imitator's (learner's) environment, and use state-action tuples from this expert policy. Note that these oracle experts NOT used in I2L (Algorithm 1), and are only for the purpose of measurement.
Wi(p" (s, a), ((s. a)) A separate Wasserstein critic is trained using (s, tuples from the oracle experts described above and the trajectories in buffer B. This critic is NOT used in 12L since we don't have access oracle experts, and is only for the purpose of measurement.
fwr We select the AIRL discriminator parameters from particular training iteration. The same parameters are then used to calculate the LHS, and the RHS for different distributions.
I. The Lipschitz constant is unknown and hard to estimate for complex, non-linear fa- We plot the lower bound for a few values: {0.5, 1.5).
Figure 8 shows the gap between the original objective and the lower bound for all our experimental settings, Te = Pol (top row), and TT # T (next rows). We observe that the gap generally reduces is updated over the iterations of I2L (Algorithm 1). A better lower bound in turn leads to improved gradients for updating the AIRL discriminator fa, ultimately resulting in more effective policy gradients for the imitator.
12

---

Published as a conference paper at ICLR 2020
Original db
Lower bound (with L- C.5)
Lower bound (wit
Lower Dound (with 2.51
Half-Cheetah
Walker2d
Hopper
25
0.0
Estimated
2
-2.
.2.5
55
2
5.0
115
2.5
2.5 Timestaps (millions)
Timesteps (millions)
Timesteps (millions)
Timesteps (millions)
Original bound 0.5)
(with
Half-Cheetah (Gravity x0.5)
Walker2d (Gravity x0.5)
Hopper (Gravity x0.5)
Ant (Gravity x0.5)
25
value
0.0
0.0
-2.
M
550
80
-15
80
2.5 Tirn resesteps (millions)
2.5 Timesteps (millions)
Timesteps (millions)
Timesteps (millions)
Ongiial nbgertve
Lawar-bourd (with 0.5)
Lower-bound (with L-
Lowar-bund (wit) 1.5
Ha Cheetah (Density x2)
Walker2d (Density x2)
Hopper (Density x2)
Ant (Density x2)
0.0
0.0
2
Estimated
-25
225
550
50
-15
-155
2.5
2.5 Timesteps (millions)
2.5 Timesteps (millions)
2.5 Timesteps (millions)
Timesteps (millions)
Lover-bourd
Lowen ind (with
Lawer sower-ocund (wtti
Half Cheetah (Friction x3)
Walker2d (Friction x3)
Happer (Friction x3)
Ant (Friction x3)
Felimatnd valuee
0.0
-2.5
5500
80
.15
2.5 Timesteps (m Illlons))
2.5 Timesteps (milllons)
2.5 Timesteps (millions)
2.5 Timesteps (millions)
Figure 8: Gap between the original objective and the lower bound for all our experimental settings, Tap 7 (top row), and Trpp j T (next 3 rows).
7.5 EMPIRICAL WASSERSTEIN DISTANCES
each iteration of 12L, we update the Wasserstein critic & using states from state-only expert demonstration, and states in the buffer B (Line 6, Algorithm 1). The objective is to obtain the Wasserstein distance:
Wilp" Wi(p"(s), p(s)) = sup Ex-pr [g6(s)) - E.-s(1((s)]
lls.ll.51
In Figure 9
we plot the empirical estimate W; of this distance over the course of training. To get the estimate at any time, the current critie parameters ó and buffer trajectories used to calculate
Fig-p- [s)] (s))
We show the values for all experimental settings, Texp PPol
13

---

Published as a conference paper at ICLR 2020
(top row), and T * TPol (next rows). It can be that Wi(p"(s), ((s)) generally decreases over time in all situations. This is because our objective for optimizing e (or updating the buffer B) is to minimize this Wasserstein estimate. Please Appendix 773 for more details. The fact that the buffer state-distribution gets closer to the expert's, together with the availability of actions the buffer which induce those states in the imitator MDP (TPol), enables successfully AIRL. for imitation under mismatched transition dynamics.
Half-Cheetah
Hopper
Feimated W. dieanne
3.00
1.75
2.5
1.50
2.55
2.0
1.25
250
1.5
100
222
1.0
0.75
2.5 Timesteps ps (millions)
Times:eps liil)
Timesteps (millions)
Timesteps (millions)
Wild' (sl, M6l1
Half-Cheetah (Gravity x0.5)
Walker2d (Gravity
Hopper (Gravity x0.5)
Ant (Gravity
Estimated Ws distance
3.00
Yor
3,0
25
20
2.75
1.5
2.9
20
2.50
10
2.25
1.5
2.8
2.00
2.5
2.5 Timesteps (millions)
Timesteps (millions)
Timesteps (milllons)
Timesteps (millions)
Chir-Cheetah (Density x2)
(Density
Hopper (Density x2)
(Density x2)
Estimated Ws distance
20
2.7
2.5
3.0
2.6
1.5
20
2.9
2
2.5
2
2.5 Timesteps (millions)
2.5 Timesteps (millions)
2.5 Timesteps (mil lons)
Timesteps (millions)
Wilp"ls,, Stsi)
Half-Cheetah (Priction x3)
Wa ker2d (Friction x3)
Happer (Friction x3)
Ant (Friction x3)
Estimated Wi distance
3.0
2.8
2.0
2.8
1.5
1.0
2.6
1.0
08
2.5 Timesteps (millions)
2.5
2'5 Timesteps (millions)
2
2/5 Timesteps (millions)
Timesteps (millions)
Figure 9: Estimate of the Wasserstein distance between p² (s) and p(s), for all experimental settings, Tex = Tol (top row), and Txp + pol (next rows).
7.6 COMPARISON WITH BCO
Figure I0 comppres 12L with BCO Torabi et al. 2018a) when the expert and imitator dynamics are same (top row) and under mismatched transition dynamics with low gravity, high density and high
14

---

Published as a conference paper at ICLR 2020
friction settings (next 3 rows). BCO proceeds by first learning inverse dynamics model in the imitator's 's environment, p(als, s'), to predict actions from state-transitions. This model is learned via supervised learning trajectories generated by exploratory policy. The inverse model is then used to infer actions from the state-transitions in state-only expert demonstrations. 'he imi- tator policy is trained with Behavioral Cloning (BC) using these inferred actions. We implement the BCO(a) version from the paper since it is shown to be better than vanilla BCO. We observe the barring two situations (Ant with no dynamics mismatch, and Ant with 2x density), BCO(a) is unsuccessful in learning high-return policies. This is potentially due to the difficulties in learn- ing a robust inverse dynamics model, and the compounding error problem inherent BC. Similar performance for BCO(a) is also reported by Torabi et al. (2018b).
IZL (state-only)
BCO
Half-Cheetah
Walker2d
Hopper
Ant
6000
3000
4000
Average Return
4000
2000
2000
2000
2000
2000
1000
-000
2.5
0
2.5 Timesteps (millions)
2.5 Timesteps (millions)
2.5 Timesteps (millions)
Timesteps (millions)
12L (state-only)
BCO
Half- Cheetah (Gravity x0.5)
Walker2d (Gravity x0.5)
Hopper (Gravity x0.5)
Ant (Gravity x0.5)
6000
3000
Ave Return
4000
2000
4000
2000
2000
2000
1000
1000
2.5
5
0
2.5
0
2.5 Timesteps (millions)
2.5 Timesteps (millions)
Timesteps (millions)
Timestess (millions)
(state-only)
BCO
Half-Cheetah (Density x2)
Walker2d (Density x2!
Hopper (Density x2)
Ant (Density x2)
4000
4000
Average Return
2000
2000
2000
2000
1000
1000
2.5
0
2.5 Timesteps (millions)
2.5 Timesteps (millions)
2.5 Timesteps (m (millions)
Timesteps (millions)
12L (state-only)
Half-Cheetah (Friction x3)
Walker2c (Friction x3)
Hopper (Friction
(Friction x3)
Average Return
6000
3000
1500
4000
4000
2000
1000
2000
2000
1000
500
2.5 Timesteps (millions)
2.5
0
2.5 Timesteps (millions)
2.5 Timesteps (millions)
Timesteps (millions)
Figure 10: Comparison between I2L and BCO.
15

---

Published as a conference paper at ICLR 2020
7.7
COMPARISON WITH GAIFO
2L (State en
GAILS (state
GAIFO (state only)
(PPO tewordsiin ym))
Half-Cheetah
Walker2d
Hopper
Ant
6000
4000
Averaye Return
6000
3000
3000
4000
4000
2000
2000
2000
2000
1000
1000
0
2.5 Timesteps (millions)
2.5 Timesteps (millions)
2.5 Timesteps (millions)
2.5 Timesteps (millions)
12L (state onyl
GAILS (state onty)
GAFO (state only)
R. (PPO w/ reouurd (oC)
Half-Cheetah (Grav ty x0.5)
Walker2d (Gravity x0.5)
Hopper (Gravity x0.5)
Ant (Gravity x0.5)
6000
3000
Average Retum
4000
2000
4000
2000
2000
2000
1000
1000
2.5 Timesteps (millions).
2.5 Timesteps (millions)
Timesteps (millions...
Timesteps (millions)
(ttat en
GAL (otae ontr)
GNFD (sean only)
(2P0 rww.rrsh in (90)
Half Cheelah (Densiyy x2)
Walker2d (Densily x2)
Hupper (Densily x2)
Anl (Densily x2)
4000
Average Return
3000
4000
2000
2000
2000
1000
2000
1000
2.5
2.5
0
2.5
2.5 Timesteps (millions)
Timesteps (millions)
Timesteps (millions)
Timesteps (millions)
(state-orty)
(stac-oonly
(stato only)
in (ro
Half-Cheetah (Friction x3)
WalkerZd (Friction x3)
Hopper (Friction x3)
Ant (Friction x3)
3000
3000
Average Return
6000
4000
3000
2000
2000
2000
2000
1000
1000
0
2.5
Timesteps (millia 1s)
Timesteps (millions)
Timesteps (millions)
Timesteps (millions)
Figure 11: Comparison between I2L and two baselines derived from GAIL. The final performance of agent trained with PPO using real rewards in Tool is also shown.
16

---

Published as a conference paper at ICLR 2020
7.8
COMPARISON WITH GAIL-SA AND AIRL
CALL
Half-Cheetah
Walker2d
Hopper
Ant
6000
Return
6000
3000
4000
4000
4000
2000
oc
3000
2000
2000
2000
1000
1000
2.5 Timestaps (millions)
2.5 mimetess (millions)
0
2.5
2.5 Timesteps (millions)
Timesteps (millions)
I2L (state only)
ont
ABL
(o1)
Half-Cheetah (Gravity x0.5)
Walker2d (Gravity x0
Hopper (Gravity x0.5)
Ant (Gravity x0.5)
6000
3000
Average Retura
4000
2000
4000
2000
2000
2000
1000
1000
2.5 Timesteps lons)
Timesteps (millions) ns)
Timesteps (millions)
Timesteps (millions)
12L (ilate-only)
GALL istate-action
ARL (state action)
AL (PPO rewords in ym
Half-Cheetah (Density x2)
Walker2d (Density
Hopper (Density
(Density
4000
Average Return
3000
4000
3000
2000
2000
2000
1000
2000
1000
1000
2.5 Timesteps (millions)
2.5 Timesteps (millions)
2.5 Timestaps (millions)
Timesteps (millions)
2L (stato only
GAILSA Istate action)
AIR (rrate-acton) action
RL IPPD vo rewards (M))
Half-Cheetah (Friction
Walker2d (Friction x3)
Happer (Friction x3)
Ant (Friction
3000
3000
Average Return
6000
4000
4000
2000
2000
2000
2000
1000
1000
2.5 Timesteps (millions)
2.5
2.5 Timesteps (millions)
Timesteps (millions)
Timesteps (millions)
Figure 12: Comparison between 12L and baselines using expert actions: GAIL-SA and AIRL
17


Reviews:
[{"review_id": "SJg24Pj6tS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an imitation method, I2L, that learns from state-only demonstrations generated in an expert MDP that may have different transition dynamics than the agent MDP. I2L modifies the existing adversarial inverse RL algorithm: instead of training the disciminator to distinguish demonstrations vs. samples, I2L trains the discriminator to distinguish samples that are close (in terms of the Wasserstein metric) to the demonstrations vs. other samples. This approach maximizes a lower bound on the likelihood of the demonstrations. Experiments comparing I2L to a state-only GAIL baseline show that I2L performs significantly better under dynamics mismatch in several low-dimensional, continuous MuJoCo tasks.\n\nOverall, I enjoyed reading this paper. A few comments:\n\n1. It would be nice to include a behavioral cloning (e.g., BCO) baseline in the experiments. Your point in Section 4 that BC can suffer from compounding errors is well taken, but in my experience, BC can perform surprisingly well on some of the MuJoCo benchmark tasks, even from a single demonstration trajectory. Prior work showing relatively poor results for BC on MuJoCo tasks usually sub-samples demonstrations to intentionally exacerbate the state distribution shift encountered by BC.\n\n2. It would be nice to discuss potential failure cases for I2L. For example, how dependent is the method on the diversity of trajectories \\tau generated by the agent in line 5 of Algorithm 1? Are there conditions in which training the critic network to approximate the Wasserstein metric is harder than prior methods like Stadie et al. (2017) and Liu et al. (2018)?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJeATxLnYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe manuscript considers the problem of imitation learning when the system dynamics of the agent are different from the dynamics of the expert. The paper proposes Indirect Imitation Learning (I2L), which aims to perform imitation learning with respect to a trajectory buffer that contains some of the previous trajectories of the agent. The trajectory buffer has limited capacity and adds trajectories based on a priority-queue that prefers trajectories that have a similar state distribution to the expert. Similarity is hereby measured by the score of a WGAN-critic trained to approximate the W1-Wasserstein distance between the previous buffer and the expert distribution. By performing imitation learning with respect to a trajectory buffer, state-action trajectories of the agent's MDP are available, which enables I2L to apply AIRL (Fu et al. 2017). By using those trajectories for the transition buffer that have state-marginals close to the expert's trajectory, I2L produces similar behavior compared to the expert. I2L is compared to state-only GAIL, state-action-GAIL and AIRL on four MuJoCo tasks with modified dynamics compared to the expert policy. The experiments show that I2L may learn significantly better policies if the dynamics of agent and the expert do not match.\n\nDecision:\nI think that the submission is below borderline in its current state. The main reason for rejecting would be the insufficient justification of some algorithmic choices, improper presentation of the experiments and the description of MaxEnt-IRL, which seems quite wrong in my opinion. However, I think that the work is interesting and sufficiently novel and could be accepted if the mentioned issues were adequately addressed.\n\nSupporting Arguments:\n- Novelty / Significance: Imitation learning from state-only observations under dynamic mismatch is an important problem and a promising approach for training robots by non-experts. The idea of performing imitation learning with respect to carefully updated trajectory buffer seems simple and effective. Although self-imitation has been applied in reinforcement learning (references in submission), I am not aware of a similar approach in imitation learning.\n\n- Soundness / Correctness:\n1. The description of MaxEnt-IRL seems quite wrong. The paper claims in Section 2.1. that MaxEnt-IRL maximizes the likelihood of the policy by optimizing the advantage function and thus only learns shaped rewards. However, the referenced paper (Ziebart et al. 2008) optimizes the weights of a linear reward function which in general does not correspond to the advantage of the learned policy. Also, the policy is not (only) proportional to the exponentiated advantage but equal to it and the normalizer of the trajectory distribution (Eq. 1) would be therefore 1.\n\n2. I think that the selection of trajectories for the buffer is not sufficiently well motivated. I2L is derived based on a lower bound but it seems that it does not even ensure improvement on that lower bound. It is not fully clear to me how the trajectory buffer is updated (see \"Clarity\") but it does not seem to ensure that the Wasserstein distance decreases compared to the last iteration. The trajectories are chosen greedily, i.e., without considering the remaining trajectories in the buffer which can be especially problematic for multimodal demonstrations. \n\n- Clarity / Presentation:\nThe paper is well-written in general and only has few typos. It is not clear to me how exactly the trajectory buffer is updated. The submission merely states that the buffer is a priority-queue structure of fixed lengths, where the trajectory priorities are given by the average state score based on the Wasserstein critic. This description leaves several open questions (see \"Questions\") and further does not seem well motivated.\n\n\n- Evaluation:\nThe presentation of the experimental results seems odd. For the imitation learning experiments (no dynamic mismatch) the (apparently) same runs of I2L are compared to GAIL-S (Table 1) and GAIfO (Table 2) in separate tables. For the experiments under dynamic mismatch, the comparison with GAIL-S are shown with learning curves (Figure 3-5) and shaded error bars, whereas the results of GAIfO are only shown in in terms of final mean performance in Table 2. This presentation may make the impression or hiding learning curves or confidence intervals. Both tables could be removed by adding a single curve to each of the plots in Figure 3-5 and 7. It is also not clear to me, why only the mean of the final performance for AIRL and SA-GAIL is shown in Figure 3-5. Instead, the performance of the expert--which is currently not shown--would be better suited as a baseline level. The paper also does not seem to mention the number of evaluations and the meaning of the error region in the figures. Furthermore, some hyper-parameters, e.g., the architectures for the AIRL reward/value-function networks are not presented.\n\nQuestions:\n- Please precisely state when the trajectories are removed from the trajectory buffer and the heap of the priority queue. May the same trajectories be used during several iterations?\n- Please elaborate: under which assumptions does the update of the trajectory buffer ensure that the W1-distance of the individual trajectories decreases with respect to the expert's trajectories? \n\n\nMinor comments:\nTypo: \"upto a constant\"\n\nPost-Rebuttal Update:\nI increased my rating to weak accept because the authors addressed my main concerns by a) giving additional information on the update of the buffer, b) improving the presentation of the experiments, c) fixing the description of MaxEnt-IRL and d) by showing empirically that both the lower bound increases / Wasserstein-distance decreases.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lITN63FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission considers the imitation learning with environment change. In the new environment (where we we aim to conduct imitation learning), the expert demonstrations are unavailable, making the objective function (5) unavailable. To deal with this, the authors derive a lower bound to replace (5). \n\nIn the experiment section, the performance of the proposed method is clearly demonstrated. Environment change is a challenging case for existing imitation learning methods, while the proposed one works. \n\nWhile empirically, the performance of the proposed method is justified, I am curious how tight the bound used to replace (or approximate) (5) is. The submission offers few discussions on the error may induced by replacing (5) with the lower bound. Will such an error decrease or converge to 0, as we iterate according to the algorithm? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the setting of imitation learning from state observations only, where the system dynamics under which the demonstrations are performed differs from the target environment. The paper proposes to circumvent this dynamics shift with an algorithm whereby the target policy is trained to imitate its own past trajectories, re-ranked based on the similarity in state occupancies as judged by a WGAN critic.

The reviewers found the paper to be clearly written and enjoyable. The paper improved considerably through reviewers feedback. Notably, a behavior cloning from observations (BCO) baseline was added, which was stronger than the authors expected but still helped highlight the strength of the proposed method by comparison. R1 had a particularly productive multiple round exchange, clarifying the description of previous work, clarifying the details of the proposed procedure and strengthening the presentation of empirical evidence.

This work compellingly addresses an important problem, and in its final form is a polished piece of work. I recommend acceptance.
Decision: Accept (Poster)