Title: STATE-ONLY IMITATION WITH TRANSITION DYNAM-ICS MISMATCH

Abstract: Imitation Learning (IL) is a popular paradigm for training agents to achieve complicated goals by leveraging expert behavior, rather than dealing with the hardships of designing a correct reward function. With the environment modeled as a Markov Decision Process (MDP), most of the existing IL algorithms are contingent on the availability of expert demonstrations in the same MDP as the one in which a new imitator policy is to be learned. This is uncharacteristic of many real-life scenarios where discrepancies between the expert and the imitator MDPs are common, especially in the transition dynamics function. Furthermore, obtaining expert actions may be costly or infeasible, making the recent trend towards state-only IL (where expert demonstrations constitute only states or observations) ever so promising. Building on recent adversarial imitation approaches that are motivated by the idea of divergence minimization, we present a new state-only IL algorithm in this paper. It divides the overall optimization objective into two subproblems by introducing an indirection step and solves the subproblems iteratively. We show that our algorithm is particularly effective when there is a transition dynamics mismatch between the expert and imitator MDPs, while the baseline IL methods suffer from performance degradation. To analyze this, we construct several interesting MDPs by modifying the configuration parameters for the MuJoCo locomotion tasks from OpenAI Gym 1 .

1. INTRODUCTION:

In the Reinforcement Learning (RL) framework, the objective is to train policies that maximize a certain reward criterion. Deep-RL, which combines RL with the recent advances in the field of deeplearning, has produced algorithms demonstrating remarkable success in areas such as games (Mnih et al., 2015; Silver et al., 2016) , continuous control (Lillicrap et al., 2015) , and robotics (Levine et al., 2016) , to name a few. However, the application of these algorithms beyond controlled simulation environments has been fairly modest; one of the reasons being that manual specification of a good reward function is a hard problem. Imitation Learning (IL) algorithms (Pomerleau, 1991; Ng et al., 2000; Ziebart et al., 2008; Ho & Ermon, 2016) address this issue by replacing reward functions with expert demonstrations, which are easier to collect in most scenarios.

The conventional setting used in most of the IL literature is the availability of state-action trajectories from the expert, τ := {s 0 , a 0 , . . . s T , a T }, collected in an environment modeled as a Markov decision process (MDP) with transition dynamics T exp . These dynamics govern the distribution over the next state, given the current state and action. The IL objective is to leverage τ to train an imitator policy in the same MDP as the expert. This is a severe requirement that impedes the wider applicability of IL algorithms. In many practical scenarios, the transition dynamics of the environment in which the imitator policy is learned (henceforth denoted by T pol ) is different from the dynamics of the environment used to collect expert behavior, T exp . Consider self-driving cars as an example, where the goal is to learn autonomous navigation on a vehicle with slightly different gear-transmission characteristics than the vehicle used to obtain human driving demonstrations. We therefore strive for an IL method that could train agents under a transition dynamics mismatch, T exp = T pol . We assume that other MDP attributes are the same for the expert and imitator environments.

Beyond the dynamics equivalence, another assumption commonly used in IL literature is the availability of expert actions (along with the states). A few recent works (Torabi et al., 2018a; b; Sun et al., 2019) have proposed "state-only" IL algorithms, where expert demonstrations do not include the actions. This opens up the possibility of employing IL to situations such as kinesthetic teaching in robotics and learning from weak-supervision sources such as videos. Moreover, if T exp and T pol differ, then the expert actions, even if available, are not quite useful for imitation anyway, since the application of an expert action from any state leads to different next-state distributions for the expert and the imitator. Hence, our algorithm uses state-only expert demonstrations.

We build on previous IL literature inspired by GAN-based adversarial learning -GAIL (Ho & Ermon, 2016) and AIRL (Fu et al., 2017) . In both these methods, the objective is to minimize the distance between the visitation distributions (ρ) induced by the policy and expert, under some suitable metric d, such as Jensen-Shannon divergence. We classify GAIL and AIRL as direct imitation methods as they directly reduce d(ρ π , ρ * ). Different from these, we propose an indirect imitation approach which introduces another distribution ρ as an intermediate or indirection step. In slight detail, starting with the Max-Entropy Inverse-RL objective (Ziebart et al., 2008) , we derive a lower bound which transforms the overall IL problem into two sub-parts which are solved iteratively: the first is to train a policy to imitate a distribution ρ represented by a trajectory buffer, and the second is to move the buffer distribution closer to expert's (ρ * ) over the course of training. The first part, which is policy imitation by reducing d(ρ π , ρ) is done with AIRL, while the second part, which is reducing d(ρ, ρ * ), is achieved using a Wasserstein critic (Arjovsky et al., 2017) . We abbreviate our approach as I2L, for indirect-imitation-learning.

We test the efficacy of our algorithm with continuous-control locomotion tasks from MuJoCo. Figure 1a depicts one example of the dynamics mismatch which we evaluate in our experiments. For the Ant agent, an expert walking policy π * e is trained under the default dynamics provided in the OpenAI Gym, T exp = Earth. The dynamics under which to learn the imitator policy are curated by modifying the gravity parameter to half its default value (i.e. 9.81 2 ), T pol = PlanetX. Figure 1b plots the average episodic returns of π * e in the original and modified environments, and proves that direct policy transfer is infeasible. For Figure 1c , we just assume access to state-only expert demonstrations from π * e , and do IL with the GAIL algorithm. GAIL performs well if the imitator policy is learned in the same environment as the expert (T exp = T pol = Earth), but does not succeed under mismatched transition dynamics, (T exp = Earth, T pol = PlanetX). In our experiments section, we consider other sources of dynamics mismatch as well, such as agent-density and joint-friction. We show that I2L trains much better policies than baseline IL algorithms in these tasks, leading to successful transfer of expert skills to an imitator in an environment dissimilar to the expert.

We start by reviewing the relevant background on Max-Entropy IRL, GAIL and AIRL, since these methods form an integral part of our overall algorithm.

2. BACKGROUND:

An RL environment modeled as an MDP is characterized by the tuple (S, A, R, T , γ), where S is the state-space, and A is the action-space. Given an action a t ∈ A, the next state is governed by the transition dynamics s t+1 ∼ T (s t+1 |s t , a t ), and reward is computed as r t = R(r t |s t , a t ). The RL objective is to maximie the expected discounted sum of rewards, η(π θ ) = E p0,T ,π ∞ t=0 γ t r(s t , a t ) , where γ ∈ (0, 1] is the discount factor, and p 0 is the initial state distribution. We define the unnormalized γ-discounted state-visitation distribution for a policy π by ρ π (s) = ∞ t=0 γ t P (s t =s|π), where P (s t =s|π) is the probability of being in state s at time t, when following policy π and starting state s 0 ∼ p 0 . The expected policy return η(π θ ) can then be written as E ρπ(s,a) [r(s, a)], where ρ π (s, a) = ρ π (s)π(a|s) is the state-action visitation distribution (also referred to as the occupancy measure). For any policy π, there is a one-to-one correspondence between π and its occupancy measure (Puterman, 1994) .

2.1. MAXIMUM ENTROPY IRL:

Designing reward functions that adequately capture the task intentions is a laborious and error-prone procedure. An alternative is to train agents to solve a particular task by leveraging demonstrations of that task by experts. Inverse Reinforcement Learning (IRL) algorithms (Ng et al., 2000; Russell, 1998) aim to infer the reward function from expert demonstrations, and then use it for RL or planning. The IRL method, however, has an inherent ambiguity, since many expert policies could explain a set of provided demonstrations. To resolve this, Ziebart (2010) proposed the Maximum Causal Entropy (MaxEnt) IRL framework, where the objective is to learn a reward function such that the resulting policy matches the provided expert demonstrations in the expected feature counts f , while being as random as possible:

Inspired by the energy-based formulation of the maximum entropy policy described above, π θ (a|s) = exp(Q soft θ (s, a) -V soft θ (s)), recent methods (Finn et al., 2016; Haarnoja et al., 2017; Fu et al., 2017) have proposed to model complex, multi-modal action distributions using energy-based policies, π(a|s) ∝ exp(f ω (s, a)), where f ω (s, a) is represented by a universal function approximator, such as a deep neural network. We can then interpret the IRL problem as a maximum likelihood estimation problem:

max ω E τ ∼demo [log p ω (τ )]

with, p ω (τ ) = p(s 0 ) t p(s t+1 |s t , a t )e fω(st,at) Z(ω)

(1)

2.2. ADVERSARIAL IRL:

An important implication of casting IRL as maximum likelihood estimation is that it connects IRL to adversarial training. We now briefly discuss AIRL (Fu et al., 2017) since it forms a component of our proposed algorithm. AIRL builds on GAIL (Ho & Ermon, 2016) , a well-known adversarial imitation learning algorithm. GAIL frames IL as an occupancy-measure matching (or divergence minimization) problem. Let ρ π (s, a) and ρ E (s, a) represent the state-action visitation distributions of the policy and the expert, respectively. Minimizing the Jenson-Shanon divergence

min π D JS [ρ π (s, a) || ρ E (s, a)

] recovers a policy with a similar trajectory distribution as the expert. GAIL iteratively trains a policy (π θ ) and a discriminator (D ω : S × A → (0, 1)) to optimize the min-max objective similar to GANs (Goodfellow et al., 2014) :

EQUATION

GAIL attempts to learn a policy that behaves similar to the expert demonstrations, but it bypasses the process of recovering the expert reward function. Finn et al. (2016) showed that imposing a special structure on the discriminator makes the adversarial GAN training equivalent to optimizing the MLE objective (Equation 1). Furthermore, if trained to optimality, it is proved that the expert reward (up to a constant) can be recovered from the discriminator. They operate in a trajectorycentric formulation which can be inefficient for high dimensional state-and action-spaces. Fu et al. (2017) present AIRL which remedies this by proposing analogous changes to the discriminator, but operating on a single state-action pair:

EQUATION

Similar to GAIL, the discriminator is trained to maximize the objective in Equation 2; f ω is learned, whereas the value of π(a|s) is "filled in". The policy is optimized jointly using any RL algorithm with log D ω -log(1 -D ω ) as rewards. When trained to optimality, exp(f ω (s, a)) = π * (a|s) = exp(A * soft (s, a)/α); hence f ω recovers the soft advantage of the expert policy (up to a constant).

2.3. STATE-ONLY IMITATION:

State-only IL algorithms extend the scope of applicability of IL by relieving the need for expert actions in the demonstrations. The original GAIL approach could be modified to work in the absence of actions. Specifically, Equation 2 could be altered to use a state-dependent discriminator D ω (s), and state-visitation (instead of state-action-visitation) distributions ρ E (s) and ρ π θ (s). The AIRL algorithm, however, requires expert actions due to the special structure enforced on the discriminator (Equation 3), deeming it incompatible with state-only IL. This is because, even though f ω could potentially be made a function of only the state s, actions are still needed for the "filled in" π θ (a|s) component. Inspired by GAIL, Torabi et al. (2018b) proposed GAIfO for state-only IL. The motivation is to train the imitator to perform actions that have similar effects in the environment, rather than mimicking the expert action. Algorithmically, GAIL is modified to make the discriminator a function of state transitions D ω (s, s ), and include state-transition distributions ρ(s, s ).

3. INDIRECT IMITATION LEARNING (I2L):

We now detail our I2L algorithm which alters the standard IL routine (used by GAIL, AIRL) by introducing an intermediate or indirection step, through a new distribution represented by a trajectory buffer. For this section, we ignore the properties of the transition dynamics for the expert and the imitator MDPs (T exp , T pol ); they can be the same or different, I2L has no specific dependence on this. τ denotes a trajectory, which is a sequence of state-action pairs, {s 0 , a 0 , . . . , s T , a T }. We begin with the expert's (unknown) trajectory distribution, although our final algorithm works with state-only expert demonstrations.

Let the trajectory distribution induced by the expert be p * (τ ), and its state-action visitation distribution be ρ * (s, a). Using the parameterization from Equation 1, the likelihood objective to maximize for reward learning in MaxEnt-IRL can be written as (ignoring constants w.r.t ω):

EQUATION

As alluded to in Sections 2.2-2.3, if expert actions were available, one could optimize for ω by solving an equivalent adversarial min-max objective, as done in AIRL. To handle state-only IL, we proceed to derive a lower bound to this objective and optimize that instead. Let there be a surrogate policy π with a state-action distribution ρ(s, a). The following proposition provides a lower bound to the likelihood objective in Equation 4.

Proposition. Under mild assumptions of Lipschitz continuity of the function f ω , we have that for two different state-action distributions ρ * and ρ,

E (s,a)∼ρ * [f ω (s, a)] ≥ E (s,a)∼ ρ[f ω (s, a)] -LW 1 (ρ * , ρ)

where L is the Lipschitz constant, and W 1 (ρ * , ρ) is the 1-Wasserstein (or Earth Mover's) distance between the state-action distributions.

Proof. Let x := s ⊕ a denote the concatenation of state and action. Under Lipschitz continuity assumption for f ω (x), for any two inputs x ∼ X and x ∼ X , we have

f ω (x ) -f ω (x) ≤ L (x -x) 1

Let µ(X, X ) be any joint distribution over the random variables representing the two inputs, such that the marginals are ρ * (X) and ρ(X ). Taking expectation w.r.t µ on both sides, we get

E x ∼ ρ[f ω (x )] -E x∼ρ * [f ω (x)] ≤ LE µ (x -x) 1

Since the above inequality holds for any µ, it also holds for µ * = arg min µ E µ (x -x) 1 , which gives us the 1-Wasserstein distance

E x ∼ ρ[f ω (x )] -E x∼ρ * [f ω (x)] ≤ LW 1 (ρ * , ρ)

Rearranging terms,

E x∼ρ * [f ω (x)] ≥ E x ∼ ρ[f ω (x )] -LW 1 (ρ * , ρ)

We can therefore lower bound the likelihood objective (Equation 4) as:

E τ ∼p * (τ ) [log p ω (τ )] ≥ E τ ∼ p(τ ) [log p ω (τ )] -LW 1 (ρ * , ρ)

where p(τ ) is the trajectory distribution induced by the surrogate policy π. Since the original optimization (Equation 1) is infeasible under the AIRL framework in the absence of expert actions, we instead maximize the lower bound, which is to solve the surrogate problem:

EQUATION

This objective can be intuitively understood as follows. Optimizing w.r.t ω recovers the reward (or soft advantage) function of the surrogate policy π, in the same spirit as MaxEnt-IRL. Optimizing w.r.t ρ brings the state-action distribution of π close (in 1-Wasserstein metric) to the expert's, along with a bias term that increases the log-likelihood of trajectories from π, under the current reward model ω. We now detail the practical implementation of these optimizations.

Surrogate policy. We do not use a separate explicit parameterization for π. Instead, π is implicitly represented by a buffer B, with a fixed capacity of k trajectories2 . In this way, π can be viewed as a mixture of deterministic policies, each representing a delta distribution in trajectory space.

B is akin to experience replay (Lin, 1992) , in that it is filled with trajectories generated from the agent's interaction with the environment during the learning process. The crucial difference is that inclusion to B is governed by a priority-based protocol (explained below). Optimization w.r.t ω can now be done using adversarial training (AIRL), since the surrogate policy actions are available in B.

Following Equation 3, the objective for the discriminator is:

EQUATION

where π θ is the learner (imitator) policy that is trained with log D ω -log(1 -D ω ) as rewards.

Optimizing ρ. Since ρ is characterized by the state-action tuples in the buffer B, updating ρ amounts to refreshing the trajectories in B. For the sake of simplicity, we only consider the Wasserstein distance objective and ignore the other bias term, when updating for ρ in Equation 5. Note that ρ * , ρ denote the state-action visitation distributions of the expert and the surrogate, respectively. Since we have state-only demonstrations from the expert (no expert actions), we minimize the Wasserstein distance between state visitations, rather than state-action visitations. Following the approach in WGANs (Arjovsky et al., 2017) , we estimate W 1 using the Kantorovich-Rubinstein duality, and train a critic network g φ with Lipschitz continuity constraint,

EQUATION

The empirical estimate of the first expectation term is done with the states in the provided expert demonstrations; for the second term, the states in B are used. With the trained critic g φ , we obtain a s∈τ g φ (s), where |τ | is the length of the trajectory. Our buffer B is a priority-queue structure of fixed number of trajectories, the priority value being the score of the trajectory. This way, over the course of training, B is only updated with trajectories with higher scores, and by construction of the score function, these trajectories are closer to the expert's in terms of the Wasserstein metric. Further details on the update algorithm for the buffer and its alignment with the Wasserstein distance minimization objective are provided in Appendix 7.3.

Algorithm. The major steps of the training procedure are outlined in Algorithm 1. The policy parameters (θ) are updated with the clipped-ratio version of PPO (Schulman et al., 2017) . Statevalue function baselines and GAE (Schulman et al., 2015) are used for reducing the variance of the estimated policy-gradients. The priority buffer B uses the heap-queue algorithm (Appendix 7.3). The Lipschitz constant L in Equation 5 is unknown and task-dependent. If f ω is fairly smooth, L is a small constant that can be treated as a hyper-parameter and absorbed into the learning rate. Please see Appendix 7.2 for details on the hyper-parameters.

4. RELATED WORK:

There is an extensive amount of literature on IL with state-action expert demonstrations, and also on integrating IL and RL to bootstrap learning (Billard et al., 2008; Argall et al., 2009) . Our work is most closely related to state-only IL and adversarial Inverse-RL methods discussed in Section 2. Here, we mention other related prior literature. BCO (Torabi et al., 2018a ) is a state-only IL approach that learns an inverse dynamics model p(a|s, s ) by running a random exploration policy. The inverse model is then applied to infer actions from the state-only demonstrations, which in turn are used for imitation via Behavioral Cloning, making the approach vulnerable to the well-known issue of compounding errors (Ross et al., 2011) . Kimura et al. (2018) learn an internal model p(s |s) on state-only demonstrations; the imitator policy is then trained with RL using rewards derived from the model. Imitation under a domain shift has been considered in Stadie et al. (2017) ; Liu et al. (2018) . These methods incorporate raw images as observations and are designed to handle differences in context (such as viewpoints, visual appearance, object positions, surroundings) between the expert and the imitator environments. Gupta et al. (2017) propose learning invariant feature mappings to transfer skills from an expert to an imitator with a different morphology. However, the reward function for such a transfer is contingent on the assumption of time-alignment in episodic tasks. In our Algorithm 1, the adversarial training between the policy and buffer trajectories (AIRL, Line 9) bears some resemblance to the adversarial self-imitation approaches in (Guo et al., 2018; Gangwani 

5. EXPERIMENTS:

In this section, we compare the performance of I2L to baseline methods for state-only IL from Section 2.3, namely GAIL with state-dependent discriminator, denoted by GAIL-S, and GAIfO (Torabi et al., 2018b) . We do the evaluation by modifying the continuous-control locomotion task from Mu-JoCo to introduce various types of transition dynamics mismatch between the expert and the imitator MDPs (T exp = T pol ). It should be noted that other aspects of the MDP (S, A, R, γ) are assumed to be the same3 . We, therefore, use dynamics and MDP interchangeably in this section. While the expert demonstrations are collected under the default configurations provided in OpenAI Gym, we construct the environments for the imitator by changing some parameters independently: a.) gravity in T pol is 0.5× the gravity in T exp , b.) density of the bot in T pol is 2× the density in T exp , and c.) the friction coefficient on all the joints of the bot in T pol is 3× the coefficient in T exp . Figure 2 has a visual. For all our experiments and tasks, we assume a single expert state-only demonstration of length 1000. We do not assume any access to the expert MDP beyond this.

Performance when T exp = T pol . Table 1 shows the average episodic returns for a policy trained for 5M timesteps using GAIL-S and I2L in the standard IL setting. The policy learning curves are included in Appendix 7.1. All our experiments average 8 independent runs with random seeds. Both the algorithms work fairly well in this scenario, though I2L achieves higher scores in 3 out of 4 tasks. These numbers serve as a benchmark when we evaluate performance with transition dynamics mismatch. The table also contains the expert demonstration score for each task. Performance when T exp = T pol . Figures 3, 4 and 5 plot the training progress (mean and standard deviation) with GAIL-S and I2L under mismatched transition dynamics with low gravity, high density and high friction settings, respectively, as described above. We observe that I2L achieves faster learning and higher final scores than GAIL-S in most of the situations. GAIL-S degrades severely in some cases. For instance, for Half-Cheetah under high density, GAIL-S drops to 923 (compared to 5974 with no dynamics change, Table 1 ), while I2L attains a score of 3975 (compared to 5240 with no dynamics change).

Similarly, with Hopper under high friction, GAIL-S score reduces to 810 (from 2130 with no dynamics change), and the I2L score is 2084 (2751 with no dynamics change). The plots also indicate the final average performance achieved using the original GAIL (marked as GAIL-SA) and AIRL algorithms. Both of these methods require extra supervision in the form of expert actions. Even so, they generally perform worse than I2L, which can be attributed to the fact that the expert actions generated in T exp are not very useful when the dynamics shift to T pol .

Comparison with GAIfO baseline. GAIfO (Torabi et al., 2018b ) is a recent state-only IL method which we discuss in Section 2.3. Table 2 contrasts the performance of I2L with GAIfO for imitation tasks both with and without transition dynamics mismatch. We find GAIfO to be in the same ballpark as GAIL-S. It can learn good imitation policies if the dynamics are the same between the expert and the imitator, but loses performance with mismatched dynamics. Learning curves for GAIfO are included in Appendix 7.7. Furthermore, in Appendix 7.6, we compare to BCO (Torabi et al., 2018a) .

Ablation on buffer capacity. Algorithm 1 uses priority-queue buffer B of fixed number of trajectories to represent the surrogate state-action visitation ρ. All our experiments till this point fixed the buffer capacity to 5 trajectories. To gauge the sensitivity of our approach to the capacity |B|, we ablate on it and report the results in Figure 6 . The experiment is done with the low-gravity Half-Cheetah environment. We observe that the performance of I2L is fairly robust to |B|. Surprisingly, even a capacity of 1 trajectory works well, and having a large buffer (|B| = 50) also does not hurt performance much. The GAIL-S baseline on the same task is included for comparison.

Empirical measurements of the lower-bound and Wasserstein approximations. Section 3 introduces a lower bound on the expected value of a function f ω (s, a) under the expert's state-action visitation. In Appendix 7.4, we analyze the quality of the lower bound by plotting the approximationgap for the different ρ distributions obtained during training. We observe that the gap generally re-duces. Finally, in Appendix 7.5, we plot the empirical estimate of the Wasserstein distance between the state-visitations of the buffer distribution and the expert, and note that this value also typically decreases over the training iterations. The buffer B is a priority-queue structure, with a fixed capacity of K4 trajectories. Each trajectory is a set of tuples τ := {s i , a i } T i=0 . Denote the trajectories by {τ 1 , . . . , τ K }, and let {s ∈ τ i } be the collection of states in trajectory τ i . Buffer B characterizes the surrogate policy π defined in Section 3. The state-visitation distribution of π can then be written as:

EQUATION

where δ denotes the delta measure. Following Equation 7, our objective for optimizing ρ is:

min ρ W 1 (ρ * (s), ρ(s)) = min ρ sup g φ L ≤1 E s∼ρ * [g φ (s)] -E s∼ ρ[g φ (s)]

The min-max objective is optimized using an iterative algorithm. The Wasserstein critic g(φ) update is done with standard gradient descent using state samples from expert demonstrations and the buffer B. The update for ρ is more challenging since ρ is only available as an empirical measure (Equation 8). For the current iterate φ, the objective for ρ then becomes:

EQUATION

Section 3 defines the quantity 1 T s∈τi g φ (s) as the score of the trajectory τ i . Therefore, the objective in Equation 9 is to update the buffer B such that the average score of the K trajectories in it increases.

Priority-queue with priority = score. Buffer B is implemented as a priority-queue (PQ) based on Min-Heap. Let the current PQ be {τ 1 , . . . , τ K }, sorted based on score such that score(τ i ) ≤ score(τ j ), ∀i < j. Let {Γ 1 , Γ 2 , . . . } be the new trajectories rolled out in the environment using the current learner policy π θ (Line 5 in Algorithm 1). For each of these, PQ is updated using standard protocol:

Update scores of {τ 1 , . . . , τ K } in PQ using the latest critic φ for each Γ i do Calculate score(Γ i ) using the latest critic φ if score(Γ i ) > score(τ 1 ) then τ 1 ← Γ i // replace the lowest scoring buffer trajectory with the new trajectory heapify // PQ-library call to maintain the heap-invariant: score(τ i ) ≤ score(τ j ), ∀i < j end end It follows from the PQ-protocol that the average score of the K trajectories in the buffer B increases (or remains same), after the update, compared to the average score before. This aligns the update with the objective in Equation 9.

7.4. EMPIRICAL CONVERGENCE OF LOWER BOUND:

In our main section, we derive the following lower bound which connects the expected value of a function f ω under the expert's state-action visitation to the expected value under another surrogate distribution, and the 1-Wasserstein distance between the distributions:

E (s,a)∼ρ * [f ω (s, a)] ≥ E (s,a)∼ ρ[f ω (s, a)] -LW 1 (ρ * , ρ)

In this section, we provide empirical measurements on the gap between the original objective (LHS) and the lower bound (RHS). This gap depends on the specific choice of the surrogate distribution ρ(s, a). In our algorithm, ρ is characterized by trajectories in the priority-queue buffer B, and is updated during the course of training based on the protocol detailed in Appendix 7.3. Figure 8 plots the estimated value of the lower bound for these different ρ, and shows that the gap generally reduces over time. To get estimates of LHS and RHS, we need the following:

• ρ(s, a) : We take snapshots of the buffer B at periodic intervals of the training to obtain the different ρ distributions.

• ρ * (s, a) : This is the expert's state-action distribution. We train separate oracle experts in the imitator's (learner's) environment, and use state-action tuples from this expert policy. Note that these oracle experts are NOT used in I2L (Algorithm 1), and are only for the purpose of measurement.

• W 1 (ρ * (s, a), ρ(s, a)) : A separate Wasserstein critic is trained using (s, a) tuples from the oracle experts described above and the trajectories in buffer B. This critic is NOT used in I2L since we don't have access to oracle experts, and is only for the purpose of measurement.

• f ω : We select the AIRL discriminator parameters ω from a particular training iteration. The same parameters are then used to calculate the LHS, and the RHS for different ρ distributions.

• L : The Lipschitz constant is unknown and hard to estimate for a complex, non-linear f ω .

We plot the lower bound for a few values: {0.5, 1.0, 1.5}.

Figure 8 shows the gap between the original objective and the lower bound for all our experimental settings, T exp = T pol (top row), and T exp = T pol (next 3 rows). We observe that the gap generally reduces as ρ is updated over the iterations of I2L (Algorithm 1). A better lower bound in turn leads to improved gradients for updating the AIRL discriminator f ω , ultimately resulting in more effective policy gradients for the imitator. 

7.5. EMPIRICAL WASSERSTEIN DISTANCES:

In each iteration of I2L, we update the Wasserstein critic φ using the states from the state-only expert demonstration, and states in the buffer B (Line 6, Algorithm 1). The objective is to obtain the 1-Wasserstein distance:

W 1 (ρ * (s), ρ(s)) = sup

g φ L ≤1

E s∼ρ * [g φ (s)] -E s∼B [g φ (s)]

In Figure 9 , we plot the empirical estimate Ŵ1 of this distance over the course of training. To get the estimate at any time, the current critic parameters φ and buffer trajectories are used to calculate Ês∼ρ * [g φ (s)] -Ês∼B [g φ (s)]. We show the values for all our experimental settings, T exp = T pol (top row), and T exp = T pol (next 3 rows). It can be seen that Ŵ1 (ρ * (s), ρ(s)) generally decreases over time in all situations. This is because our objective for optimizing ρ (or updating the buffer B) is to minimize this Wasserstein estimate. Please see Appendix 7.3 for more details. The fact that the buffer state-distribution gets closer to the expert's, together with the availability of actions in the buffer which induce those states in the imitator MDP (T pol ), enables us to successfully use AIRL for imitation under mismatched transition dynamics. . BCO proceeds by first learning an inverse dynamics model in the imitator's environment, p(a|s, s ), to predict actions from state-transitions. This model is learned via supervised learning on trajectories generated by an exploratory policy. The inverse model is then used to infer actions from the state-transitions in state-only expert demonstrations. The imitator policy is trained with Behavioral Cloning (BC) using these inferred actions. We implement the BCO(α) version from the paper since it is shown to be better than vanilla BCO. We observe the barring two situations (Ant with no dynamics mismatch, and Ant with 2× density), BCO(α) is unsuccessful in learning high-return policies. This is potentially due to the difficulties in learning a robust inverse dynamics model, and the compounding error problem inherent to BC. Similar performance for BCO(α) is also reported by Torabi et al. (2018b) . 

:

Code for this paper is available at https://github.com/tgangwani/RL-Indirect-imitation

k = 5 in all our experiments

Since state-only IL does not depend on expert actions, A can also be made different between the MDPs without requiring any modifications to the algorithm.

CONCLUSIONIn this paper, we presented I2L, an indirect imitation-learning approach that utilizes state-only expert demonstrations collected in the expert MDP, to train an imitator policy in an MDP with a dissimilar transition dynamics function. We derive a lower bound to the Max-Ent IRL objective that transforms it into two subproblems. We then provide a practical algorithm that trains a policy to imitate the distribution characterized by a trajectory-buffer using AIRL, whilst reducing the Wasserstein distance between the state-visitations of the buffer and expert, over the course of training. Our experiments in a variety of MuJoCo-based MDPs indicate that I2L is an effective mechanism for successful skill transfer from the expert to the imitator, especially under mismatched transition dynamics.

K =

in our experiments

Figures and Tables:

Figure 1: (a) Different amount of gravitation pull is one example of transition dynamics mismatch between the expert and the imitator MDPs. (b) An expert policy π * e trained in T exp transfer poorly to an environment with dissimilar dynamics T pol (gravity 0.5×). (c) IL performance with GAIL degrades when T exp = T pol , compared to the conventional IL setting of imitating in the same environment as the expert.

Figure 2: Environments for training an imitator policy are obtained by changing the default Gym configuration settings, one at a time.

Figure 3: Training progress for I2L and GAIL-S when the imitator and expert MDPs differ in the configuration of the gravity parameter. Gravity in T pol is 0.5× the gravity in T exp .

Figure 4: Training progress for I2L and GAIL-S when the imitator and expert MDPs differ in the configuration of the density parameter. Density of the bot in T pol is 2× the density in T exp .

Figure 5: Training progress for I2L and GAIL-S when the imitator and expert MDPs differ in the configuration of the friction parameter. The friction coefficient on all the joints of the bot in T pol is 3× the coefficient in T exp .

Figure 6: Ablation on capacity of buffer B using low-gravity Half-Cheetah.

Figure 7: Training progress for I2L and GAIL-S when the imitator and expert MDPs are the same.

Figure 8: Gap between the original objective and the lower bound for all our experimental settings, T exp = T pol (top row), and T exp = T pol (next 3 rows).

Figure 9: Estimate of the Wasserstein distance between ρ * (s) and ρ(s), for all our experimental settings, T exp = T pol (top row), and T exp = T pol (next 3 rows).

Figure 10: Comparison between I2L and BCO.

Comparing

Table content: <table/>

Peer review (iclr2020):
Summary: The manuscript considers the problem of imitation learning when the system dynamics of the agent are different from the dynamics of the expert. The paper proposes Indirect Imitation Learning (I2L), which aims to perform imitation learning with respect to a trajectory buffer that contains some of the previous trajectories of the agent. The trajectory buffer has limited capacity and adds trajectories based on a priority-queue that prefers trajectories that have a similar state distribution to the expert. Similarity is hereby measured by the score of a WGAN-critic trained to approximate the W1-Wasserstein distance between the previous buffer and the expert distribution. By performing imitation learning with respect to a trajectory buffer, state-action trajectories of the agent*s MDP are available, which enables I2L to apply AIRL (Fu et al. 2017). By using those trajectories for the transition buffer that have state-marginals close to the expert*s trajectory, I2L produces similar behavior compared to the expert. I2L is compared to state-only GAIL, state-action-GAIL and AIRL on four MuJoCo tasks with modified dynamics compared to the expert policy. The experiments show that I2L may learn significantly better policies if the dynamics of agent and the expert do not match. Decision: I think that the submission is below borderline in its current state. The main reason for rejecting would be the insufficient justification of some algorithmic choices, improper presentation of the experiments and the description of MaxEnt-IRL, which seems quite wrong in my opinion. However, I think that the work is interesting and sufficiently novel and could be accepted if the mentioned issues were adequately addressed. Supporting Arguments: - Novelty / Significance: Imitation learning from state-only observations under dynamic mismatch is an important problem and a promising approach for training robots by non-experts. The idea of performing imitation learning with respect to carefully updated trajectory buffer seems simple and effective. Although self-imitation has been applied in reinforcement learning (references in submission), I am not aware of a similar approach in imitation learning. - Soundness / Correctness: 1. The description of MaxEnt-IRL seems quite wrong. The paper claims in Section 2.1. that MaxEnt-IRL maximizes the likelihood of the policy by optimizing the advantage function and thus only learns shaped rewards. However, the referenced paper (Ziebart et al. 2008) optimizes the weights of a linear reward function which in general does not correspond to the advantage of the learned policy. Also, the policy is not (only) proportional to the exponentiated advantage but equal to it and the normalizer of the trajectory distribution (Eq. 1) would be therefore 1. 2. I think that the selection of trajectories for the buffer is not sufficiently well motivated. I2L is derived based on a lower bound but it seems that it does not even ensure improvement on that lower bound. It is not fully clear to me how the trajectory buffer is updated (see *Clarity*) but it does not seem to ensure that the Wasserstein distance decreases compared to the last iteration. The trajectories are chosen greedily, i.e., without considering the remaining trajectories in the buffer which can be especially problematic for multimodal demonstrations. - Clarity / Presentation: The paper is well-written in general and only has few typos. It is not clear to me how exactly the trajectory buffer is updated. The submission merely states that the buffer is a priority-queue structure of fixed lengths, where the trajectory priorities are given by the average state score based on the Wasserstein critic. This description leaves several open questions (see *Questions*) and further does not seem well motivated. - Evaluation: The presentation of the experimental results seems odd. For the imitation learning experiments (no dynamic mismatch) the (apparently) same runs of I2L are compared to GAIL-S (Table 1) and GAIfO (Table 2) in separate tables. For the experiments under dynamic mismatch, the comparison with GAIL-S are shown with learning curves (Figure 3-5) and shaded error bars, whereas the results of GAIfO are only shown in in terms of final mean performance in Table 2. This presentation may make the impression or hiding learning curves or confidence intervals. Both tables could be removed by adding a single curve to each of the plots in Figure 3-5 and 7. It is also not clear to me, why only the mean of the final performance for AIRL and SA-GAIL is shown in Figure 3-5. Instead, the performance of the expert--which is currently not shown--would be better suited as a baseline level. The paper also does not seem to mention the number of evaluations and the meaning of the error region in the figures. Furthermore, some hyper-parameters, e.g., the architectures for the AIRL reward/value-function networks are not presented. Questions: - Please precisely state when the trajectories are removed from the trajectory buffer and the heap of the priority queue. May the same trajectories be used during several iterations? - Please elaborate: under which assumptions does the update of the trajectory buffer ensure that the W1-distance of the individual trajectories decreases with respect to the expert*s trajectories? Minor comments: Typo: *upto a constant* Post-Rebuttal Update: I increased my rating to weak accept because the authors addressed my main concerns by a) giving additional information on the update of the buffer, b) improving the presentation of the experiments, c) fixing the description of MaxEnt-IRL and d) by showing empirically that both the lower bound increases / Wasserstein-distance decreases.