State Alignment-based Imitation Learning
Authors: Fangchen Liu
Zhan Ling
Tongzhou Mu
Hao Su
Venue:  (2020)
Keywords: Imitation learning
Reinforcement Learning

Abstract: Consider an imitation learning problem that the imitator and the expert have different dynamics models. Most of existing imitation learning methods fail because they focus on the imitation of actions. We propose a novel state alignment-based imitation learning method to train the imitator by following the state sequences in the expert demonstrations as much as possible. The alignment of states comes from both local and global perspectives. We combine them into a reinforcement learning framework by a regularized policy update objective. We show the superiority of our method on standard imitation learning settings as well as the challenging settings in which the expert and the imitator have different dynamics models.

Published as a conference paper at ICLR 2020
STATE ALIGNMENT-BASED IMITATION LEARNING
Fangchen Liu
Zhan Ling
Tongzhou Mu
Hao Su
University of California San Diego La Jolla, CA 92093, USA {fliu, z6ling, t 3mu, haosu)eeng, ucsd.
ABSTRACT
Consider an imitation learning problem that the imitator and the expert have differ- ent dynamics models. Most of the current imitation learning methods fail because they focus on imitating actions. We propose a novel state alignment based imita- tion learning method to train the imitator to follow the state sequences in expert demonstrations as much possible. The state alignment comes from both lo- cal and global perspectives and we combine them into a reinforcement learning framework by regularized policy update objective. We show the superiority of our method on standard imitation learning settings and imitation learning settings where the expert and imitator have different dynamics models.
INTRODUCTION
Learning from demonstrations (imitation learning, abbr. as IL) is basic strategy to train agents for solving complicated tasks. Imitation learning methods can be generally divided into two categories: behavior cloning (BC) and inverse reinforcement learning (IRL). Behavior cloning (Ross et al., 2011b) formulates supervised learning problem to learn policy that maps states actions using demonstration trajectories. Inverse reinforcement learning (Russell 1998, Ng et al. 2000) tries to find a proper reward function that can induce the given demonstration trajectories. GAIL (Ho & Ermon 2016) and its variants (Fu al., Qureshi al. 2018 Xiao et al. 2019) are the recently proposed IRL-based methods, which uses a GAN-based reward to align the distribution of state- action pairs between the expert and the imitator.
Although state-of-the-art BC and IRL methods have demonstrated compelling performance in stan- dard imitation learning settings, e.g. control tasks (Ho & Ermon 2016 Fu et al. Qureshi et al.] 2018 Xiao et al., 2019) and video games (Aytar et al. (2018)), these approaches are developed based on strong assumption: the expert and the imitator share the same dynamics model; specifically, they have the same action space, and any feasible state-action pair leads to the same next state in proba- bility for both agents. The assumption brings severe limitation in practical scenarios: Imagine that robot with a low speed limit navigates through maze by imitating another robot which moves fast, then, it is impossible for the slow robot to execute the exact actions as the fast robot. However, the demonstration from the fast robot should still be useful because it shows the path go through the maze.
We are interested in the imitation learning problem under a relaxed assumption: Given an imita- tor that shares the same state space with the expert but their dynamics may be different, train the imitator to follow the state sequence in expert demonstrations as much as possible. This is a more general formulation since it poses fewer requirements on the experts and makes demonstra- tion collection easier. Due to the dynamies mismatch, the imitator becomes more likely to deviate from the demonstrations compared with the traditional imitation learning setting. Therefore, it is very important that the imitator should be able resume to the demonstration trajectory by itself. Note that neither BC-based methods nor GAIL-based IRL methods have learned to handle dynamics misalignment and deviation correction.
To address the issues, we propose a novel approach with four main features: 1) State-based. Com- pared to the majority literature in imitation learning, our approach is state-based rather than action-based. Not like BC and IRI. that essentially match state-action pairs between the expert and the imitator, we only match states. An inverse model of the imitator dynamics is learned to recover the action; 2) Deviation Correction.
state-based 8-VAF (Higgins al.] 2017) is learned as the prior for the next state to visit. Compared with ordinary behavior cloning. this VAE-based next state predictor advise the imitator to return the demonstration trajectory when it deviates. The

---

Published as a conference paper at ICLR 2020
robustness benefits from VAE's latent stochastic sampling; 3) Global State Aligument. While the VAE can help the agent to correct trajectory to some extent, the agent may still occasionally enter states that are far away from demonstrations, where the VAE has no clue how to correct it. So we have add global constraint to align the states in demonstration and imitation. Inspired by GAIL that uses reward to align the distribution of state-action pairs, we also formulate an IRL problem whose maximal cumulative reward is the Wasserstein Distance between states of demonstration and imitation. Note that we choose not to involve state-action pairs as in GAIL(Ho & Ermon 2016), or state-state pairs as in an observation based GAIL (Torabi et al., 2018a), because our state-only formulation imposes weaker constraints than the two above options. thus providing more flexibility to handle different agent dynamics; 4) Regularized Policy Update. We combine the prior for next state learned from VAE and the Wasserstein distance-based global constraint from IRL. in a unified framework, by imposing Proximal Policy Optimization algorithm.
Becllback--eibble divergence based regularizer to the policy update in the
To empirically justify our ideas, we conduct experiments in two different settings. We first show that our approach achieve similar better results on the standard imitation learning setting, which assumes the same dynamics between the expert and the imitator. We then evaluate our approach in the more challenging setting that the dynamies of the expert and the imitator are different. In number of control tasks, either change the physies properties of the imitators or cripple them by changing their geometries. Existing approaches either fail or can only achieve very low rewards, but our approach can still exhibit decent performance. Finally, we show that even for imitation across agents of completely different actuators. still possible for the state- aate-aliinment based method to work. Surprisingly, a point mass and an ant in MuJoCo (Todorov et al. 2012) can imitate each other to navigate in maze environment.
Our contributions can be summarized as follows:
Propose to use a state alignment based method in the imitation learning problems where the expert's 's and the imitator's dynamics are different.
Propose a local state alignment method based on 8-VAE and a global state alignment method based on Wasserstein distance.
Combine the local alignment and global alignment components into a reinforcement learn- ing framework by regularized policy update objective.
2 RELATED WORK
Imitation learning is widely used in solving complicated tasks where pure reinforcement learning might suffer from high sample complexity, like robotics control (Le et al. 2017: Ye & Alterovitz
2017 Pathak et al.) 2018), autonomous vehicle (Fu et al. Pomerleau 1989), ). and playing video game
(Hester et al. 2018 Pohlen et al. 2018 Aytar et al. (018a). Behavioral cloning (Bain & Sommut,
1999) is straight- forward method learn policy in supervised way. However, behavioral
cloning suffers from the problem of compounding errors as shown by (Ross & Bagnelll 2010)), and
this can be somewhat alleviated by interactive learning, such as DAGGER (Ross et al.] 201 1b).
Another important line in imitation learning inverse reinforcement learning (Russell [9988] Ng
jet al. 2000, Abbeel Abbeel & Ng. 20044 Ziebart et al. 2008 Fu et al.), which finds a cost function under which the expert is uniquely optimal.
Since IRL can be connected to min-max formulations, works like GAIL, SAM (Ho & Ermon 2016
Blondé & Kalousis 2018) utilize this to directly recover policies. Its connections with GANs (Good-
fellow et | 2014) also lead to f-divergence minimization (Ke et al. 2019; Nowozin et al. 2016)
and Wasserstein distance minimization (Xiao et al., 2019). One can also extend the framework from matching state-action pairs to state distribution matching, such as Torabi et al. (2018a); Sun et al. (219); Schroecker & Isbell (017). Other works (Aytar al. 2018b Liu et al. 2018 Peng al., 2018) also learn from observation alone, by defining reward on state and using IRL to solve the tasks. Works like (Lee et al. 2019; Lee et al.) also use state- based reward for exploration. Torabi et al. (2018b); [dwardds et al. (2018) will recover actions from observations by learning an inverse model or latent actions. However, our work aims to combine the advantage of global state distribu- tion matching and local state transition alignment, which combines the advantage of BC and IRL. through a novel framework.
3 BACKGROUNDS
Variational Autoencoders Kingma & Welling (2013); Rezende et al. (2014) provides a frame- work to learn both probabilistie generative model (x (z) well approximated posterior
2

---

Published as a conference paper at ICLR 2020
Local Alignment
Global Alignment
Imitator's state Expert's state
min (5))
(5)) - syll
Wasserstein distance
Figure 2: Visualization of state alignment
distribution 4s (z x). 3-VAE is variant VAE that introduces an adjustable hyperparameter the original objective: C(O, 6; x, 3) E (x/x) [log pa (x (x) 33Dx (go
(1)
Larger 3 will penalize the total correlation (Chen et 2018) to encourage disentangled latent representations, while smaller 3 often results in sharper and more precise reconstructions.
Wasserstein distance The Wasserstein distance between two density functions p(z) and g(x)) with support a compact metric space (M, d) has alternative form due Kantorovich-Rubent duality (Villani 2008):
W(p. 4) sup (x(x) (Egt)) [o(z)]
Here, L1 is the set of all I-Lipschitz functions from M R. Compared with the prevalent KL- divergence and its extension, the f-divergence family, Wasserstein distance has a number advan- lages theoretically and numerically. Please refer to Arjovsky et al. (2017) and Sooooon (2018) for detailed discussion.
4 SAIL: STATE ALIGNMENT BASED IMITATION LEARNING
4.1 OVERVIEW
Our imitation learning method is based state align- ment from both local and global perspectives. For lo- cal alignment, the goal is to follow the transition of the demonstration much as possible, and allow the turn to the demonstration trajectory whenever the im- itation deviates. To achieve both goals, we use a 8- state
inverse astion
VAE nost state
VAE (Higgins al., (017) generate the (Figure For global alignment, we set up an ob- jective to minimize the Wasserstein distance between the states in the current trajectory and the demonstra- tions (Figure Right). There has to be framework to naturally combine the local alignment and global alignment components. We resort the reinforcement learning framework by encoding the local alignment as policy prior and encoding the global alignment as reward over states. Using Proximal Policy Optimiza- tion (PPO) by SSchulman et al. (2017) as the backbone RL solver, derive regularized policy update. To maximally exploit the knowledge from demon- strations and reduce interactions with the environment, we adopt pre-training stage to produce a good initialization based on the same policy prior induced by the local alignment. Our method is summarized in Algorithm| [ of our method in details.
VAE nost astion
Using VAE state predic- tive model will be more self-correctable because of the stochastic sampling mech- anism. But this won't happen when VAE to predict actions.
Figure 1:
In the rest parts of this section, we will introduce all the components
4.2 LOCAL ALIGNMENT BY STATE PREDICTIVE VAE
To align the transition of states locally, we need predictive model to generate the next state which the agent should target at. And then train inverse dynamics model to recover the cor-

---

Published as a conference paper at ICLR 2020
Algorithm SAIL: State Alignment based Imitation Learning
Require: Expert trajectories тe : [81, (1, $2, a2....] ~ Tex initial policy я, inverse dynamics model
g. discriminator Φ, total episode T, memory capacity 5
1: if Imitator and Expert have the same dynamics model then
2: Pre-train using Te and transitions collected by random policy
3: else
4:
Pre-train g using transitions collected by random policy
5: end if
6: Pre-train VAE using Te. and obtain the policy prior
Pre-train VAE and obtain policy prior
7: Pretrain ı using policy prior as described in Sec 4.5
8: while episode < T do
9: while |r| S do
is the collected trajectories
10:
Collect trajectory {(s, a, s', done)} using क
11:
Update using (4)
Add {(s, a, s' done)} to
13:
end while
14:
Train & using maxsec, Eg-T, [p(s)] - E Eawr(o(s))
Calculate Wasserstein Distance
Update inverse dynamics model g
16:
Update policy using (5)
17:
end while
responding action, so as to provide a direct supervision for policy. It is worth-noting that, while training an inverse dynamics model is generally challenging, it is not so hard if we only focus on the agent dynamics, especially when the low-dimensional control states are accessible in many practical scenarios. The problem of how to learn high-quality inverse/forward dynamics models is an active research topic.
Instead of using an ordinary network to memorize the subsequent states, which will suffer from the same issue of compounding errors as behavioral cloning (Ross & Bagnell 2010:: Ross et 201 propose to use VAE to generate the next state based the following two reasons. First, as shown in (Dai al. 2018), VAE more robust to outliers and regularize itself find the support set of data manifold, so it will generalize better for unseen data. Second, because of the latent stochastic sampling, the local neighborhood of data point will have almost the same prediction, which is self-correetable when combined with a precise inverse dynamics model as illustrated Figure
We can also use a VAE to generate action based on the current state. But if the agent deviated from the demonstration trajectory little bit, this predicted action is not necessarily guide the agent back to the trajectory, as shown in Figure And in Sec 5.3.2. we conduct experiments to compare the state predictive VAE and the action predictive VAE.
Instead of the vanilla VAE, we use 3-VAE to balance the KL penalty and prediction error, with formulation shown in (I. In Sec we discuss the effects of the apper-prrameter /3 in different experiment settings one of the ablation studies.
4.3 GLOBAL ALIGNMENT BY WASSERSTEIN DISTANCE
Due to the difference of dynamics between the expert and the imitator, the VAE-based local align- ment cannot fully prevent the imitator from deviating from demonstrations. In such circumstances, still need to assess whether the imitator making progress in learning from the demonstra- tions. We, therefore, seek to control the difference between the state visitation distribution of the demonstration and imitator trajectories, which is global constraint.
Note that using this global constraint alone will not induce policies that follow from the demon- stration. Consider the simple case of learning an imitator from experts of the same dynamics. The expert takes cyclic actions. If the expert runs for 100 cycles with a high velocity and the imitator runs for only 10 cycles with a low velocity within the same time span, their state distribution would still roughly align. That is why existing work such as GAIL. aligns state-action occupancy measure. However, shown later, state-based distribution matching will be combined with the local alignment component, which will naturally resolve this issue. The advantage of this state-based distribution matching over state-action pair matching as in GAIL or -ate-next-sstte pair matching in (Torabi et al. 2018a) is that the constraint becomes loosened.
4

---

Published as a conference paper at ICLR 2020
We use IRL approach to achieve the state distribution matching by introducing reinforcement learn- ing problem. Our task is to design the reward train an imitator that matches the state distribution of the expert.
Before introducing the reward design, first explain the computation of the Wasserstein distance between the expert trajectories (Te) and imitator trajectory {f} using the Kantorovich duality:
W)
sup ESS (0()) Be-r-Blle dEL1
where Φ is the Kantorovich's potential, and serves the discriminator in WGAN (Arjovsky al. 2017). Ø is trained with a gradient penalty term as WGAN-GP introduced in (Gulrajami et al. 2017)
After the rollout of imitator policy is obtained, the potential é will be updated by 33. Assume
transition among imitation policy rollout of length T is (8i, 8141). To provide dense signal every timestep, we assign the reward as:
(1+1)) - ((((11)) BesT ((8)
(4)
We now explain the intuition of the above reward. By solving
3 those states of higher probability
in demonstration will have larger & value. The reward in 4 will thus encourage the imitator to visit such states.
Maximizing the curriculum reward will be equivalent to
T
Banr, (,,r,(((s))
J(xT = Σ E. 1=1
s(+1)
W((T. 7)
T
1=1
In other words, the optimal policy this MDP best matches the state visitation distributions w.r.t Wasserstein distance.
Compared with AIRL (Fu et al.) that also defines rewards on states only, our approach indeed enjoys certain advantages in certain cases. We provide a theoretical justification in the Appendix [
4.4 REGULARIZED PPO POLICY UPDATE OBJECTIVE
As mentioned in the second paragraph of Sec 4.3 the global alignment has to be combined with local alignment. This is achieved by adding a prior to the original clipped PPO objective.
We maximize the following unified objective function:
(TTO = CLIP (0) - -DDLL (Te((1ss) Po) pa)
(5)
We will explain the two terms in detail. LCLIP (0) denotes the clipped surrogate objective used in the original PPO algorithm:
T CLIP (0) = By min To(als) A, clip (a|s)
Te(als) Team (als)
)
e, -,1,, A. A.
(6)
where Å, is an estimator of the advantage function at timestep t. The advantage function calcu- lated based on reward function described in Sec 4.3
The DEL
(5) serves regularizer keep the policy close to learned policy prior p.. This policy prior Pa is derived from the state predictive VAE and an inverse dynamics model. Assume the 3-VAE is (sx) St+1 and the inverse dynamics model is (ine St+1) a. To solve the case when the agents have different dynamics, learn a state prediction network and use learned inverse dynamics to decode the action. We define the action prior as
Pa(at x exp (-1 9imv ginn (st, f(s.)) at
where the RHS is a pre-defined policy prior, a Gaussian distribution centered at ginn (St, ((x)). controls how strong the action prior is when regularizing the policy update, which is a hyper- parameter. Note that the inverse model can be further adjusted during interactions.
5

---

Published as a conference paper at ICLR 2020
LOLIP is computed through the advantage A, and reflects the global alignment. The policy prior is obtained from the inverse model and local / VAE, which makes the kt serve as a local alignment constraint. Furthermore, our method can be regard as combination of BC and IRL because our KL- divergence based action prior encodes the BC policy and we update the policy leveraging reward.
We would note that our state-alignment method augments state distribution matching by taking relationships of two consecutive states into account with robustness concern.
4.5 PRE-TRAINING
We pretrain the state predictive VAE and the inverse dynamics model, and then obtain the policy prior in 77. which is Gaussian distribution. For pre-training, we want to initialize PPO's Gaussian policy и by this prior Pa. by minimizing the KL-divergence between them. Practically, we use direct supervision from Vinn((:z [8t f(:)) and in ( directly train both the mean and variance of the policy network, which is more efficient during the pre- pre-training stage. During the online interaction, the update rule of PPO's policy is by optimizing (5), and the variance will be further adjusted for all the dimensions of the action space.
EXPERIMENTS
We conduct two different kinds of experiments to show the superiority of our method. In Sec 5.1 we compare our method with behavior cloning (Bain & Sommut, 1999), GAIL (Ho & Ermon 2016). and AIRL (Fu et al.) in control setting where the expert and the miitator have different dynamics model, e.g., both of them are ant robots but the imitator has shorter legs. In Sec [5.] we further we conduct ablation study
evaluate in the traditional imitation learning setting. Finally, in See to show the contribution of the components.
5.1 IMITATION LEARNING ACROSS AGENTS OF DIFFERENT ACTION DYNAMICS
5.1.1 ACTORS OF MODIFIED PHYSICS AND GEOMETRY PROPERTIES
We create environments using MuJoCo (Todooov al. 2012) by changing properties of ex- perts, such as density and geometry of the body. We choose environments, Ant and Swimmer, and augment them to 6 different environments: HeavyLLigh/Dissblel Ant/Swimmer. The Heavy/Light agents have modified density, and the disabled agents have modified head/tail/leg lengths. The demonstrations are collected from the standard Ant-v2 and Swimmer-v2 More descriptions of the environments and the demonstration collection process can be founded in the Appendix.
We then evaluate method them.
(a) DisabledAnt
(b) LightAnt
(c) HcavyAnt
(d) DisabledSwimmer
(e) LightS wimmer
(I) HeavyS Swimmer
Figure 3: Comparison with BC, GAIL and AIRL when dynamics are different from experts.

---

Published as a conference paper at ICLR 2020
Figure demonstrates the superiority of methods all the baselines. Our approach is the most stable all the 6 environments and shows the leading performance each of them. GAIL seems to be the most sensitive to dynamics difference. AIRL, which is designed to solve imitation learning for actors of different dynamics, based environments (DisabledSwimmer and HeavySwimmer) that have relatively lower dimensional action space (2D for swimmer versus 8D for ants).
perform par with our method in two swimmer-
Interestingly, the stability and performance of vanilla behavior cloning are quite reasonable in 4 of the environments, although it failed to move about in the DisabledAnt and HeavyAnt environments. For these two tasks, the agent will reach dangerous states by cloning actions, yet our method will not approach these states by using state-based imitation. In the other four games, BC agents do die but just move less efficiently, so they have a sub-optimal yet still reasonable score.
5.1.2 ACTORS OF HETEROGENEOUS ACTION DYNAMICS
We consider extremely challenging setting that the imitator and demonstrator are functionally different. One typical example of expert/imitator pair in practice would be a human and a humanoid robot. We consider a much simplified version but with similar nature MuJoCo. In this task, even if the state space cannot be exactly matched, there are still some shared dimensions across the state space of the imitator and the actor, e.g., the location of the center of mass, and the demonstration should still teach the imitator in these dimensions.
Point and an Ant in
We use the same setting as many hierarchical RL papers, such as HIRO and Near-Optimal RL (Nachum et al. 2018a b). The agent need to reach a goal position in maze, which represented by (x.y) coordinates. We also know that the first two dimensions of states agent. The prior knowledge includes: (1) the goal space (or the common space that need to be matched) (2) the projection from the state space to the goal space (select the first two dimensions of the states).
the position of the
(a) Original Ant Figure 4: Imitation Learning of Actors with Heterogeneous Action Dynamics.
(b) Disabled Ant
(c) PointMaze
(d) AntMaze
The first task is that the Ant should reach the other side of the maze from several successful demon- strations of Point robot. As shown in Figure 4(c) and Figure 4(d) the maze structure for the ant and point mass exactly the same.
To solve this problem, we first pre-train an VAE on the demonstrations, and use this VAE to propose the next "subgoal" for the Ant. This VAE is trained on the goal space (i.e. the first dimensions) of the Point robot's 's trajectory. Then we train an inverse model for Ant, which will generate an action based on the Ant's current state (high dimensional) and goal predicted by VAE (2 dimensional).
Our performance is shown in Figure (jc.. After 1M training steps, the agent has success rate of 0.8 to reach the other side of the maze.
5.2 ACTORS OF THE SAME DYNAMICS (STANDARD IMITATION LEARNING)
We also evaluate our algorithm on 6 non-trivial control tasks in MuJoCo: Swimmer, Hopper, Walker, Ant, HalfCheetach, and Humanoid. We first collect demonstration trajectories with Soft Actor- Critic, which can learn policies that achieve high scores in most of these environment For com-
For LightSwimmer 3(c) AIRL meets MuloCo numerical exception for several trials.
We collect near-optimal demonstration Swimmer using TRPO due the limited performance of SAC.

---

Published as a conference paper at ICLR 2020
parison, we evaluate method against baselines: behavior cloning, GAIL, and AIRI Also, to create even stronger baselines for the cumulative reward and imitator run uuntime sample complex- ity, we initialize GAIL with behavior cloning, which would obtain higher scores in Swimmer and Walker. Lastly, to evaluate how much each algorithm depends the amount of demonstrations, sampled demonstration trajectories of ten and fifty episodes.
Table depicts representative results in Hopper and HalfCheetalfi The advantage of meth- ods over BC should be attributed to the inherent data augmentation by VAE. On Hopper-v2, we are significantly better with 10 demos but are just on par if the demos are increased to 50. On HalfChectah-v2. the demo cheetah runs almost perfectly ( 12294 scores); in other words, the demo provides limited instruction when the imitator is even slightly off the demo states, thus the robustness from VAE becomes critical.
Table 1: Performance on Hopper-v2 and HalfCheetah--2 Hopper-v2
HallC hallCheetthh-22
# Demo
10
Expert
3566
= 1.24
12294.22
273,59
BC
1318.76 + 804.36
3525.87 + 160.74
971.42 + 249.62
4813.20 + 1949.26
GAIL
3372.66 + 130.75
3363.97 + 262.77
474.42 + 389.30
-175.83 + 26.76
BC-GAIL
3132.11 1 520.65
3130.82 1 554.54
578.85 1 934.34
1597.51 1 1173.93
AIRL
3.07 0.02
3.31 0.02
-146.46 1: 23.57
755.46 1 10.92
Our init
3412.58 + 450.97
3601.16 + 300. 14
1064.44 + 227.32
7102.29
+ 910.54
Our final
3539.56 130.36
3614.19 1 150.74
1616.34 1 180.76
8817.32
1 860.55
5.3
ABLATION STUDY
5.3.1
COEFFICIENT 3 IN 8- B-E
3-VAE introduces an additional parameter to the original VAE. It controls the variance of the ran- domly sampled latent variable sampling. which subsequently affects the reconstruction quality and robustness. Theoretically, a smaller 3 leads to better state prediction quality, with the cost of losing the deviation correction ability (Dai al.) 2018).
To empirically show the role of beta and check the sensitivity of our algorithm with respect to beta, evaluate VAE in settings of both the imitator has the same dynamics and has different dynam- ics. We select HalfCheetah-v2 and HeavyAnt as an example. For HalfCheetah-v2 we pretrain the inverse dynamics and VAE using given demonstrations so that the initial performance will tell the quality of the VAE's prediction. For DisabledAnt, pretrain the dynamics with random trials, which results in forward/inverse dynamics estimation of less accuracy. In this case, we examine both its initialized performance and final performance. The results are shown in Table We find that for B in [0.01, 0. the performance is better. Specifically, when the imitator different from the expert, a smaller will result in poor performance as overfits the demonstration data.
We also compare our method with an ordinary MLP trained by MSE loss. We find out that VAE outperforms MIP in all settings. Note that the MI.P-based approach is very similar the state-based behavior cloning work of (Torabi et al. 2018b).
5.3.2 ACTION PREDICTIVE 3- 3-VAE
In Figure [ we mentioned that a VAE to predict the next action is less favorable. To justify the claim, we compare a VAE-based BC with vanilla BC that both predict actions, as shown in Table Experiments show that VAE-BC even outperformed by vanilla BC, especially when f is larger than 0.001. Compared with the last line in Table we can conclude that VAE is more useful when predicting state, which consolidates that the advantage really comes from state-based approach but not only the robustness of VAE.
5.3.3 EFFECT OF WASSERSTEIN DISTANCE AND KL REGULARIZATION
In our policy update process, we use Wasserstein distance with KL regularization to update the policy. To analyze their effects on the performance, we use HalfCheetah-v2 and Humanoid-v2 with
'AIRL and EAIRL E et al. 2018) have similar performance, and we only compare to AIRL. *Results for other environments can be founded in the Appendix.
8

---

Published as a conference paper at ICLR 2020
Analyze the role of VAE coefficient. The "None" item means replacing VAE with an Table 2:
ordinary network with linear layers.
Enviromments
HallCheetab-50
HalrCheetah--00
HeaavAAnt-mmiial
HeavyAnt-Finnl
2007.86
1289.21
258.91
282.13
2653.04
1151.93
1149.65
1502.68
7102.29
1797.44
1219.34
5208.45
5933.28
2215.71
987.72
4850.62
0.01
5893.17
1982.62
740.54
1921.26
0.005
4415.04
1369.57
320.54
399.31
None
4759.69
1123.79
359.15
-62.13
Table Compare behavior cloning to variational behavior cloning
Environments
B
HalfChectah 50
Hopper-50 50
0.1
230..5 +: 13.26
203.87 + 14.39
0.01
1320.04 + 15.43
438.10 + 20.43
0.001
3306.91 + 12.51
3303.72 + 10.46
None
4813.20
1949.26
3525.87 6.74
1000
200
3010
2009
Stuze
Stape
(a) HallCheetah-v2
(b) Humanoid-v2
(c) AntMaze
Figure 5: (a), (b) show the effects of Wasserstein distance and KL regularization HalfCheetah-v2 and Humanoid-v2 given 20 demonstration trajectories. And (c) presents the result on Antmaze.
20 expert trajectories. For each environment, they use the same pretrained inverse model and VAE, thus they have the same behavior after pretraining.
As shown in Figure [ja)(b), Wasserstein distance combined with KL regularization performs the best. Wasserstein objective is used in inverse RL based mechanism that would significantly penalize the exploration when the agent deviates from the demonstration far away. However, using this objective alone lacks constraints over consecutive states, thus performing the worst. The KL objective adds constraints extrapolate to states when the imitator deviates far from the demo (green line gradually fails as in Fig (b)), but this is the scenario when the Wasserstein distance would not favor, thus the reward from the Wasserstein distance will push the imitator back the demonstration states.
consecutive states using VAE prior; however, VAE is unable to
CONCLUSION
We proposed SAIL,, flexible and practical imitation learning algorithms that use state alignment from both local and global perspective. We demonstrate the superiority of our method using MuJoCo environments, especially when the action dynamics are different from the demonstrations.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-firs international conference on Machine learning, pp. 1. ACM, 2004.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv: 1701:0770..0787. 2017.
Wasserstein
gan.
arXiv preprint

---

Published as a conference paper at ICLR 2020
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, pp. 2930-2941, 2018a.
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, pp. 2930-2941, 2018b.
Michael Bain and Claude Sommut. A framework for behavioural cloning. Machine intelligence, 15 (15):103, 1999.
Lionel Blondé and Alexandros Kalousis. Sample-efficient imitation learning via generative adver- sarial nets. arXiv preprint (rXivv/809.02066 2018.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp- 2610-2620, 2018.
Bin Dai, Yu Wang. John Aston, Gang Hua, and David Wipf. Connections with robust pea and the role of emergent sparsity variational autoencoder models. The Journal of Machine Learning Research, 19(1):1573-1614 1614,, 2018.
Ashley D Edwards, Himanshu Sahni, Yannick Schroecker, and Charles L Isbell. Imitating latent policies from observation. arXiv preprint 1rXiv: 07914, 2018
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. ICIR 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in neural information processing systems, pp. 5767-5777, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abheel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv: 180100090 2018.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations, In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR, 2(5):6, 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pp. 4565 4573, 2016.
Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization. arXiv preprint arXiv: 12888, 2019.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. 1rXiv:1312.61 14, 2013.
arXiv preprint
Hoang Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-agent imitation learning. In Proceedings of the 34th International Conference on Machine Learning-Volume Volume 70, pp. 1995- 2003. JMLR. org, 2017.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Ruslan Salakhutdinov, and Sergey Levine. State marginal matching with mixtures of policies.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdi- nov. Efficient exploration via state marginal matching. arXiv preprint arXiv: arXiv:/906.00 naae:99660.55 05274, 2019.
10

---

Published as a conference paper at ICLR 2020
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learn- ing to imitate behaviors from raw video via context translation. In 2018 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 1118-1123 IEEE, 2018.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arXiv:1810001257, 2018a.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3303-3313, 2018b.
Andrew Y Ng. Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume I, pp. 2, 2000.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, pp. 271-279, 2016.
Deepak Pathak, Parsa Mahmoudich, Michael Luo, Pulkit Agrawal, Dian Chen, Fred Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. inter- national conference learning representations, 2018.
Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. Sfv: Rein- forcement learning of physical skills from videos. ACM Trans. Graph., 37(6), November 2018.
Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron, Hado van Hasselt, John Quan, Mel Večerík, et al. Observe and look further: Achieving consistent performance on atari. arXiv preprint 180iv: 11593, 2018.
Dean A Pomerleau. Alvinn: An autonomous land vehicle a neural network. In Advances in neural information processing systems, pp. 305-313, 1989.
Ahmed H Qureshi, Byron Boots, and Michael C Yip. Adversarial imitation via variational inverse reinforcement learning. arXiv preprint / 004,06444, 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.444 2014.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-668, 2010.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635, la.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction no-regret online learning. In Proceedings of the fourteenth international ference on artificial intelligence and statistics, pp. 627-635, 2011b.
Stuart Russell. Learning agents for uncertain environments. In COLT, volume 98, pp. 101-103, 1998.
Yannick Schroecker and Charles L Isbell. State aware imitation learning. In Advances in Neural Information Processing Systems, pp. 2911-2920, 2017.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrXiv:1707.06347 2017.
Justin Solomon. Optimal transport on discrete domains. AMS Short Course on Discrete Differential Geometry, 2018
Wen Sun, Anirudh Vemula, Byron Boots, and Andrew Bagnell. Provably efficient imitation learn- ing from observation alone. arXiv preprint arXiv: 1905. 10948, 2019.
11

---

Published as a conference paper at ICLR 2020
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Faraz. Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06156 2018a.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprini 1805:0180001954 2018b.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Huang Xiao, Michael Herman, Joerg Wagner, Sebastian Ziesche, Jalal Etesami, and Thai Hong Linh. Wasserstein adversarial imitation learning. arXiv preprint arXiv: 13, 2019.
Gu Ye and Ron Alterovitz. Guided motion planning. In Robotics research, pp. 291-307. Springer, 2017.
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. 2008.
12

---

Published as a conference paper at ICLR 2020
A LEARNING ACROSS DIFFERENT ENVIRONMENTS
PointMaze & AntMaze As shown in Figure point mass or an ant is put in 24 x 24 U-maze. The task is make the agent reach the other side of U-maze with the demonstration from the point mass. The ant trained to reach a random goal in the maze from a random location, and should reach the other side of the maze. The state space of ant is 30-dim, which contains the positions and velocities.
HeavyAnt Two times of original Ant's density, Two times of original gear the armature.
LightAnt One tenth of original Ant's density.
DisabledAnt Two front legs are quarters of original Ant's legs.
HeavySwimmer 2.5 times of original Swimmer's 's density.
LightSwimmer One twentieth of original Swimmer's density.
DisabledSwimmer Make the last joint 1.2 times longer and the first joint 0.7 times of the original length
The exact results of these environments are listed Table 4 S All the statistics are calculated from 20 trails.
Table 4: Performance on modifeid Swimmer DisabledSwimmer
LightSwimmer
HeavySwimmer
BC
249.09 = 1.53
277.99 1 3,41
253.95 2.5
GAIL
228.46 = 2.02
-4.11 + 0.51
254.91 + 1.35
AIRL
283.42 = 3.69
67.58 = 25.09
301.27 = 5.21
(AuL(
287.71 1 2.31
342.61 6.14
286.4 3.2
Table 5: Performance modified Ant
DisabledAnt
HeavyAn 550.6 + 77.62
LightAnt
BC
1042.45
75.13
4936,59
53.42
GAIL
-1033.54 254.36
-1089.34 174.13
-971.74
123.14
AIRL
-3252.69 153.47
-62.02
-626.44
104.31
SAIL(Ours)
3305.71 67.21
5608.47 57.67
4335.46 82.34
B IMITATION BENCHMARK EXPERIMENT: SETTINGS AND RESULTS
We six MuJoCo Todorov al. 2012) control tasks. The name and version of the environments are listed in Table 6 which also list the state and action dimension the tasks with expert perfor- mance and reward threshold to indicate the minimum score to solve the task. All the experts are trained by using SAC (Haarnoja et al. 2018) except Swimmer-v2 where TRPO (Schulman al. 2013) get higher performance.
Table 6: Performance benchmark control tasks
Environment
State Dim
Action Dim
Reward threshold
Expert Performance
Swimmer-v2
8
360
332
Hopper-v2
11
3800
3566
Walker2d-v2
17
4924
Ant-v2
I1I
6000
6157
HalfCheetahh-2:
17
4800
12294
Humanoid-v2
376
17
1000
5187
The exact performance of all methods are list in Table 7
12
We compare GAIL (Ho & method
Ermon 2016), behavior cloning, GAIL with behavior clomigg initilization and AIRL containing. Means and standard deviations are calculated from 20 trajectories after the agents con- verge and the number total interactions with environments is less than one million environment steps.
13

---

Published as a conference paper at ICLR 2020
Table 7: Performance on Swimmer-v2 with different trajectories
Swimmer-v2
# Demoo
5
10
20
50
Expert
332.88
1.24
328.85 2.26
331.17 2.4
332.17
2,4
330.65 + 2.42
GAIL
304.64 = 3.16
271.59
1 11.77
36.16 1 5.99
246.73 1 5.76
BC-GAIL
313.80 + 3.42
326.58
7.87
294.93 + 12.21
313.68 + 9.99
AIRL
332.11 = 2.57
338.43
3.65
335.67
= 2.72
340.08 + 2.70
Our imit
332.36 1 3.62
335.78
0.34
0.34
336.23
2.53
334.03 1 2.11
linal
332.22
3.21 339.67
336.18
1.87
336.31 3.20
Table 8: Performance Hopper-v2 with different trajectories Hopper-v2
# Demo
I0
Expert
3566
1.24
BC
1471.40 + 637.25
1318.76 + 804.36
1282.46 + 772.24
3525.87 + 160.74
GAIL
3300.32 + 331.61
3372.66
+ 130.75
3201.97
295.27
3363.97
262.77
BC-GAIL
3122.23
358.65
3132.11
520.65
3111.42
22 414.28
3130.82 2 554.54
AIRL
4.12 0.01
3.07
0.02
4.11 0.01
3.31 0.02
Our init
2322.49
300.93
3412.58
450.97
3314.03
310.32
3601.16
300.14
Our final
3092.26
+ 670.72
3539.56
130.36
3516.81
+ 280.98
3610.19 + 150.74
Table 9: Performance on Walker2d-v2 with different trajectories
Walker2d-v2
#Demo
5
10
20
50
Expert
5070.97
209.19
BC
1617.34 1 693.63
4425.50
930.62
4689.30
1. 372.33
4796.24
490.05
GAIL
1307.21 1 388.55
692.16
1: 145.34
1991.58
446.66
751.21 1: 150.18
BC-GAIL
3454.91 = 792.40
2094.68
= 1425.05
3482.31 + 828.21
2896.50 = 828.18
AIRL
-7.13 0.11
-7.39 0.09
13.44:0.15 0.13
-4.64 + 0.09
Our init
1859.10 + 720.44
2038.90 + 260.78
4509.82 + 1470.65
4757.58 + 880.45
Our final
2681.20 = 530.67
3764.14 470.01
4778.82 + 760.34
4780.73 = 360.66
Table 10: Performance on Ant-v2 with different trajectories
Ant-v2
Demo
10
20
50
Expert
6190.90
1 254.18
BC
3958.20 + 661.28
3948.88 + 753.41
5424.01 + 473.05
5852,79
572.97
GAIL.
340.02 + 59.02
335.25 + 89.19
314.35 + 52.13
284.18 + 32.40
BC-GAIL
-1081.30 + 673.65
-1177.27 + 618.67
-13618.45 + 4237.79
-1166.16 16 + 1246.79
AIRL
-839.32 1: -301.54
-386.43
156.98
-586.07 1: 145.43
-393.90 1 145.13
Our mit
1150.82 200.87
3015.43
300.70
5200.58
870.74
5849.88
890.56
Our final
1693.59 = 350.74
3983.34
+ 250.99
5980.37
- 420.16
5988.65
470.03
Table 11: Performance on HalfCheetah v2 with different trajectories
HallCheetah-vv2
# #emo
5
10
20
50
Expert
12294.22
22 + 208 41
BC
225.42 +: 147.16
971.42
1: 249.62
2782.76 1 939.67
4813,20 1 1949.26
GAIL
-84.92
43.29
474.42
1: 389.30
-116.70 1: 34.14
-175.83
26.76
BC-GAIL
1362.59
1255.57
578.85
= 934.34
3744.32
1471.90
1597.51
1173.93
AIRI.
782.36 + 48.98
-146.46
+ 23.57
1437.25
25.45
755.46 + 10.92
Our mit
267.71 + 90.38
1064.44
227.32
32001.0
+ 520.04
7102,74 + 910.54
Our final
513.66 + 15.31
1616,34
+ 180.76
6059.27 + 344.41
8817.32 1 860.55
14

---

Published as a conference paper at ICLR 2020
Table 12: Performance Humanoid-v2 with different trajectories
Humanoid-v2
Demo
5
10
20
50
Expert
5286.2 21
145.98
BC
1521.55.1
272.14
3491.07.1
518.64
4686.05
1:355.74 74
4746.88
1605.61
GAIL
485.924 27.59
486.44
+27.18
477.15+t 22.07
481.14 24.37
BC-GAIL
363.68 +44.44
410.03 +33.07
487.99- 30.77
464.91 #33.21
AIRL
79.72 1: 4.27
87.15 = 5.01
-1293.86 1: 10.70
84.84 1 6.46
Our init
452.31 = 190.12
1317.63 = 110,45
4610.25
= 2750.86
4776.83 = 1320.46
Our final
1225.58 = 210.88
2190.43 = 280.18
4716.91 1680.29
4780.07 + 700.01
C
AND NETWORK ARCHITECTURE
When we pretrain the policy network with our methods, we choose 3 = 0.05 3-EEE. We use Adam with learning rate 3e-4 the basic optimization algorithms for all the experiments. The policy network and value network used in the algorithms all use a three-layer relu network with hidden size 256. We choose o = 0.1 in the policy prior for all the environments.
D COMPARISON WITH AIRL FUU ET AL.) FROM THEORETICAL PERSPECTIVE
Here illustrate the theoretical advantage of our SAIL algorithm over AIRL certain scenarios by an example.
The theory of AIRL shows that it is able to recover the groundtruth reward of an MDP up to constant if the reward of this MDP is define states only, when the adversarial learning reaches the equilibrium. Next we show a basic case that violates the theoretical assumption of AIRL but can be solved by our algorithm.
d21
du
2
az
412
Figure 6: Two-ring MDP with deterministic transition
6 the states and transition of an MDP. The demonstration policy jumps back and forth between and 82 periodically. Because algorithm has the action prior (local aligament), it is clear that we can solve this problem. The dynamics of many periodic games, such as Walker and HalfCheetah in MuJoco, are extension of this two-ring graph.
It easy to show that it is impossible for the adversarial game in AIRL solve this problem cquilibrium. According to Sec of et al.) the reward family of AIRL is parameterized as
fo 5') = g(s) + >h(s') - h(ss)
(8)
For simplicity of notation, let d(s) = g(s) - h(s) and (s)) = 7h(s), then fo (s. ss) o(s) (b(s')
(9)
In other words, the reward of AIRL is decomposible to the sum of two functions defined on states only.
Again, for simplicity, we omit the arguments functions but use subscripts to represent states. For example, f12 = f($1, s2) and Φ1 = (((1). Then,
f12 Φi + t/2, fei のz + v1. f=2= 4/2
f1= + w
(10)
Assume that ATRL has reached the equilibrium and learned the optimal policy, then must be true that f12 > fui and f21 > f22 (otherwise, there exists other optimal policies). But f12 > fu implies that 4/2 1/1, while f21 f22 implies that vi > viz, which is contradiction.
15


Reviews:
[{"review_id": "rkgo-346tS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper seeks a solution to the problem of performing imitation learning when the dynamics of the demonstrator are different from the dynamics of the imitator. The authors present a novel approach that combines global alignment by minimizing the Wasserstein distance between state occupancies with local alignment via a state-predictive VAE and inverse dynamics model. The experimental results support the claims that the method works for different dynamics and the proposed approach usually outperforms existing imitation learning methods.\n\nThe problem of dealing with different dynamics between a demonstrator and imitator is an important, but often overlooked problem in imitation learning. The combination of the global and local alignment is novel, nicely motivated, and ablation studies demonstrate that both are needed for good performance. Given the extensive experimental results showing the efficacy of this method I recommend that the paper be accepted. \n\nHowever, I feel that the paper can still be improved. Below are some of my questions and suggestions.\n\nThe success of BC is interesting. Why does it do so well? This seems to violate the motivation of the paper that using (s,a) for imitation learning won't work if the dynamics change. \n\nI thought the experiments for different action dynamics was very nice. The paper mentions that even state-spaces cannot be matched between the point mass and Ant. How do you know what part of the state space to imitate? Do you assume that is prior knowledge? Is this something that could be learned or inferred? How?\n\nHow is the potential updated. Eq(3) doesn't give an update rule. From later discussion it appears you mean take the \\phi that results in the supremum. Is this value learned? How is the optimization performed to solve Eq(3)?\n\nHow would you make the policy prior in Eq(7) work if actions are discrete? What if actions are multidimensional, with different ranges where some actions are not important? What is sigma?\n\nIn the RL community there is interest in transfer learning when the dynamics change. If the reward were observable, would the current approach be potentially useful for boosting the performance of transfer learning in RL? \n\nRelated Work:\n\nThe authors cite AIRL, but could do a better job distinguishing between AIRL and the current work. AIRL also tries to learn a state-based reward that is disentangled from the dynamics. Are there theoretical reasons why this work is better? Why does the proposed method work so much better in practice. On a related note, the results for the disabled ant seem much lower than those presented in the original AIRL paper. Why is this?\n\nThe authors do not mention the work by Brown et al. \"Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations\", ICML 2019. This work also learns only from observations and does not require any pretraining. How is the current work different?\n\nThe authors cite the work by Ayatar on imitation learning from observing YouTube videos. This method is also state-based and uses a similar state occupancy matching reward that would also work with different dynamics. How is the current method different. Would the current method work on domains such as learning to play Atari from raw visual trajectories?\n\nTypos:\n\n\"... able to resume to the demonstration trajectory by itself.\"\n--maybe say \" able to return to the demonstration trajectory by itself.\"\n\n\"... pairs as in an observation-based GAIL (Ho and Ermon 2016). I think this should be Torabi et al. instead.\n\nPage 5 \"state predictive VAE and an inverse *dynamics*\"\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1gjKT42Kr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of claims:\n\nThe paper proposes an imitation learning method that aims to align state distributions rather than state-action distributions to account for cases where the imitator dynamics differ from expert dynamics. They achieve this by two objectives: one local, the other global. The local objective aligns the next state to be close to the expert's next state in each transition by first training a VAE on the expert demonstrations, and using the trained VAE in conjunction with a pretrained inverse dynamics model to compute the action that the imitator needs to imitate. The global objective tries to do a global alignment of states encountered in the imitator and expert trajectories, by minimizing the Wasserstein distance between the two trajectory distributions. The paper claims that using these two objectives results in a method that outperforms existing inverse reinforcement learning and behavior cloning approaches in settings where the imitator and expert dynamics differ.\n\nDecision:\n\nI recommend the paper to be rejected. I have three main reasons for my decision (with more details in the next section):\n1. The paper is very poorly written : A lot of details are missing in the paper, notation is not standardized,  related work is just a list of previous papers without any context on how the proposed method is related, previous methods are referred to without any citations, and quite a few blanket statements which are not substantiated.\n2. Incomplete approach description: Quite a few components of the approach are not explained (or even discussed), no intuition provided for the choices made in the approach, the concept of different dynamics is not formalized, some technical inconsistencies in the algorithm, no formal problem statement (which would really help in standardizing notation), and most claims made about the approach are not justified or substantiated\n3. Poor experiments: Experiments are not well chosen to reflect the premise and claims of the paper, little to no details given for how the baseline approaches were trained, no details on policy parameterization, and missing comparison with baseline approaches in some experiments\n\nComments:\n\n(1) Problem setup: \n(a)Problem setup is very vague and not formalized. \n(b) Differing dynamics could mean several things: different agent dynamics (like different action spaces; different actuators; etc.), different environment dynamics (different moving obstacles in the world;) etc. \n(c) Basically, different dynamics can mean a lot more than what was accounted for in the paper\n\n(2) Blanket Statements: \n(a) The authors keep saying that their framework is more flexible without any justification as to why, \n(b) \"simply train an inverse dynamics model\"- training an inverse dynamics model can be very hard especially when environment dynamics are stochastic/when the inverse model is multimodal, \n(c) \"constraint becomes loosened\" - this statement doesn't make any sense without more explanation,\n(d) Several other blanket statements about existing approaches\n\n(3) Notation : \n(a) Notation was never standardized in the paper, \n(b) what is \\phi? what is the input-output of \\phi? \n(c) What is \\theta_old? What is \\sigma? \n(d) There are a lot of things that needed explanation, especially in the algorithm\n\n(4) Related Work : \n(a) The related work section is just a dump of citations without giving any context for where the proposed work lies in the spectrum of these works. How does it compare? Why is it better/worse? \n(b) Missing related work that was publshed in ICML 2019 that has a very similar approach in matching state distributions (\"Provably efficient Imitation Learning from Observations Alone\" or FAIL) and works very well.\n\n(5) Motivation : \n(a) The approach, in general, needs better motivation. The toy example in the introduction was good but the experiments did not reflect the complexity of that example. \n(b) Try to have a running example in the paper that will help you motivate the approach better.\n\n(6) Background : \n(a) The background section is very minimal and lacks any details necessary. \n(b) I had to read the beta-VAE paper to understand what it does. \n(c) The section also lacks any minimal background in IL/RL and the notation could also have been standardized in this section\n\n(7) Figures: \n(a) All the figures in this paper could use a lot of improvement in terms of descriptive captions, more informative legends, bigger fonts, descriptive text, and more figures as well\n\n(8) Algorithm : \n(a) The algorithm was not referenced anywhere in the text, \n(b) No definition for \\tau, details on pretraining inverse dynamics model lacking in both text and algorithm, \n(c) *Policy prior is used to pretrain policy before the VAE was trained!* which doesn't make any sense since the policy prior is obtained using the VAE, \n(d) the equation at the end of Sec 4.3 has r(s, a) whereas reward is defined as r(s_t, s_{t+1}) but they are not equivalent when dynamics are stochastic\n\n(9) Experiments: \n(a) What is AIRL? No reference was given. \n(b) Why is keeping the variance of the policy constant reasonable? How do you come up with the value? \n(c) How do you pretrain the VAE, invserse dynamics model? \n(d) The setup of making the ant's legs smaller or body heavier seems very artificial. I am sure its easy to come up with more realistic setups in navigation domains, for example. Try to use more realistic experiments in the future. \n(e) For results in Fig 3, is AIRL, GAIL also pretrained with VAE or by BC? Seems like SAIL was pretrained but the others weren't since SAIL starts off with a high score at the start. \n(f) For Sec 5.1.2, comparison with baseline approaches are missing. Legends for the plots are terribly small.\n\nConceptual questions:\n1. How do you account for cases where due to differing dynamics, states reached by expert in demonstrations are unreachable by the imitator? \n2. How do you account for cases where the environment dynamics changes between expert and imitator?\n3. Why does using Wasserstein distance make sense? Why not other f-divergences? Also, matching global distributions can be very misleading if you have states that are visited multiple times in the same trajectory. FAIL recommends matching state distribution at each time-step instead and is much more stable\n4. How is this different from GAIL where we match state visitation distribution (instead of state-action visitation distribution?)\n\nThings to improve:\n1. Writing needs to be improved a lot\n2. Better experiments - more realistic domains\n3. Approach needs to be explained more formally", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeDsmmIqS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review for \"State alignment-based imitation learning.\"\n\nSummary: \n\nThis paper addresses the problem of learning from demonstrations where the demonstrator may potentially have different dynamics from the learner. To address this, the paper proposes to align state-distributions (rather than state-action distributions) between the demonstrations and the learner. To alleviate issues arises from doing this alone, they also propose to use a local learned prior to guide their policy to select actions that take it back to the original demonstrations. The paper shows a number of experiments on control tasks in support of their claims. \n\nPros:\n+ The problem that the paper solves is fairly relevant, and some experiments (such as cross morphology imitation learning) are promising in concept. \n+ The paper is mostly well written (save some small improvements that could be made in clarity) and can be followed. \n+ The paper presents a series of experiments across several agents and some other baselines.\n\nCons (and primary concerns): \n\n1) The idea of matching state distributions in the context of learning behaviors is not new. In particular, [1] also uses similar ideas of matching state distributions in the imitation learning context, if via different machinery. [4] points towards such ideas as well (noted on page 43). Works such as [2, 3] also use this idea in the context of Reinforcement Learning. Further, ideas of deviation correction in the imitation learning domain have been addressed before in [5]. The paper would benefit from a more thorough treatment of these related works, and how the proposed work differs from these. \n\n2) The choice of approach (in particular, the use of the Wasserstein distance to match state distributions, and the manner of learning a local prior by training an autoregressive Beta VAE) are lacking motivation, and it is unclear if or why these choices are the best way to approach the problem. \n\n3) While the paper presents a large number of comparisons, the analysis of the relative performance of the proposed approach against the baselines is lacking. For example, in section 5.1.1., vanilla BC seems to do very well - why is it the proposed approach only marginally outperforms BC on several of these tasks? In section 5.2, why is SAIL able to outperform other IL techniques on same-dynamics tasks? What about SAIL provides this performance benefit? Similarly, in section 5.3.3, what is it about the Wasserstein objective and the KL that together enables good learning? This ablation seems crucial to assessing the paper, and is lacking a deeper analysis. Further, the relevance of section 5.3.1 is questionable - as no new insight is provided over the original Beta-VAE paper. \n\nOther Concerns: \n\n1) The paper ultimately uses a form of prior that is defined over actions, and not states (so that it may be used in the KL divergence term). How is the choice of form of prior made? It is unclear why it is better to have a prior learned over states converted to actions via eq. 7, versus a similarly designed prior over actions. \n\n2) It is unclear why the expression of the reward function (Eq. 4) is necessary - if it is possible to compute the Wasserstein distance (and hence the cummulative reward), it is possible to update the policy purely from this cummulative reward. \nIs the function of the per-timestep reward simply to provide a denser signal to the policy optimization? \n\n3) The authors claim to introduce a \"unified RL framework\" in their regularized policy objective. It appears that this is simply the addition of the KL between the policy and the prior $p_a$ to the global alignment objective (subsumed into $L_{CLIP}$), hence the reviewer questions whether this can indeed be treated as a novel contribution of the paper. \n\n4) The problem this paper addresses (and the fundamental thesis for its approach) is that action-predictive methods are likely to suffer from deviation from the original demonstrations, as compared to state-predictive methods. \nWhat purpose does section 5.3.2 serve beyond reiterating this point? \n\n5) State-matching (and the implied use of inverse models) means the feasibility of retrieved actions is not guaranteed, as compared to models that predict actions directly. \n\nMinor Points: \n\n1) Explaining the various phases of training (as observed in the algorithm) would be useful. \n\n2) Discussing how states are compared in the cross morphology experiments (Section 5.1.2.) would also be useful. \n\nCited Literature: \n\n[1] State Aware Imitation Learning, Yannick Shroecker and Charles Isbell, https://papers.nips.cc/paper/6884-state-aware-imitation-learning.pdf\n[2] Efficient Exploration via State Marginal Matching, Lisa Lee et. al., https://arxiv.org/abs/1906.05274\n[3] State Marginal Matching With Mixtures Of Policies, Lisa Lee et. al., https://spirl.info/2019/camera-ready/spirl_camera-ready_25.pdf\n[4] An Algorithmic Perspective on Imitation Learning, Takayuki Osa et. al., https://arxiv.org/pdf/1811.06711.pdf\n[5] Improving Multi-step Prediction of Learned Time Series Models, Arun Venkatraman et. al., https://www.ri.cmu.edu/pub_files/2015/1/Venkatraman.pdf\n\nInitial Decision: Weak reject\n\n#######\nPost Rebuttal Comments: \n\nConsidering the authors' motivations of approach and the additional analysis provided in the comments below, I change my decision to weak accept. I would like to encourage the authors to include the details listed below in their paper. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper seeks to adapt behavioural cloning to the case where demonstrator and learner have different dynamics (e.g. human demonstrator), by designing a state-based objective. The reviewers agreed the paper makes an important and interesting contribution, but were somewhat divided about whether the experiments were sufficiently impactful. They furthermore had additional concerns regarding the clarity of the paper and presentation of the method. Through discussion, it seems that these were sufficiently addressed that the consensus has moved towards agreeing that the paper sufficiently proves the concept to warrant publication (with one reviewer dissenting).

I recommend acceptance, with the view that the authors should put a substantial amount of work into improving the presentation of the paper based on the feedback that has emerged from the discussion before the camera ready is submitted (if accepted).
Decision: Accept (Poster)