Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Review for "State alignment-based imitation learning."

Summary: 

This paper addresses the problem of learning from demonstrations where the demonstrator may potentially have different dynamics from the learner. To address this, the paper proposes to align state-distributions (rather than state-action distributions) between the demonstrations and the learner. To alleviate issues arises from doing this alone, they also propose to use a local learned prior to guide their policy to select actions that take it back to the original demonstrations. The paper shows a number of experiments on control tasks in support of their claims. 

Pros:
+ The problem that the paper solves is fairly relevant, and some experiments (such as cross morphology imitation learning) are promising in concept. 
+ The paper is mostly well written (save some small improvements that could be made in clarity) and can be followed. 
+ The paper presents a series of experiments across several agents and some other baselines.

Cons (and primary concerns): 

1) The idea of matching state distributions in the context of learning behaviors is not new. In particular, [1] also uses similar ideas of matching state distributions in the imitation learning context, if via different machinery. [4] points towards such ideas as well (noted on page 43). Works such as [2, 3] also use this idea in the context of Reinforcement Learning. Further, ideas of deviation correction in the imitation learning domain have been addressed before in [5]. The paper would benefit from a more thorough treatment of these related works, and how the proposed work differs from these. 

2) The choice of approach (in particular, the use of the Wasserstein distance to match state distributions, and the manner of learning a local prior by training an autoregressive Beta VAE) are lacking motivation, and it is unclear if or why these choices are the best way to approach the problem. 

3) While the paper presents a large number of comparisons, the analysis of the relative performance of the proposed approach against the baselines is lacking. For example, in section 5.1.1., vanilla BC seems to do very well - why is it the proposed approach only marginally outperforms BC on several of these tasks? In section 5.2, why is SAIL able to outperform other IL techniques on same-dynamics tasks? What about SAIL provides this performance benefit? Similarly, in section 5.3.3, what is it about the Wasserstein objective and the KL that together enables good learning? This ablation seems crucial to assessing the paper, and is lacking a deeper analysis. Further, the relevance of section 5.3.1 is questionable - as no new insight is provided over the original Beta-VAE paper. 

Other Concerns: 

1) The paper ultimately uses a form of prior that is defined over actions, and not states (so that it may be used in the KL divergence term). How is the choice of form of prior made? It is unclear why it is better to have a prior learned over states converted to actions via eq. 7, versus a similarly designed prior over actions. 

2) It is unclear why the expression of the reward function (Eq. 4) is necessary - if it is possible to compute the Wasserstein distance (and hence the cummulative reward), it is possible to update the policy purely from this cummulative reward. 
Is the function of the per-timestep reward simply to provide a denser signal to the policy optimization? 

3) The authors claim to introduce a "unified RL framework" in their regularized policy objective. It appears that this is simply the addition of the KL between the policy and the prior $p_a$ to the global alignment objective (subsumed into $L_{CLIP}$), hence the reviewer questions whether this can indeed be treated as a novel contribution of the paper. 

4) The problem this paper addresses (and the fundamental thesis for its approach) is that action-predictive methods are likely to suffer from deviation from the original demonstrations, as compared to state-predictive methods. 
What purpose does section 5.3.2 serve beyond reiterating this point? 

5) State-matching (and the implied use of inverse models) means the feasibility of retrieved actions is not guaranteed, as compared to models that predict actions directly. 

Minor Points: 

1) Explaining the various phases of training (as observed in the algorithm) would be useful. 

2) Discussing how states are compared in the cross morphology experiments (Section 5.1.2.) would also be useful. 

Cited Literature: 

[1] State Aware Imitation Learning, Yannick Shroecker and Charles Isbell, https://papers.nips.cc/paper/6884-state-aware-imitation-learning.pdf
[2] Efficient Exploration via State Marginal Matching, Lisa Lee et. al., https://arxiv.org/abs/1906.05274
[3] State Marginal Matching With Mixtures Of Policies, Lisa Lee et. al., https://spirl.info/2019/camera-ready/spirl_camera-ready_25.pdf
[4] An Algorithmic Perspective on Imitation Learning, Takayuki Osa et. al., https://arxiv.org/pdf/1811.06711.pdf
[5] Improving Multi-step Prediction of Learned Time Series Models, Arun Venkatraman et. al., https://www.ri.cmu.edu/pub_files/2015/1/Venkatraman.pdf

Initial Decision: Weak reject

#######
Post Rebuttal Comments: 

Considering the authors' motivations of approach and the additional analysis provided in the comments below, I change my decision to weak accept. I would like to encourage the authors to include the details listed below in their paper. 
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper seeks a solution to the problem of performing imitation learning when the dynamics of the demonstrator are different from the dynamics of the imitator. The authors present a novel approach that combines global alignment by minimizing the Wasserstein distance between state occupancies with local alignment via a state-predictive VAE and inverse dynamics model. The experimental results support the claims that the method works for different dynamics and the proposed approach usually outperforms existing imitation learning methods.

The problem of dealing with different dynamics between a demonstrator and imitator is an important, but often overlooked problem in imitation learning. The combination of the global and local alignment is novel, nicely motivated, and ablation studies demonstrate that both are needed for good performance. Given the extensive experimental results showing the efficacy of this method I recommend that the paper be accepted. 

However, I feel that the paper can still be improved. Below are some of my questions and suggestions.

The success of BC is interesting. Why does it do so well? This seems to violate the motivation of the paper that using (s,a) for imitation learning won't work if the dynamics change. 

I thought the experiments for different action dynamics was very nice. The paper mentions that even state-spaces cannot be matched between the point mass and Ant. How do you know what part of the state space to imitate? Do you assume that is prior knowledge? Is this something that could be learned or inferred? How?

How is the potential updated. Eq(3) doesn't give an update rule. From later discussion it appears you mean take the \phi that results in the supremum. Is this value learned? How is the optimization performed to solve Eq(3)?

How would you make the policy prior in Eq(7) work if actions are discrete? What if actions are multidimensional, with different ranges where some actions are not important? What is sigma?

In the RL community there is interest in transfer learning when the dynamics change. If the reward were observable, would the current approach be potentially useful for boosting the performance of transfer learning in RL? 

Related Work:

The authors cite AIRL, but could do a better job distinguishing between AIRL and the current work. AIRL also tries to learn a state-based reward that is disentangled from the dynamics. Are there theoretical reasons why this work is better? Why does the proposed method work so much better in practice. On a related note, the results for the disabled ant seem much lower than those presented in the original AIRL paper. Why is this?

The authors do not mention the work by Brown et al. "Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations", ICML 2019. This work also learns only from observations and does not require any pretraining. How is the current work different?

The authors cite the work by Ayatar on imitation learning from observing YouTube videos. This method is also state-based and uses a similar state occupancy matching reward that would also work with different dynamics. How is the current method different. Would the current method work on domains such as learning to play Atari from raw visual trajectories?

Typos:

"... able to resume to the demonstration trajectory by itself."
--maybe say " able to return to the demonstration trajectory by itself."

"... pairs as in an observation-based GAIL (Ho and Ermon 2016). I think this should be Torabi et al. instead.

Page 5 "state predictive VAE and an inverse *dynamics*"



review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary of claims:

The paper proposes an imitation learning method that aims to align state distributions rather than state-action distributions to account for cases where the imitator dynamics differ from expert dynamics. They achieve this by two objectives: one local, the other global. The local objective aligns the next state to be close to the expert's next state in each transition by first training a VAE on the expert demonstrations, and using the trained VAE in conjunction with a pretrained inverse dynamics model to compute the action that the imitator needs to imitate. The global objective tries to do a global alignment of states encountered in the imitator and expert trajectories, by minimizing the Wasserstein distance between the two trajectory distributions. The paper claims that using these two objectives results in a method that outperforms existing inverse reinforcement learning and behavior cloning approaches in settings where the imitator and expert dynamics differ.

Decision:

I recommend the paper to be rejected. I have three main reasons for my decision (with more details in the next section):
1. The paper is very poorly written : A lot of details are missing in the paper, notation is not standardized,  related work is just a list of previous papers without any context on how the proposed method is related, previous methods are referred to without any citations, and quite a few blanket statements which are not substantiated.
2. Incomplete approach description: Quite a few components of the approach are not explained (or even discussed), no intuition provided for the choices made in the approach, the concept of different dynamics is not formalized, some technical inconsistencies in the algorithm, no formal problem statement (which would really help in standardizing notation), and most claims made about the approach are not justified or substantiated
3. Poor experiments: Experiments are not well chosen to reflect the premise and claims of the paper, little to no details given for how the baseline approaches were trained, no details on policy parameterization, and missing comparison with baseline approaches in some experiments

Comments:

(1) Problem setup: 
(a)Problem setup is very vague and not formalized. 
(b) Differing dynamics could mean several things: different agent dynamics (like different action spaces; different actuators; etc.), different environment dynamics (different moving obstacles in the world;) etc. 
(c) Basically, different dynamics can mean a lot more than what was accounted for in the paper

(2) Blanket Statements: 
(a) The authors keep saying that their framework is more flexible without any justification as to why, 
(b) "simply train an inverse dynamics model"- training an inverse dynamics model can be very hard especially when environment dynamics are stochastic/when the inverse model is multimodal, 
(c) "constraint becomes loosened" - this statement doesn't make any sense without more explanation,
(d) Several other blanket statements about existing approaches

(3) Notation : 
(a) Notation was never standardized in the paper, 
(b) what is \phi? what is the input-output of \phi? 
(c) What is \theta_old? What is \sigma? 
(d) There are a lot of things that needed explanation, especially in the algorithm

(4) Related Work : 
(a) The related work section is just a dump of citations without giving any context for where the proposed work lies in the spectrum of these works. How does it compare? Why is it better/worse? 
(b) Missing related work that was publshed in ICML 2019 that has a very similar approach in matching state distributions ("Provably efficient Imitation Learning from Observations Alone" or FAIL) and works very well.

(5) Motivation : 
(a) The approach, in general, needs better motivation. The toy example in the introduction was good but the experiments did not reflect the complexity of that example. 
(b) Try to have a running example in the paper that will help you motivate the approach better.

(6) Background : 
(a) The background section is very minimal and lacks any details necessary. 
(b) I had to read the beta-VAE paper to understand what it does. 
(c) The section also lacks any minimal background in IL/RL and the notation could also have been standardized in this section

(7) Figures: 
(a) All the figures in this paper could use a lot of improvement in terms of descriptive captions, more informative legends, bigger fonts, descriptive text, and more figures as well

(8) Algorithm : 
(a) The algorithm was not referenced anywhere in the text, 
(b) No definition for \tau, details on pretraining inverse dynamics model lacking in both text and algorithm, 
(c) *Policy prior is used to pretrain policy before the VAE was trained!* which doesn't make any sense since the policy prior is obtained using the VAE, 
(d) the equation at the end of Sec 4.3 has r(s, a) whereas reward is defined as r(s_t, s_{t+1}) but they are not equivalent when dynamics are stochastic

(9) Experiments: 
(a) What is AIRL? No reference was given. 
(b) Why is keeping the variance of the policy constant reasonable? How do you come up with the value? 
(c) How do you pretrain the VAE, invserse dynamics model? 
(d) The setup of making the ant's legs smaller or body heavier seems very artificial. I am sure its easy to come up with more realistic setups in navigation domains, for example. Try to use more realistic experiments in the future. 
(e) For results in Fig 3, is AIRL, GAIL also pretrained with VAE or by BC? Seems like SAIL was pretrained but the others weren't since SAIL starts off with a high score at the start. 
(f) For Sec 5.1.2, comparison with baseline approaches are missing. Legends for the plots are terribly small.

Conceptual questions:
1. How do you account for cases where due to differing dynamics, states reached by expert in demonstrations are unreachable by the imitator? 
2. How do you account for cases where the environment dynamics changes between expert and imitator?
3. Why does using Wasserstein distance make sense? Why not other f-divergences? Also, matching global distributions can be very misleading if you have states that are visited multiple times in the same trajectory. FAIL recommends matching state distribution at each time-step instead and is much more stable
4. How is this different from GAIL where we match state visitation distribution (instead of state-action visitation distribution?)

Things to improve:
1. Writing needs to be improved a lot
2. Better experiments - more realistic domains
3. Approach needs to be explained more formally
review_title: Official Blind Review #1

Meta-review: This paper seeks to adapt behavioural cloning to the case where demonstrator and learner have different dynamics (e.g. human demonstrator), by designing a state-based objective. The reviewers agreed the paper makes an important and interesting contribution, but were somewhat divided about whether the experiments were sufficiently impactful. They furthermore had additional concerns regarding the clarity of the paper and presentation of the method. Through discussion, it seems that these were sufficiently addressed that the consensus has moved towards agreeing that the paper sufficiently proves the concept to warrant publication (with one reviewer dissenting).

I recommend acceptance, with the view that the authors should put a substantial amount of work into improving the presentation of the paper based on the feedback that has emerged from the discussion before the camera ready is submitted (if accepted).
Decision: Accept (Poster)