Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes an imitation method, I2L, that learns from state-only demonstrations generated in an expert MDP that may have different transition dynamics than the agent MDP. I2L modifies the existing adversarial inverse RL algorithm: instead of training the disciminator to distinguish demonstrations vs. samples, I2L trains the discriminator to distinguish samples that are close (in terms of the Wasserstein metric) to the demonstrations vs. other samples. This approach maximizes a lower bound on the likelihood of the demonstrations. Experiments comparing I2L to a state-only GAIL baseline show that I2L performs significantly better under dynamics mismatch in several low-dimensional, continuous MuJoCo tasks.

Overall, I enjoyed reading this paper. A few comments:

1. It would be nice to include a behavioral cloning (e.g., BCO) baseline in the experiments. Your point in Section 4 that BC can suffer from compounding errors is well taken, but in my experience, BC can perform surprisingly well on some of the MuJoCo benchmark tasks, even from a single demonstration trajectory. Prior work showing relatively poor results for BC on MuJoCo tasks usually sub-samples demonstrations to intentionally exacerbate the state distribution shift encountered by BC.

2. It would be nice to discuss potential failure cases for I2L. For example, how dependent is the method on the diversity of trajectories \tau generated by the agent in line 5 of Algorithm 1? Are there conditions in which training the critic network to approximate the Wasserstein metric is harder than prior methods like Stadie et al. (2017) and Liu et al. (2018)?
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The submission considers the imitation learning with environment change. In the new environment (where we we aim to conduct imitation learning), the expert demonstrations are unavailable, making the objective function (5) unavailable. To deal with this, the authors derive a lower bound to replace (5). 

In the experiment section, the performance of the proposed method is clearly demonstrated. Environment change is a challenging case for existing imitation learning methods, while the proposed one works. 

While empirically, the performance of the proposed method is justified, I am curious how tight the bound used to replace (or approximate) (5) is. The submission offers few discussions on the error may induced by replacing (5) with the lower bound. Will such an error decrease or converge to 0, as we iterate according to the algorithm? 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary:
The manuscript considers the problem of imitation learning when the system dynamics of the agent are different from the dynamics of the expert. The paper proposes Indirect Imitation Learning (I2L), which aims to perform imitation learning with respect to a trajectory buffer that contains some of the previous trajectories of the agent. The trajectory buffer has limited capacity and adds trajectories based on a priority-queue that prefers trajectories that have a similar state distribution to the expert. Similarity is hereby measured by the score of a WGAN-critic trained to approximate the W1-Wasserstein distance between the previous buffer and the expert distribution. By performing imitation learning with respect to a trajectory buffer, state-action trajectories of the agent's MDP are available, which enables I2L to apply AIRL (Fu et al. 2017). By using those trajectories for the transition buffer that have state-marginals close to the expert's trajectory, I2L produces similar behavior compared to the expert. I2L is compared to state-only GAIL, state-action-GAIL and AIRL on four MuJoCo tasks with modified dynamics compared to the expert policy. The experiments show that I2L may learn significantly better policies if the dynamics of agent and the expert do not match.

Decision:
I think that the submission is below borderline in its current state. The main reason for rejecting would be the insufficient justification of some algorithmic choices, improper presentation of the experiments and the description of MaxEnt-IRL, which seems quite wrong in my opinion. However, I think that the work is interesting and sufficiently novel and could be accepted if the mentioned issues were adequately addressed.

Supporting Arguments:
- Novelty / Significance: Imitation learning from state-only observations under dynamic mismatch is an important problem and a promising approach for training robots by non-experts. The idea of performing imitation learning with respect to carefully updated trajectory buffer seems simple and effective. Although self-imitation has been applied in reinforcement learning (references in submission), I am not aware of a similar approach in imitation learning.

- Soundness / Correctness:
1. The description of MaxEnt-IRL seems quite wrong. The paper claims in Section 2.1. that MaxEnt-IRL maximizes the likelihood of the policy by optimizing the advantage function and thus only learns shaped rewards. However, the referenced paper (Ziebart et al. 2008) optimizes the weights of a linear reward function which in general does not correspond to the advantage of the learned policy. Also, the policy is not (only) proportional to the exponentiated advantage but equal to it and the normalizer of the trajectory distribution (Eq. 1) would be therefore 1.

2. I think that the selection of trajectories for the buffer is not sufficiently well motivated. I2L is derived based on a lower bound but it seems that it does not even ensure improvement on that lower bound. It is not fully clear to me how the trajectory buffer is updated (see "Clarity") but it does not seem to ensure that the Wasserstein distance decreases compared to the last iteration. The trajectories are chosen greedily, i.e., without considering the remaining trajectories in the buffer which can be especially problematic for multimodal demonstrations. 

- Clarity / Presentation:
The paper is well-written in general and only has few typos. It is not clear to me how exactly the trajectory buffer is updated. The submission merely states that the buffer is a priority-queue structure of fixed lengths, where the trajectory priorities are given by the average state score based on the Wasserstein critic. This description leaves several open questions (see "Questions") and further does not seem well motivated.


- Evaluation:
The presentation of the experimental results seems odd. For the imitation learning experiments (no dynamic mismatch) the (apparently) same runs of I2L are compared to GAIL-S (Table 1) and GAIfO (Table 2) in separate tables. For the experiments under dynamic mismatch, the comparison with GAIL-S are shown with learning curves (Figure 3-5) and shaded error bars, whereas the results of GAIfO are only shown in in terms of final mean performance in Table 2. This presentation may make the impression or hiding learning curves or confidence intervals. Both tables could be removed by adding a single curve to each of the plots in Figure 3-5 and 7. It is also not clear to me, why only the mean of the final performance for AIRL and SA-GAIL is shown in Figure 3-5. Instead, the performance of the expert--which is currently not shown--would be better suited as a baseline level. The paper also does not seem to mention the number of evaluations and the meaning of the error region in the figures. Furthermore, some hyper-parameters, e.g., the architectures for the AIRL reward/value-function networks are not presented.

Questions:
- Please precisely state when the trajectories are removed from the trajectory buffer and the heap of the priority queue. May the same trajectories be used during several iterations?
- Please elaborate: under which assumptions does the update of the trajectory buffer ensure that the W1-distance of the individual trajectories decreases with respect to the expert's trajectories? 


Minor comments:
Typo: "upto a constant"

Post-Rebuttal Update:
I increased my rating to weak accept because the authors addressed my main concerns by a) giving additional information on the update of the buffer, b) improving the presentation of the experiments, c) fixing the description of MaxEnt-IRL and d) by showing empirically that both the lower bound increases / Wasserstein-distance decreases.
review_title: Official Blind Review #1

Meta-review: This paper addresses the setting of imitation learning from state observations only, where the system dynamics under which the demonstrations are performed differs from the target environment. The paper proposes to circumvent this dynamics shift with an algorithm whereby the target policy is trained to imitate its own past trajectories, re-ranked based on the similarity in state occupancies as judged by a WGAN critic.

The reviewers found the paper to be clearly written and enjoyable. The paper improved considerably through reviewers feedback. Notably, a behavior cloning from observations (BCO) baseline was added, which was stronger than the authors expected but still helped highlight the strength of the proposed method by comparison. R1 had a particularly productive multiple round exchange, clarifying the description of previous work, clarifying the details of the proposed procedure and strengthening the presentation of empirical evidence.

This work compellingly addresses an important problem, and in its final form is a polished piece of work. I recommend acceptance.
Decision: Accept (Poster)