Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The idea of the paper is to learn a distance function between observed and the agent’s behaviors. Once they have the distance function, they can learn the agent’s policy efficiently given a single demonstration of each task. In their formulation, the distance function and the policy are jointly learned. 

The idea is reasonable and the performance outperforms baselines like GAIL and VAE. However, the paper is not-well written with many relevant equations defined in the supplementary material. The unsupervised data labeling part seems Adhoc with many details in the supplementary material. I wonder if the process stable or not. How many lower than the average performance of the proposed method as shown in F.g 4 are caused by unsupervised data labeling?

In Fig. 4b, the manual performance is very strong once converged. Although the proposed method initially reaches high reward, after twice many iterations the manual performance even outperforms the proposed method on average many times. Hence, I am not very convinced about the proposed method will be the best-picked method in practice.

Overall, I think the idea is good. But the paper is poorly written and I concern the most about the stability of the unsupervised data labeling process. The experimental results are also not super convincing. Hence, I recommend for weak rejection.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper presents an imitation learning method that deploys previously well-studied techniques such as siamese networks, inverse RL, learning distance functions for IRL and tracking.

+ the paper studies an important problem of IL using visual data.
+ I found the ablation studies in the appendix quite useful in understanding the efficiency of the proposed method.

-In terms of novelty, the proposed approach is a combination of several past works so the technical novelty is limited. Additionally, it is not clear how impactful the proposed method can be given that it is only tested on a synthetic domain which is the same as the train domain. So, from the current experimental results it is not clear if this approach would be effective to be applied in a real system (e.g. robots) on the practical side. 

-There are not enough evaluation done to compare with the most updated state-of-the-art baselines. The evaluations are done on just a single synthetic domain with a single character. Therefore, the train and test videos are very similar. 


review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper presents visual imitation with reinforcement learning (VIRL), an algorithm for learning to imitate expert trajectories based solely on visual observations, and without access to the expert’s actions.  The algorithm is similar in form to GAIL and its extensions, learning a reward function which captures the similarity between an observed behavior and the expert's demonstrations, while simultaneously using reinforcement learning to find a policy maximizing this reward, such that the learned policy will replicate the demonstrated behavior as well as possible.  A key feature of this method is that the learned reward function is defined by a learned distance metric, which evaluates the similarity between the agent's current trajectory, and the nearest demonstrated expert trajectory.

The network describing the distance metric is recurrent, such that the distance is defined between trajectories rather than individual states.  The distance function network is trained via a negative sampling approach, where expert trajectories are randomly reordered to produce examples that dissimilar to the expert trajectories.   The distance network also defines a variational autoencoder, and the reconstruction of the target trajectories is treated as an auxiliary task to help train better representations of the trajectory space.

While previous work has considered the problem of visual imitation learning, the approach taken here is novel in its architecture and loss function, and significantly outperforms the baselines in terms of the similarity between the resulting behavior and the expert behavior.

The clarity of the technical presentation could be improved, however.  In particular, it would be helpful for the reader if the definitions of the negative sampling loss and the autoencoder losses were given before the combined loss, and if we saw the form of the loss for both positive and negative sequence pairs.  Equation 4 could also be made explicit, with the full summation term included.
review_title: Official Blind Review #2

Meta-review: The main concern raised by reviewers is limited novelty, poor presentation, and limited experiments. All the reviewers appreciate the difficulty and importance of the problem. The rebuttal helped clarify novelty, but the other concerns remain.
Decision: Reject