Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The approach described in the paper uses a relatively complicated architecture, with multiple losses and a very processed training set, but it achieves strong results when compared to two of the most similar published methods - TCNs (Time-Contrastive Networks) and GAIfO (Generative adversarial imitation from observations). A set of ablations is presented which is not exhaustive, but which is adequate to understand the comparative importance of the different parts of the approach. Overall, I think that there are some rough edges to the paper which could be improved, but the contribution of the paper is enough to warrant publication.

One problem is that the method is poorly explained in some places. Some of the terms need to be explained earlier in the paper - Siamese Network or Siamese Loss is never properly explained and TCN is used in the abstract without reference or definition. At the beginning of section 3, it is stated that the 'Siamese network triplet loss' is used in the proposed method, but Eq. 3 shows a contrastive loss with margin, not a triplet loss. Adding to the problems with clarity, there are a number of obvious typos which are distracting to the reader and make it clear that this is not a fully polished submission:

typos - oultine -> outline; Advisarial -> Adversarial; primariliy -> primarily; subsquent -> subsequent; independanlty -> independently; kinematiccally -> kinematically 
fragments: 'Including recent Sim2Real quadreped robots and a huanoid with 38 DoF, which is a
particularly challenging problem domain.'; 'Where states and actions are discrete.'

The paper claims that the method can train an agent to do imitations from noisy visual data from single demonstrations, but this is not clearly shown by the experiments. The domains where the agent performs well (judging by the videos available on the linked website) are simple and cyclic and have little or no variation. It is unclear whether the agent would be able to 1-shot imitate a novel demonstration which was not within the training set already. On the more challenging domains, the videos show that the agent imitates the expert for a very short timespan before diverging. However, visual imitation is very difficult and the proposed method does perform better than the other baselines.

The paper would have more impact if the authors had shown whether the trained approach could be used for better performance or learning speed on new tasks where expert data was not available, perhaps by training a new policy on top of the LSTM, or distilling to a new agent and then finetuning.

Overall, the paper is exciting because it shows the value of the complex architecture for solving a challenging problem. The experiments are convincing and show that this is a promising approach which may be useful in a real-world domain.

Pros: 
- The method is well-designed and is a natural extension on existing work (TCNs, Siamese nets, GAIfO). There is enough novelty for the work to be published.
- The results show that the approach works well on a number of different continuous control environments
- The comparisons to other published methods, the baselines and ablations, are well chosen.
- The supplementary contains additional details on training, and additional analysis, which is valuable

Cons:
- The text is poorly written in places - Acronyms and terms need to be explained, and spelling and grammar need to be proofed. However, this is an easy fix that should not prevent the paper from being published.
- The authors have not adequately explained whether the model is actually capable of 1-shot imitation. It is not clear from the domains whether the agent has simply memorized the full data distribution.
- The model is limited if there is no way to reuse the model without demonstrations.
review_title: Good results in a challenging setting: one-shot imitation from visual demonstration 

final_score: rating: 5: Marginally below acceptance threshold
review_content: SUMMARY:
The authors propose an extension of recent imitation from observation techniques called VIRL that explicitly incorporates Siamese network and LSTM encodings in order to attempt to better overcome some of the challenges of imitation learning from visual observations. They evaluate their algorithm in several simulation domains and find reasonable results.


STRENGTHS:
	(S1) The paper considers an important and timely problem in trying to improve upon the state-of-the-art in imitation learning from visual observations.
	(S2) The paper proposes an interesting new technique that, in some ways, seems to perform very well in experiments.


WEAKNESSES:
	(W1) The motivation for the technique in the sense of being for single-shot learning is not clear. Learning a distance metric between behaviors would seem to be applicable for learning from _many_ demonstrations -- in what way is the proposed approach specialized for single-shot learning?
	(W2) The presentation of the experimental results is, at times, incomplete and confusing. For example:
		(a) The ablation study shows lines for random/max sequence lengths, but (as best I can tell), the methods that generated these curves are never discussed.
		(b) The ablation study presented in Figures 5a and 5b generates more questions that it answers. For example, how is it that "(EESP) + pretrain" gets the lowest loss in 5b but does very poorly in 5a?
		(c) The experimental setup is unclear to the point where its doubtful that a reader could reproduce the results. In particular, at the beginning of Section 5, it's not clear exactly how the demonstrations are generated. The authors state that the simulation environments are "new and have all been updated to take motion capture data and produce view video data that can be used for training RL agents," but do not describe in any detail how the mocap data was processed to actually generate the demonstration video data.


RECOMMENDATION STATEMENT:
While the paper considers and important problem and may indeed describe a very promising new approach, the current experiments section is in need of a major rework before the story is clear.


QUESTIONS FOR AUTHORS:
	(Q1) How, exactly, were the demonstrations from mocap data produced?


MINOR COMMENTS:
	(MC1) The authors need to state in the abstract that their experiments are done on _simulated_ agents.
	(MC2) The authors state in the introduction that "the fundamental problem of imitation learning is how to align a demonstration in space and time with the agent's own space." Respectfully, don't think this is true. It is perhaps "a" fundamental problem, but certainly not "the" fundamental problem. Even with aligned state spaces, the general problem imitation learning poses several other challenges such as how to learn robust policies, how to incorporate multiple demonstrations if they are available, etc. The authors should revise the wording of this statement.
	(MC3) The caption of Figure 4 does not address (d)-(g).
	(MC4) The legend in Figure 4a says "GAILfO" but the text says the comparison algorithm is "GAIfO."
	(MC5) The text in Figures 4 and 5 is so small that it is unreadable. The figures need to be reformatted to address this.
	(MC6) The authors should reference and discuss how their work relates to / differs from a very similar recent paper [https://arxiv.org/pdf/1909.13392.pdf]
review_title: Interesting method, but experiments section is lacking

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### Paper summary

This paper proposes using a recurrent siamese network to learn distance functions between observed behaviours and agent behaviours, and the use of this distance for reinforcement learning. Experiments on a set of simulated walking tasks show that this approach works reasonably well.

Unfortunately, I am recommending rejection, as I do not think the contributions of this work are significant enough to warrant publication at ICLR. Siamese recurrent networks are already well established techniques for distance learning ([Chopra et al. 05](http://yann.lecun.com/exdb/publis/pdf/chopra-05.pdf), [Pei et al. 16](https://arxiv.org/pdf/1603.04713.pdf)), so it seems the primary contribution here is a method of artificially adding noise to help distinguish expert and demonstrated behaviours in training, an idea already explored to a large extent by TCNs and D-REX. 

### Pros
- The proposed approach performs well when compared against GAILfO and learns better distance functions than a TCN.
- The results look good, and I really like the zombie walk.

### Cons

- The ideas in this work are very close to T-Rex [Brown et al. 19a](https://arxiv.org/abs/1904.06387) and D-Rex [Brown et al. 19b](https://arxiv.org/pdf/1907.03976.pdf) which uses a pairwise trajectory ranking formulation, which essentially learns the distance between trajectories, much like a siamese network. Brown et al. 19b also explore artificial trajectory ranking by adding noise to trajectories. 
- The proposed approach relies on a number of heuristics and connected neural components and seems to perform well in walking tasks, but makes no real theoretical contribution.

### General comments and queries

- The paper seems to have been quite rushed and in need of some refinement when it comes to presentation. Specifically, please use a spell checker - there are numerous typos where letters are missing from words alongside spelling errors (eg. Advasarial,  quadreped, etc...), and check figure sizes to make sure legends are legible. The accompanying website is also full of broken links and unfilled template information.
- Some sweeping statements need toning down. Eg. in the introduction it is claimed that *The fundamental problem of imitation learning is how to align a demonstration in space and time with the agent’s own state.* which is arguable, I've always thought the fundamental problem in imitation learning is determining which part of a demonstration is essential and which is ancillary.
- I'd recommend a discussion and comparison with [D-REX Brown et al. 19b](https://arxiv.org/pdf/1907.03976.pdf)
- Equation 4 - how were the weights for these heuristic losses chosen?




review_title: Limited contributions, needs some polish

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Summary**: This paper studies the problem of visual imitation learning: given a video of an expert demonstration, take actions to reproduce that same behavior. The proposed method learns a distance metric on videos and uses that distance metric as a reward function for RL. Experiments show that this method does recover reasonable behaviors across a range of simulated robotic tasks.  Compared with prior methods, the main contribution of this work is that the distance metric is parametrized and trained as a siamese network.

**Novelty and Significance**: While the exact method seems novel, it is very similar to a number of prior methods, most notably GAIL. At a high-level the main difference is that this paper uses a siamese network to parametrize the discriminator, and employs a few types of data augmentation. If the experiments had shown that this architectural choice made a significant improvement in performance, and was useful inside a range of imitation learning frameworks (e.g., GAIL, AIRL, and Value Duce), then I think it'd represent a significant contribution. As is, the paper has not convincingly shown that this architectural choice is critical for significantly improved performance.


**Experiments:**
* I found it challenging to assess the experimental results without any quantitative comparisons with baselines (besides TCN). I'd highly recommend comparing against recent imitation learning methods. For example, some reasonable baselines would be
  * GAIL or InfoGAIL
  * Zero-shot visual imitation
  * Imitating latent policies from observation
  * Generative adversarial imitation from observation (Though this method is discussed, it's not compared against in the figures.)
  * Learning robust rewards with adversarial inverse reinforcement learning
  * Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning
  * Imitation learning via off-policy distribution matching
* The videos of the humanoid imitation behavior on the accompanying website actually look quite poor. The videos for other domains seemed to have been rendered incorrectly.
* I found it somewhat irksome that the method is entitled "visual imitation" but the learned policy doesn't take visual inputs ("Note that we experimented with using visual features as the state input for the policy as well; however, this resulted in poor policy quality.")



**Clarity:**
* The introduction makes imitation learning sound like a new problem. I'd recommend clarifying the relationship with prior work earlier in the introduction.
* I found Fig 1 hard to follow, largely because it's unclear what the method is supposed to produce. One idea is to explicitly say something like: "We aim to learn a distance function (left) and then use that distance function as a reward function for RL (right). The distance function is learned by ..."
* I found Eq 4 hard to parse. It might be helpful to label each term (using \underbrace{}) with its semantic meaning.
* Reward Calculation: One thing that's unclear here is which trajectories are used for computing the reward function. I'm guessing that the agent's trajectory is compared to an expert trajectory, but I don't think this is ever stated explicitly. If so, it's unclear how to compute the reward function when multiple expert trajectories are given.


Overall, I give this paper a score of 4/10, primarily because of the lack of discussion and comparison with prior work. I would consider increasing my score if the paper compared against recent visual imitation learning methods (see above). The proposed method also seems quite similar to GAIL; if possible, it'd be great to formally explore the connections between these two methods. Finally, I'd recommend focusing the related work section to only discuss the most related works (see list above), but to discuss the exact differences between these prior methods and the proposed method in more depth.

**Questions for discussion**:
* Precisely, what are the differences between this method and GAIL? (I want to make sure I didn't miss an important difference)
* For the experiments in "3D Robot Video Imitation", is the expert demonstration provided as an RGB video, a mocap trace, or something else?
* Where is the comparison with GAIfO shown? ("We have compared our method to GAIfO")
* "that takes into account demonstration ordering" -- In a Markovian environment with a Markovian expert, shouldn't comparisons based on state-action pairs be sufficient? Is the assumption that the expert's motion is not Markovian?
* "Mutual information loss (”Siamese Network triplet loss”)" -- Eq 3 doesn't look like a mutual information objective. Can you explain precisely how maximizing Eq 3 leads to maximizing mutual information?


**Minor comments:**
* Abstract: No need to define "GAIfO" as an acronym in the abstract. Instead, just use "GAIL without access to actions."
* "In nature ... their movements" -- Please add citations.
* "formulating ... is challenging" -- How is this different from the large body of prior work on imitation/apprenticeship learning?
* "The fundamental problem ..." -- This is only true for *trajectory*-based imitation learning, not *transition*-based imitation learning (e.g., BC, GAIL, AIRL).
* "re-use" -> "reuse"
* "Additionally, While" -> "Additionally, while"
* "Including, ..." This sentence is missing a subject.
* "Markov Decision Processes" -> "Markov decision processes"
* When defining a trajectory, use \langle and \rangle.
* "it often suffers from distribution mismatch" -- Cite DAGGER, or something similar.
* "behaviour Ng" -- Use \citep{} for references that are not used as nouns.
* Eq 2: Can you define the distribution over $h$? (I assume it will be p(h | x)).
* Eq 3: Perhaps cite prior methods that use this contrastive margin loss (e.g., FaceNet)
* Define VIRL = "Visual Imitation with RL" in S3. It took me awhile to figure out what VIRL was when it was first mentioned in S4.
* "is an active research area" -- I agree, but using citations from '04 and '09 doesn't make this area sound particularly "recent."
* "the goal is to" -- The goal of what? Of all "good distance functions"?
* "state-based metrics ... image based inputs" -- Aren't image-based metrics a special case of state-based metrics? For example, the [Ho & Ermon] citation for state-based metrics can be applied to images.
* "Additional works..., none of these ..." -> "While additional works ..., none of these ..."

----------------------------
**Update after author response**: Thanks to the authors for answering my questions and for incorporating feedback into the paper. Through discussion, I think we got to the crux of the method: distance functions seem to work better than classifiers for imitation learning. I think this is a really neat observation, and potentially quite important; the experiments definitely support this hypothesis. That said, I don't think the paper goes far enough in exploring this hypothesis, either experimentally or theoretically. There are a number of confounding variables, such as data, loss function, and architecture, which each will need to be accounted for. I therefore stand by my previous vote (4) to reject the paper. With more thorough experiments and ablations, I think this will make a fantastic submission to a future conference.
review_title: Review of "Visual Imitation with Reinforcement Learning using Recurrent Siamese Networks"

Meta-review: The paper considers the problem of learning to imitate behaviors from visual demonstrations, without access to expert actions. Consistent with recent approaches, the proposed method uses a neural network to measure the similarity between visual demonstrations and the agent's behavior, and employs this metric as a reward in RL. The primary contribution is the use of a recurrent siamese network that is trained to measure the distance between motions, as a means of better dealing with the challenges of imitation learning from a small number of (as few as one) noisy visual demonstrations. Experiments on a variety of simulated domains show that the proposed approach achieves reasonable results.

The paper was reviewed by four knowledgeable referees, who read the author responses and engaged in extensive discussion. The reviewers agree that learning to imitate behaviors from a small amount of noisy demonstrations is a challenging and important problem that is of significant interest. The proposed method nicely extends existing approaches to visual imitation learning, and the results reveal that the method performs well in a variety of continuous control domains. The reviewers raise several concerns regarding the clarity of the technical presentation and the sufficiency of the experimental evaluation. The authors have made a significant effort to address these concerns in their responses and updates to the paper, which the reviewers very much appreciate. However, some of the reviewers' primary concerns regarding clarity and the thoroughness of the experimental evaluation remain. This work has the potential to make a really nice contribution and the authors are encouraged to take this feedback into account for any future version of the manuscript.
Decision: Reject