Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper talks about solving an interesting problem of meta-imitaion learning from watching human video demonstration. The paper proposes a A-CycleGAN approach to learn a latent representation for human-robot video correspondence and a self-adaptive meta imitation learning under imagined latent space to evaluate the quality of translated data. The proposed methods in the paper achieves comparable performance to the state-of-the-art method.
main_review: Strengths:
- Paper is well-written.
- The paper only uses human video demonstration and no robot demo/states and actions to learn a task seems unique to me.
- Paper situates the problem very well and presents good series of prior work to solve the problem addressed in the paper.
- Good range of experiments

Weakness:
- In the introduction section, the authors mention about challenging tasks, could you please explain in one line what is the difficulty of the tasks that you are attempting the robot to learn?
- It will be good add a description of "imagined videos" that has been reference in the paper multiple times. Maybe add in the Method section. 
- I encourage the authors to submit a video of experiment results for both simulated tasks and real-world pushing task.
- A question asked in the Introduction section para 2, is not clear to me. Could you please re-write the sentence with more clarity?
summary_of_the_review: Based on paper methodology contributions and above merits (explained in Main Review section).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers the problem of robot imitation learning from video demonstration, without the need of accessing to an explicit representation of the action of the demonstrator. 
The authors introduce an approach for meta-imitation learning that aims to traslate human videos to robot demonstrations from a single video demonstration. The correspondence between human and robot is learned end-to-end exploiting a generative architectures, A-CycleGAN, extension of CycleGAN. The method is experimentally assessed on different visual tasks.
main_review: + The task is very relevant in the robotics community
+ The paper is well structured
+ The experimental analysis is convincing

- The reviewer finds some part a bit unclear. Here is a tentative summary of the main concerns:
   * Not fully clear to the reviewer how supervision works in A-CycleGAN (on the robot side)
   * Not fully clear to the reviewer how information on action and on environment is disentangled and exploited (Sec. 4.1)
   * The reviewer fails in gathering an intuition about the interplay between the different modules in the architecture. In particular, the Inverse 
      Dynamic Model lacks details, and the Meta-Imitation Learning, one of the core parts of the work, has a description which is a but 
      compressed. 
summary_of_the_review: I find the topic very interesting and the video-based approach is certainly among the most relevant in robotics. However, I find the clarity of the presentation could be improved to fully appreciate the essence of the approach
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the problem of meta imitation learning of robot control policies from only video demonstrations of humans performing the tasks. The goal is to make data collection simpler by not requiring robot demonstrations which can be expensive to acquire. The method proposed by the authors combines multiple components. Specifically, a CycleGAN which translates human demo videos into robot demo videos, an Inverse Dynamics Model which infers robot action from consecutive video frames, a method to weigh the training loss according to the quality of translated images, and finally the same learning algorithm from DAML is used. 
main_review: Strengths

- This paper tackles a very challenging problem and accordingly presents an elaborately designed approach that brings together components from several state-of-the-art methods from slightly different areas. As a result, the overall approach is pretty novel.

Weaknesses

- The motivation of this paper is to reduce data collection complexity by only requiring human demonstrations, since robot demonstrations may require kinematic teaching / teleoperation which can be difficult to collect. However, while the proposed method doesn’t require robot demonstrations, it does require pairwise human and robot videos for training the CycleGAN component. My assumption is that this is also difficult to collect in practice, so I think the value of the proposed method is significantly diminished. 

- In terms of the experiments, three tasks are presented, with two in simulation, where demonstrations are from a Fetch robot and the learner is a Sawyer robot, and one task in the real world that learns from human videos. While the simulated robot-to-robot experiments demonstrate some aspects of the algorithm, they are less powerful results. With only one task in the target setup, I feel the overall experimental results are not sufficiently rigorous. 

- In addition, the final success rate achieved by the algorithm for the real-world pushing task is 56.3%, which is a lot lower than what DAML achieves as shown in the DAML paper (88.9%) on a highly similar task. While in the present paper, the proposed algorithm matches DAML’s performance (56.3 vs 58.8%), both success rates are low. I wonder if there are any structural limitations of the proposed approach that prevents it from achieving a higher rate. Consider running experiments with more training data; would the proposed algorithm still be able to match DAML’s performance?

- The writing of the paper needs significant polishing.
summary_of_the_review: This paper studies a very challenging problem and presents a novel method, but the experiments are not convincing enough.

----

UPDATED the overall recommendation after authors' response. see details in thread.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the meta-imitation learning from human videos problem, which aims to learn a new task by watching a few demonstrations of a human performing the task. While DAML [1], one of the relevant prior approaches to this problem, requires paired demonstrations in the human domain and robot domain for training, this approach only requires human videos and a small amount of paired data. The approach proposed in this paper translates human videos into the robot domain with a CycleGAN. Simultaneously, an inverse dynamics model is trained in the robot’s observation domain with randomly collected robot data to recover the actions from the translated demonstrations. Because the quality of the translations can be quite varied, higher-quality translations, measured based on the SSIM and Brenner gradient, are prioritized over lower-quality ones in the meta-imitation learning procedure, implemented by the DAML algorithm.
main_review: Strengths:
- This method extends the previously proposed DAML algorithm, requiring only human video demonstrations (and also a small amount of paired data). This is appealing especially when it’s difficult to collect large amounts of demonstrations in the robot domain, e.g., through teleoperation. The resulting method performs comparably to the DAML algorithm, which requires more robot demonstration data.

- The experiments study the relevant ablations of the method, including using a decoupled CycleGAN and inverse dynamics model, using the translated images instead of the latent representation, and using a non-adaptive loss in DAML. The results show that all of the choices in their method make it stronger.

Weaknesses:
- The experiments lack comparisons to other methods besides DAML, which operates on a different set of assumptions. Since the success rates for all methods are quite low, it would be good to understand where the upper bound is. The method proposed in [1], like DAML, requires robot demonstrations as well, but may perform better than DAML. Another potential comparison is AVID [2]. Specifically, we can use all of the random robot data to train a visual dynamics model, and the rest of the data to train CycleGAN. One difference is that AVID solves multi-stage tasks and assumes the beginning of each stage is specified by the user; their reward function for planning is then defined by learned success classifiers. Instead, we could derive a simple dense reward function from the translated video, e.g., to match the translated video frame-by-frame or just to match the final frame as measured by the MSE. 

- The SSIM and Brenner gradients are heuristic measures of the quality of the translated videos, and do not directly measure how accurate they are with respect to the true corresponding video in the robot domain. Since the meta-IL component operates on the latent representation and not the generated images themselves, it is an even more indirect measure. However, the non-adaptive loss does lead to worse results, so it empirically seems to work better. I’m curious how well these two heuristics actually correlate with the translation error. This could be evaluated on the small dataset of paired videos -- compute the MSE of the translated video and true robot video pair, and then measure the correlation with \beta_qua.

- It would also be nice to include some qualitative examples of the translated videos.

Other questions:
- The method still requires a small amount of paired human and robot data, which could consist of random actions (in their experiments, this is <= 100 trajectories). How does varying this number affect the performance of the method?

- The success rates for all methods are quite low. It’s also surprising that DAML, which leverages paired demonstrations, performs similarly to/worse than A-CycleGan + DAML, which doesn’t. How does the meta-policy learned by the different methods perform on the training shapes? 

- Compared to DAML, the main mode of failure is due to control, i.e., not executing the right sequence of actions to complete the task, which may be explained by the prediction error in the inverse dynamics model. The chosen action space in the experiments is 3D - I’m curious how well it scales to higher dimensional action spaces, e.g., if the tasks required joint space control. 


[1] Dasari et al. Transformers for One-Shot Visual Imitation. CoRL 2020.

[2] Smith et al. AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos. RSS 2020.

summary_of_the_review: The paper tackles a challenging and compelling problem: meta-imitation learning of human demos with a small amount of paired robot demos. However, the experiments lack relevant comparisons to contextualize the significance of the proposed method. The heuristics used in the adaptive loss could also be better analyzed, e.g., by looking at their correlation with the MSE between the translated videos and true robot video.
review_title: null

Meta-review: This is an interesting paper working on the difficult problem of learning from video demonstration. The authors provided convincing experimental solutions for visual representation, domain adaptation, and imitation. It would be a nice ICLR paper.
Decision: Accept (Poster)