Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In this paper, the author investigates how to utilize large-scale human video to train dexterous robot manipulation skills. To leverage the information from the Internet videos, the author proposes a handful of techniques to pre-process the video data to extract the action information. Then the network is trained on the extracted hand data and deployed to the real robot with some human demonstration collected by teleoperation for fine-tuning. Experiments show that the proposed pipeline can solve multiple manipulation tasks.

strengths_and_weaknesses: **Strength**

- The direction explored in this paper is important. Utilizing the internet video data for robot learning is well motivated. Especially considering the similarity between human and multi-finger hands, this direction looks very promising. 

- The authors perform experiments with multiple real-world tasks with pick and place, pushing, and rotating objects.

**Weakness**

- Although the objective of this paper is very impressive, the experiments can not support the introduction and there are multiple overclaims.

- Section 4 is titled VideoDex: Learning Dexterity from Youtube. However, I can not find any evidence that the author utilizes YouTube data for learning dexterous manipulation. As mentioned in the Section on Retargeting Wrist Pose, ORB SLAM and the camera’s acceleration data are used to compute the camera pose trajectory. This information is not readily available in the YouTube data. The experiments and methods are misaligned with this claim.

- In the introduction line 42, the author mentioned that our key insight is to combine these visual and action priors from passive data with the physical constraints of how robots should move in the world. However, the method does not consider the surroundings of the human hand, and the detection results itself is not accurate. How to incorporate physical information into the training data?

- Missing literature discussion on previous learning from video works:

    [1] *DexMV: Imitation Learning for Dexterous Manipulation from Human Videos, 2021*: This paper focuses also on how to learn dexterous manipulation from human videos. The reviewer understands that this literature paper uses simulated tasks while the authors focus on the real robot settings. But it seems that similar pipelines are also used in this paper: estimating the human hand, retargeting, and learning from retargeted hand pose.

    [2] *The Surprising Effectiveness of Representation Learning for Visual Imitation, 2021*: This paper also focuses on how to leverage the video data for better learning. It also uses a GoPro camera to collect a video of each trajectory, which is the same as the Ego4D dataset used in this paper. It shows that by learning from this video data, the final manipulation performance can be improved a lot.

    These literature works use very similar methods to achieve robot learning. The novelty claims of this paper can also be found in this literature.

- Missing details for Re-targeting Wrist Pose
The detection module FrankMocap is a 2D hand detector, it is not clear how the author can get 3D keypoints from the hand model in the camera frame. Also, this section is important in the whole technical approach, it is better to provide visualization of the final retargeted robot. A hand wrist pose and robot arm should also be visualized in Figure 3 if they are used in the training. If the wrist pose and arm joint pose is not used, how to pretrain the action prior?

- Missing details about transforms
In the equation, it is not clear why the author uses T and M to denote pose simultaneously. What are the differences? If M is also a $SE(3) $transformation, how to compute the position part of the $M_{World}^{C_1}$? Besides, the reviewer can not find any information about how the $T_{Robot}^{World}$ is determined heuristically in both the main paper and supplementary.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper studies an important problem of learning manipulation skills from YouTube videos. The objective is of vital importance to the field. However, the methods described in the paper are not able to support the claim. The manipulation tasks are diverse but lacking in technical difficulty, which can also be achieved by parallel gripper. More investigations are necessary.

## Updates (after rebuttal)

The updated version removes the over-cliam of learning from YouTube and utilize visual RGB-D slam techniques to learn from image only. The reviewer believes that this is an important direction in imitation learning on dexterous manipulation. Although the experiments of this work is not so solid, i.e. missing intermediate evaluation results of the pretraining data generated from videos, the overall technical pipelines sounds reasonable to me. The reviewer raise the score to weak accept under the promise of the author that:

Shows intermediate visualization/results of the generated data for pretraining. The current experiments only show the final learning performance of manipulation tasks. It is unknown that what is the quality of these generated data from videos and how the quality of these data will influence the final performance in a more strict way. It will be preferable if the author can show some visualization of the generated VideoDex data in a more straight-forward way. For example, visualize the robot hand-arm trajectory and compared with the original internet video.
issues: - Clarification for the weakness part listed in the previous section.
- More explanation about the transformation used in Eq5: 
  - How to heuristically compute $T_{Robot}^{World}$
  - How to compute the translation term for $M_{C_1}^{World}$
  - How to get the 3D keypoints used to compute $M_{C_t}^{Wrist}$ by PnP
- Link in supp is broken with 404 error: https://videodex.github.io/download
- Clarification on the benefits of using dexterous hand for the task defined in the paper compared to normal parallel gripper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors demonstrate a system in which they combine a few different components to get interesting supervised-learned open loop behavior of real robot hands doing several different tasks. In particular the most notable part of the approach is using videos of human hands as an “action prior” which informs their supervised mapping.
strengths_and_weaknesses: # Strengths

- Good core idea. The overall idea of using action priors from human videos, via hand tracking, to make robots work better, is a good idea. There are a lot of closely related works, but I think they are well referenced in this paper. 
- Good execution on several key parts. The execution details of handling moving cameras with camera pose tracking, together with per-frame hand tracking, seems to be well done. I also like just using R3M features out of the box, this is smart and interesting to see external validation. 
- Results of real robots with hands doing a variety of things. 

# Weaknesses

There are various unscientific elements of this paper in its current form.

While the work is interesting, I can’t recommend a strong accept for a paper in this form. Hopefully the list below will help the authors improve both this work and their future work.

If the authors can address all of the following weaknesses in their rebuttal, which I think is all doable and within scope to do in a rebuttal, I’d be happy to move from weak accept to strong accept. 

1. It seems like the authors are not very upfront about the fact that this method does not produce closed loop policies. Only on the last page or two is it mentioned that the whole method is open loop. This is fine to study the task of (i) inputting an image of a scene and (ii) outputting an open loop trajectory, but, it of course is very limiting. The tasks are carefully chosen such that they don’t require any closed loop feedback. This aspect of their approach is not what most researchers in the field would expect… so a common experience of a researcher would be to look over the first handful of pages of this paper, and only at the last page or so realize that this is an open loop method. Please just make this clear up front.
2. Several false statements in the introduction:
  - “ To build such robotic agents that can operate anywhere, we need access to a lot of successful robot interaction data in many environments.” —> not necessarily true… This is a reasonable hypothesis, but one that isn’t tested in this paper, and it can’t be stated as a fact.
  - “ However, deploying inexperienced real world robots to collect experience must require constant supervision which is in feasible.” —> also not necessarily true… but also a very reasonable hypothesis. Just need to say “may require” instead. 
  - “Most of the inefficiency in robot learning is due to the exponentially large action space.” —> an opinion, and can’t be stated as fact.
3. “NDPs can produce safe and smooth trajectories” … yes, but this is a meaningless statement. They *can* also produce trajectories that are completely unsafe. There is nothing about NDPs/DMPs that provides safety other than a bit of smoothness that may arguably help. But there is nothing that helps here with the presence of obstacles in the environment, or humans, etc. This statement probably only serves to confuse/mislead inexperienced readers, please remove/fix.
4. The paper mentions a “physical” prior as a key component, but this is just that it uses Dynamic Movement Primitives it seems. I’m not sure this is the best way to communicate this. Line 191 also says physically-aware NDPs… they don’t know anything about contact physics… maybe just say second order system or dynamical system or something, maybe physically-inspired, but not physically-aware. And whenever it says, for example line 269, “baselines without a physical prior” it should just be instead clear that this just means they don’t use DMPs.
5. Line 213 “ is VideoDex able to perform general purpose manipulation?” Since the method is open loop, the answer is no. That’s fine, and the results are still impressive, but should be clarified… this is not something that needs to be empirically evaluated, it’s just a result of the formulation.
6. It’s very confusing that citation 44 is used open loop… this isn’t an intention of the method. Also, is the RNN version closed loop over time? It’s not clear. And if it’s not? … I’m not sure how the RNN would be any different if it’s not used sequentially over time. 
7. Please state exactly how many demonstrations were used for the different experiments. 
8. In the conclusion… “ this is because training RL in the real world is difficult due to hardware limitations.” Yes, but this isn’t reason to make the used behavior cloning method open loop instead of closed loop.


## Minor

Don’t worry about these too much but I mention these as opportunities to improve the paper further. 

- Ego4D is not cited on page 2 (mentioned but not cited)
- HR() is not defined in an equation. Also, I would recommend not using two letters for a math symbol… it looks like a matrix H multiplied by a matrix R
- Why use ORBSLAM3 rather than COLMAP for the poses? Already running colmap for the calibration. 



robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Please see the start of the Weaknesses section where I explain the recommendation. 
issues: Please see the list in Weaknesses.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: VideoDex pretrains a policy network with videos, with gyroscope and accelerometer data, of humans performing a task, then fine-tunes with demonstrating trajectories collected by teleoperating the robot.  In order to train with the human data, they use the approach from [49] for mapping human pose to robot pose and use ORBSLAM3[55] to account for the camera motion.  They feed the image data, labeled with the outputted pose, into a ResNet18[15] backbone initialized with R3M's[6] features and use a Neural Dynamic Policy (NDP) [13] network to generate actions.  The paper demonstrates that using human data allows improved performance on 6/7 tasks.
strengths_and_weaknesses: Pros
The paper presents a theoretically simple method of learning from videos of humans.

The method is demonstrated on 7 different tasks, outperforming the baselines without human data on 6 of them.


Cons
The writing of the paper is somewhat scattered.

The analysis of why the proposed approach using NDP rather than a MLP works better with human data could be stronger. 

The paper needs to be much clearer that it relies on gyroscope and accelerometer data from the human videos, which is a barrier to truly using internet-scale data.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: I think this paper has promise, but could use some additional work to solidify its claims and clean up its presentation. 
issues: Line 47, Line 301 - The claim that the approach "learns from passive videos on the internet" is a little too strong.  Epic-Kitchens is a large and visually challenging source of data, but it is still curated and collected with a consistent methodology, making there be a significant gap between using it and using videos from the wilds of the internet.  Additionally, the proposed method makes use of gyroscope and accelerometer data for the human videos, which is not universally available for internet videos.

Line 84 - The related work section on learning manipulation from watching videos of humans only mentions a single recent paper.  There has been several years of work in this area by different groups that approach the problem in a variety of different ways. Here are some examples of other work that should be included and discussed.
Learning Robot Activities from First-Person Human Videos Using Convolutional Future Regression - https://arxiv.org/abs/1703.01040v2
Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller - https://arxiv.org/abs/1911.09676
Learning Predictive Models From Observation and Interaction - https://arxiv.org/abs/1912.12773
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos - https://arxiv.org/abs/1912.04443
Visual Imitation Made Easy - https://arxiv.org/abs/2008.04899
Model-Based Inverse Reinforcement Learning from Visual Demonstrations - https://arxiv.org/abs/2010.09034
Reinforcement Learning with Videos: Combining Offline Observations with Interaction - https://arxiv.org/abs/2011.06507
Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos - https://arxiv.org/abs/2101.07241
Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos - https://arxiv.org/abs/2103.16817

Line 145 - Figure 4?

Line 179 - What are the differences from [49] that makes the retargetting network "similar" rather than just using the method from [49]?

Line 181 - Figure 3?

Line 218 - Should this be 120 demonstrations per task?  The appendix says 120-175 demonstrations per task and I'm not sure what the "hour" is measuring.

Table 1 - How many trials were the different policies evaluated on?  It would also be good to include some measure of variation in the results. That would help to show that the differences in performance are actually statistically meaningful.

Table 1 - What are the success criteria for the tasks?  For instance, the example for "Push" in the video looks more like the robot bumped into the object rather than executing a sustained pushing motion, but the table reports a 100% success rate.

Line 233 - Why is CQL not run on the rest of the tasks?  

Line 289 - This experiment modifies both network architecture and network parameters.  A clearer experiment would be to compare ResNet18 with R3M features to ResNet18 with ImageNet features.  Also, the comparison between the performance with VGG and with MVP isn't fair, since VGG is an architecture that is 8 years out of date and should perform worse on pretty much any task, no matter the initialization.

Appendix - Which augmentations were used and which augmentations were not used?

One important experiment that I would like to see is the performance of VideoDex pretrained on human data from task A, then finetuned and tested on Task B.  If there was a significant gap between that and the performance when pretrained and tested on the same data, that would demonstrate that the human data is actually providing the model with useful, task-specific action priors rather than just putting it in a generally good initialization.


One of the central claims of the paper is that the use of NDP helps provide a physically grounded prior that enables the network to better deal with the noise in the trajectories/actions generated by [49]. It would interesting to compare the performance of VideoDex and VideoDex-Open pretrained with teleoperated demonstrations with different levels of noise artificially added.  If the performance gap increased as the level of noise increased, that would be convincing evidence that NDPs have some unique benefits in this setting.
review_title: null

Meta-review: metareview: This paper studies how to learn dexterous manipulation from human videos.  

In the initial review, the reviewer appreciated the direction and real-world experiment but also raised  concerns about the need of special sensor for tracking. During rebuttal, the authors effectively addressed this concern by providing additional experiment results, and reviewers were satisfied with the response. 
AC would like to recommend acceptance for this paper. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)