VideoDex: Learning Dexterity from Internet Videos
Authors: Kenneth Shaw
Shikhar Bahl
Deepak Pathak
Venue: CoRL 2022 Poster (2022)
Keywords: Dexterous Manipulation
Large Scale Robotics
Imitation Learning

Abstract: To build general robotic agents that can operate in many environments, it is often imperative for the robot to collect experience in the real world.  However, this is often not feasible due to safety, time and hardware restrictions.  We thus propose leveraging the next best thing as real world experience: internet videos of humans using their hands.  Visual priors, such as visual features, are often learned from videos, but we believe that more information from videos can be utilized as a stronger prior.  We build a learning algorithm, Videodex, that leverages visual, action and physical priors from human video datasets to guide robot behavior.  These action and physical priors in the neural network dictate the typical human behavior for a particular robot task.   We test our approach on a robot arm and dexterous hand based system and show strong results on many different manipulation tasks, outperforming various state-of-the-art methods. For videos and supplemental material visit our website at https://video-dex.github.io.

VideoDex: Learning Dexterity from Internet Videos
Kenneth Shaw Carnegie Mellon University
Shikhar Bahl
Deepak Pathak
Abstract: To build general robotic agents that can operate in many environments, it often imperative for the robot to collect experience in the real world. However, this is often not feasible due to safety, time, and hardware restrictions. We thus propose leveraging the next best thing as real-world experience: internet videos of humans using their hands. Visual priors, such as visual features, are often learned from videos, but we believe that more information from videos can be utilized as a stronger prior. We build learning algorithm, VideoDex, that leverages visual, action, and physical priors from human video datasets to guide robot behavior. These actions and physical priors in the neural network dictate the typical human behavior for a particular robot task. We test our approach on a robot arm and dexterous hand-based system and show strong results on various manipulation tasks, outperforming various state-of-the-art methods. For videos and supplemental material visit our website at https /tt/://vvdeoeeee deriddeo-ee..... dex.gittubl.. github.thh..... io
Keywords: Dexterous Manipulation, Large Scale Robotics, Imitation Learning
1 Introduction
The long-standing dream of many roboticists is to see robots autonomously perform diverse tasks in diverse environments, To build robot that can operate anywhere, many methods rely on successful robotic interaction data train on. However, deploying inexperienced. real-world robots collect experience may require constant supervision which is infeasible. This poses a chicken-and-ege problem for robot learning because to collect experience safely, the robot already needs to be experienced. How do we get around this deadlock?
Fortunately, there is plenty of real-world human interaction videos on the internet. This data can potentially help bootstrap robot learning by side-stepping the data collection-trainnng loop. This insight of leveraging human videos to aid robotics is not new and has seen immense attention from the community at large [1, 2, 3]. However, most of the prior work tends to use human data as mechanism for pretraining just the visual representation [4, 6, 7, 8], much like how deep learning has been used as pretraining tool in related areas of computer vision [9, 10] and natural language processing [11, 12]. Although pretraining visual representations can aid in efficiency, we believe that a large part of the inefficiency stems from very large action spaces. For continuous control, learning this exponential in the number of actions and timesteps, and even more difficult for high degree-of ofree-foff-rrommmmmm robots (shown in Figure 1). Dexterous hands are one such class of high degree of freedom robots that have the possibility to provide great contact for the grasping and manipulation of different objects. Their similarity to human hands makes learning from human video advantageous.
In this work, we study how to go beyond using internet human videos merely as a source of visual pretraining (i.c. visual priors), and leverage the information of how humans move their limbs to guide train robots on how they should move (i.e. action priors). However, guiding robot motions using human videos requires understanding the scene in 3D, figuring out human intent, and transferring from human to robot embodiment. First, 3D human estimation works decently well in general human videos which we can leverage to gather 3D understanding. Second, there have been large-scale datasets that break down the human intent via crowdsourcing labels [2, 1]. Finally, to handle the embodiment transfer, we use human hand to robot hand retargeting as an energy function to pretrain the robot action policy. Our key insight is to combine these visual and action priors from human videos with prior on how robot should move in the world [13, 14] (i.e., physical prior, using second order dynamical system) to obtain dexterous robot policies that can act in the real world. We
"Equal contribution, order decided by coin flip.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Human Videos
VexeoDex
visual prior
physical prior
b
ST
action prior
D--ommiin Demonstrations
Figure I: We re-target human videos as an action prior, use pretrainined embeddings as a visual prior, and use Neural Dynamical Policies (NDPs) [13] as a physical prior to complete many different tasks on a robotic hand.
call this approach, VideoDex. To enhance real-world performance, we mix the experience obtained from massive internet data with few in-domain demonstrations.
In summary, VexeoDex is robot learning algorithm that incorporates visual, action, and physical priors into single open-loop policy by learning from passive videos contained in human activity datasets from the internet. VideoDex then only needs to adapt to real world tasks using a few in- domain examples. We find that VideoDex outperforms many state-of-the-art rohot learning methods on seven different real-world manipulation tasks on a high DOF multi-fingered robotic arm-hand system well
1-DOf gripper robotic system.
2 Related Work
Learning for Dexterity Reinforcement learning (RL) with an engineered reward function can show dexterous simulation results [15, 16] but requires lots of data, especially in high DOF dexterous manipulation. This requires simulators [17, 18], which cannot model physics properly. making real- world transfer difficult. Behavior cloning is approach [19, 20] that work safely. DIME [21] involves using nearest neighbor matching of image representations with demonstrations to determine actions. Qin et al. [22] teleoperates and learns policies in simulation, followed by Sim2Real transfer. DexMV[23] uses collected human hand videos for robot hand imitation learning. DIP [24] learns hand-object affordances and priors for RL initialization using curated video datasets.
Learning from Videos and Large-Scale Datasets There are many curated datasets from internet human videos. for example, FreiHand [25] for hand poses, 100 Days of Hands [26] for hand-object interactions, Saanentinn-Somethiien
[3] for semantically similar interactions, Human3.66 3.6 [27] and the CMU Mocap Database [28] for Human pose estimation. Epic Kitchens [2), ActivityNet datasets [29], or YouCook [30] are action-driven datasets we focus on for dexterous manipulation.
Learning Action from Videos Detecting humans, estimating poses of different body parts, or understanding the dynamics and interactions related to human motion is commonly studied problem. One can model human hands using the MANO [31] model and the human body using SMPL, SMPL- X [32, 33] models. There are many efforts in human pose estimation such [34, 35, 36). We focus on Frank [36] for our project as it is robust for online videos. Traditionally, teleoperation approaches have employed hand markers with gloves for motion capture [37] or VR settings [38]. Without gloves, Li et. al. [39] used depth images and a paired human-robot dataset for teleoperation. and Handa et. al. [40] designed system that mimics the functional intent of the human operator to perform object manipulation tasks.
Robot Learning by Watching Humans Recent works have leveraged human datasets learn cost functions [41, 42, 43], learn action correspondences [44] both in a paired [45] and unpaired manner [46]. This data can also be used to extract explicit actions by leveraging structure in the collection (such as saacher-grabbe tools [47]) or prediction of future hand and object locations [48], as well as keypoint detectors [49]. This can also be used to build representations for robot learning [6, 50]. R3M [6] trains on the Ego4D [1] dataset using a temporal alignment loss between language labels and video frames. We build on top of previous efforts this area, where we combine visual representations trained on human activity data, with action driven representations.

---

CHEEZIT
BLD
Figure 2: The collection of train objects (left) and test objects (right) used for experimentation.
Background
3.1 Neural Dynamic Policies
Neural Dynamic Policies (NDPs) [13, 14, 51], produce smooth and safe open-loop trajectories. When using them as network backbone, they be rolled out trajectories of arbitrary lengths which enables the use of varying-length human videos. NDPs can be described with the Dynamic Movement Primitive equation [52, 53, 54, 55]:
(1) where is the coordinate frame of the robot, g is the desired goal in the given coordinate frame, fw is a radial basis forcing function, a time variable, and ax 3 are global constants. NDPs use the robot state, scene, and a NN to output the goal y and shape parameters of the forcing function sw-
= a
9) (x.
3.2 Learning from Watching Humans
Recently, Sivakumar et al. [56] introduced Robotic Telekinesis, a pipeline that teleoperates the Allegro Hand [57] using a single RGB camera. Leveraging work in monocular human hand and body pose estimation [36], hand and body modeling [31, 32, 33], and human internet data, Robotic Telekinesis real-time re-targets the human hand and body to the robot hand and arm. Due to its efficiency and case of use, we leverage Sivakumar et al. [56]'s approach for demonstration collection.
We borrow the human hand to robot hand retargeting method from Robotic Telekinesis [56] that manually defines key vectors of and of between palms and fingertips on both the human and robot hand. They build an energy function E, which minimizes the distance between human hand poses (3,0) and robot hand poses q. o is a scale parameter. Therefore, the energy function is defined as:
E (Bh, On), 9) Σ 11 (a. 2f
Sivakumar et al. [56] train an MLP HR(.) implicitly minimize this energy function in conditioned on knowing human poses (3,0). For more details, we refer the readers to Sivakumar et al. [56].
4 Learning Dexterity from Human Videos
We learn general-purpose manipulation by utilizing large-scale human hand action data prior robot experience. We leverage not only visual priors of the scene's 's appearance but also leverage important aspects of the human hand's motion, intent, and interaction. To do this, we re-target the human video data to trajectories from the robot's embodiment and point of view. By pretraining policies with these human hand trajectories, we learn action priors how the robot should behave. However, it's notoriously difficult to leverage these noisy human video detections. Therefore, we must also employ policy with physical priors learn smooth and robust policies that do not overfit noise. We explain insights and our method used to leverage action priors in the sections below.
4.1 Visual Priors from Human Activity Data
Many previous works [6, 7, 8] have tackled visual priors and representations for robot learning. These networks often encode some form of semantic visual priors into the pretrained network from human video internet datasets. We use the encoder from Nair et al. [6] as useful visual initialization for policy. Nair et al. [6] is trained on a visual-language alignment as well as a temporal consistency loss. Our network takes human video frames and processes them using the publicly released ResNet18 [58] encoder, Es from R3M [6]. The output of this network is our visual representation for learning.
3

---

use internet vidcos as pscudo-robot experience, we re-target human hand detections from 3D MANO model [31] to 16 DoF robotic hand (LEAP) embodiment and we retarget the wrist from the moving camera to the xArm6 [59] embodiment. Videos at https: petts://viiee htt////vieodeeeee desooodex.ge titbhu..
Figure 3:
4.2 Action Priors from Human Activity Data
While visual pretraining aids in semantic understanding, human data contains lot more information about how to interact with the world. VideoDex uses action information to pretrain an action prior, a network initialization that encodes information about the typical actions for particular task.
However, training robot policies on human actions are difficult, as there is a large embodiment gap between humans and robots described in Handa al. [40] and Sivakumar et al. [56] Thus, must re-targe re-target the motion the human to the robot embodiment to use it in training. This problem is solved using three main components. First, we detect human hands in videos. Second, hand poses H to robot finger joints H,.. Finally, we convert human wrist pose P to robot arm pose extract
project
P.. H, and P, define the trajectory of the human in the robot's frame, from which actions to pretrain our policy network with the action prior. See Figure for a summary of the stages.
Action and Hand Detections First, we must detect the right actions the human is completing. To expedite development, we use the action annotations from the EpicKitchens dataset [2] but an action detection network such as [60] can be used. Now, must detect the hand. VideoDex first computes a crop around the operator's 's hand using OpenPose [61] and the result is passed to FrankMocap [36] obtain hand shape (5) and pose parameters (0) of the 3D MANO model [31]. These parameters are passed through a low pass filter and subsequently used in ve-targeting to the robot.
Re-targeting Wrist Pose In this section, show how to compute the transformation that describes the wrist pose in the robot frame denoted as Wririst Roobot First, to calculate !WWrist where C2 is the camera frame timestep i we leverage the Perspective-n-poine algorithm [62]. This takes 2D keypoint outputs (u, v.) by the hand detection model and 3D keypoints from the hand model (æ;, Us. 2;) and computes Mc. rist To accurately obtain camera intrinsics for PnP, COLMAP is used [63].
In human egocentric video datasets, the position
camera not fixed and we must compensate for this movement. Specifically, compute the transformation between the camera pose in the first frame C1 and all other frames in the trajectory, Ct. We call this transform MC. To estimate this, run monocular SLAM, specifically ORBSLAM3 [64].
Computing wrist poses in the first camera coordinate frame is important but this is still not in the robot frame because the robot is always upright. To be able transform the human trajectory in the robot's frame, we must find the vector that is parallel to gravity in the camera's 's frame, ap. Thus recover object segmentations for surfaces that are parallel to the floor such as tables, floors, counters, and similar synonyms using a state-of-the-art object detector (Detic [65]). Then an estimated depth map from RGB frames only using Adabins [66] computed. This way, the method does not rely on the long-term contiguity of video like most SLAM approaches. We then use depth map portions that correspond to the relevant objects and calculate a surface normal vector. We estimate ap using this normal vector and the following equations:
pitch tan
- (x / U2cc 22)00)
(3)
roll = tan
Mann-(yyan (yAAc// 42000 + 2200)
(4)
Detailed ablations on the parameterization of the initial pitch of the predicted trajectory ( (o) are provided in Section 6. In SLAM, we also remove the dependency gyroscope data by assuming

---

Robot Hand
TAP
IULIIA
Camera Trajectory
Hand Detections
Robot Trajectory
Learn Robot Policies
Figure 4: To use human videos as an action prior for training policies, we re-target them to the rohot embodiment. The detected human fingers are converted to the robot fingers using learned energy function. The wrist is re-targeted using the detections and camera trajectory and transformed to the robot
that the scaling factor is 1.0. This is acceptable because the trajectory is rescaled to the robot frame later. Therefore, this wrist re-targeting approach uses only 2D images from human videos.
Since the robot has workspace limits, and we would also like to center the starting pose of the robot, heuristically compute orld orld which rescales and rotates the human trajectory in the world frame The final function obtain M Wrist can be described as:
wrist into the rohot trajectory TR
M Writt
T Worrd Robott
M World" M Murrit
(5)
Robot
Re-targeting Hand Pose Human hands also in a different embodiment compared to that of robot hands, like our 16 DOF LEAP Hand [67] Similarly, to Sivakumar et al. [56), we use H() to map hand poses robot hand poses. Given human detected pose The we obtain I = H(Tx) using a similar re- targeting network to Sivakumar al. [56], and get human hand trajectories: T in the robot's embod- iment. We use TR denote the combined hand and
Algorithm 1 Procedure for VideoDex
Require: Human videos Vilk (length T), policy To. demonstrations DI.N- Human detection Suumnn [36].
for h = 1...K do
for t 1...f do
Pose parameters 0,, 3: fruman (1+) Get wrist pose wi from 3. 4 and 5, Hand pose h4 = (0,, B.)
wrist trajectories: TR .TR visualization.
hand wrist See Figure 3 for
end for
Store all he, uy into robot trajectory T
F = [of 11. w()
4.3 Learning with Human Videos
Optimize La 1| +kll1
end for
We must design an open-loop policy æ that lears first from the re-targeted human trajectories (the action prior) and then from real robot trajectories collected teleoperation. Naively, training a neural network policy on will lead to overfitting to noisy hand detections. To cireumvent this, we first use visual priors from the visual ResNet-based [58] encoder provided by Nair al. [6]. Eg. Then, we introduce physical prior to the network, the physically-nnsiired
Store policy weights 9₁ to initialize no while not converged do for v = 1.... do
Tv. Ii. D.
Fr = (I [f. w()
Optimize Lo = I| fnlli
end for end while
Neural Dynamic Policies [13, 14).
We construct æ with the following setup. We first process the first scene image / with the visual encoder Eg. Then the extracted features E& are used condition an NDP for the wrist and hand separately. fwria and fhand. Concretely, each NDP operates by processing the input features with small MLP which outputs w, which are the trajectory shape and goal parameters. The forward integrator of the NDP outputs an open-loop trajectory for the hand and the wrist, îg. We use the following loss function:
Σ LOSSI,(TR [Shana (Es (I)),, S Swris) (ss
Training Methodology: We use between 500-3000 video clips of humans completing the same task category as the robot will from the Epic Kitchens dataset [2]. For example, in pick, there are

---

Figure 5: Tasks used in experiments. From left to right: pick, rotate, open, cover, uncover, place and push. Sec bttpa://videor dex. github io for videos of these tasks.
close to 3000 video clips of humans picking items. These are retargeted to the robot domain and used to pretrain the network with the human action prior of the pick task. Then, the final policy æ is trained on a few teleoperated demonstrations of pick on the real robot. The full training takes about 10 hours on a single 2080Ti GPU. More training details can be found network consists of the R3M [6] initialized ResNet-18 18 [58]. We process these features with a 3-layer MLP with a hidden layer size of 512, which are then processed by NDP [13] networks.
appendix and in Algorithm Our
5 Experimental Setup
We perform thorough real world experiments on manipulation tasks, specifically many tasks that require dexterity. See our webpage for result videos. We aim to answer the following questions. (1) Is VexeoDex able to perform general purpose open-loop manipulation? (2) How much does the action prior of VideoDex help? (3) How much does the physical prior of the NDPs in VideoDex help? (4)
What important design choices are there (visual priors, physical priors, or training setup)?
Task Setup We pretrain action priors on retargeted Epic Kitchens data for seven robot tasks. Then, collect about 120-175 demonstrations for each of these tasks pick, the goal is to pickup an object. In rotate, the agent grasps and rotates the object in place. In cover and uncover, the goal is cover uncover pan/plate with soft cloth object. Push involves flickingpokking an object with the fingers. In place, the robot has to pick up an object and place it into a plate, pan or pot. In open we open three different drawers. Our testing procedure consists of unseen locations and objects. Details on the tasks and objects are in the supplemental.
setup train the policy. In
While robot hands can provide great dexterity, we also investigate whether 2-finger grippers can benefit from action priors. The internet data is converted to where the closed human hand is a closed 2-nigrer gripper, and the open human hand is an open 2-nineer gripper. We collect separate demonstrations on the real-robot using the 2-finger gripper from xArm [59]. Separate action priors trained for the 16 DoF LEAP Hand and the 2-finger gripper.
Results
3.48
13
First, evaluate the need for initial- ization with the action priors obtained from the human internet videos. f, The baseline without internet pre-training is called BC-NDP. It uses the same physi- cal prior and visual network initializa- tion, without the initialization from On. We also compare the effect the action prior on 2-finger gripper policies. Sec- ond, we compare against two standard open-loop behavior cloning approaches introduced in recent benchmarks [51]. BC-open uses a 2 layer MLP instead of the NDP network. BC-RNN, uses an RNN to pre-process the visual features and then a two-stream, 2 layer MIP for wrist and hand trajectories. We try an offline RL ablation CQL [68], where we
3.46
3:35
9.25
Gownd Trahh
180
135
340
39
Thrastop Vanilla Visual Prior 0
Initialization with 0yunan
Figure 6: Networks initialized using action priors on human data without further training closer ground truth robot trajectories than networks only initialized using visual priors

---

Pick
Rotate
Open
Cover
Uncover
Place
Push
train
test
train
test
train
test
train
test
train
test
train
test
train
test
BC-XDP [14]
0.64
0.38
0.94
0.56
0.90
0.60
0.78
0.58
0.88
0.82
0.70
0.35
1.00
0.71
BC-Open[51]
0.50
0.44
0.72
0.38
0.80
0.40
0.44
0.58
1.00
0.91
0.40
0.25
1.00
0.93
BC-RUN [51]
0.56
0.31
0.78
0.50
0.90
0.50
0.56
0.42
0.88
0.75
0.70
0.50
1.00
1.00
VideoDex
0.83
0.77
0.85
0.71
0.80
0.80
0.75
0.63
0.96
0.92
0.89
0.80
1.00
1.00
Table 1: We present the results of train objects and test objects for Videodex and haselines as descrihed above.
use the demonstrations as sparse reward. We train a behavior cloning policy with the action prior from human videos without the physical prior of the NDP. We call this VideoDex-BC-Open We ablate the type of visual representation and prior use by trying an initialization using the VGG16 network [69] (VideoDex- VGG) and the MVP network [7] [70] (VideoDex-MV) based representation trained for robot learning. We ablate the need for a two stream policy, instead training a single NDP for both hand and wrist. AadeoDex-Singll (around 50 demonstrations, 5-7 per variant only), we train policy called GeleeDex-Constrailee
To see VideoDex works with fewer demonstrations
We analyze the results of our experiments and the guiding questions discussed in Section 5. We present the results of our findings as a 0-1 success rate in Table 1 and the result of the ablations we ran on the place task in Table 4.
Effect of Action Priors We firstly compare VideoDex against methods that do not employ an action prior trained on human data, explained in Section 5. For almost all of the tasks, VideoDex either outperforms baselines or has a similar performance, especially for held out ob- jects/instances. We believe that one of the key aspects of VideoDex generalizing to test objects is the action prior pretraining on human videos. This can be seen in Figure 6. Without ever training on the robot demonstrations, the trajectories initialized using the action prior pretrained network 8% (left) are much closer to the ground truth trajectories of a network that is initialized using only a visual prior such as the encoder from Nair et al. [6] (right). From the results, we see that VideoDex-BC-Oper with action priors (Table 4) outperforms BC-Open. Having physical prior added (BC-NDP) tends to help, but it is not the case for every task. We suspect that some tasks require smoother behavior than others. Additionally, in Table 4 our offline RL baseline, CQL [68] does not perform as well as the rest of the approaches, even
perforrnrf the Behavior Cloning setup. Qualitatively, we see much less smooth and less safe execution with this method, thus we only perform on one task (place). Note that we use the same visual prior for this as well.
Place
Open
Pick
Hand vs 2-Finger Gripper We compare whether the action priors from VideoDex also help in the more general 1-DOF gripper setting. In Table 2, we find that in the 1-DOF setting, VideoDex still improves performance on these tasks. This is because the pri- ors from human internet videos still encode typical wrist trajectory behaviors as well as when the gripper should close for each task.
1-DOF BC-0pen[51]
0.62
0.69
0.71
1-DOF VideoDex
0.69
0.82
0.77
Table 2:
We compare how the 1-DOF xArm gripper performs using Videodex. [59] Separate demonstrations were collected using this gripper.
Initial Pose Computation Comparison We com- pare three different ways estimate Op or MC₁ World" the vector that points parallel to gravity. These methods contrast with VideoDex which uses the sur- face normal of objects that are typically parallel with the floor to calculate the direction of gravity. Vi feeoDex-Fixedd assumes that ap is [0,0]. This is reasonable as we are not relying on robots ex- actly mimic the human but get a general action prior. Vi deeDex-Randde 15-45 degrees, which is the typical egocentric camera angle. VideoDex- IMU uses the internal image stabilization sensor data to estimate the upright vector. None of these approaches use gyroscope data in SLAM, as we assume that the scaling factor is 1.0. In Table 3, we present the results of these experiments. The performance degrades when randomizing or setting M to a fixed value, in all three of the tasks, but still comparable to or better than our baselines that do not use any human action data. A possible explanation for the fact that
Place
Cover
Uncover
VideaDex-Fiss 0.55
0.50
0.77
EideoDex-Randon
0.45
0.63
0.85
VideoDex-IMU
0.70
0.67
0.90
VideoDex
0.80
0.63
0.92
Table 3: Ablations that compare the different ways of calculating the initial pitch of the cam- era with respect to gravity, on test objects. This enables us to transform human trajectories to he upright like the rohot is.
randomizesa, in the range of
7

---

VideoDex- Surffeeeee performed better than our VideoDex-INe is that the sensor data may be noisy and estimating surface normals from visual features is more robust.
Effect of Physical Priors and Architectural Choices We compare different types of physical priors in Table 1 and in Table 4. In general (BC-NDP) tends to outperform baselines without physical prior, except for BC-RNN in a couple of tasks. BC-RNN performs less aggressive behavior, which allowed it to efficiently grasp more objects. In Table it's shown that an important physical prior is to treat the wrist and the hand in more disentangled manner, as the performance for VideoDex-Singg tends to drop compared to BC-NDP and VideoDex-BC-Open (Behavior Cloning with our action prior pretraining). The two stream architecture aids in learning, as it allows the policy to disentangle the actions of the wrist and the hand. This is important as the same grasp might be used for picking objects in many different locations, and similarly, it is possible to localize many objects and perform completely different types of interactions.
Generalization with Less Data We limit VexeoDex to a max- imum of 5 and 10 teleoperated demonstrations per variant (we have 12-15 variants in our setup), As shown in Table I, even with 5 instances per variant, we still see 30% success rate for unseen objects. Empirically, the policies generally go to the right area but are not able to grasp objects properly. With less robot experience, VideoDex outperforms which demonstrates that action priors also boosts sample efficiency.
Train
Test
Baselines:
BC-NDP [14]
0.70
0.35
BC-Open [51]
0.40
0.25
BC-RIN [51]
0.70
0.50
CQL [68]
0.40
0.20
No Physical Prior:
Vi Bideppe VideoDex-Singll
0.50
0.50
Effeet of Visual Priors We compared using our approach with MVP (VideoDex-MVP) [7] and VGG (VideoDex-VGG) [69] and their performance was below VideoDex using Nair et al. [6]. This is likely because both encoders are much larger than the ResNet18 8 [58] we use and require a lot more training time than feasible on human videos. However, VideoDex-NVP still performs better than VideoDex-VGG, which indicates that using visual prior trained on human data does in fact help, as Xiao et al. [7] trained the representation in self-supervised fashion on videos and use the embeddings to perform roboties tasks in simulation. We see in Table I, that while visual priors are important, action priors are more impactful.
0.50
0.30
Visual Prior Ablation:
VideoDex-VGG VideoDex-MVP
0.20
0.20
0.40
0.20
Constrained Data:
VideaDex-Gonst-S
0.80
0.60
VideoDex-Const-10
0.50
0.30
VideoDex (ours)
0.90
0.70
Table 4: We present the results of the ablations discussed in Section 5. These are all performed on the place task.
Choice of Robotic Hand In our experiments, we also tried using the Allegro Hand [57]. We found that the Allegro had higher inaccuracy in control and more hardware failures as compared to LEAP Hand. LEAP Hand outperformed the Allegro Hand 7 - 12% on average in all experiments, thus we use it for our setup [67].
7 Discussion and Limitations
Although we see strong results on the held-out objects, VideoDex has several limitations and scope for future work. First, we focus on curated human video datasets, such as EpicKitchens [2], but only use these as a convenience to expedite our process. It is possible to filter internet videos of humans according to tasks using action detectors and then processing them with VideoDex. We also use camera data in VideoDex but show that with heuristic driven approach it is possible to obtain similar or better results. Second, we rely on off-the-shelf human hand detection modules that very often have erroneous 6D pose detections, especially when the hand is interacting with objects. Second, the action priors rely on the arm trajectory as well as the hand trajectory retargeting which must be recomputed for each different set of robot parameters and embodiment. Finally, our method of behavior cloning in the real world is currently open-loop, so it cannot react to changes in the environment. This is because closed-loog behavior cloning is difficult to keep safe in the real world. Similarly, when running closed-loog RL it is difficult to guarantee the safety of the system. We leave this to future work, to train policies that can react to changes in the real world.
Acknowledgments
We thank Aditya Kannan and Shivam Duggal for assisting in robot data collection. We thank Aravind Sivakumar, Russell Mendonca, Jianren Wang, and Sudeep Dasari for fruitful discussions. KS is supported by NSF Graduate Research Fellowship under Grant No. DGE2140739. The work is supported by Samsung GRO Research Award and ONR N00014-22-1-20%.
8

---

References
[1] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, Hamburger, H. Jiang. M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[2] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray. Scaling egocentric vision: The epic-kitchens dataset. European Conference on Computer Vision (ECCV), 2018.
[3] R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, Yianilos, M. Mueller-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic. The "something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[4] L. Pinto, D. Gandhi, Y. Han, Y.-L. Park, and A. Gupta. The curious robot: Learning visual representations via physical interactions. In ECCV, 2016.
[5] Sermanet, C. Lynch, Chebotar, Hsu, Jang. Schaal, and S. Levine. Time-contrastive networks: Self-supervised learning from video. In ICRA, 2018.
[6] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint 2rXiv:2003 2022.
[7] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173, 2022.
[8] A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. In NeurlPS, pages 9191-9200, 2018.
[9] K. He, G. Gkioxari, Dollar, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[10] T. Chen, S. Korbllihh, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020. URL https: httts::///pproee /ttpc//eioc............. mlr. press/v119/ chen20j html.
[11] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss,, G. Krueger, T. Henighan, R. Child, A. Ramesh, M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, Sigler, Litwin, Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. 2020.
[12] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.044 2018
[13] Bahl, M. Mukadam, A. Gupta, and D. Pathak. Neural dynamic policies for end-to-end sensorimotor learning. In NeurlPS, 2020.
[14] S. Bahl, A. Gupta, and D. Pathak. Hierarchical neural dynamic policies. RSS, 2021.
[15] D. Kalashnikov, A. Irpan, Pastor, Ibarz, A. Herzog, E. Jang, D. Quillen, Holly, M. Kalakr- ishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:1806.1000 10293, 2018.
[16] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. JMLR, 2016.
[17] E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. In IROS, 2012.

---

[18] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108, 10470, 2021.
[19] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 1. Morgan- Kaatfan,, 1988. URL https://prooeedingg
wtw.eeeeeinnnnn.nniiii...e. neurips wwterips.ccpaaapeeee wtt.lcccppppppccc. wttppppe//////iaaa//// pdf.
[20] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P.
Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba. End to end learning for self-driving cars, 2016. URL httpp://///. //tp://///xiivvi/ai org/ abs//////// 1604. 07316.
[21] S. P. Arunachalam, S. Silwal, B. Evans, and L. Pinto. Dexterous imitation made easy: A learning-base framework for efficient dexterous manipulation, 2022. URL https://arxii: org/abs/ 2abb//2203. 13251.
[22] Y. Qin, H. Su, and X. Wang. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation, 2022. URL https://arxiv..rrgg wttp:/iwww.rgr////////// 2204. 12490.
[23] Y. Qin, Y.-H. Wu, S. Liu, H. Jiang, R. Yang, Y. Fu, and X. Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. arXiv preprint arXiv:2108 05877, 2021.
[24] P. Mandikal and K. Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. In Conference on Robot Learning, pages 651-661. PMLR, 2022.
[25] C. Zimmermann, D. Ceylan, J. Yang, B. Russell, M. Argus, and T. Brox, Freihand: dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 122- 2019.
[26] D. Shan, J. Geng, M. Shu, and D. F. Fouhey. Understanding human hands contact at internet scale. In Proceedingy of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 9869-988 6899-8888, 2020.
[27] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3, 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments, IEEE transactions on pattern analysis and machine intelligence,
2013.
[28] Cmu graphics lab motion capture database.
ca. cmu, cmu.edu/.
[29] B. G. Fabian Caba Heilbron, Victor Escorcia and J. C. Nicbles. Activitynet: A large-scale video benchmark for human activity understanding, CVPR, pages 961-970, 2015.
[30] P. Das, C. Xu, R. F. Doell, and J. J. Corso. A thousand frames in just a few words: Lingual description of videos through latent topies and sparse object stitching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2634-2641, 2013.
[31] J. Romero, D. Tzionas, and M. J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6), Nov. 2017.
[32] M. Loper, N. Mahmood, Romero, G. Pons-Moll, and M. J. Black. Smpl: A skinned multi- person linear model. ACM transactions graphics (TOG), 34(6):1-16. 2015.
[33] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10975-10985. 2019.
[34] Wang, F. Mueller, F. Bernard, S. Sorli, O. Sotnychenko, N. Qian, M. Otaduy, D. Casas, and C. Theobalt. Rgb2hands: real-time tracking of 3d hand interactions from monocular rgb video. ACM Transactions on Graphics (TOG), 39(6):1-16, 2020.
[35] A. Kanazawa, M. J. Black, D. W. Jacobs, and Malik. End-to-end recovery of human shape and pose. CoRR,
2017. URL //axiiv///xxv....// org/abs/////// /1722 06584.
10

---

[36] Y. Rong, T. Shiratori, and H. Joo. Frankmocap: A monocular 3d whole-body pose estimation system via regresssi and integration. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 1749-1759, October 2021.
[37] S. Han, B. Liu, R. Wang, Y. Ye, C. D. Twigg, and K. Kin. Online optical marker-based hand tracking with deep labels. ACM Transactions on Graphics (TOG), 37,4):1-10, 2018.
[38] V. Kumar and E. Todorov. Mujoco haptix: A virtual reality system for hand manipulation. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Hlumanoids), pages 657-663, 2015, doi:10. 10::10.110910 D0D.UMAAAIIDSS..1..7.6...
[39] S. Li, X. Ma, H. Liang, M. Görner, P. Ruppel, B. Fang, F. Sun, and J. Zhang. Vision-based teleoperation of shadow dexterous hand using end-to-end deep neural network. In 2019 International Conference on Robotics and Automation (ICRA), pages 416-422. IEEE, 2019.
[40] A. Handa, K. Van Wyk, W. Yang, J. Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox. Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164-9170, 2020. doi: 100110/// 9197124.
[41] L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manipulation concepts from instructions and human demonstrations. The International Journal of Robotics Research, 40(12-14), 2021.
[42] A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from" in-the-wild" human videos. arXiv preprint arXiv:2103. 2021.
[43] S. Bahl, A. Gupta, and D. Pathak. Human-to-robot imitation in the wild. RSS, 2022.
[44] K. Schmeckpeper. O. Rybkin, K. Daniilidis, S. Levine, and C. Finn. Reinforcement learning with videos: Combining offline observations with interaction. arXiv preprint arXiv:201 I. 1.06507, 2020.
[45] Sharma, D. Pathak, and A. Gupta. Third-person visual imitation learning via decoupled hierarchical controller. arXiv preprint arXiv:1911.00666. 2019.
[46] L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. In RSS, 2020.
[47] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto. Visual imitation made easy. arXiv preprint htl::2008004899 2020.
[48] Lee and M. S. Ryoo. Learning robot activities from first-person human videos using convolu- tional future regression. In CVPR Workshops, pages 1-2, 2017.
[49] N. Das, S. Bechtle, T. Davchev, D. Jayaraman, A. Rai, and F. Meier. Model-based inverse reinforcement learning from visual demonstrations. arXiv preprint arXiv:2010.09034 2020.
[50] J. Pari, N. Muhammad, S. Arunachalam, L. Pinto, et al. The surprising effectiveness of representation learning for visual imitation. arXiv preprint a2.00:21 2021.
[51] S. Dasari, J. Wang, Hong, Bahl, Y. Lin, S. Wang, A. Thankaraj, S. Chahal, B. Calli, S. Gupta, et al. Rb2: Robotic manipulation benchmarking with twist. In NeurIPS Datasets and Benchmarks Track (Round 2). 2021.
[52] A. J. Ijspeert, Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal. Dynamical movement primitives: Learning attractor models for motor behaviors. Neural Computation, 2013.
[53] M. Prada, A. Remazeilles, A. Koene, and S. Endo. Dynamic movement primitives for human- robot interaction: Comparison with human behavioral observation. In International Conference on Intelligent Robots and Systems, 2013.
[54] S. Schaal. Dynamic movement primitives-a framework for motor control in humans and humanoid robotics. In Adaptive motion of animals and machines. Springer, 2006.
11

---

[55] P. Pastor, H. Hoffiaann, T. Asfour, and S. Schaal. Learning and generalization of motor skills by learning from demonstration. ICRA, 2009.
[56] A. Sivakumar, K. Shaw, and D. Pathak. Robotic telekinesis: Learning a robotic hand imitator by watching humans youtube, 2022.
[57] Allegro hand. https://wwz hand.h.ttsss/w.....okrroooooooo http://wwwworrroooooo/coo//o/rra
[58] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385. 2015. URL. htt..tttp//aarra..og//aa////////
[59] xarmó by http://////..aaattooo.....aaoooa robot..
[60] M. Contributors. Openmmlab's next generatio video understanding toolbox and benchmark. https://githut. com/ httpm//ppermmmaammaaiioiooo 2020.
[61] Z. Cao, G. Hidalgo, I. Simon, S.-E. Wei, and Y. Sheikh. Openpose: realtime multi colli-person. 2d pose estimation using part affinity fields. IEEE transactions pattern analysis and machine intelligence, 43(1):172-1886 2019.
[62] M. A. Fischler and R. C. Bolles. Random sample consensus: paradigm for model fitting with applications image analysis and automated cartography. Commun. ACM, 24(6):381-39) jun 1981. ISSN 0001-0782. doi: 151.11114/35866.33866666 URL https://dooi.. 1r1/11..14// 018669.358630
[63] L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise View Selection for Unstructured Multi-View Stereo. In European Conference on Computer Vision (ECCV), 2016 6.
[64] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós. Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Transactions on Robotics, 37(6):1874-189)) 2021.
[65] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and Misra. Detecting twenty-thousand classes using image-level supervision. In ECCV, 2022.
[66] S. F. Bhat, I. Alhashim, and P. Wonka. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4009 4018, 2021.
[67] K. Shaw and D. Pathak. Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning. in Submission, ICRA, 2023.
[68] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning, Advances in Neural Information Processing Systems, 33:1 1199-199, 2020.
[69] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
[70] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[71] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 Ieee, 2009.
[72] A. Handa, K. Van Wyk, W. Yang, J. Liang, Y.-.. Chao, Q Wan, S. Birchfield, N. Ratliff, and D. Fox. Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164-9177. IEEE, 2020.
[73] M. I. Takuma Seno. d3rlpy: An offline deep reinforcement library. In NeurIPS 2021 Offline Reinforcement Learning Workshop, December 2021.
12


Reviews:
[{"review_id": "SRKc6nse5y-Y", "reviewer": "Reviewer_nwhC", "summary": "The authors demonstrate a system in which they combine a few different components to get interesting supervised-learned open loop behavior of real robot hands doing several different tasks. In particular the most notable part of the approach is using videos of human hands as an “action prior” which informs their supervised mapping.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- Good core idea. The overall idea of using action priors from human videos, via hand tracking, to make robots work better, is a good idea. There are a lot of closely related works, but I think they are well referenced in this paper. \n- Good execution on several key parts. The execution details of handling moving cameras with camera pose tracking, together with per-frame hand tracking, seems to be well done. I also like just using R3M features out of the box, this is smart and interesting to see external validation. \n- Results of real robots with hands doing a variety of things. \n\n# Weaknesses\n\nThere are various unscientific elements of this paper in its current form.\n\nWhile the work is interesting, I can’t recommend a strong accept for a paper in this form. Hopefully the list below will help the authors improve both this work and their future work.\n\nIf the authors can address all of the following weaknesses in their rebuttal, which I think is all doable and within scope to do in a rebuttal, I’d be happy to move from weak accept to strong accept. \n\n1. It seems like the authors are not very upfront about the fact that this method does not produce closed loop policies. Only on the last page or two is it mentioned that the whole method is open loop. This is fine to study the task of (i) inputting an image of a scene and (ii) outputting an open loop trajectory, but, it of course is very limiting. The tasks are carefully chosen such that they don’t require any closed loop feedback. This aspect of their approach is not what most researchers in the field would expect… so a common experience of a researcher would be to look over the first handful of pages of this paper, and only at the last page or so realize that this is an open loop method. Please just make this clear up front.\n2. Several false statements in the introduction:\n  - “ To build such robotic agents that can operate anywhere, we need access to a lot of successful robot interaction data in many environments.” —> not necessarily true… This is a reasonable hypothesis, but one that isn’t tested in this paper, and it can’t be stated as a fact.\n  - “ However, deploying inexperienced real world robots to collect experience must require constant supervision which is in feasible.” —> also not necessarily true… but also a very reasonable hypothesis. Just need to say “may require” instead. \n  - “Most of the inefficiency in robot learning is due to the exponentially large action space.” —> an opinion, and can’t be stated as fact.\n3. “NDPs can produce safe and smooth trajectories” … yes, but this is a meaningless statement. They *can* also produce trajectories that are completely unsafe. There is nothing about NDPs/DMPs that provides safety other than a bit of smoothness that may arguably help. But there is nothing that helps here with the presence of obstacles in the environment, or humans, etc. This statement probably only serves to confuse/mislead inexperienced readers, please remove/fix.\n4. The paper mentions a “physical” prior as a key component, but this is just that it uses Dynamic Movement Primitives it seems. I’m not sure this is the best way to communicate this. Line 191 also says physically-aware NDPs… they don’t know anything about contact physics… maybe just say second order system or dynamical system or something, maybe physically-inspired, but not physically-aware. And whenever it says, for example line 269, “baselines without a physical prior” it should just be instead clear that this just means they don’t use DMPs.\n5. Line 213 “ is VideoDex able to perform general purpose manipulation?” Since the method is open loop, the answer is no. That’s fine, and the results are still impressive, but should be clarified… this is not something that needs to be empirically evaluated, it’s just a result of the formulation.\n6. It’s very confusing that citation 44 is used open loop… this isn’t an intention of the method. Also, is the RNN version closed loop over time? It’s not clear. And if it’s not? … I’m not sure how the RNN would be any different if it’s not used sequentially over time. \n7. Please state exactly how many demonstrations were used for the different experiments. \n8. In the conclusion… “ this is because training RL in the real world is difficult due to hardware limitations.” Yes, but this isn’t reason to make the used behavior cloning method open loop instead of closed loop.\n\n\n## Minor\n\nDon’t worry about these too much but I mention these as opportunities to improve the paper further. \n\n- Ego4D is not cited on page 2 (mentioned but not cited)\n- HR() is not defined in an equation. Also, I would recommend not using two letters for a math symbol… it looks like a matrix H multiplied by a matrix R\n- Why use ORBSLAM3 rather than COLMAP for the poses? Already running colmap for the calibration. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Please see the start of the Weaknesses section where I explain the recommendation. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please see the list in Weaknesses.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "Ag2_JoEOC3SO", "reviewer": "Reviewer_KbvG", "summary": "In this paper, the author investigates how to utilize large-scale human video to train dexterous robot manipulation skills. To leverage the information from the Internet videos, the author proposes a handful of techniques to pre-process the video data to extract the action information. Then the network is trained on the extracted hand data and deployed to the real robot with some human demonstration collected by teleoperation for fine-tuning. Experiments show that the proposed pipeline can solve multiple manipulation tasks.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\n- The direction explored in this paper is important. Utilizing the internet video data for robot learning is well motivated. Especially considering the similarity between human and multi-finger hands, this direction looks very promising. \n\n- The authors perform experiments with multiple real-world tasks with pick and place, pushing, and rotating objects.\n\n**Weakness**\n\n- Although the objective of this paper is very impressive, the experiments can not support the introduction and there are multiple overclaims.\n\n- Section 4 is titled VideoDex: Learning Dexterity from Youtube. However, I can not find any evidence that the author utilizes YouTube data for learning dexterous manipulation. As mentioned in the Section on Retargeting Wrist Pose, ORB SLAM and the camera’s acceleration data are used to compute the camera pose trajectory. This information is not readily available in the YouTube data. The experiments and methods are misaligned with this claim.\n\n- In the introduction line 42, the author mentioned that our key insight is to combine these visual and action priors from passive data with the physical constraints of how robots should move in the world. However, the method does not consider the surroundings of the human hand, and the detection results itself is not accurate. How to incorporate physical information into the training data?\n\n- Missing literature discussion on previous learning from video works:\n\n    [1] *DexMV: Imitation Learning for Dexterous Manipulation from Human Videos, 2021*: This paper focuses also on how to learn dexterous manipulation from human videos. The reviewer understands that this literature paper uses simulated tasks while the authors focus on the real robot settings. But it seems that similar pipelines are also used in this paper: estimating the human hand, retargeting, and learning from retargeted hand pose.\n\n    [2] *The Surprising Effectiveness of Representation Learning for Visual Imitation, 2021*: This paper also focuses on how to leverage the video data for better learning. It also uses a GoPro camera to collect a video of each trajectory, which is the same as the Ego4D dataset used in this paper. It shows that by learning from this video data, the final manipulation performance can be improved a lot.\n\n    These literature works use very similar methods to achieve robot learning. The novelty claims of this paper can also be found in this literature.\n\n- Missing details for Re-targeting Wrist Pose\nThe detection module FrankMocap is a 2D hand detector, it is not clear how the author can get 3D keypoints from the hand model in the camera frame. Also, this section is important in the whole technical approach, it is better to provide visualization of the final retargeted robot. A hand wrist pose and robot arm should also be visualized in Figure 3 if they are used in the training. If the wrist pose and arm joint pose is not used, how to pretrain the action prior?\n\n- Missing details about transforms\nIn the equation, it is not clear why the author uses T and M to denote pose simultaneously. What are the differences? If M is also a $SE(3) $transformation, how to compute the position part of the $M_{World}^{C_1}$? Besides, the reviewer can not find any information about how the $T_{Robot}^{World}$ is determined heuristically in both the main paper and supplementary.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper studies an important problem of learning manipulation skills from YouTube videos. The objective is of vital importance to the field. However, the methods described in the paper are not able to support the claim. The manipulation tasks are diverse but lacking in technical difficulty, which can also be achieved by parallel gripper. More investigations are necessary.\n\n## Updates (after rebuttal)\n\nThe updated version removes the over-cliam of learning from YouTube and utilize visual RGB-D slam techniques to learn from image only. The reviewer believes that this is an important direction in imitation learning on dexterous manipulation. Although the experiments of this work is not so solid, i.e. missing intermediate evaluation results of the pretraining data generated from videos, the overall technical pipelines sounds reasonable to me. The reviewer raise the score to weak accept under the promise of the author that:\n\nShows intermediate visualization/results of the generated data for pretraining. The current experiments only show the final learning performance of manipulation tasks. It is unknown that what is the quality of these generated data from videos and how the quality of these data will influence the final performance in a more strict way. It will be preferable if the author can show some visualization of the generated VideoDex data in a more straight-forward way. For example, visualize the robot hand-arm trajectory and compared with the original internet video.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Clarification for the weakness part listed in the previous section.\n- More explanation about the transformation used in Eq5: \n  - How to heuristically compute $T_{Robot}^{World}$\n  - How to compute the translation term for $M_{C_1}^{World}$\n  - How to get the 3D keypoints used to compute $M_{C_t}^{Wrist}$ by PnP\n- Link in supp is broken with 404 error: https://videodex.github.io/download\n- Clarification on the benefits of using dexterous hand for the task defined in the paper compared to normal parallel gripper.\n\n"}}, {"review_id": "8x5jVyTS5XS", "reviewer": "Reviewer_1pwQ", "summary": "VideoDex pretrains a policy network with videos, with gyroscope and accelerometer data, of humans performing a task, then fine-tunes with demonstrating trajectories collected by teleoperating the robot.  In order to train with the human data, they use the approach from [49] for mapping human pose to robot pose and use ORBSLAM3[55] to account for the camera motion.  They feed the image data, labeled with the outputted pose, into a ResNet18[15] backbone initialized with R3M's[6] features and use a Neural Dynamic Policy (NDP) [13] network to generate actions.  The paper demonstrates that using human data allows improved performance on 6/7 tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\nThe paper presents a theoretically simple method of learning from videos of humans.\n\nThe method is demonstrated on 7 different tasks, outperforming the baselines without human data on 6 of them.\n\n\nCons\nThe writing of the paper is somewhat scattered.\n\nThe analysis of why the proposed approach using NDP rather than a MLP works better with human data could be stronger. \n\nThe paper needs to be much clearer that it relies on gyroscope and accelerometer data from the human videos, which is a barrier to truly using internet-scale data.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this paper has promise, but could use some additional work to solidify its claims and clean up its presentation. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Line 47, Line 301 - The claim that the approach \"learns from passive videos on the internet\" is a little too strong.  Epic-Kitchens is a large and visually challenging source of data, but it is still curated and collected with a consistent methodology, making there be a significant gap between using it and using videos from the wilds of the internet.  Additionally, the proposed method makes use of gyroscope and accelerometer data for the human videos, which is not universally available for internet videos.\n\nLine 84 - The related work section on learning manipulation from watching videos of humans only mentions a single recent paper.  There has been several years of work in this area by different groups that approach the problem in a variety of different ways. Here are some examples of other work that should be included and discussed.\nLearning Robot Activities from First-Person Human Videos Using Convolutional Future Regression - https://arxiv.org/abs/1703.01040v2\nThird-Person Visual Imitation Learning via Decoupled Hierarchical Controller - https://arxiv.org/abs/1911.09676\nLearning Predictive Models From Observation and Interaction - https://arxiv.org/abs/1912.12773\nAVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos - https://arxiv.org/abs/1912.04443\nVisual Imitation Made Easy - https://arxiv.org/abs/2008.04899\nModel-Based Inverse Reinforcement Learning from Visual Demonstrations - https://arxiv.org/abs/2010.09034\nReinforcement Learning with Videos: Combining Offline Observations with Interaction - https://arxiv.org/abs/2011.06507\nLearning by Watching: Physical Imitation of Manipulation Skills from Human Videos - https://arxiv.org/abs/2101.07241\nLearning Generalizable Robotic Reward Functions from \"In-The-Wild\" Human Videos - https://arxiv.org/abs/2103.16817\n\nLine 145 - Figure 4?\n\nLine 179 - What are the differences from [49] that makes the retargetting network \"similar\" rather than just using the method from [49]?\n\nLine 181 - Figure 3?\n\nLine 218 - Should this be 120 demonstrations per task?  The appendix says 120-175 demonstrations per task and I'm not sure what the \"hour\" is measuring.\n\nTable 1 - How many trials were the different policies evaluated on?  It would also be good to include some measure of variation in the results. That would help to show that the differences in performance are actually statistically meaningful.\n\nTable 1 - What are the success criteria for the tasks?  For instance, the example for \"Push\" in the video looks more like the robot bumped into the object rather than executing a sustained pushing motion, but the table reports a 100% success rate.\n\nLine 233 - Why is CQL not run on the rest of the tasks?  \n\nLine 289 - This experiment modifies both network architecture and network parameters.  A clearer experiment would be to compare ResNet18 with R3M features to ResNet18 with ImageNet features.  Also, the comparison between the performance with VGG and with MVP isn't fair, since VGG is an architecture that is 8 years out of date and should perform worse on pretty much any task, no matter the initialization.\n\nAppendix - Which augmentations were used and which augmentations were not used?\n\nOne important experiment that I would like to see is the performance of VideoDex pretrained on human data from task A, then finetuned and tested on Task B.  If there was a significant gap between that and the performance when pretrained and tested on the same data, that would demonstrate that the human data is actually providing the model with useful, task-specific action priors rather than just putting it in a generally good initialization.\n\n\nOne of the central claims of the paper is that the use of NDP helps provide a physically grounded prior that enables the network to better deal with the noise in the trajectories/actions generated by [49]. It would interesting to compare the performance of VideoDex and VideoDex-Open pretrained with teleoperated demonstrations with different levels of noise artificially added.  If the performance gap increased as the level of noise increased, that would be convincing evidence that NDPs have some unique benefits in this setting.  \n"}}]

Author rebuttal:

Decision: Accept (poster)