DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video
Authors: Priyanka Mandikal
Kristen Grauman
Venue: CoRL2021 Poster (2021)
Keywords: dexterous manipulation
learning from observations
learning from demonstrations
computer vision

Abstract: Dexterous multi-fingered robotic hands have a formidable action space, yet their morphological similarity to the human hand holds immense potential to accelerate robot learning. We propose DexVIP, an approach to learn dexterous robotic grasping from human-object interactions present in in-the-wild YouTube videos. We do this by curating grasp images from human-object interaction videos and imposing a prior over the agent's hand pose when learning to grasp with deep reinforcement learning. A key advantage of our method is that the learned policy is able to leverage free-form in-the-wild visual data.  As a result, it can easily scale  to new objects, and it sidesteps the standard practice of collecting human demonstrations in a lab---a much more expensive and indirect way to capture human expertise. Through experiments on 27 objects with a 30-DoF simulated robot hand, we demonstrate that DexVIP compares favorably to existing approaches that lack a hand pose prior or rely on specialized tele-operation equipment to obtain human demonstrations, while also being faster to train.

Dex Learning Dexterous Grasping with Human Hand Pose Priors from Video
Priyanka Mandikal 1.2 and Kristen Grauman 1.2
Department of Computer Science, The University of Texas at Austin Faccbook Al Research
(mandikal, grw.man) Baraamman@eaaaa @cs. utexas. edu
Abstract: Dexterous multi- culli--fineeed robotic hands have formidable action space, yet their morphological similarity to the human hand holds immense potential accelerate robot learning. We propose Dex VIP, approach to learn dexterous robotic grasping from human-object interactions present in-the-wild YouTube videos. We do this by curating grasp images from human-object interaction videos and imposing a prior over the agent's hand pose when learning to grasp with deep reinforcement learning. A key advantage of our method is that the learned policy is able to leverage free-form in-the-wild visual data. As result, can easily scale to new objects, and it sidesteps the standard practice of collecting human demonstrations a lab a much more expensive and indirect way to capture human expertise. Through experiments on 27 objects with 30-DoF simulated robot hand, we demonstrate that Dex VIP compares favorably to existing approaches that lack hand pose prior rely specialized rele-operation equipment to obtain human demonstrations, while also being faster to train.
Keywords: dexterous manipulation, learning from observations, learning from demonstrations, computer vision
Introduction
Many objects in everyday environments are made for human hands. Mugs have handles grasp; the stove has dials to push and turn; a needle has small hole through which to weave a tiny thread. In order for robots to assist people in human-centric environments, and in order for them to reach new levels of adept manipulation skill, multi-fingered dexterous robot hands are of great interest as a physical embodiment [1, 2, 3, 4, 5, 6, 7]. Unlike common end effectors like parallel jaw grippers or suction cups, dexterous hand has the potential to execute complex behaviors beyond pushing, pulling, and picking, and to grasp objects with complex geometries in functionally useful ways [8, 9].
The flexibility of a dexterous robotic hand, however, comes with significant learning challenges. With articulated joints offering 24 to 30 degrees of freedom (DoF), the action space is formidable. At the same time, interacting with new objects having unfamiliar shapes demands a high level of generalization. Both factors have prompted exciting research in deep reinforcement learning (RL), where an agent dynamically updates its manipulation strategy using closed-loop feedback control with visual sensing while attempting interactions with different objects [2, 5, 7].
To mitigate sample complexity- since many exploratory hand pose trajectories will yield no reward current methods often incorporate imitation learning [1, 2, 3, 4. 10]. With imitation, expert (human) demonstrations provided by teleoperation in virtual reality [11, 12], mocap [1, 13], or kinesthetic manipulation of the robot's body [14, 15] are used steer the RL agent towards desirable state-action sequences. Such demonstrations can noticeably accelerate robot learning.
However, the existing paradigms for human demonstrations have inherent shortcomings. First, they require some degree of specialized setup: a motion glove for the human demonstrator to wear, a virtual reality platform matched to the target robot, high precision hand and arm tracker, and/or physical access to the robot equipment itself. This in turn restricts demonstrations to lab environments, assumes certain expertise and resources, and entails repeated overhead to add new manipulations
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Watch
Act
We learn dexterous grasping by watching human-object interactions in YouTube how-to videos. Using hand poses extracted from a repository of curated human grasp images (left), train dexterous robotic agent to learn to grasp objects in simulation (right). The key benefits include improved grasping performance and the ability to quickly scale the method to new objects.
Figure I: Main idea:
or objects. Second, there is an explicit layer of indirection: in conventional methods, a person does not do task with their own hands, but instead enacts proxy under the constraints of the hardware/software used to collect the demonstration. For example, in VR, the person needs to watch a screen to judge their success in manipulating simulated hand and may receive limited or no force feedback 12, 16]. Similarly, advanced visual teleoperation systems that observe the bare hand [13] still separate the person's ss hand from the real-world object being manipulated. In kinesthetic demonstration, the human expert is furthest removed, manually guiding the robot's end effector, which has very different embodiment [15, 17].
In light of these challenges, we propose to learn dexterous robot grasping policies by watching people interact with objects in video (see Fig. 1). The main idea is to observe people's hand poses as they use objects in the real world in order to establish a 3D hand pose prior that robot might attempt to match during functional grasping. Rather than enlist special purpose demonstrations (e.g., record videos at lab tabletops), we turn to on-the-wild Internet videos as the source of the visual prior. Our method automatically extracts human hand poses from video frames using state-of-the-art computer vision technique. We then define a deep reinforcement learning model that augments grasp success reward with reward favoring human-like hand poses upon object contact, while also preferring to grasp the object around affordance regions predicted from an image-based model. In short, by watching video of people performing everyday activities with variety of objects, our agents learn how to approach objects effectively using their own multi- finle--fingeee hand, while also accelerating training.
Aside from providing accurate policies and faster learning, our approach has several conceptual advantages. First, it removes the indirection discussed above. There is no awkwardness or artificiality of VR, kinesthetic manipulations, etc,; people in the videos are simply interacting with objects in the context of their real activity. This also promotes learning functional grasps those that prepare the object for subsequent use-as opposed to grasps that simply lift an object in an arbitrary manner. New demonstrations are also easy to curate whenever new objects become of interest, since it is a visual model focuses on 3D hand
matter of downloading additional video. In addition, because pose, it tolerates viewpoint variation and is agnostic to the complex visual surroundings in training data (e.g.. variable kitchens, clutter, etc.). Finally, the proposed method requires only visual input from the human data no state-action sequences.
We demonstrate our approach trained with frames from YouTube how-to videos to learn grasps for 30-DoF robot in simulation for variety of 27 objects. The resulting policy outperforms state-of-the art methods for learning from demonstration and visual affordances [2, 18), while also being 20% more efficient to train. The learned behavior resembles that of natural human-object interactions and offers an encouraging step in the direction of robot learning from in-the-wild Internet data.
2 Related Work
Learning to grasp objects Early grasping work explicitly reasons about an object's 3D shape against the gripper [19, 20], whereas learning methods often estimate an object or hand pose followed by model-based planning [21, 8], optionally using supervised learning on visual inputs to predict successful grasps [21, 22, 23, 24, 25]. Most planning methods cater to simple non non-dexterous end- effectors like parallel jaw grippers or suction cups that make a control policy easier to codify but need not yield functional grasps. Rather than plan motions to achieve grasp. reinforcement learning (RL) methods act a closed loop with sensing, which has the advantage of dynamically adjusting to object conditions [26, 27, 28]. Only limited work explores RL for dexterous manipulation [5. 7, 18). Our work addresses functional dexterous grasping with RI., but unlike the existing methods it primes agent behavior according to videos of people.
2

---

Imitation and learning from demonstration To improve sample complexity, imitation learning
from expert demonstrations is often used, whether for non-dexterous [29, 30, 31, 32, 33, 15] or dexterous [1, 2, 3, 4] end effectors. Researchers often use demonstrations to explore dexterous manipulation in simulation [2. 3]. and recent work shows the promise of sim2real transfer [11, 1 7]. Like learning from demonstrations (LfD), our approach aims to learn from human experts, but unlike traditional LID, our method does so without full state-action trajectories, relying instead only on a visual prior for "good" hand states. Furthermore, our use of in-the-wild video the source of human behavior is new and has all the advantages discussed above.
Imitating visual observations Learning to imitate observations [34] relaxes the requirement of capturing state sequences in demonstrations. This includes ideas for overcoming viewpoint differences between first and third person visual data [33, 30, 31], multi-task datasets to learn correspondences between video of people and kinesthetic trajectories of robot arm [15], few-shot learning [29, 32], and shaping reward functions with video [33, 35, 15, 31]. However, none of the prior work uses in-the-wild video to learn dexterous grasping as we propose. By connecting real video of human- object interactions to dexterous robotic hand, our method capitalizes on both the naturalness of the demonstrations as well as the near-shared embodiment. Furthermore, unlike our approach, the existing (almost exclusively non-dexterous) methods require paired data for the robot and person's ' visual state spaces [33, 15, 3]. assume the demonstrator and robot share a visual environment (e.g., a lab tabletop) [29, 32, 30, 31, 3], and/or tailor the imitation for specific object [3].
Visual object affordances As dual hand pose. priors for the object regions that afford an interaction can also influence grasping. Vision methods explore learning affordances from visual data [36, 37, 38), though they stop short of agent action. Visual affordances can successfully influence pick and place robot [39, 40] and help grasping with simple grippers [9, 21, 23, 24] or dexterous hand [8, 18]. Object-centric affordances predicted from images also benefit the proposed model, but they are complementary our novel contribution-thh the hand-pose prior learned from video.
Estimating hand poses Detecting human hands and their poses is explored using a variety of visual learning methods [41, 42, 43, 44, 45]. Many methods jointly reason about the shape of the object being grasped [43, 9, 46, 47, 46]. Recent work provides large-scale datasets better understand human hands [48, 49, 50]. We rely on the statc-of-the -fet---rtr FrankMocap method [44] to extract 3D hand poses from video. Our contribution not a computer vision method to parse pose, but rather a machine learning framework to produce dexterous robot grasping behavior.
3 Approach
We consider the task of dexterous grasping with an articulated 30- R0-DoF multi-fingeret robotic hand. Our goal is leverage abundant human interaction videos to provide the robot with a prior over meaningful grasp poses. To this end, we propose DEX, an approach to learn Dexterous grasping using Video Informed Pose priors. We first lay out the formulation of the reinforcement learning problem for dexterous grasping (Section 3.1). Then, we describe how we leverage human hand pose priors derived from in-the-wild YouTube videos for this task (Section 3.2).
3.1 Reinforcement Learning Framework for Dexterous Grasping
Background Our dexterous grasping task is structured reinforcement learning (RL) problem, where an agent interacts with the environment according to a policy order to maximize specified reward (Fig. 2). At each time step I, the agent observes the current observation on and samples an action ae from its policy т. It then receives a scalar reward R&+1 and next observation o:+1 from the environment. This feedback loop continues until the episode terminates at T time steps. The goal of the agent is to determine the optimal stochastic policy that maximizes the expected sum of rewards.
Observations Our task setup consists of robotic hand positioned above tabletop, with an object of interest resting on the table. At the start of each episode, we sample an object and randomly rotate it from its canonical orientation. The observations of (Fig. 2, green block) at each time step t are a combination of visual and motor inputs to the robot. The visual stream is from an egocentric hand-mounted camera. It consists of an RGB image of the scene 1;" and the corresponding depth map DJ. Additionally, we provide a binary affordance map A that inferred from I using an affordance prediction network [18] to guide the agent towards functional grasp regions the object. The motor
3

---

I,
D,"
a1
A
Actor
Environment
RGB
Depth
M
A Affordance
Robot Pose
P
Critie
Pi+2
Reward
far
Proprioception
P.
d,
Per
Hand-Obj
Human
Estimated
Target Robot Pose
Grusp Image
Human Pose
Observations
Policy
Pose Prior
Environment
Figure 2: Overview of DEX We use grasp poses inferred from Internet video to train a dexterous grasping policy. An actor-critic network (blue) processes sensory observations from visual and motor streams (green) to estimate agent actions. Human hand pose priors derived from how-to videos (red) encourage the agent explore worthwhile grasp poses via an auxiliary reward (purple).
inputs are a combination of the robot proprioception Pr and the hand-object contact distances di- P. comprises the robot joint angles or pose pi and angular velocities of of the actuator, while di is the pairwise distance between the object affordance regions and contact points on the hand. The latter assumes the object tracked in 3D once its affordance region is detected, following [18]. The agent also has 21 touch sensors 7" spread uniformly across the palm and fingers.
Action space At each time step t, the policy processes observations of and estimates actions a--30 continuous joint angle values values-which are applied at the joint angles in the actuator. The robotic manipulator we consider is the Adroit hand [51], 30-DoF position-controlled dexterous hand. With a five-fingered 24-DoF actuator attached to a 6-DoF arm, the morphology of the robot hand closely resembles that of the human hand. This congruence opens up exciting avenue infuse grasp pose prior learned from human-object interaction videos, as we present later in Sec. 3.2.
Feature and policy learning We adopt actor-critic model for learning the grasping policy. The visuo-motor observations of are processed separately using two neural networks, fv and fas (Fig. 2, blue block). Specifically, the visual inputs encompassing (T4. D4. AA concatenated and fed to a three-layer CNN fv that encodes them to obtain a visual embedding Vz. The motor stream comprised of PP" d; is processed by two-layer fully connected network fir that encodes them to a motor embedding M, Finally V. and M, are concatenated and fed to the actor and critic networks estimate the policy distribution To (a lo," and state values (o()), respectively, each time step. To(a
The resulting policy outputs a 30-D unit-variance Gaussian whose mean is inferred by the network; we sample from this distribution obtain the robot's next action af.
We train the complete RL network with PPO [52] using a reward that encourages successful grasping, touching object affordance regions, and mimicking human hand poses,
will detail below.
Robot hand simulator We conduct experiments in MuJoCo [53], a physics simulator commonly used in robotics research. Due to lack of access to real robotic hand, we perform all experiments simulation. The successful transfer of dexterous policies trained purely simulation to the real learning in research today. In addition,
world [11, 7, 6] supports the value of simulator- conduct numerous experiments with noisy sensing and actuation settings that might occur in the real world to illustrate the robustness of the policy to non-ideal scenarios (see Sec. 4 and Supp.).

---

Human grasp images and poses
Robot grasp poses and actions
Ps
Pa
Po-
Figure 3: Human hand pose priors informing the agent action. Each row shows three example images and extracted human hand poses for each object category (left) and the corresponding consensus robot hand pose rewarded by our method and its application by the agent in action (right).
3.2 Object-Specific Human Hand Pose Priors from Video
We now describe how we leverage human-object interaction videos to enable robust dexterous grasping. The morphological similarity between the human hand and the dexterous robot holds immense potential to learn meaningful pose priors for grasping. To facilitate this, we first curate a human-object interaction dataset of video frames to infer 3D grasp hand poses for variety of objects. We then transfer these poses to the robotic hand using inverse kinematics, and finally develop a novel RL reward that favors human-used poses when learning grasp. The proposed approach allows us to leverage readily available Internet data to learn robotic grasping.
Object dataset We consider household objects often encountered in daily life activity (and hence online instructional videos) and for which 3D models are available. We acquire 3D object models from multiple public repositories: ContactDB [37], 3DNet [54], YCB [55], Free 3D [56], and 3D Warchouse [57]. We specifically include the 16 ContactDB objects with one-hand grasps used in recent grasping work to facilitate concrete comparisons [18]. We obtain a total of 27 objects to be used for training the robotic grasping policy, all 16 from ContactDB plus 11 additional objects.
Video frame dataset We use the HowTo1001 dataset [58] to curate images containing human- object grasps for the objects of interest. HowTo100M a large-scale dataset consisting of 13.6 M instructional YouTube videos across categories such as cooking, entertainment, hobbies and crafts, etc. We focus on videos consisting of commonly used household objects-tools and kitchen utensils such as mug, hammer, jug, etc. The idea is to capture objects in active use during natural human interactions so that we can obtain functional hand poses. The grasp images contain the object in its canonical upright position (c.g.. pan stove), which is also the initial vertical orientation of the object on the tabletop in the simulator. Using the above criteria, we curate an object interaction repository In of 715 video frames from How 100M where the human hand is grasping total objects, to yield on average 26 grasp images per object. For instance, to collect expert data for grasping a pan,
of the 27
curate grasp images from task ids such as "care for nonstick pans", "buy cast iron pans", etc. While we found it effective to use simple filters based on the weakly labeled categories and specific task ids in HowTo100M4 the curation step could be streamlined further by deploying vision methods for detecting hands, actions, and objects in video [48, 59, 58).
Target hand pose acquisition We propose to use the obtained HowTo.10M grasp images I, to provide learning signal for robotic grasping. To that end, for each image we first infer its 3D human hand pose p" In particular, we employ FrankMocap [44] to estimate 3D human hand poses (see Fig. 3, left). FrankMocap is a near real-time method for 3D hand and body pose estimation from monocular video; it returns 3D joint angles for detected hands in each frame. While alternative pose estimation methods could be plugged in here, we use FrankMocap in our implementation due to its efficiency and good empirical performance. We keep right hand detections only, since robot is one-handed; we leave handling bimamual grasps for future work.
This step yields collection of different hand poses per object for variety of objects found in the videos. Let P(c) = [p2...... P
be the set of human hand poses associated with object class e. The poses within object class are often quite consistent since the videos naturally portray people
5

---

using the object in its standard functional manner, e.g., gripping pot handle elicits the same pose for most people. However, some objects elicit a multi-modal distribution of poses (e.g., a knife held with or without an index finger outstretched). See Fig. 3. In order to automatically discover the "consensus" pose for an object, we next apply k-medoid clustering on each set P(c). We consider the medoid hand pose of the largest cluster be the consensus target hand pose ph- and use its associated target robot pose P--- (obtained using joint re-targeting mechanism described in Supp. Sec. D) during policy learning for object c.
reward function To exert the video prior's influence in our RL formulation, we incorporate an auxiliary reward function favoring robot poses similar the human ones in video. In this way, the reward function not only signals where to grasp a particular object, but also guides the agent on how grasp effectively. To realize this, we combine three rewards: Rsace (positive reward when the object is lifted off the table), Rass (negative reward denoting the hand-affordance contact distance obtained from [18], see Supp. Sec. F), and-mos notably-Rppse positive reward when the agent's pose Pł matches the target grasp pose P-- for that object. Our total reward function is:
R = aBscce + BRass + Boose + Tj Rentropp
(I)
where a, B, 7. 17 are scalars weighting the rewards that are set by validation, and Rentrops rewards entropy over the target action distribution to encourage the agent to explore the action space. Through Rass. the agent is incentivized explore areas of the object within the affordance region, while Rpose encourages the agent to reach hand states that are most suitable for grasping those regions.
For Rpose, we compute the mean per-joint angle error time between the robot joints p4 and the target grasp pose pi-.. We ignore the azimuth and elevation values of the arm in the pose error since they are specific to the object orientation in 1h, which may be different from the robot's viewpoint I", This provides more flexibility during object reaching to orient the arm while trying to match the hand pose alone. Furthermore, Rpose is applied only when 30% of the robot's touch sensors are activated. This encourages the robot assume the target hand pose once it is close to the object and in contact with it. Following [18], Rass is computed as the Chamfer distance between points on the hand and points on the inferred object affordance region. See Supp. Sec. B for reward function details.
The proposed approach permits imitation by visual observation of the human how-to videos, yet without requiring access to the state-action trajectories of the human activity. Its mode of supervision is therefore much lighter than that of conventional teleoperation or kinesthetic teaching, as we will see in results. Furthermore, because our model can incorporate priors for new objects by obtaining new training images, it scales well to add novel objects.
4 Experiments
We present experiments to validate the impact of learning pose priors from video and to gauge DEXVIP'S performance relative to existing methods and basclines.
Compared methods We compare to the following methods: (1) CoM: uses the center of mass of the object as a grasp location prior, since this location can lead to stable grasps [60]. We implement
a this by penalizing the land-CoM distance for Raff in Eqn 1, and removing Rpose. (2) TOUCH:
uses only the touch sensors T" on the hand to positively reward the agent +1 for object interaction when 30% of them are activated, but imposes no supervision on the hand pose. (3) GRAFF [18]: a state-of-the-e RL grasping model that trains a policy to grasp objects at inferred object-centric affordance regions. Unlike our method, GR AFF does not enforce any prior over the agent's hand pose. All the above three RL methods use the same architecture as our model for the grasping policy, allowing for a fair comparison. (4) DAPG [2]: is hybrid imitation+RL model that uses motion-glov demonstrations collected from a human expert in VR. It is trained with object-specific mocap demonstrations collected by [18] for grasping ContactDB object (25 demos per object). For objects beyond ones in ContactDB, we use demos from the object most similar in shape in ContactDB.
Metrics We report four metrics: (1) Grasp Success: when the object is lifted off the table by the hand for at least the last 50 time steps (a quarter of the episode length) to allow time to reach the object and pick it up. (2) Grasp Stability: the firmness with which the object is held by the robot, discounting grasps in which the object can easily be dropped. We apply perturbation forces of Newton six orthogonal directions on the object after an episode completes. If the object continues to be grasped

---

CoM
Touch
DAPG
GRAFF
DexVIP (Ours)
No Noise
Noise
80%
80%
60%
60%
Sucoess
40%
40%
20%
0%
0%
40M
BOM
120M
Success
Stability
Punctionaiiy
Posture
Samples
Figure 5: Grasping success and learning speed. Left: The proposed model outperforms all the baselines, including recent methods for learning from demonstrations (DAPG [2]) or favoring visual affordance regions (GRAFF [18].. Relative results remain stable under noisy sensing and actuation (shaded bars). Right: Our model trains faster than the others, showing that the Internet videos of people help kickstart learning even before the agent begins attempting its own grasps.
by the agent, the grasp is deemed stable. (3) Functionality: the percentage of successful grasps in which the hand lies close to the GT affordance region. This metric evaluates the utility of the grasp for post-grasp functional use. (4) Posture: the distance between the target human hand pose pa, and the agent hand pose after a successful grasp pf. It tells us how human-like the learned grasps are. We normalize all metrics on a [0, 100%] scale, where higher better. We evaluate 100 episodes per object with the objects placed at different initial orientations ranging from [0,180%] We report the mean and standard deviation of the metrics across all models trained with four random seeds.
Implementation details The visual encoder fv has filters of size [8. (8.4.3] and a bottleneck 512-D and uses ReLU activations. The motor encoder fM has dimensions [512,512]. For the hand-object contacts, we use 10 and 20 uniformly sampled points on the hand and affordance region, respectively, A single
following [18]. The entire network optimized using Adam with learning rate of 5e grasping policy is trained on all the curated objects for 150M agent steps with an episode length of 200 time steps. The coefficients in the reward function (Eq. 1) are set as: a = 1, B = 1, = 1. 0.001. We train for four random seed initializations. Further details provided in Supp. Sec. E.1.
Grasping policy performance We take the policy trained on all 27 objects and first evaluate it on the 16 objects from ContactDB [37]. Since these objects have ground truth affordances (used when training GRAFF and our model) and mocap demonstrations (used when training DAPG), they represent the best case scenario for the existing models to train with clean expert data. Note that our method always uses hand poses inferred from YouTube videos, even for the ContactDB objects.
Fig (left) shows the results. DIX consis- tently outperforms all the methods on all met- rics. The grasp success and stability rates ex- perience significant boost even compared to
pan
mug
teapot
Imiife
cup
hammer
CoM
GRAFF [18], which utilizes object affordances but does not enforce any constraints on the hand pose. The Functionality values are similar as both the methods encourage the agent to grasp the object at the affordance regions. Our method also scores well on the Posture metric, indicat- ing that the learned policies indeed demonstrate human-like behavior during grasping. See Supp. Sec. G and Sec. E.2 for additional re- sults and Sec. C for TSNE plots illustrating our model's himan-like poses.
Touch
DAPG
DeXVIP GRAPF
Fig. shows grasp policies from sample episodes. We can see that DEX is able to grasp objects with a human-like natural posture compared to the other methods. Contrast this with CoM or GRAFF, which only uses object affordances: those agents often end up in un-
Figure Grasping performance. Example frames for the grasping task. Our DEXVIP policy guided by pose-priors is able to successfully grasp objects in natural human-like poses, while the other methods may either generate unusual poses or fail to grasp effectively. Please Supp. video.
7

---

usual poses which reduce their applicability to post-grasp functional tasks using the object. Failure cases arise when some objects are in orientations not amenable to the target hand pose. For instance. a knife with the handle on the left would ideally be picked up differently and then re-oriented for use.
Training speed Fig. 5 (right) shows the training curves. Our method learns successful policies 20% faster than the next best method, GRAFF [18]. This underscores how the hand pose priors enable our agent approach objects in easily graspable configurations, thus improving sample efficiency.
Expert data: teleoperation vs. YouTube video Compared to traditional demonstrations, a key advantage of our video-based approach is the ease of data collection and scalability to new objects. We analyze the time taken to collect demonstrations for DAPG [2] versus the time taken to curate videos for DEXVIP. On average, it takes 5 minutes to collect single demonstration for DAPG owing to the complex setup, while a video or image for DEXVIP is collected in a few seconds. Fig. 6 quantifies the impact of the efficiency of human demonstrations for our model (trained with video frames) compared to traditional state-action demonstrations (trained with VR demos). Plotting success rates as function of accumulated demo experience, we see how quickly the proposed image- based supervision translates to grasping success on an increasing number of new objects, whereas with traditional demonstrations reaching peak performance takes much longer. This highlights the significant gains that can be realized by shifting from tele-op to video supervision for robot learning.
This efficiency also means objects are quick to add. To illustrate, we further evaluate DEXVIP and DAPG on all 11 non-ContactDDB objects, for which mo- cap demonstrations are not available. DAPG's success rate drops from 59% to 50%-2 15% relative drop in performance-while DEXVIP experiences a marginal 4% drop from 68% to 65%, remaining comparable to its performance on ContactDB. Since DAPG is trained on sub-optimal demonstrations, this precludes it from generalizing well to objects for which expert data is ab- sent. DEXVIP. on the other hand, is able to benefit from easily available Internet images to scale up supervision.
80%
vidoc
teleoperation
60%
Success
40%
20%
0%%
10
20 30
Demonstration Time (hrs)
Demo-time vs success rate. Dex VIP benefits from easily available In- ternet data to scale up supervision.
Figure 6:
Ablations Next investigate how different compo- nents of the reward influence the dexterous grasping policy. Using only Rafs, the success rate is 60% on ContactDB. When we successively add touch and hand pose priors to the reward function (Eq. 1), we obtain success rates of 63% and 68% respectively. Thus our full model is the most effective.
Noisy sensing and actuation Finally, to mimic non-ideal real-world scenarios, we induce multiple sources of noise into agent's sensory and actuation modules during training and testing following prior work [11. 7, 6, 18]. These include Gaussian noise on the robot's proprioception P.", object tracking di. and actuation af, well as pixel perturbations the image observations 47. Under heavy noise, DEXVIP still yields a grasp success rate of 64%, even outperforming noise-free models of the other methods (Fig. 5, shaded bars). This encouraging result in noise-induced simulation environment lends support for potentially transferring the learned policies to the real world [6, 7, 61] were we to gain access to dexterous robot. Please see Supp. Sec. A for more details.
5 Conclusion
We proposed an approach to learn dexterous robotic grasping from human hand pose priors derived from video. By leveraging human-object interactions in YouTube videos, we showed that our dex- terous grasping policies outperformed methods that did not have access to these priors, including two state-of-the-an models for RL-based grasping- Key advantages of our approach are 1) humans are observed directly doing real object interactions, without the interference of conventional demon- stration tools; and 2) expert information from video sources scales well with new objects. This is an encouraging step towards training robotic manipulation agents from weakly supervised and casily scalable in-the-wild expert data available on the Internet. In the future, we are interested in expanding the repertoire of tasks beyond grasping to learn fine-grained manipulation skills from human interaction videos.
8

---

Acknowledgments
UT Austin is supported by DARPA L2M, the IFML NSF AI Institute, and NSF IIS -1514118. K.G. paid as a research scientist at Facebook Al Research. We would like to thank the reviewers for their valuable feedback.
References
[1] A. Gupta, C. Eppner, S. Levine, and P. Abbeel. Learning dexterous manipulation for a soft robotic hand from human demonstrations. In International Conference on Intelligent Robots and Systems (IROS), 2016.
[2] A. Rajeswaran, Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Robotics: Science and Systems (RSS), 2018.
[3] D. Jain, A. Li, S. Singhal, A. Rajeswaran, V. Kumar, and E. Todorov. Learning deep visuomotor policies for dexterous hand manipulation. In International Conference on Robotics and Automation (ICRA). 2019.
[4] H. Zhu, A. Gupta. A. Rajeswaran, S. Levine, and V. Kumar. Dexterous manipulation with deep reinforce-
ment learning: Efficient, general, and low-cost. In International Conference on Rohotics and Automation
(ICRA), 2019.
[5] A. Nagabandi, K. Konoglie, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. Conference on Robot Learning (CoRL), 2019.
[6] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's 's cube with a rohot hand. arXiv preprint arXiv: 1910.07113, 2019. [7] OpenAL, M. Andrychowicz. B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki. A. Petron, M. Plappert. G. Powell, A. Ray. et al. Learning dexterous in-hand manipulation. The International Journal of Rubotics Research, 2020.
[8] S. Brahmbhatt, A. Handa, J. Hays, and D. Fox. ContactGrasp: Functional multi-finger grasp synthesis from contact. IROS, 2019.
[9] Kokic, D. Kragic, and Bohg. Learning task-oriented grasping from human activity datasets. JEEE Robotics and Automation Letters, 2020.
[10] I. Radosavovic, X. Wang, I.. Pinto, and J. Malik. State-only imitation lcarning for dexterous manipulation.
arXiv:2004.00 040000004460w,,220 2020.
[11] Y. Zhu, Z. Wang. J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool. J. Kramár, R. Hadsell, N. de Freitas, ct al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv: 184204440.09 2018.
[12] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforce-
ment learning with demonstrations. arXiv: 10089, 2017.
[13] A. Handa, K. Van Wyk, W. Yang. J. Liang, Y.W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox. Dexpilot: Vision based telcoperation dexterous robotic hand-arm system. ICRA, 2020.
[14] M. Vecerik, T. Hester, J. Scholz, F. Wang. O. Pietquin, B. Piot, N. Heess, T. Rothorl, T. Lampe, and
M. Riedmiller. Leveraging demonstrations for deep reinforcement learning robotics problems with sparse rewards. arXiv: 7007070877777777 2018.
[15] P. Sharma, L. Mohan, L. Pinto, and A. Gupta. Multiple interactions made casy (mime): Large scale demonstrations data for imitation. In Conference on Robot Learning (CoRL), 2018.
[16] V. Kumar and E. Todorov. Mujoco haptix: A virtual reality system for hand manipulation. In IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), 2015.
[17] P. Sharma, D. Pathak, and A. Gupta. Third- person visual imitation learning via decoupled hicrarchical
controller. In NeurlPS, 2019.
[18] P. Mandikal and K. Grauman. Dexterous robotic grasping with object-centric visual affordances. In arXiv preprint Xrx:::2009.0 01409, 2020.
[19] M. Ciocarlie, C. Goldfeder, and P. Allen. Dimensionality reduction for indeeindeeendent dexterous robotic
grasping. In IROS, 2007.
[20] A. Bicchi and V. Kumar. Robotic grasping and contact: A review. In IEEE International Conference on Robotics and Automation, 2000.
[21] S. Levine, Pastor, A. Krizhevsky, Ibarz, and D. Quillen. Learning hand-eye aand-eye coordination for robotic grasping with deep learning and large-scale seale data collection. IJRR, 2018.
[22] C. Wu, Chen, Q Cao. J. Zhang, Y. Tai, I. Sun, and K. Jia. Grasp proposal networks: An end-to-end solution for visual learning of robotic grasps. In NeurIPS, 2020.
[23] Lenz, H. Lee, and A. Saxena. Deep learning for detecting robotic grasps. The International Journal of
Robotics Research, 2015.
[24] J. Redmon and A. Angelova. Real-time grasp detection using convolutional neural networks. In IEEE
International Conference on Robotics and Automation (ICRA), 2015.
[25] J. Mahler, J. Liang. S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. 2017.

---

[26] D. Kalashnikov, A. Irpan. P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen. E. Holly, M. Kalakrishnan, V. Vanhoucke, al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on Robot Learning (CoRL). 2018.
[27] D. Quillen, E. Jang, O. Nachum, C. Finn, J. Ibarz, and S. Levine. Deep reinforcement learning for vision-bascd based robotic grasping: A simulated comparative evaluation of off-policy methods. In 2018 IEEE International Conference
Robotics and Automation (ICRA). 2018.
[28] H. Merzić, M. Bogdanović, D. Kappler, L.. Righetti, and J. Bohg. Leveraging contact forces for learning grasp. In International Conference on Robotics and Automation (ICRA), 2019.
[29] C. Finn, T. Yu, T. Zhang, P. Abheel, and S. Levine. One-shot visual imitation learning via meta-lcarning.
In Conference on Robot Learning (CoRL), 2017.
[30] B. Stadie, P. Abbeel. and I. Sutskever. Third-person imitation learning. In ICLR, 2017.
[31] Y. Liu, A. Cupta, P. Abbeel, and S. Levine. Imitation from observation: Learning to imitate behaviors from
raw vidco via context translation. In ICRA, 2018.
[32] D. Pathak, P. Mahmoudich, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell, Zero-shot visual imitation. In ICLR, 2018.
[33] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, and S. Levine. Time-contrastive networks:
self-supervised learning from video. In ICRA, 2018.
[34] F. Torabi, G. Warnell, and P. Stone. Recent advances in imitation learning from observation. In IJCAI,
2019.
[35] W. Goo and S. Nickum. One-shot learning of multi-step tasks from observation via activity localization in auxiliary video. ICRA, 2019.
[36] T.-T. Do, A. Nguyen, D. G. C. lan Reid, and N. G. Tsagarakis. Affordancenet: An end-to-end deep learning
approach for object affordance detection. In ICRA, 2017.
[37] S. Brahmbhatt, C. Ham, C. C. Kemp, and J. Hays. Contactdb: Analyzing and predicting grasp contact via thermal imaging. In IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[38] T. Nagarajan. C. Feichtenhofer, and K. Grauman. Grounded human-object interaction hotspots from video. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
[39] H. Wu, Z. Zhang, H. Cheng, K. Yang. J. Liu, and Z. Guo. Learning affordance space in physical world for vision-based robotic object manipulation. In International Conference on Robotics and Automation (ICRA), 2020.
[40] L. Yen-Chen A. Zeng, S. Song, P. Isola, and T.-Y. Lin. Learning to see before lcarning to act: Visual pre-training for manipulation. In IEEE ICRA, 2020.
[41] M. Cai, K. M. Kitani, and Y. Sato. Understanding hand-object manipulation with grasp types and object attributes. In Conference on Robotics Science and Systems (RSS), 2016.
[42] Y. Zhang. N. Boddeti, and K. Kitani. Gesture-based bootstrapping for egocentric hand segmentation.
arXiv:1612.0288,, 2018.
[43] Y. Hasson, G. Varol, D. Tzionas, I. Kalevatykh, M. J. Black. I. Laptev, and C. Schmid. Learnnng joint reconstruction of hands and manipulated objects. In CVPR, 2019.
[44] Y. Rong, T. Shiratori, and H. Joo. Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration. arXiv preprint arXiv:2008.00 08332, 2020.
[45] Y. Zhou, M. Habermann, W. Xu, I. Habibic, C. Thcobalt, and F, Xu. Monocular realtime hand shape and motion capture using multi-modal data. In CVPR, 2020.
[46] Y.W. Chao, W. Yang. Y. Xiang. P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal,
S. Birchfield, J. Kautz, and D. Fox. DexYCB: A benchmark for capturing hand grasping of objects. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
[47] Z. Cao, I. Radosavovic,, A. Kanazawa, and J. Malik. Reconstructing hand-object interactions in the wild.
2xiv:2002..9856611 2020.
[48] D. Shan, J. Geng, M. Shu, and Fouhey, Understanding human hands in contact at internet scale. In CVPR, 2020.
[49] S. Brahmbhatt, C. Tang, C. D. Twigg, C. Kemp, and J. Hays. ContactPose: A dataset of grasps with object contact and hand pose. In The European Conference on Computer Vision (ECCV), 2020.
[50] O. Taheri, N. Ghorbani, M. Black, and D. Tzionas. Grab: A dataset of whole-body human grasping of objects. In ECCV, 2020.
[51] V. Kumar, Z. Xu, and E. Todorov. Fast, strong and compliant pneumatic actuation for dexterous tendon- driven hands. In IEEE international conference on robotics and automation, 2013.
[52] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms.
arXiv preprint xrviv: 1707,06347. 2017.
[53] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ
International Conference on Intelligent Robots and Systems, 2012.
[54] W. Wohlkinger, A. Aldoma Buchaca, R. Rusu, and M. Vincze. 3DNet: Large-Scale Object Class Recognition from CAD Models. In International Conference on Robotics and Automation (ICRA), 2012.
[55] B. Calli, A. Singh, J. Bruce, A. Walsman, K. Konolige. S. Srinivasa, P. Abbeel, and A. M. Dollar. hale-cmu-berkelee dataset for robotic manipulation research. IJRR, 2017.
[56]
www.free3d.com.
[57]
www.3dwarchouse.cm www.://uusss.etttu..o....m
10

---

[58]
A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi. I. Laptev, and J. Sivie. HowTo100M: Learning
Text- Video Embedding by Watching Hundred Million Narrated Video Clips. In ICCV, 2019. Bojanowski, R. Lajugie, F. Bach, I. Laptev, Ponce, C. Schmid, and Sivie. Weakly supervised action labeling in vidcos under ordering constraints. ECCV, 2014.
[59]
[60] D. Kanoulas, J. Lec, D. G. Caldwell, and N. G. Tsagarakis. Center-of-man mass based grasp pose adaptation
using 3d range and force/torque sensing. International Journal of Humanoid Robotics, 2018. Tobin, Fong, A. Ray. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring
[61]
deep neural networks from simulation real world, IROS, 2017.


Reviews:
[{"review_id": "ptuT-bEBsrI", "reviewer": "Reviewer_HJKw", "summary": "This paper presents DexVIP, a method for learning a policy for dexterous grasping. Grasping with a dexterous, multi-fingered robot hand is a challenging problem; while the hand’s high degrees of freedom (DOF) make it possible to perform behaviors that are beyond simple parallel-jaw grippers, the large action space presents significant learning challenges. The key contribution of DexVIP is to use human hand pose priors when training the grasping policy, to improve sample complexity and produce human-like grasps. Hand pose priors are first extracted using FrankMocap [1] from in-the-wild video demonstrations of humans grasping an object similar to the desired object. By using video demonstrations, DexVIP does not require expensive or difficult-to-use hardware for collecting demonstrations. The human hand pose priors are incorporated as the “R_pose” term in the reward function (Eq. 1) to encourage the robot hand to use the same grasp pose as the human demonstrator. The method is otherwise identical to GRAFF [2], a state-of-the-art dexterous grasping approach. The paper presents experiments with a simulated 30-DOF hand evaluating DexVIP against a number of baselines: GRAFF, a hybrid imitation+RL approach named DAPG [3], and alternative reward terms such as grasping towards the object center of mass and rewarding contact with the object. The paper also provides additional experiments comparing video-based vs. teleoperation-based demonstrations, ablations, and the effect of training with noise. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The proposed method helps pave the way towards dexterous robot manipulation\n- The data used for training the pose prior can be collected from in-the-wild videos and is hence easy to collect and easily scalable. This data involves people directly interacting with objects, as opposed to the more indirect demonstration approaches of VR, teleoperation, or kinesthetic teaching.\n- The proposed method is somewhat invariant to viewpoint variation, since the reward focuses on the pose of the individual hand joints (regardless of the position / orientation of the hand itself)\n- The paper evaluates DexVIP against existing methods and shows that DexVIP outperforms them on all evaluation metrics. Specifically, DexVIP slightly outperforms other methods on grasp success (% grasps that lifted the object for more than 50 timesteps), stability (% grasps robust to perturbations), and functionality (% grasps close to a ground-truth affordance region of the object). DexVIP achieves significant outperformance on posture (normalized distance of robot grasp pose to human grasp pose), a metric its additional reward term is designed to maximize.\n- DexVIP achieves better performance with less training time (Fig 5, right) and less demonstration time (Fig 6) compared to the alternative methods. \n- The paper provides ablation experiments demonstrating the importance of the hand pose reward term, as well as experiments showing that DexVIP is robust to observation and actuation noise during training.\n- The paper is well-structured and written clearly. \n\nMajor Weaknesses:\n\n**Scalability is limited by ContactDB data:**\nThe paper argues in lines 49-59, 65-66, and 303-326 that DexVIP scales better to new objects because of its use of video demonstrations for extracting hand pose priors (“New demonstrations are also easy to curate whenever new objects become of interest, since it is a matter of downloading additional video.”). However, the reward used for DexVIP still requires R_aff, the object affordance regions. If the object affordance maps require expensive human demonstrations with thermal cameras (on 3D-printed objects) for each object or object class, then this argument does not hold; the scalability would still be limited by the ability to collect this affordance data. The following additional experiments should be performed to measure the true scalability of DexVIP:\n\n1) Train DexVIP without the affordance reward term and show the performance (this would show how scalable the method is based only on Youtube data)\n\n2) Assuming that DexVIP makes use of the “Image affordance prediction model” from GRAFF (and this should be explicitly stated in the DexVIP paper), experiments should be added to show the performance of DexVIP on unseen object instances and unseen object classes for which ContactDB data has not been obtained. These experiments would show that additional thermal ContactDB data is not added for new instances or classes, and that the proposed method can rely on the generalization of the affordance prediction, as well as the scalability of Youtube videos. For the novel object instances, I believe that this can be achieved by training and evaluating only on the 11 non-ContactDB objects.This experiment is nearly done in the section “Expert data: teleoperation vs. YouTube video”, but that experiment only compares the performance of DexVIP and DAPG instead of comparing all methods). For novel object classes, an additional experiment should be added to show the scalability of DexVIP to novel classes for which ContactDB data is not available.\n\n**Limited technical novelty:**\nDexVIP has limited technical novelty, as it is mostly a combination of existing methods: GRAFF + FrankMocap.\n\n**Minimal performance improvement:**\nThe results show only a small improvement over GRAFF, especially in grasping success, stability, and functionality, which are the most important metrics.  The ablation experiments show \nthat adding the hand pose prior increases grasp success rate from 63 to 68%, a relatively small increase. There is a large improvement in “posture”, but this metric is the least important because it doesn’t really matter whether the robot performs a task in a human-like way, as long as the task is successfully completed. \n\n\n\nMinor Weaknesses\n- Lines 271-272 state: “We take the policy trained on all 27 objects and first evaluate it on the 16 objects from ContactDB.” This seems like an odd procedure; it seems that a more straightforward evaluation would be to just train and evaluate on the 16 objects from ContactDB rather than training on 27 objects and only evaluating on a subset of them.\n- Is the state space between the human and robot hands the same?  This isn’t discussed; if not, how do they bridge the morphological gap for computing the R_pose reward term? It is somewhat surprising to me that FrankMocap and the Adroit hand happen to use the same state space.\n- DexVIP requires both hand pose priors extracted from video for its R_pose metric, as well as object affordance maps for the object affordances metric R_aff. For the 16 ContactDB training objects, the ground truth object affordance maps used in training are obtained from human demonstrations using thermal cameras. For non-ContactDB training objects, it is unclear from the paper how the object affordance maps are obtained. It is crucial to explain how these object affordance maps are obtained. \n- Fig. 6: The caption is not very descriptive compared to the other figure captions. \n- Supp. line 20: For the details in the supplement on the experiment with noisy observations and actions, is the 0.1 variance value in radians? This value does not seem to be used in [4] or [5] for similar noise-injected experiments. \n- Are the objects in the in-the-wild video demonstrations and the object meshes matched manually? This could be clarified in the paper.\n- For the analysis in Appendix C, it would be interesting to see how often grasps that are similar according to standard taxonomies (https://www.is.mpg.de/uploads_file/attachment/attachment/256/grasp_taxonomy.pdf) are grouped together.\n\n[1] Y. Rong, T. Shiratori, and H. Joo. Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration. arXiv preprint arXiv:2008.08324, 2020.\n\n[2] P. Mandikal and K. Grauman. Dexterous robotic grasping with object-centric visual affordances. In arXiv preprint arXiv:2009.01439, 2020.\n\n[3] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Robotics: Science and Systems (RSS), 2018.\n\n[4] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kram r, R. Hadsell, N. de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv:1802.09564, 2018. \n\n[5] OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Extracting hand pose priors from in-the-wild video demonstrations seems promising for learning dexterous grasping with human-like hand poses. However, the current version of the approach seems to rely on other types of demonstrations such as thermal human demonstrations on 3D-printed objects for collecting ground truth object affordance maps, which weakens the claim made in the paper that DexVIP scales better for learning new objects. Further, the technical novelty is limited and the performance improvement compared to prior work is minimal.\n\n# Post-rebuttal update\n\nI would like to clarify that my concerns about the evaluation are not based on a misunderstanding.  The issue is that the 16 objects that were used in the main evaluation in the paper (Figure 5) were in the *training set* for the affordance prediction model.  Because these objects were in the training set, this evaluation does not demonstrate the generalization of the affordance prediction model; contactDB data was acquired and used for these specific objects that were used in the evaluation of Figure 5. The use of contactDB data significantly limits the scalability of the method, and it is unclear from Figure 5 whether the proposed method would still outperform the baselines on objects for which contactDB data is not available.  This is an issue because Figure 5 is the only figure in the main text in which DexVIP is compared to all of the baselines, and it is the only figure in the main text in which DexVIP is evaluated on a full range of metrics.\n\nThis issue is mostly rectified in Supp Sec. G.4 and Figure F. I would recommend to replace Figure 5 with Figure F in the final version of the paper. The downside of the approach used in Figure F, though, is that the functionality metric can no longer be included. Ideally, the paper should evaluate DexVIP on ContactDB objects that are in the affordance prediction model's test set (objects not used for affordance prediction training) so that the functionality metric can also be included, while still demonstrating generalizability to objects beyond those in the affordance prediction model's training set.\n\nOne small typo: the main text refers to \"Supp Sec H\", but the sections in the supplement only goes up to G.\n\nOverall, I think the authors have significantly improved the paper during the rebuttal and addressed most of my concerns; with these updates, I think that the paper represents a solid contribution and thus I will update my recommendation to weak accept. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- Run an ablation removing the object affordance reward R_aff \n- Evaluate all methods only on the 11 objects not in ContactDB\n- Evaluate all methods only on object classes not in ContactDB\n- Clarify how object affordance maps are obtained for non-contact DB objects\n- Clarify the difference between the state space between the human and robot hands\n"}}, {"review_id": "afk2gB4rMGU", "reviewer": "Reviewer_8fsL", "summary": "This paper proposes to guide an RL algorithm for grasping tabletop household objects with a hand pose prior derived from in-the-wild images of humans using those objects. Experiments performed in simulation show that this accelerates the learning process and makes the grasps more functional and stable.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The paper is very well written and was a joy to read!\n- It presents an interesting new idea - guiding grasping algorithms with YouTube video data - and evaluates it thoroughly (albeit only in simulation).\n- Models the grasping activity as a series of actions rather than the end state only (which is the formulation used by most grasping papers).\n- The affordance input can also be manually generated at test time to \"steer\" the grasp towards a desired area of the object.\n- Code will be publicly released.\n\n# Clarification\n- How is demo time calculated in Fig 6? Is it the total length of all videos from which images are chosen for DexVIP?\n\n# Weaknesses\n- Lack of real world robotic experiments is an obvious weakness, but I don't think that alone should be grounds for rejection this year.\n- Effort spent in selecting an appropriate demo image may be quite high. L307 mentions that a demonstration image is collected in a few seconds. But that seems unrealistic, given that an image has to be chosen where the object is upright, the grasp is complete (not approaching), good visibility for FrankMocap, etc. Automating this process, especially detecting when the grasp is complete, might not be as straightforward as claimed. Please add some discussion and visualizations on if the proposed clustering method allows some of these considerations to be relaxed. Why not get the grasp pose from a virtual application that allows the Adroit hand joints to be interactively changed by a user (sort of like virtual kinesthetic teaching)? The paper discusses that normal kinesthetic teaching disembodies the demonstrator, but inaccuracies at a similar level might also be expected from image-based 3D pose estimation.\n- It has been noted in the literature that sim2real transfer with RGB image inputs is difficult. Have you considered a policy which does not use RGB images i.e. depth + affordance only?\n\n# Post Rebuttal\nI will keep my \"strong accept\" rating because the rebuttal addresses my concerns. Especially the verification that the lack of RGB image input does not affect performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper introduces and novel idea and evaluates it as thoroughly as possible in simulation. The lack of real robot experiments and discussion on alternative demonstration methods like virtual kinesthetic teaching are drawbacks, but in my opinion not enough to warrant rejection.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "- Please discuss virtual kinesthetic teaching and challenges likely to be faced while performing sim2real transfer of the policy.\n- If possible, show (preliminary) results with a policy that does not rely on RGB image input, because that modality might cause issues with sim2real transfer to a real robot."}}, {"review_id": "1alO9N_lYRj", "reviewer": "Reviewer_Vrhm", "summary": "This work proposes an approach to learn dexterous robotic grasping from human-object interactions recorded in the video. Target hand poses are predicted from the video leveraging an existing method. The distances between target hand poses and current hand poses are used as a reward signal to train a deep reinforcement learning algorithm in the simulation. Besides the target hand poses, the affordance of each object is pre-calculated. The distances between the hand and affordance regions are also used as a reward signal. The proposed pipeline enables a 30-Dof hand to grasp 27 objects in the simulation achieving 68% success rates. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) This work proposes to leverage the people's hand poses from the video to train dexterous robotic grasping policy.\n2) Extensive comparison experiments are conducted to demonstrate the effectiveness of the proposed approaches.\n3) The paper is well-organized and easy to follow. \n\nWeaknesses:\n1) Some important information is missing. a) how to calculate the difference between the target hand poses and the current hand poses since the target hand poses returned from the FrankMocap are based on a different hand.  b) how to determine whether there are objects in the video dataset which are similar to the objects that needed grasping.\n2) The sim2 real gap remains unclear. What mass values are assigned to these objects? What friction coefficients are used? Are these values sensitive in the learned policy?\n3) The comparison experiments on functionality and posture are kind of unfair. Because these attributes are used during the training process. For example, the proposed method is optimized to grasp under similar postures. Of course, it achieves the highest score in the posture metric.  \n4) It's unclear that how the prediction errors of the target pose affect the learned policy and how the similarity between the object in the video and the object to be grasped affects the learned policy. \n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work proposes to leverage the people's hand poses from the video to train dexterous robotic grasping policy. The proposed pipeline enables a 30 DoF hand to grasp 27 objects in the simulation with 68% grasp success rates. However, some important explanations of the proposed method are missing. It's unclear how the errors in the human pose prior affect the grasping performances. I would be happy to increase the evaluation score if these concerns are addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "1) In Line 137, \"Additionally, we provide a binary affordance map At that is inferred from I0 using an affordance\", Is the affordance map inferred from \"I_0\" or \"I_t\"\n2) In Line 141, \"is the pairwise distance between the object affordance regions and contact points on the hand\". How are these affordance regions defined? How to calculate the distance between a region on the object with a contact point on the hand? Are the number of regions varying between different objects? \n3) In Line 189-190, \"we curate an object interaction repository Ih of 715 video frames from HowTo100M where the human hand is grasping one of the 27 total objects, to yield on average 26 grasp images per object.\" The object in the video and the object for the simulation are not exactly identical. How to make sure that the object in the video and the object for the simulation are similar enough? Are they verified automatically or by human experts?\n4) In Line 197-200, \"In particular, we employ FrankMocap [44] to estimate 3D human hand poses (see Fig. 3, left). FrankMocap is a near real-time method for 3D hand and body pose estimation from monocular video; it returns 3D joint angles for detected hands in each frame\"\nIn FrankMocap[40], the 3D joints angles are based on the SMPL-X hand model (Expressive body capture: 3d hands, face,\nand body from a single image. Pavlakos et al., CVPR 2019).  How to make sure that the designs and joints of SMPL-X hand model are similar to the hand used in the Mujoco simulation? There are 20 joints in SMPL-X hand while there are 24 joints in the hand used in this work. How to calculate the similarity between the poses of two hands?\n5) In Line 209-210, \"we next apply k-medoid clustering on each set P(c). We consider the medoid hand pose of the largest cluster to be the consensus target hand pose\" What metric space is used for applying k-medoid clustering here?\n6) In Line 224-225, \"For Rpose, we compute the mean per-joint angle error at time t between the robot joints prt and the target grasp pose\". A similar question is in Issue 4).  The two hands have different joint designs. How to measure their error?\n7) In Ablation, how do the prediction errors of the target pose affect the learned policy? how does the similarity between the object in the video and the object needed grasping affect the learned policy?\n8) In Ablation, why only success rates are reported? How about the rest three metrics including stability, functionality and posture?\n\n "}}]

Author rebuttal:

Meta-review: Reviewer Vrhm provides a good summary of the paper, "This work proposes an approach to learn dexterous robotic grasping from human-object interactions recorded in the video. Target hand poses are predicted from the video leveraging an existing method. The distances between target hand poses and current hand poses are used as a reward signal to train a deep reinforcement learning algorithm in the simulation. Besides the target hand poses, the affordance of each object is pre-calculated. The distances between the hand and affordance regions are also used as a reward signal. The proposed pipeline enables a 30-Dof hand to grasp 27 objects in the simulation achieving 68% success rates". 

Two reviewers have recommended the paper be accepted and one has recommended a rejection. Overall, reviewers like the idea of using youtube videos to improve grasping, which I agree is an interesting area of research.  But there are several issues in the proposed method. In particular, reviewer HJKw has well summarized them: 

- Run an ablation removing the object affordance reward R_aff
- Evaluate all methods only on the 11 objects not in ContactDB
- Evaluate all methods only on object classes not in ContactDB
- Clarify how object affordance maps are obtained for non-contact DB objects
- Clarify the difference between the state space between the human and robot hands

In my opinion, addressing these issues in the rebuttal is necessary. In addition, Vrhm raises concerns about "However, some important explanations of the proposed method are missing. It's unclear how the errors in the human pose prior affect the grasping performances. I would be happy to increase the evaluation score if these concerns are addressed.". Please address these concerns. 

Next, it would be good to point out on what object types is the difference between GRAFF and the proposed method the maximum. I agree with reviewers Vrhm and HJKw that the grasp success rate and not posture is the appropriate metric. 

Furthermore, reviewers have raised concerns about sim2real transfer. It would be worthwhile to conduct experiments with noisy observation and actuation to instill confidence that the method has a chance to work in the real world. Also, the details of physical parameters used in simulation requested by a reviewer are also critical. 

**Post Rebuttal Update**
The concerns of the reviewers have been addressed. All reviewers vote to accept the paper which I agree with. 
Decision: Accept (Poster)