DEFT: Dexterous Fine-Tuning for Hand Policies
Authors: Aditya Kannan
Kenneth Shaw
Shikhar Bahl
Pragna Mannam
Deepak Pathak
Venue: CoRL 2023 Poster (2023)
Keywords: Dexterous Manipulation
Reinforcement Learning
Learning from Videos

Abstract: Dexterity is often seen as a cornerstone of complex manipulation. Humans are able to perform a host of  skills with their hands, from making food to operating tools. In this paper, we investigate these challenges, especially in the case of soft, deformable objects as well as complex, relatively long-horizon tasks. Although, learning such behaviors from scratch can be data inefficient. To circumvent this, we propose a novel approach,  DEFT (DExterous Fine-Tuning for Hand Policies), that leverages human-driven priors, which are executed directly in the real world. In order to improve upon these priors, DEFT involves an efficient online optimization procedure. With the integration of human-based learning and online fine-tuning, coupled with a soft robotic hand, DEFT demonstrates success across various tasks, establishing a robust, data-efficient pathway toward general dexterous manipulation.  Please see our website at https://dexterousfinetuning.github.io for video results.

DEFT: Dexterous Fine-Tuning for Hand Policies
Aditya Kannan'
Kenneth Shaw*
Shikhar Bahl
Pragna Mannam
Deepak Pathak
Carnegie Mellon University
Pick Cup
Pour
Open Drawer
Pick Spoon
Stir Spoon
Scoop Grape
Pick Grape
Flip Bagel
Squeeze Lemon
Figure 1: We present DEFT, novel approach that can learn complex, dexterous tasks in the real world in an efficient manner. DEFT manipulates tools and soft objects without any robot demonstrations.
Abstract: Dexterity often seen cornerstone of complex manipulation. Hu- mans able perform host of skills with their hands, from making food to op- erating tools. In this paper, investigate these challenges, especially in the case of soft, deformable objects well complex, relatively long-horizon tasks. However, learning such behaviors from scratch can be data inefficient. To circumvent this, we propose novel approach, DEFT (DExterous Fine- Tine-Tuning for Hand Policies), that leverages numan-driven priors, which are executed directly in the real world. In order to improve upon these priors, DEFT involves an efficient online optimization procedure. With the integration of human-based learning and online fine-tur tne-ntuing, coupled with a soft robotic hand, DEFT demonstrates success across various tasks, establishing robust, data-efficient pathway toward general dexterous manipula- tion. Please see our website at https: /ttps://dexterous finetuning gi github... io for video results.
Keywords: Dexterous Manipulation, Learning from Videos
1 Introduction
The longstanding goal of robot learning is to build robust agents that can perform morizoo tasks autonomously. This could for example mean a self-improving robot that can build furniture or an agent that can cook for us. A key aspect most tasks that humans would like to perform that they require complex motions that are often only achievable by hands, such as hammering a nail or using screwdriver. Therefore, we investigate dexterous manipulation and its challenges the real world.
A key challenge in deploying policies in the real world, especially with robotic hands, is that there exist many failure modes. Controlling dexterous hand is much harder than end-effectors due to larger action spaces and complex dynamics. To address this, one option is to improve directly in the real world via practice. Traditionally, reinforcement learning (RL) and imitation learning (IL) techniques have been used deploy hands-on tasks such as in-hand rotation or grasping. This is
"Equal contribution, order decided by coin flip.
7th Conference Robot Learning 2023), Atlanta, USA.

---

Contact Location
Contact Map Decoder
Image Encoder
Predicing Graup afforiazes
Rea World Flae Taning
Grasp Prediction Network
"Pick Spatula' Antotation
Text Eneader
Grasp Pose
Deploycest
Figure 2: Left: DEFT consists of two phases: affordance model that predicts grasp parameters followed by online fine-tuning with CEM. Right: affordance prediction setup predicts grasp location and pose.
case as setups are often built so that either easy to simulate the real world or robust to practice. However, the real world contains tasks that one cannot simulate (such as manipulation of soft objects like food) difficult settings in which the robot cannot practice (sparse long-horizon tasks like assembly). How can we build an approach that can scale to such tasks?
There are several issues with current approaches for practice and improvement in the real world. Robot hardware often breaks, especially with the amount of contact to learn dexterous tasks like operating tools. We thus investigate using a seft anthropomorphic hand [1], which can easily run in the real world without failures or breaking. This soft anthropomorphic hand is well-suited to our approach is flexible and gently handle object interactions. The hand does not get damaged by the environment and is robust to continuous data collection. Due its human-like proportions and morphology, retargeting human hand grasps to robot hand grasps is made simpler.
Unfortunately, this hand difficult to simulate due to its softness. Directly learning from scratch is also difficult we would like to build generalizable policies, and not practice for every new setting. To achieve efficient real-world learning, we must learn prior for reasonable behavior explore using useful actions. Due to recent advances in computer vision, propose leveraging human data to learn priors for dexterous tasks, and improving on such priors in the real world. We aim to use the vast corpus of internet data to define this prior. What is the best way to combine human priors with online practice, especially for hand-based tasks? When manipulating an object, the first thing one thinks about is where on the object to make contact, and how to make this contact. Then, we think about how move our hands after the contact. In fact, this type of prior has been studied in computer vision and robotics literature visual affordances [2, 3, 4, 5, 7, 8, 9]. Our approach, DEFT, builds grasp affordances that predict the contact point, hand pose at contact, and post contact trajectory. To improve upon these, we introduce sampling-basee approach similar to the Cross Cross-Entropy Method (CEM) to fine-tune the grasp parameters in the real world for variety of tasks. By learning a residual policy [10, 1 11], CEM enables iterative real-world improvement in less than an hour.
In summary, our approach (DEFT) executes real- world learning on a soft robot hand with only few trials in the real world. To facilitate this efficiently, videos. We introduce 9 challenging tasks (as seen in Figure 1) that are difficult even for trained operators to perform. While our method begins to show good success on these tasks with real-world fine-tuning, more investigation is required to complete these tasks more effectively.
train priors on human motion from internet
Related Work
Real-world robot learning Real-world manipulation tasks can involve a blend classical and learning- methods. Classical approaches like control methods or path planning often use hand-crafted features objectives and can often lack flexibility in unstructured settings [12, 13, 14]. On the other hand, data-driven approaches such deep reinforcement learning (RL) can facilitate complex behaviors in various settings, but these methods frequently rely on lots of data, privileged

---

reward information and struggle with sample efficiency [15, 16, 17, 18, 19]. Efforts have been made to scale end-to-end RL [20, 21, 22, 23, 24, 25] to the real world, but their approaches are not yet efficient enough for more complex tasks and action spaces and are reduced to mostly simple tasks even after a lot of real-world learning. Many approaches try to improve this efficiency such as by using different action spaces [26]. goal relabeling [27]. trajectory guidance [28], visual imagined goals [21], turiosity-driver exploration [29]. Our work focuses on learning prior from human videos in order to learn efficiently in the real world.
Learning from Human Motion The field of computer vision has seen much recent success in human and object interaction with deep neural networks. The human hand is often parametrized with MANO, a 45-dimensional vector [30] of axes aligned with the wrist and a 10-dimensional shape vector. MANOtorch from [31] aligns it with the anatomical joints. Many recent works detect MANO in monocular video [32, 33, 34]. Some also detect objects as well as the hand together [6, 35J. We use FrankMocap detect the hand for this work. There are many recent datasets including the CMU Mocap Database [36] and Human3.66 [37] for human pose estimation, 100 Days of Hands [6] for hand-object interactions, FreiHand [38] for hand poses, Comething-Sometiine
[39] for semantic interactions. ActivityNet datasets [40], or YouCook [41] are action-driven datasets that focus on dexterous manipulation. We use these three datasets: [42] is a large-scale dataset with human-object interactions, [43] for curated human-object interactions, and [44] which has many household kitchen tasks. In addition to learning exact human motion, many others focus on learning priors from human motion. [45, 46] learn general priors using contrastive learning on human datasets.
Learning for Dexterous Manipulation With recent data-driven machine learning methods, roboti- cists are now beginning to learn dexterous policies from human data well. Using the motion of human can be directly used to control robots [47, 48, 49]. Moving further, human motion in internet datasets can be retargeted and used directly to pre-train robotic policies [50, 51J. Additionally, using human motion as a prior for RL can help with learning skills that are human-like [52. 53, 54]. Without using human data as priors, object reorientation using RL has been recently successful in a variety of settings [55, 56]. Similar work in robot dogs which do not have an easy human analog to learn from, these methods rely significant training data from simulation with zero-shot transfer [57, 58).
Soft Object Manipulation Manipulating soft and delicate objects in robot's environment has been a long-standing problem. Using the torque output on motors, either by measuring current or through torque sensors, is useful feedback to find out how much force a robot is applying [59, 60]. Coupled with dynamics controllers, these robots can learn not to apply too much torque to the environment around them [61, 62, 63]. A variety of touch sensors [64, 65, 66, 67] have also been developed to feel the environment around it and can be used as control feedback. Our work does not rely on touch sensors. Instead, we practice in the real world learn stable and precise grasps.
3 Fine-Tuning Affordance for Dexterity
The goal of DEFT to learn useful, dexterous manipulation in the real world that can generalize many objects and scenarios. DEFT learns in the real world and fine-tunes robot hand-to-object interaction in the real world using only few samples. However, without any priors useful behavior, the robot will explore inefficiently. Especially with nigh-dimensional robotic hand, we need a strong prior to effectively explore the real world. We thus train an affordance model on human videos that leverages human behavior to learn reasonable behaviors the robot should perform.
3.1 Learning grasping affordances
To learn from dexterous interaction in a sample efficient way, we use human hand motion as a prior for robot hand motion. We aim to answer the following: (1) What useful, actionable information can we extract from the human videos? (2) How can human motion be translated to the robot embodiment to guide the robot? In internet vidcos, humans frequently interact with wide variety of objects. This
3

---

Contact Location
Grasp
Post Grrsp Trajectory
Figure 3: We produce three priors from human videos: the contact location (top row) and grasp pose (middle row) from the affordance prior; the post-grasp trajectory (bottom row) from a human demonstration of the task.
data is especially useful in learning object affordances. Furthermore, of the major obstacles in manipulating objects with few samples is accurately grasping the object. A model that can perform strong grasp must learn where and how to grasp. Additionally, the task objective is important in determining object affordances-humee
often grasp objects different ways depending on their goal. Therefore, we extract three items from human videos: the grasp location, human grasp pose, and task.
Given video clip V {vi, U2, .... JT the first frame t2 where the hand touches the object is found using an off-the-shelf hand-object detection model [6]. Similar to previous approaches 3). a set of contact points are extracted to fit Gaussian Mixture Model (GMM) with centers
[3, 4, 5J, {μI. Pe..... } Detic [68] is used to obtain a cropped image of containing just the object in the initial frame in to condition the model. We use Frankmocap [34] to extract the hand grasp pose P in the contact frame 0 as MANO parameters. We also obtain the wrist orientation w in the camera frame. This guides our prior to output wrist rotations and hand joint angles that produce stable grasp. Finally, we acquire
description T describing the action occurring in V.
We extract affordances from three large-scale, egocentric datasets: Ego4D [42] for its large scale and the variety of different scenarios depicted, HOI4D [8] for high-quality human-object interactions, and EPIC Kitchens [44] for its focus on kitchen tasks similar to our robot's. We learn cask-conditioned affordance model f that produces (f,, Wwissts P) = f (w] T). We predict ğ in similar fashion to [3]. First, we use pre-trained visual model [69] encode 2 into latent vector z.. Then we pass 2n through a of deconvolutional layers get a heatmap and use spatial softmax estimate
To determine Owisst and P. we use 2u and an embedding of the text description 21 g(T), where g is the CLIIP text encoder [70]. Because transformers have seen success in encoding various mul- tiple modes of input, transformer encoder T to predict Owriat = T( 2r). Overall, we train our model to optimize
Parameter
Dimensions
Description
J Ownis P
XYZ grasp location in workspace Wrist grasp rotation (euler angles) Finger joint angles soft hand
3
16
Table 1: Parameters that fine-tuned the real world. The affor- dance model predicts 45-dimensional hand joint pose P. which retargeted to a 16-dimensional soft hand pose.
L = >ullæ All2 do 1wwist - @wislll + Apl -
(1)
At test time, we generate a crop of the object using Segment-Anything [71] and give our model task description. The model generates contact points on the object, and we take the average as our contact point. Using depth camera, model outputs MANO parameters [30] that are designed to describe human hand joints, we retarget
determine the 3D contact point navigate to. While the

---

Realsense
Hand
7
Object
3
Figure 4: Left: Workspace Sctup. We place an Intel RealSense camera above the robot to maintain egocentric viewpoint, consistent with affordance model's training data. Right: Thirteen objects used our experiments.
these values to produce similar grasping poses on our robot hand a similar manner to previous approaches [72, 48]. For more details, we refer readers to the appendix.
In addition these grasp priors, need task-specific post-contact trajectory to successfully execute a task. Because it is challenging to learn complex and high-frequency action information from purely offline videos, we collect one demo of the human doing the robot task (Figure 3) separate from the affordance model f. We extract the task-specific wrist trajectory after the grasp using [34]. We compute the change in wrist pose between adjacent timesteps for the first 40 timesteps. When deployed for fine-tuning, we execute these displacements for the post-grasp trajectory. Once we have this prior, how can the robot improve upon it?
3.2 Fine-tuning via Interaction
The affordance prior allows the robot to narrow down its learning behavior small subset of all possible behav- iors. However, these affordances are not perfect and the robot will often- times still not complete the task. This is partially due to morphology differ- ences between the human and robot hands, inaccurate detections of the hu- man hands, or differences in the task setup. To improve upon the prior, practice learning residual policy for the grasp parameters in Table 1.
Algorithm Fine- TuneiTTunng Procedure for DEFT
Require:
conditoond affordance model f. task description T. post-grasp trajectory т. parameter distribution 2, residual cVAE policy æ. E' number of clites, M number of warm-up episodes, N total iterations.
D
(0, I...N 1x,0 <- initial image
& f(E,oo
Sample D
Exccute grasp from EN + ck. then trajectory 7
Collect reward Rui reset environment
k >
M then
Order traj indices 11, f2,.... Ω {Ee,, Eig.... fig)
.......,. is based on rewards
Residual policies have been used pre- viously to efficiently explore in real world [3]. They use the prior starting point and explore nearby, Let the grasp location, wrist rotation, grasp pose, and trajectory from our af- fordance prior be &. During training we sample noise e ~ D where D is initialized to N (( o²) (for a small σ). We rollout trajectory parameterized by & c. We collect R, the reward for each & = f(x) + e, where v is the image. First, we execute an initial number of M warmup episodes with actions sampled from D, recording a reward R; based on how well the trajectory completes the task. For each episode afterward, rank the prior episodes based on the reward RG and extract the sampled noise from the episodes with the highest reward (the 'elites' Q). We fit D to the elite episodes to improve the sampled noise. Then sample actions from D. execute the episode, and
Fit to distribution residuals
end if
end for
Fit x() as VAE to Ω
5

---

Herations
Affordance Prior
Afforlance Model
Time
X
X
X
X
PR
X
Figure 5: Qualitative results showing the finetuning procedure for DEFT. The model learns hold the spatula flip the bagel after 30 CEM iterations.
record the reward. By repeating this process we can gradually narrow the distribution around the desired values. In practice, we use M = 10 warmup episodes and total of N 30 episodes total for each task. This procedure is shown in Algorithm 1. See Table 4 for more information.
At time, we could take the mean values of the top N trajectories for the rollout policy. However, this does not account for the appearance of different objects, previously unseen object configurations, or other properties in the environment. To generalize to different initializations, we train VAE [74, 75, 76, 77] to output residuals ó, conditioned on an encoding of the initial image @(L,.0) and affordance model outputs E from the top ten trajectories. We train an encoder q=]|5]. 93) where = =(((I,)), as well as decoder p( 2 Cÿ) that learns to reconstruct residuals 6z. At test time, our residual policy π (To.E) E) samples the latent 2 ~ No0,I) I) and predicts å = =[(z, (To. {)). Then we rollout the trajectory determined by the parameters Ç + 8. Because the VAE is conditioned on the initial image, we generalize to different locations and configurations of the object.
4 Experiment Setup
We perform variety of experiments answer the following: 1) How well DEFT learn and improve in the real world? 2) How good is our affordance model? 3) How can the experience collected by DEFT be distilled into policy? 4) How can DEFT be used for complex, soft object manipulation? Please see our website at http: bestp://dexten /ttp:///xxxtuuusssee finetuning. github.. io for videos.
Task Setup We introduce tabletop tasks, Pick Cup, Pour Cup, Open Drawer, Pick Spoon, Scoop Grape, Stir Spoon, Pick Grape, Flip Bagel, Squeeze Lemon. Robotic hands are especially well-suited for these tasks because most of them require holding curved objects or manipulating objects with tools to succeed. For all tasks, we randomize the position of the object on the table, as well as use train and test objects with different shapes and appearances to test for generalization. To achieve real-world learning with the soft robot hand, we pretrain an internet affordance model as a prior for robot behavior. As explained Section 3, we train one language-conndtioone model on all data, At test time, we use this initialization for our real-world fine-tuning. The fine-tuning is done purely in the real world. An operator runs 10 warmup episodes of CEM, followed by 20 episodes that continually update the noise distribution, improving the policy. After this stage, we train residual VAE policy that trains on the top ten CEM episodes to predict the noise given the image and affordance outputs. We evaluate how effectively the VAE predicts the residuals on each of the tasks by averaging over trials. Because it takes less than an hour to fine-tune for one task, we are able to thoroughly evaluate our method on 9 tasks, involving over 100 hours of real-world data collection.
Hardware Setup We use a 6-DOF UFactory xArmó robot arm for all our experiments. We attach to 16-DOF Soft Hand using a custom, 3D-printed base. We use a single, egocentric RGBD camera to capture the 3D location of the object in the camera frame. We calibrate the camera so that the predictions of the affordance model can be converted and executed in the robot frame. The flexibility of the robot hand also makes robust collisions with objects unexpected contact with the environment. For the arm, the arm's dynamics controller senses that the arm collided aggressively with the environment.
that it stays above the tabletop. The job will be terminated if

---

Method
Pick cup
Pour cup
Open drawer
Pick spoon
Scoop
Grape
Stir Spoon train
train
test
train
test
train
test
train
test
train
test
test
Real-World Only
0.0
0.1
0.2
0.1
0.1
0.0
0.7
0.3
0.0
0.0
0.3
0.0
Affordance Model Unly
0.1
0.4
0.5
0.5
0.0
0.3
DEFT
0.8
0.8
0.8
0.9
0.5
0.4
0.8
0.6
0.7
0.3
0.8
0.5
Table 2: We present the results of method as well compare them other baselines: Real-world learning without internet priors used as guidance and the affordance model outputs without real-world learning. evaluate the success of the methods on the tasks over 10 trials.
Ours
No Fine-Tuming
No Affordance Model
Pick Cup
Pour
Open Drawer
0.8
0.8
0.8
Supcess Rate
0.6
Success Rate
0.6
Success Rate
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
10
15
20
25
30
10
15
20
25
30
10
15
20
25
30
CEM Rollouts
CEM Rollouts
CEM Rollouts
Pick Spoon
Scoop
Stir
0.8
Success Rate
0.6
Success Rate
0.6
Success Rate
0.6
0.4
0.4
0.4
0.2
0.2
0.0
0.0
0.0
30
30
30
CEM Rollouts
CEM Rollouts
CEM Rollouts
Figure 6: Improvement results for 6 tasks: pick cup, pour, open drawer, pick spoon, scoop, and stir. We see
steady improvement in method as more CEM episodes are collected.
Results
Effect of affordance model We investigate the role of the affordance model and real-world fine- tuning (Table 2 and Figure 6). In the real-world only model, provide few heuristics in place of the affordance prior. We detect the object in the scene using a popular object detection model [71] and let the contact location prior be the center of the bounding box. We randomly sample the rotation angle and use a half-closed hand as the grasp pose prior. With these manually provided priors, the robot has difficulty finding stable grasps. The main challenge was finding the correct rotation angle for the hand. Hand rotation is very important for many tool manipulation tasks because it requires not only picking the tool but also grasping in a stable manner.
Zero-shot model execution We explore the zero-shot performance of our prior. Without applying any online fine-tuning to our affordance model, we rollout the trajectory parameterized by the prior. While our model is decent on simpler tasks, the model struggles on tasks like stir and scoop that require strong power grasps (shown in Table 2). In these tasks, the spoon collides with other objects, so fine-tuning the prior to hold the back of the spoon is important in maintaining a reliable grip throughout the post-grasp motion. Because DEFT incorporates real-world experience with the prior, it is able sample contact locations and grasp rotations that can better execute the task.
Human and automated rewards We ablate the reward function used evaluate episodes. Our method queries the operator during the task reset process assign a continuous score from 0 to 1 for the grasp. Because the reset process requires human-in-the-loog regardless, this adds little marginal cost for the operator. But what if we would like these rewards to be calculated autonomously? We use the final image collected in the single post-grasp human demonstration from Section as the goal image, We define the reward to the negative embedding distance between the final image of the episode and the goal image with either an R3M [69] or a ResNet [78] encoder. The model learned from ranking trajectories with R3M reward is competitive with DEFT in all but one task, indicating that using visual reward model can provide reasonable results compared human rewards.
7

---

Method
Pour Cup train
Open Drawer test
Pick Spoon train
test
train
test
Reward Function:
R3M Reward
0.0
0.0
0.4
0.5
0.5
0.4
Resnet18 Imagenet Reward
0.1
0.2
0.3
0.4
0.2
Policy Ablation:
DEFT w/ MLP
0.0
0.0
0.5
0.0
0.6
0.5
DEFT w/ Transformer
0.4
0.5
0.1
0.4
0.5
DEFT w/ Direct Parameter est.
0.1
0.1
0.1
0.0
0.3
0.0
DEFT
0.8
0.9
0.5
0.4
0.8
0.6
Table 3: Ablations for (I) reward function type, (2) model architecture, and (3) parameter estimation.
Model Architecture We
Fne-Tuzing
investigate different models
0.4
Pick Graps
0.6
Flip Bapsl
Squeeze Lemon
and training architectures for the policy trained on the
Socces Rate
Suncas Rai:
Success Rute
0.6
0.4
rollouts (Table 3). When we replace the conditional VAE with MLP that predicts residuals, the model has dif- ficulty learning the grasp rotation to effectively pour cup. We find that the MLP cannot learn the multi-modality of the successful data properly. Our transformer ablation an offline method similar to [79] where in addition to the image and affordance model outputs, we condition on the reward outputs and train a transformer to predict the residual. At test time the maximum reward is queried and the output is used in the rollout. While this method performs well, we hypothesize that the transformer needs more data match DEFT. Finally, train VAE to directly estimate instead of the residual. This does not effectively distill the information from the affordance prior without the training time allotted. As result, it often makes predictions that are far from the correct grasp pose.
02
0.0
6.0
10
15
20
2.5
30
10
15
200
25
10
15
20
25
30
CIM Rollouts
CEM Rollouts
CEM Rollouts
Figure 7: We evaluate DEFT three difficult manipulation tasks.
Performance complex tasks and soft object manipulation We investigate the performance of DEFT on more challenging tasks. Tasks involving soft objects cannot be simulated accurately, while our method is able to perform reasonably on food manipulation tasks shown in Figure 7.
Of the three tasks, our method has the most difficulty with the Pick Grape task. Because grapes are small, the fingers must curl fully to maintain stable grasp. limitation of our hand is that the range of its joints does not allow it close the grasp fully and result, it has difficulty in consistently picking small objects. This also makes it challenging to hold heavy objects like the spatula in Flip Bagel, but with practice DEFT learns maintain stable grasp of the spatula. For Squeeze Lemon, DEFT develops grasp that allows it to apply sufficient pressure above the juicer. Specifically, our method takes advantage of the additional fingers available for support in hands.
6 Discussion and Limitations
In this paper, we investigate how to learn dexterous manipulation in complex setups. DEFT aims to learn directly the real world. order to accelerate real-world fine-tuning, we build an affordance prior learned from human videos. We are able efficiently practice and improve in the real world via our online fine-tuning approach with soft anthropomorphic hand, performing variety of tasks (involving both rigid and soft objects). While our method shows some success these tasks, there are some limitations to DEFT that hinder its efficacy. Although we are able to learn policies for the high-dimensional robot hand, the grasps learned are not very multi-modal and do not capture all of the different grasps humans are able to perform. This is mainly due to noisy hand detections in affordance pretraining. As detection models improve, we hope to be able to learn a more diverse set of hand grasps. Second, during finetuning, resets require human input and intervention. This limits the real-world learning we can do, as the human has be constantly in the loop reset the objects. Lastly, the hand's 's fingers cannot curl fully. This physical limitation makes it difficult to hold thin objects tightly. Future iterations of the soft hand can be designed to grip such objects strongly.
8

---

Acknowledgments
We thank Ananye Agarwal and Shagun Uppal for fruitful discussions. KS is supported by the NSF Graduate Research Fellowship under Grant No. DGE2140739. The work is supported in part by ONR N00014-22-1-2096 ONR MURI N00014-22-1-2773, and Air Force Office of Scientific Research (AFOSR) FA9550-23-1-0747.
References
[1] P. Mannam, K. Shaw, D. Bauer, J. Oh, D. Pathak, and N. Pollard. A framework for designing anthropomorphic soft hands through interaction, 2023.
[2] D. Fouhey, X. Wang, and A. Gupta. In defense of the direct perception of affordances. arXiv preprint (.05:v5500.0000 2015.
[3] S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak. Affordances from human videos versatile representation for robotics. 2023.
[4] M. Goyal, S. Modi, R. Goyal, and S. Gupta. Human hands as probes for interactive object understanding. In CVPR, 2022.
[5] T. Nagarajan, C. Feichtenhofer, and K. Grauman. Grounded human-object interaction hotspots from video. In ICCV, 2019.
[6] D. Shan, J. Geng, M. Shu, and F. Fouhey. Understanding human hands contact at internet scale. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9869-9878, 2020.
[7] S. Liu, S. Tripathi, S. Majumdar, and X. Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In CVPR, 2022.
[8] Y. Liu, Y. Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and L. Yi. Hoi4d: A egocentric dataset for category-level human-object interaction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21013-21022, June 2022.
[9] X. Wang, R. Girdhar, and A. Gupta. Binge watching: Scaling affordance learning from sitcoms. In CVPR, 2017.
[10] Davchev, K. S. Luck, M. Burke, F. Meier, Schaal, and S. Ramamoorthy, Residual learning from demonstration. arXiv preprint 2xXiv:2008.07622 2020.
[11] T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine. Residual reinforcement learning for robot control. In ICRA, 2019.
[12] S. Karaman, M. R. Walter, A. Perez, E. Frazzoli, and S. Teller. Anytime motion planning using the rrt. ICRA, 2011.
[13] Kuffner and M. LaValle. Rit-connect: An efficient approach single-query path planning. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 2, pages 995- IEEE, 2000.
[14] M. Mukadam, X. Yan, and B. Boots. Gaussian process motion planning. In 2016 IEEE international conference on robotics and automation (ICRA), pages 9-15. IEBE, 2016.
[15] J. Kober and J. Peters. Policy search for motor primitives in robotics. Advances in neural information processing systems, 21, 2008.

---

[16] J. Peters, K. Mulling, and Y. Altun. Relative entropy policy search. In Twenty-Fove Fourth AAAI Conference on Artificial Intelligence, 2010.
[17] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. ICLR, 2016.
[18] I. Popov, N. Heess, Lillicrap, R. Hafner, Barth-Maron, M. Vecerik, T. Lampe, Tassa, T. Frez, and M. Riedmiller. Data-efficient deep reinforcement learning for dexterous manipula- tion. arXiv preprint arXiv:/704.03077 2017.
[19] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In ICML, 2017.
[20] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen. Learning hand-eye coordination for robotic grasping with large-scale data collection. In ISER, 2016.
[21] A. V. Nair, V. Pong, M. Dalal, S. Bahl, Lin, and S. Levine. Visual reinforcement learning with imagined goals. In NeurlPS, pages 9191-9200, 2018.
[22] Agrawal, A. Nair, P. Abbeel, Malik, and Levine. Learning to poke by poking: Experiential learning of intuitive physics. NIPS, 2016.
[23] T. Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1352-1361. JMLR. org, 2017.
[24] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, F. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-basee robotic manipulation. arXiv preprint arXiv: 10293, 2018.
[25] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08221 08212 2021.
[26] R. Martin-Martin M. A. Lee, R. Gardner, S. Savarese, J. Bohg, and A. Garg. Variable impedance control in end-effector space: An action space for reinforcement learning in contact-rich tasks. IROS, 2019.
[27] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, Tobin, Abbeel, and Zaremba. Hindsight experience replay. NeurlPS, 2017.
[28]
Levine and V. Koltun. Guided policy search. In ICML, 2013.
[29] R. Mendonca, S. Bahl, and D. Pathak. Alan: Autonomously exploring robotic agents in the real world. arXiv preprint arXiv:2302.06604 2023.
[30] J. Romero, D. Tzionas, and M. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions Graphics, (Proc. SIGGRAPH Asia), 36(6). Nov. 2017.
[31] L. Yang, X. Zhan, K. Li, W. Xu, J. Li, and C. Lu. CPF: Learning a contact potential field to model the hand-object interaction. In ICCV, 2021.
[32] Wang, F. Mueller, F. Bernard, S. Sorli, O. Sotnychenko, N. Qian, M. A. Otaduy, D. Casas, and Theobalt. Rgb2hands: real-time tracking of 3d hand interactions from monocular rgb video. ACM Transactions on Graphics (TOG), 39(6):1-16. 2020.
[33] A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik. End-to-end recovery of human shape and pose. CoRR,
2017. URL http: /tatpi///rxiv..... org/abs/ abs//112 06584.
10

---

[34] Y. Rong. T. Shiratori, and H. Joo. Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 1749-1759, October 2021.
[35] Y. Ye, A. Gupta, and S. Tulsiani. What's in your hands? 3d reconstruction of generic objects in hands. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3895-3905, 2022.
[36] Cmu graphics lab motion capture database. http://mocap cs. cmu, cmu.edu/
[37] C. lonescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions pattern analysis and machine intelligence,
2013.
[38] C. Zimmermann, D. Ceylan, J. Yang, B. Russell, M. Argus, and T. Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813-822, 2019.
[39] R. Goyal, S. Ebrahimi Kahou, V. Michalski, Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic. The "something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[40] B. G. Fabian Caba Heilbron, Victor Escorcia and J. C. Niebles. Activitynet: A large-scale video benchmark for human activity understanding, In CVPR, pages 961-970, 2015.
[41] P. Das, C. Xu, R. F. Doell, and J. J. Corso. A thousand frames in just few words: Lingual description of videos through latent topics and sparse object stitching. In Proceedings of the IEEE conference computer vision and pattern recognition, pages
2013.
[42] K. Grauman, A. Westhury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[43] Y. Liu, Y. Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and L. Yi. Hoi4d: A 4d egocentric dataset for category-level human objant interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21013-21022 June 2022.
[44] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV), 2018.
[45] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030. 2022.
[46] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 22611 2022.
[47] A. Handa, K. Van Wyk, W. Yang, J. Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox. Dexpilot: Aision-based teleoperation of dexterous robotic hand-arm system. In ICRA, 2020.
[48] A. Sivakumar, K. Shaw, and D. Pathak. Robotic telekinesis: learning robotic hand imitator by watching humans on youtube. arXiv preprint arXiv:2202.10445 2022.
[49] F. O. H. to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single- Camera Teleoperation. Qin, yuzhe and su, hao and wang, xiaolong, 2022.

---

[50] K. Shaw, S. Bahl, and D. Pathak. Videodex: Learning dexterity from internet videos. In CoRL, 2022.
[51] Mandikal and K. Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. In Conference on Robot Learning, pages 651-661. PMLR, 2022.
[52] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: (ttt::179.00877 2017.
[53] X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4))1-14, 2018.
[54] P. Mandikal and K. Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. In Conference on Robot Learning (CoRL), 2021.
[55] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, al. Learning dexterous in-hand manipulation. I/RR, 2020.
[56] T. Chen, M. Tippur, S. Wu, V. Kumar, E. Adelson, and P. Agrawal. Visual dexterity: In-hand dexterous manipulation from depth. arXiv preprint arXiv:2211. 2022.
[57] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. Conference on Robot Learning, pages 403-415. PMLR, 2023.
[58] G. B. Margolis, G. Yang, K. Paigwar. T. Chen, and Agrawal. Rapid locomotion via reinforce- ment learning. arXiv preprint arXiv:2205.028.. 2022.
[59] T. Yoshikawa. Dynamic manipulability of robot manipulators. Transactions of the Society of Instrument and Control Engineers, 21(9):970-99) 975, 1985.
[60] H. Asada and J.-J. Slotine. Robot analysis and control. John Wiley & Sons, 1991.
[61] K. M. Lynch and F. C. Park. Modern robotics. Cambridge University Press, 2017.
[62] C. Liu and M. Tomizuka. Designing the robot behavior for safe human robot interactions. In Trends in Control and Decision-Making for Human-Robot Collaboration Systems, pages 241-270. Springer, 2017.
[63] O. Khatib. A unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE Journal on Robotics and Automation, 3(1):43-53, 53, 1987.
[64] Z. Si. T. C. Yu, K. Morozov, J. McCann, and W. Yuan. Robotsweater: Scalable, generalizable, and customizable machine-knitted tactile skins for robots. arXiv preprint arXiv:2303.02285. 2023.
[65] W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017.
[66] R. Bhirangi, T. Hellebrekers, C. Majidi, and A. Gupta. Reskin: versatile, replaceable, lasting tactile skins. arXiv preprint arXiv:211 2021.
[67] S. Sundaram, P. Kellnhofer, Y. Li, J.-Y. Zhu, A. Torralba, and W. Matusik. Learning the signatures of the human grasp using a scalable tactile glove. Nature, 569(7758), 2019. doi: 10. 10.1038/s41586-09 -19-6-1---122
[68] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra. Detecting twenty-thousan classes using image-level supervision. arXiv preprint arXiv:2201.02600 2022.
12

---

[69] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. /2601, 2022.
[70] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021. URL https://arxiv org abs/ 2103. 2103.00020.
[71] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.02644 2023.
[72] A. Handa, K. Van Wyk, W. Yang, Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox. Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164-9170. IEEE, 2020.
[73] Bahl, A. Gupta, and D. Pathak. Human-to-robot imitation the wild. In RSS, 2022.
[74] K. Sohn, X. Yan, and H. Lee. Learning Structured Output Representation using Deep Condi- tional Generative Models. In NeurlPS, 2015.
[75] D. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference machine learning, pages 1286. PMLR, 2014.
[76] D. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference deep generative models. arXiv preprint arXiv:1401.4882, 2014.
[77] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312,1114 2013.
[78] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CoRR, abs/ hass1552..03885 2015. URL http:://araa org/abs/1512 aa//ab///1/5 0ates/151...30 03385.
[79] L. Chen, Lu, A. Rajeswaran, Lee, Grover, M. Laskin, Abbeel, Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint 2XXi::2006.01355, 2021.
[80] D. P. Kingma and Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: Mtt::4412.69850 2014.
13

---

A
Video Demo
We provide video demos of our system at httpss//daxterros--fieuuuut wtw.:iitnininnnnngggubbb.. io.
B DASH: Dexterous Anthropomorphiccc Soft Hand
Recently introduced, DASH (Dexterous Anthropomorphic Soft Hand) [1] is four- cour-fingered anthro- pomorphic soft robotic hand well-suited for machine learning research use. Its human-like size and form factor allow us to retarget human hand grasps to robot hand grasps easily as well as perform human-like grasps. Each finger is actuated by 3 motors connected to string-like tendons, which deform the joints closest to the fingertip (DIP joint), the middle joint (PIP joint), and the joint the base of the finger (MCP joint). There one motor for the finger to move side-to-side the MCP joint, one for the finger move forward the MCP joint, and one for PIP and DIP joints. The PIP and DIP joints are coupled to motor and move dependently. While the motors do not know the end-effector positions of the fingers, we learn a mapping function from pairs of motor angles and visually observed open-loop finger joint angles. These models are used to command the finger joint positions learned from human grasps.
C MANO Retargeting
For MANO parameters, the axis of each of the joints is rotation aligned with the wrist joint and translated across the hand. However, our robot hand operates on forward and side-to-side joint angles. To translate the MANO parameters to the robot fingers we extract the anatomical consistent axes of MANO using MANOTorch. Once these axes are extracted, each axis rotation represents twisting (not possible for human hands), bending, and spreading. We then match these axes to the robot hand. The spreading of the human hand's fingers (side-to-side motion the MCP jCitoi maps to the side-to-side motion at the robot hand's base joint. The forward folding at the base of the human hand (forward motion at the MCP joint) maps to the forward motion at the base of the robot hand's finger. Finally, the bending of the other two finger joints on the human hand, PIP and DIP, map to the robot hand's PIP and DIP joints. While the thumb does not have anatomically the same structure, we map the axes in the same way. Other approaches rely on creating an energy function to map the human hand to the robot hand. However, because the soft hand is similar anatomy and size to a human hand, it does not require energy functions for accurate retargeting.
D Affordance Model Training
We use data from Ego4D [42], EpicKitchens- 100 [44], and HOI4D [8]. After filtering for clips of sufficient length, clips that involve grasping objects with the right hand, and clips that have language annotations, we used 64666 clips from Ego4D, 9144 clips from EpicKitchens, and 2707 clips from HOI4D. In total, use dataset of 76517 samples for training our model.
For our contact location model, we use the visual encoder from [69] to encode the image as a 512-dimensional vector. We use the spatial features of the encoder to upsample the latent before applying a spatial softmax to return the contact heatmap. This consists of three deconvolutional layers with 512, 256, and 64 channels that order.
To predict wrist rotation and grasp pose, we use the language encoder from [70] to compress the language instruction to a 512-dimensionnl vector. We concatenate the visual and language latents and pass them through a transformer with eight heads and six self-attention layers. We pass the result of the transformer through an MLP with hidden size 576, and predict a vector of size 48: the first dimensions are the axis-angle rotations; the last 45 dimensions are the joint angles of the hand. These correspond to the parameters output by Frankmocap [34], which we used to get ground truth hand pose in all the datasets. The images used from the training datasets well the ground truth labels released here.
14

---

We jointly optimize the L2 loss of the contact location μ4, the wrist rotation wwisst and grasp pose P. The weights we used for the losses are U = 1.0, > = 0.1,, Ap = 0.1. We train for 70 epochs with an initial learning rate of 0.0002, and a batch size of 224. We used the Adam optimizer [80] with cosine learning rate scheduler. We trained on a single NVIDIA RTX A6000 with 48GB RAM.
E
Fine- Tone-Tuning Parameters
Below the values of the parameters used for the CEM phase of DEFT.
Parameter
Value
Description
E
10
Number of clites for CEM
M
10
Number of warm-up episodes
N
30
Total number of CEM episodes
de
0.02
Initial contact location Standard Deviation (meters)
0.2
Initial wrist rotation Standard Deviation (culer angle radians) Initial soft hand joints Standard Deviation
ap
0.05
Table 4: Values for fixed parameters in fine-tuning Algorithm 1.
F
Success Criteria for Tasks
We define the criteria for success in each of our 9 tasks as follows:
Pick Cup: Cup must leave table surface and stay grasped throughout trial.
Pour Cup: Cup must be grasped throughout trial and also rotate so that the top of the cup is at a lower height than the base.
Open Drawer: Drawer is initially slightly open that it can be grasped. By the end of the episodes, the drawer should be at least centimeter more open than it was at the beginning.
Pick Spoon: The spoon must not be in contact with the table at the end of the trial.
Stir Spoon: The spoon base must rotate around the jar/pot least 180 degrees while grasped.
Scoop Grape: The spoon must hold a grape at the end of the trial while being held by the soft hand.
Pick Grape: All grapes must be held by the hand above the table surface. In particular, any single grape falls due to weak stem, this is considered failure.
Flip Bagel: The side of the bagel that is facing up at the end the trial should be opposite the side facing up at the beginning.
Squeeze Lemon: The lemon should be grasped securely on top of the juicer.
15


Reviews:
[{"review_id": "t14Agp0R1m", "reviewer": "Reviewer_1dz2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present DEFT, a method that learns useful dexterous manipulation in the real world that can generalize to many objects and scenarios. It first learns prior affordances from human videos and fits a transformer to jointly predict the pose, orientation, and affordance. It then finetunes the policies extracted from human videos through online interaction by a CEM-like trajectory optimization method in the real world. As a result it can control dexterous hands to solve 9 real-world tasks with limited online interactions.", "summary_of_recommendation": "I think the paper presents a nice system for real-world dexterous hand manipulation. I appreciate the authors’ effort but I feel details are missing now and need clarification before acceptance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- please provide the details for CEM and VAE training.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength: \n\n- The method combines the prior knowledge extracted from the human videos and the online data for fine-tuning, achieving dexterous manipulation in the real world. I appreciate real-world experiments. The result looks impressive to me, and the method is quite promising. I was not aware such simple CEM methods were applied in the real world. I think the paper contains sufficient contribution for acceptance.\n\nWeakness:\n\n- However, despite the authors' efforts to build and implement the system, I found essential details missing in the paper, making it hard to provide a fair evaluation. I tried to find the details in the appendix as promised in Line 185 of the main text, which confused me about the CEM and VAE part. What are the differences between the “rollouts” and “episodes”? Are they the same concepts? How do you train the VAE with only 10 data samples? Won’t this cause overfitting? How many rollouts are required for each task? I hope the authors can provide more details regarding the implementation.\n- I wonder if CEM is the best choice for online-finetuning. Before the details are clarified, I am worried that online fine-tuning will cost too much or can not generalize well to different initialization. Methods like iterative residual policy sound more data-efficient than the CEM and may benefit from large-scale video datasets. I’d hear the authors’ thoughts on this.\n- The method contains a complex pipeline for extracting suitable priors for grasping, but I need to see ablation studies regarding the visual parts. The authors should provide more analysis details, at least in the appendix, to help readers better understand its contribution.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "NSgUPqZCWl6", "reviewer": "Reviewer_Tfey", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an approach for achieving dexterous manipulation using a high dimensional robotic hand. The approach is based on fine-tuning manipulation behavior. The initial manipulation trajectory is obtained through a learned affordance model that is trained on video data and essentially handles grasp generation, together with one human demo of the task for obtaining the post-grasping movement.\nThis initial manipulation trajectory is then optimized by performing sampling-based optimization (CEM-like) on the real system. The result of the online optimization using CEM is subsequently distilled into a residual variational autoencoder. The VAE allows directly estimating residual terms when facing a previously unseen and slightly modified testing environment.\nThe authors subsequently present an experimental evaluation of their method, illustrating the effect of policy fine-tuning, as well as presenting ablations for different reward functions and ablations of the policy.", "summary_of_recommendation": "The paper addresses a relevant robotics problem and introduces an interesting pipeline. However, in its current form, it is missing essential details in almost all of the sections, and some points need to be clarified to understand its potential impact. I, therefore, currently lean towards rejecting the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- many details regarding the method need to be clarified, as otherwise it is impossible to reproduce the method and the experimental results\n- would it be possible to open source the code as well as the affordance dataset? This would heavily benefit the paper\n- why is an additional demonstration needed for the post-grasp movement?\n- it would be great if the authors could provide additional details on whether the text input to the affordance model actually impacts the grasping pose", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- the paper addresses a very relevant problem in robotics\n- paper proposes an interesting overall pipeline \n\nWeaknesses:\n- the paper lacks clarity in quite some locations, i.e., many crucial details of the method are unclear, and also no further details are provided in the appendix (i.e., dimensions of variables, number of training data, number of samples for evaluation, etc.). It thus feels that it would be impossible to reproduce the results given the current state of the paper. That said, it also feels that the appendix is incomplete.\n- I believe the experiments are missing some important and relevant ablations. I.e., how much does the affordance prior change / react to the text task description? I.e., is the text description always something like \"grasp object x\" or does it also contain information about the task that has to be completed after grasping the object, i.e., flip the object?\n- I was unclear why the human wrist trajectories could not be recovered from the video data. Why was one explicit human demonstration needed for the post-grasp trajectory?\n- the paper would substantially improve its impact if the code and data for training the models, as well as the data from the experiments would be open-sourced \n\n\nThis paper investigates a relevant problem in robotics, achieving dexterous manipulation using complicated, high-dimensional hardware, i.e., robotic hands.\n\nGenerally speaking, the introduction section could be made more concise. Moreover, some statements in the introduction could be underlined with citations. Also, there exist better arguments why we need multi-fingered hands for dexterous robotic manipulation than tying shoelaces. For the latter case, I would argue that a bimanual manipulator is inevitable and perhaps more important.\n\nAdditionally, the authors should better highlight in the related work section how this work differs from existing works and where the contributions are. This, in particular, holds true for the section on soft object manipulation, where I feel that the authors should comment or at least briefly mention how they actually achieve soft object manipulation, as they do not use touch sensors or direct force feedback, but rather soft, compliant hardware.\n\nI also feel that section 3 does not contain enough information for it to be a section on its own. Therefore, I propose adding this information at the beginning of the method section.\n\nRegarding the method, in my opinion, the authors might need to create a better overview figure depicting the individual components. While Figure 2 goes in this direction, it is not referenced in the text and thus might be overlooked in the current version of the manuscript.\nMore critically, the method section is missing crucial details regarding the method. In particular, I would appreciate it if the authors added the information in which spaces the individual components live. E.g., do the contact points $\\mu$ with centers $\\mu_1 ... \\mu_k$ live in the space of pixel coordinates, or are these already 3D coordinates. Second, how many dimensions does the hand grasp pose P have? Third, is the wrist orientation $\\theta_{wrist}$ a single scalar or actually a 3D rotation? Also, I was generally wondering why the contact points mu have to be estimated at all. Would not the hand grasp pose P and the wrist orientation be sufficient?\nLastly, the post-grasp trajectory parameterization was also unclear to me. Is it defined as a set of waypoints (if yes, how many?) or differently? Moreover, as mentioned previously, it would be interesting to know about the versatility of the text labels, and it would be even more interesting to see experiments on whether the task description actually influences the grasp pose when trying to achieve different tasks using the same tool. \nWhile it is possible to recover some of this information from the related and cited works, it is absolutely essential that the authors provide all this information in the paper.\n\nMoreover, the authors mention that they extract affordances from three large-scale egocentric datasets. It would here again be interesting whether the authors trained on all the data from the datasets or whether they focussed on specific samples / scenarios that are related to the settings presented in the paper. Ideally, the authors would release the data that they trained their affordance model on with the paper.\n\nMoreover, the authors mention in the methods section that for the post-grasp movement they \"collect one demo of the human doing the task\". As for instance the videos of the EPIC kitchens dataset, in theory, also contain the trajectory of the hand to complete the task, I was wondering why this information has to be provided separately and is not extracted from the dataset. It would be great if the authors could comment on that.\n\nIn line with the question about the dimensionality of the method's components, I was also wondering about the dimensionality of the search space of the sampling-based optimization that is introduced in Section 4.2. Moreover, it would also be good to know how the authors decided on the initial noise scale for the sampling-based optimization. They also do not provide any information on how many trajectories / rollouts are done per episode.\n\nIn the results section, it was again unfortunately unclear to me how the success percentages have been obtained. I assume that they are the average of repeating the task x times, however, this information is not specified in the paper.\n\nMoreover, in Table I, the authors differentiate between train and test scenarios. What was unclear to me is whether the \"train\" scenarios relate to evaluating the final noise distribution obtained by the sampling-based CEM optimization or whether \"train\" relates to evaluating the trained VAE simply in the same scenario that it was trained in. Depending on this information, the drop in performance between train and test that can be observed for DEFT could either be due to the change in the scene or due to the VAE having problems learning the residuals. It would thus be important to have this kind of information.\nAdditionally, while the authors mention that they evaluate the method in different scenarios compared to training, it would be nice if they could provide pictures / videos that showcase how different the scenarios are.\n\nNext, it felt to me that the author's statement at the beginning of the experiment setup section, i.e., \"over all our experiments, we collect data for several thousands of rollouts for over 100 hours of real-world data collection\" somehow even seems to contradict with the statement in the introduction of \"the method enables iterative real-world improvement in less than one hour\". It would thus be great if the authors could differentiate between the timely effort to refine the policies on all tasks and the time needed for performing the thorough evaluations.\n\nFinally, I really appreciated the honest statement at the end of the introduction (i.e., \"While our method begins to show good success on these tasks ... more evaluation is required to complete these tasks more effectively\"). Nevertheless, a similar statement should be added in the actual limitations and discussion section.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "GVvXrnQLb8", "reviewer": "Reviewer_g2GV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a novel method (DEFT) that leverages human-driven priors to quickly learn grasping policies that can immediately be deployed in the real world on an anthropomorphic robot hand. The method is evaluated on 9 real-world tasks, and is shown to outperform baselines that do not include either the real-world finetuning or affordance model.", "summary_of_recommendation": "The work proposes a novel method (DEFT) that can be trained and finetuned quickly in the real world, and showcases impressive results on certain real-world tasks, including the “Pour Cup” task involving liquids. However, the experimental setup and evaluation method lacks some key details and motivation, and further discussion on the intended scope of the method is needed.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Questions:\n- How is success determined? Based on the visualized rollouts, it does not seem clear to me that some tasks (e.g.: “Stir Spoon” or “Squeeze Lemon” tasks) are actually being completed, at least semantically.\n- It seems the crux of the problem is effectively executing a given grasp and maintaining grasp stability throughout a desired trajectory. Is DEFT expected to be robust to unstable grasps or is this a limitation of the method?\n- What is the motivation for selecting these specific 9 tasks? How does these specific tasks showcase both the advantages of an anthropomorphic hand as well as the strength of the proposed method?\n\nTechnical Concerns:\n- L208: Syntax errors / typos\n- The 9 listed tasks in L192-L193 and Table 1 do not match the tasks shown in Figure 1. I assume “Scoop Sugar” and “Stir Straw” are outdated?\n- Scoop Grape task missing from website", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The proposed method is very sample efficient, and can fine-tune in under an hour in the real-world.\n- The method showcases impressive results in some of the tasks; most notably the “Pour Cup” task which highlights the advantages of an anthropomorphic hand.\n- The majority of the technical details are well-specified and easy to follow.\n- The visualized rollouts are very helpful for qualitatively evaluating the resulting learned policy.\n\n\nWeaknesses:\n- I am not convinced that the proposed task suite is most suitable to showcase the advantages of using an anthropomorphic hand. With the exception of “Pour Cup” or “Squeeze Lemon”, one could argue that all of the tasks could be completed with a parallel jaw gripper, and perhaps with greater stability as well. Justification of the tasks chosen would help clarify the scope of the proposed method and strengthen the overall argument.\n- A central argument of the paper is that the proposed method improves anthropomorphic dexterity; however, the visualized rollouts do not seem to suggest that the robot consistently learns stable grasps. In many tasks, such as “Flip Bagel”, “Stir Spoon”, and “Pick Spoon”, the grasped object looks as if it is on the verge of dropping, and in many cases some of the fingers seem to have no impact on the resulting grasp. This may be a limitation of the specific hardware used, but in any case it would helpful to explicitly address these obvious visual discrepancies.\n- Currently, it is unclear how DEFT is expected to perform on more complex tasks involving soft object manipulation. While there is a brief discussion in L253-L255, further discussion may be necessary – as that section stands, it does not provide any concrete insight into DEFT’s performance under these settings.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a method to learn policies for a high-dof anthropomorphic hand.  The method has a couple key ingredients including learning affordances from videos of humans, and a CEM-type optimization method for finetuning the robot policy.

The reviews for this paper had considerable detail, and although all 3 reviewers started at weak reject, all 3 reviewers updated to weak accept after the rebuttal.  In initial reviews, reviewers pointed to various strengths of the method, including the ability to use prior knowledge from human videos, addressing high-dof dexterous manipulation, and reasonably data-efficient real-robot behavior.  A common weakness area pointed out was the omission of key details, including on baseline comparisons and ablations concerning the effect of the affordance prior.  Reviewer g2GV also pointed out that many of these tasks could have been done with a parallel jaw gripper, which perhaps did not best highlight the capabilities of the hand.  The authors had robust discussion with the reviewers, and all reviewers updated to weak accept after understanding the answers to their questions.  Reviewers found the responses satisfactory given the constraints of rebuttal time, and of the used robot hardware.

The topic is very relevant to core problems of robot learning, and the results are encouraging and would be of considerable value for the community to see at CoRL.

As requested by reviewer Tfey and promised by the authors, the authors should release the datasets used for training. Also, it would certainly be nice for the authors to open source the code.  Also, as a minor suggestion to consider for future, the 6dof motion of the arm end-effector appears a bit jerky, and might be helped by having some high-rate smoothing and/or interpolation between different commands chosen by the learned robot policy.
Decision: Accept (Poster)