Dexterous Functional Grasping
Authors: Ananye Agarwal
Shagun Uppal
Kenneth Shaw
Deepak Pathak
Venue: CoRL 2023 Poster (2023)
Keywords: Functional Grasping
Sim2real

Abstract: While there have been significant strides in dexterous manipulation, most of it is limited to benchmark tasks like in-hand reorientation which are of limited utility in the real world. The main benefit of dexterous hands over two-fingered ones is their ability to pickup tools and other objects (including thin ones) and grasp them firmly in order to apply force. However, this task requires both a complex understanding of functional affordances as well as precise low-level control. While prior work obtains affordances from human data this approach doesn't scale to low-level control. Similarly, simulation training cannot give the robot an understanding of real-world semantics. In this paper, we aim to combine the best of both worlds to accomplish functional grasping for in-the-wild objects. We use a modular approach. First, affordances are obtained by matching corresponding regions of different objects and then a low-level policy trained in sim is run to grasp it. We propose a novel application of eigengrasps to reduce the search space of RL using a small amount of human data and find that it leads to more stable and physically realistic motion. We find that eigengrasp action space beats baselines in simulation and outperforms hardcoded grasping in real and matches or outperforms a trained human teleoperator. Videos at https://dexfunc.github.io/.

Dexterous Functional Grasping
Ananye Agarwal Shagun Uppal Carnegie Mellon University
Kenneth Shaw Deepak Pathak
Figure We use a single policy trained in simulation pickup and grasp ohjects like hammers, drills, saucepan, staplers and screwdriver in diffcrent positions and orientations. An affordance model based on matching DINOv2 features is used to localize the object and move above the relevant region of the object. A blind reactive policy then picks up the object and moves it inside the palm firm grasp that post-grasp motions like drilling, hammering, etc can executed. Videos at https / //texfunec.c//////////////////ee gi github... io/.
While there have been significant strides in dexterous manipulation, most if is limited benchmark tasks like in-hand reorientation which are of limited utility in the real world. The main benefit of dexterous hands over two- fingered ones is their ability to pickup tools and other objects (including thin ones) and grasp them firmly in order to apply force. However, this task requires both a complex understanding of functional affordances as well as precise low-level control. While prior work obtains affordances from human data this approach doesn't 't scale to low- level control. Similarly, simulation training cannot give the robot an understanding of real-world semantics. this paper, we aim to combine the best of both worlds accomplish functional grasping for in-the-wild objects. We use a modular approach. First, affordances obtained by matching corresponding regions of different objects and then a low-level policy trained in sim is run to grasp it. We propose novel application of eigengrasps to reduce the search space of RL using a small amount of human data and find that it leads to more stable and physically realistic motion. We find that eigengrasp action space beats baselines in simulation and outperforms hardcoded grasping in real and matches or outperforms trained human teleoperator. Videos https: //deff////ee//ee//// github.io.
Abstract:
Keywords: Functional Grasping, Tool Manipulation, Sim2real
Introduction
The human hand has played pivotal role in the development of intelligence dexterity enabled humans to develop and use tools which in turn necessitated the development of cognitive intelligence.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

2, 4, 5] Dexterous manipulation is central to the day-to-day activities performed by humans ranging from tasks like writing, typing, lifting, eating, or tool use to perform end tasks. In contrast, the majority of robot learning research still relies on using two-fingered grippers (usually parallel jaws) or suction cups which makes them restricted in terms of the kind of objects that can be grasped and how they can be grasped. For instance, grasping a hammer using a parallel jaw gripper is not only challenging but also inherently unstable due to the center of mass of the hammer being close to the head, which makes it impossible to use it for the hammering function it is intended for. Although there are lots of recent works in learning control of dexterous hands, they are either limited to simple grasping or the tasks of in-hand reorientation [6, 7, 8, 9, 10, 11] which ignore the functional aspect of picking the object for tool use.
This paper investigates the problem of functional grasping of such complex daily life objects using low-cost dexterous multi-fingered hand. For instance, consider the sequence of events that take place when one uses hammer. First, the hammer must be detected and localized in the environment. Next, one must position their hand in suitable pose perpendicular to the handle such that a suitable grasp pose may be initiated. A hammer may be feasibly grasped from both the hammer or the head and choosing the correct pose (also known as pre-grasp pose) requires an understanding of how hammers work. Next, the actual grasping motion is executed which is a high-dimensional closed-loop operation involving first picking up the hammer from the table and then moving it with respect to the hand into a firm power grasp. Power grasp is essential to ensure the stability of the hammer during usage. Once this is done, the arm can then execute the hammering motion while the hand holds it stably grast-grasp trajectory). Notably, the act of functional grasping, which is almost muscle memory for humans, is not just a control problem but lies at the intersection of perception, reasoning, and control. How to do it seamlessly in a robot is the focus of our work.
Inspired by the above example, we approach the problem of functional grasping in three stages: predicting pre-grasp, learning low-level control of grasping, post-grasp trajectory. Out of these stages, visual reasoning is the critical piece of the first and third stage, while the second stage can be performed blind using proprioception as long as the pre-grasp pose is reasonable. To obtain the pre- grasp pose, we use a one-shot affordance model that gives pre-grasp keypoints for different objects in different orientations by finding correspondences across objects. To obtain these correspondences, we leverage a pretrained D2 model [12] which is trained using self-supervised learning on internet images. This allows us to generalize across object instances. However, a more challenging problem is how to learn the low-level control for functional grasping the task itself.
We take a sim2real approach for the grasping motion in our approach. Prior approaches to sim2real have shown remarkable success for in-hand reorientation [7, 6] and locomotion [13, 14, 15, 16). How- ever, we observe that directly applying prior sim2real methods that have shown success in locomotion or reorientation yields unrealistic finger-gu mager-gaiting results in simulation that are not transferrable the real world. This is because grasping tools typically involve continuous surface contacts and high forces while maintaining the grasping pose challenges which pose significant sim2real gap and are nontrivial to engineer reward for. We introduce an action compression scheme to leverage a small amount of human demo data to reduce the action space of the hand from 16 to 9 and constrain it to output physically realistic poses. We evaluate our approach across 7 complex tasks in both the real world and simulation and find that our approach is able to make significant progress towards this major challenge of dexterous functional grasping as illustrated in Figure 1.
2 Method: Dexterous Functional Grasping
In this paper, we aim to combine the best of both human data and large scale simulation training to accomplish dexterous functional grasping in the real world. Given an object to grasp we use an affordance model predict plausible functional grasp pose for the hand. Then, we train a blind pickup policy to pickup the object and then grasp it tightly so that the arm may execute the post-grasp trajectory. Our method is divided into three phases the pre-grasp, grasp and post-grasp (see Fig. 2)
2

---

Affordaace Cernependeness functonal pre-gr gre-grasps
Paliy
Goweos
Matton Capture
Passtra data
High-dimennional action
Pre-grasp
Grasp
Post-grasp
To get the pre-grasp pose we use a one-shot affordance model. After annotating one object we are able to get affordances for other objects in that category via feature matching. Given new object. the arm is moved to that point and oriented perpendicular to the principal component of the object mask. The sim2real pickup policy then exccuted and moves the object into power grasp. After this, post grasp trajectory can be safely exccuted.
Figure 2:
In the pre-grasp phase, an affordance model outputs region of interest of the object and we use local object geometry around that region compute a reasonable pre-grasp pose. We train a sim2real policy to execute robust grasps for pickup. However, in contrast to fingered manipulation locomotion where simple reward functions suffice, in the complex high-dimensional dexterous case it is easy to fall into local minima or execute poses in simulation that are not realizable in the real world. We therefore use human data to extract a lower-dimensional subspace of the full action space and run RL inside the restricted action space. Empirically, this leads to physically plausible poses that can transfer to real and stabler RI. training.
2.1 Pre-grasp pose from affordances
An affordance describes region of interest on the object that is relevant for the purpose of using it. This usually cannot be inferred from object geometry alone and depends upon the intended proper use of the object. For instance, by just looking at the geometry or by computing grasp metrics we could conclude that grabbing a hammer from the head or handle are both equally valid ways using it. However, because we have seen other people use it we know that the correct usage to grab the handle. This problem has been studied in the literature and one approach is use human data in the form of videos, demos obtain annotations for affordances. However, these are either not scalable too noisy to enable zero-shot dexterous grasping.
Another approach to leverage the fact that affordances across objects usually correspond. For all hammers, no matter the type the hammering, affordance will always associated with the handle. This implies that feature correspondence can be used one-shot fashion to obtain affordances. In particular, we use Hadjivelichkov al. [17], where for each object category we annotate one image object instance we simply match DINO-ViT features to find the region which matches the specified mask. Since the mask may bleed across the object boundary we take its intersection with the segment obtained using DETIC [18]. Taking the center the resulting mask gives us the keypoint (Zimg: Vimg) in image space corresponding to the pre-grasp position. To get the Zimg- we project to the points (Fimg. Ming) into the aligned depth image and then transform by camera intrinsics and extrinsics to get the corresponding point in the coordinate frame of the robot (Irobot, Mlobot 7robot). To get the correct hand orientation q the object mask obtained from DETIC and take the angle perpendicular to its largest principal component. Since there three cameras, each along x, y, axes (Fig. 7) repeat this process
from the internet with its affordance mask. To obtain the affordance mask for

---

for each camera and pick the angle that has the highest affordance matching score (see Fig. ??). This allows us to grasp objects in any direction, like upright drills and glasses.
Given the pregrasp pose, we first move the hand a point fixed offset (Iirobat: Yrobot: 2roba) + 8v where 5v is a fixed offset along the chosen grasp axis. We then move the finger joints to a pre-grasp pose with the joint positions midway between their joint limits. We found that same pre-grasp pose to work well across objects since our policy learns to adapt to the inaccuracies in the pre-grasp.
2.2 Sim2real for dexterous grasping
Once the robot is in plausible pre-grasp pose it must execute the grasp action which involves using the fingers to grip the object and then moving it into stable grasp pose. This requires high frequency closed-loop control. Further, this is typically a locally reactive behavior which can be accomplished using proprioception alone. Indeed, once we move our hand close to the object we wish to grasp we can usually pick it up even we close our eyes. However, the challenge is that learning high frequency closed loop behavior typically requires lot of interaction data which is missing from human videos and infeasible to scale via demos. In the past, sim2real has had remarkable successes in locomotion and in-hand dexterous manipulation in learning robust and reactive policies and propose to use this method here.
Dexterous manipulation however presents a unique challenge because of its high-dimensional nature. is easy for the hand to enter physically inconsistent poses or experience self collisions. Further, RL in high dimensional action spaces is unstable or sample efficient. We propose to leverage a small amount of human data restrict the action space to physically realistic poses.
Eigengrasp action space A small number of human demos are often used to guide RL. towards reasonable solutions like offline RL [19], DAPG [20]. However, the main problem with these is that they fail to learn optimal behavior from highly suboptimal demos. Further, the coverage of the demo data may be very poor which can artificially restrict the exploration space of the RL algorithm. We propose simple alternative to these approaches which works from a few demos and can discover optimal behaviors even from suboptimal data. Our insight is that we have a very weak constraint on the behavior of the RL policy. We only care that the individual hand poses are realistic and not so much about the exact sequence in which they occur. We can therefore restrict the action space such that only realistic hand poses are possible.
In particular, suppose we are given mocap dataset D = ( T where T; (x1,..... X and X; E IR. 16 is a set of joints angles of the 16 dof hand. We perform PCA on the set of all hand poses to get 9 eigenvectors eI,..., 6m where m = 9. These vectors are called eigengrasps [21] and have been classically used in grasp synthesis approaches. Here, we instead use it as a compressed action space for RL. Our policy predicts m-dimensional actions π (o,) = a, E R. The raw joint angles are then computed as a linear combination of eigenvectors (at)ee +... (a))eek- This transformation reduces the action dimension of the RL problem and decreases sample complexity in addition to enforcing realism. It also exploits the property that the convex combination of any two realistic hand poses is also likely to be realistic. Thus, doing PCA (as opposed to training a generative model) allows the policy to output hand poses that were not seen in the dataset. Empirically, we find that this stabilizes training and minimizes variation between different random seeds.
Rewards We train our policy to lift objects off the ground and them firmly grasp them in their hand. We find that a simple reward function that a combination of two terms Tihreshold and Thand-ohj enough. The first, is a binary signal incentivizing the policy to pickup the object Titreshola (4) I ((ro)((t)= 2 0.04cm] and the second sum of exponentials and an L2 distance to incentivize the object be close to the palm of the hand
3
Fobb
Thand] ) - 4 44/[rab) Thand
Thand-ohj (1) = Σ exp
di

---

where d1 = 10cm, d2 5em and d3 lem. The overall reward function is r(t) = Thand-obj (t) - 0.1 . (4reshola(()) 1. Due to the eigengrasp parameterization we do not need any additional reward shaping terms.
Policy Architecture We use recurrent policy as that maps observations o, E R.6 to actions a, E Rº. stateful policy is able to adapt to changes in environment dynamics better than a feedforward one. This allows our robot to adapt to slight errors in the pre-grasp pose from the affordance model. The policy observes the 7 dimensional target pose (position, quaternion) of the end-effector and the 16 joint angle positions of the hand.
Training environment We our policy be robust different surface properties and geome- tries and grasp them firmly. We therefore domain randomize the physical properties of the object, robot and simulation environment. We procedurally generate a set of hammers in simulation with randomized physical parameters. The hand is initialized in rough pre-grasp pose with hand joint angles zeroed out. This corresponds to a neutral relaxed pose for the hand. The end-effector pose is initialized to be close to the real world pose obtained from the affordance model. The arm is kept close to the ground for allow the grasp to execute and then spun around in a circle. Episodes are terminated if the hand object distance exceeds 20cm. This spinning motion produces tight grasps and we emergent behavior where the hand adjusts its grasp in response to changes in orientation. We also randomize physical properties of the simulation and add gaussian noise to observations and actions to simulate actuator noise (see Tab. 4).
2.3 Post-grasp trajectory
Once the object tool is firmly grasped, since it is mounted 6-dof arm it can be moved arbitrarily in space to accomplish tasks such as screwing, hammering, drilling, etc. During training and evaluation we use either motion capture trajectories or define set of keypoints and interpolate between them, but in principle these could be obtained from other sources such as internet video or third person imitation.
3 Experimental Setup
We demonstrate the performance of our method on a variety of objects, both similar and very dissimilar to the training objects like stapler, drill (light and heavy), saucepan, hammer (light and heavy). In our real world experiments, we aim understand the reliability and efficiency of our method relative to expert teleop oracle (20 hours) and hardcoded grasping primitive. The former acts upper bound the performance of the hardware while the latter is designed to show that large scale sim training yields more robust policy than a grasping hardcoded.
In simulation, we test the effectiveness of our restricted action space and policy architecture. First, we compare against an unconstrained baseline that operates the full dimensional action space. Second, we compare against policy that operates in the latent space of VAE trained on the mocap dataset. Unlike our method, since VAE is a generative model it can only output hand poses seen in the dataset and cannot extrapolate to ones. Finally, we compare to a feedforward version of our method where the RNN policy is replaced by feedforward one. This is designed to test whether recurrence helps in adaptation to domain randomization.
We experimentally validate the pre-grasp affordance matching [17] part of our pipeline separately. We compare against CLIPort [22] and CLIPSeg [23], two CLIP-based affordance prediction methods. CLIPort uses demonstration data to learn the correct affordances in a supervised fashion. CLIPSeg uses CLIP text and image features zero-shot segment object given text prompt.
5

---

Averuge Reward Drill
Success Rate
Hammer
Screwdriver
Hammer
Drill
Screwdriver
Unconstrained
213.40 + 169.37
102.12 + 36.12
121.28
96.05
0.60 + 0.55 0.09 0.11 0.46 + 0.45
VAE
140.60 + 109.24
83.34 = 43.32
117.25
76.26
0.30 + 0.44
0.08 0.18
0.25 + 0.41
Feed-forward
232.80 + 175.59
104.61 + 44.84
153.19
+ 105.83
0.60 + 0.54
0.56 = 0.52 0.21 0.19
Ours
327.40 = 11.61 129.03 + 22.58
+ 2
1.00 + 0.00 0.23 + 0.16 0.95 = 0.10
Table 1: We measure the average reward and success rate of the trained policy in simulation. For each method we train policy hold the object close to the palm while arm spins. success is counted when the arm does not drop the object at anytime. We see that our method outperforms the baselines and has significantly less variation between the runs. This is likely because the restricted action space makes the exploration problem easier and the physically plausible poses help keep the motion smooth. Each policy was trained randomized hammer but still generalizes to other different objects.
3.1 Hardware
We use the xarmó with our own custom hand pictured in Fig. 7. The arm has 6 actuated joints, while the hand has 16 joints, four on each digit (three fingers and one thumb). An overhead calibrated D435 camera facing downward is used to obtain masks and affordance regions. The hand consists of Dynamixel servos mounted in a special kinematic structure designed to maximize dexterity [24]. We use an overhead D435 camera to obtain pre-grasp end-effector poses. Both the arm and the hand run at 30Hz. To teleoperate the hand and collect human demos for cigengrasps with Steam VR lighthouses which gives fingertip and hand positions which are then retargeted to our hand as in Figure 8.
Manus VR glove
3.2 Implementation Details
We use IsaacGym [25] as simulator with IsaacGymEnvs for the environments and rl-games the reinforcement learning library. The policy contains a layer-normed GRU with 256 as the hidden state followed by an MLP with hidden states 512, 256, 128. The policy is trained using PPO with backpropagation through time truncated at 32 timesteps. We run 8192 environments in parallel and train for 400 epochs.
4 Results and Analysis
4.1 Simulation Results
We train each baseline and our method for 400 epochs over 5 seeds. We find that ours beats all other methods primarily because it is stable with respect to the seed whereas the other baselines fluctuate widely in performance across seeds resulting in a high standard deviation and lower average overall performance. Note that our method also perfectly solves the training task for all seeds. This likely due to a combination of two factors (a) the restricted action space nearly halves the action dimension (from 16 to 9), since the search space scales exponentially with action dimension this cuts down the space significantly and it is more likely that the algorithm discovers optimal behavior regardless of seed, and (b) since each hand pose is realistic and doesn't 't have self-collision it leads to smoother and more predictable dynamics in simulation allowing the policy to learn better.
The RNN policy is also better and more stable than the feedforward variant as reported in Table 1. This is because (a) an RNN can use the hidden state adapt domain randomization (b) since the hand hardware does not output joint velocities, the feedforward policy has no idea of how fast the fingers are moving which can hinder performance. The RNN on the other hand is able to implicitly capture velocity of joints in the hidden state and this helps it to learn better.
4.2 Real World Results
We choose variety of objects to compare against hammer (light and heavy), saucepan, drill (light and heavy), stapler and screwdriver. Of these, hammer and saucepan are quite similar to the

---

Hammer (unseen)
Spatula (seen)
Frying Pan (seen)
Pick success
IoU
Pick success
IoU
Pick success
IoU
CLIPort 2/10
0.034
6/10
0.15
7/10
0.15
ClipSeg
1/10
0.05
2/10
0.06
1/10
0.014
Ours
9/10
0.33
8/10
0.23
7/10
0.17
Table 3: We compare our affordance matching against CLIPort and CLIPSeg terms of pick success rate and IoU between the predicted and ground truth affordance (human-annotated), We use the simulated CLIPort dataset for both unseen and seen objects. Our method outperforms CLIPort on both seen and unseen categories. CLIPSeg fails because it does not capture object parts such the handle of the hammer.
training distribution because of the handle geometry while the drill, stapler and screwdriver have substantially different geometry. The heavy drill is especially challenging because of its narrow grip and unbalanced weight distribution. We run 10 trials per object per baseline in the real world (see Table 2). For all objects except the saucepan we execute a post-grasp trajectory where the object is picked up and waved around to test the strength of the grasp. For the saucepan we simply pick up since waving around is safety hazard. During each trial, the orientation is randomized in the range [-m, x and position randomized in the entire workspace 1m x 0.5m, the affordance model is run and the hand is moved the pre-grasp pose. Videos at https //tp////ee.///////////e//eeeeeee gi gitb.b.
We obtain the hardcoded baseline by interpo- lating between the fully open and fully closed cigengrasp over 1s. This leads to the hand quickly snapping shut before the arm rises up. We find that this baseline performs poorly and gets zero success rate many objects, espe- cially thin ones. This is because in order to successfully grasp the object the thumb must retract closer to the palm. However, the timing of this crucial, if the thumb retracts too early then the object flies back away from the hand. This the most common failure case of this baseline that observe. The hardcoded grasp succeeds for tall objects like an upright stapler or if the object happens to be in favorable pose at the time of grasping.
Success Rate f
Telcop Oracle Hardeoded
Ours
Hammer (heavy)
0.5
0.0
0.8
Hammer (light)
0.6
0.3
0.9
Sauce pan Drill (heavy)
0.9
0.3
0.9
0.9
0.2
0.5
Drill (light) Stapler Screwdriver
0.9
0.3
0.8
0.9
0.3
1.0
0.5
0.0
0.7
Table 2: We show functional grasping for a varied set of objects. We compare to a hardcoded pinch grasp and a trained teleoperator with VR glove. The hardcoded baseline fails since the fingers push the object behind. Our method is able to beat the teleop oracle on challeng- ing objects such as screwdriver, stapler and hammer.
The teleop oracle baseline was carried out with a Manus VR glove with the joints mapped one to one to the robot hand (ignoring the human pinky). This was teleoperated by trained user (0 hours of experience). This intended to serve an upper bound of hardware capability. We find our method matches or slightly lags behind the oracle for drill (light) and saucepan. Surprisingly, for stapler, screwdriver and both hammers it even exceeds the oracle baseline. This is because these objects are heavy and sit close to the ground and require very swift and forceful motion which is also very precise in order to be successfully picked out. This is very hard to execute reliably for a human being, whereas our policy is able to do it well. We also find that our method is able to complete the task in shorter time for the same reason.
4.3 Affordance Analysis
We experimentally validate the pre-grasp affordance matching part of our pipeline separately. We compare against CLIPort [22] and CLIPSeg [23] in terms of both pick success rate and IoU between the predicted and ground truth affordance CLIPort dataset for both unseen and seen objects (Table. 3). For our method, we annotate one exemplar from each category. To obtain affordance from CLIPSeg we prompt with the relevant part of the object such as "hammer handle". Note that Spatula and Frying Pan are present in the CLIPort training data while hammer is new category.
annoottedd... We run evaluation on the simulated
7

---

Our method outperforms CLIPort on both seen and unseen categories. We observe that CLIPSeg fails to localize objects or is not able to capture the functional part of the object and only has understanding of the entire object as a whole (Fig. 6). While CLIPort is able to localize objects better but often predicts bounding boxes that are not functionally correct (such as the pan part of the sauce instead of the handle in Fig. 6).
5 Related Work
In-hand dexterous manipulation: Dexterity in humans the ability manipulate objects within their hand's workspace [26, 27, 28). Accordingly, in-hand reorientation has remained standard, yet challenging task in robotics to imitate human's 's dexterity. In recent years, there has been surge of interest in this field and sim2real approaches have shown some success at reorienting objects [7, 29, 9, 30, 8] and also manipulating them [6, 31J. Other works bypass sim and directly learn in-hand manipulation through trial and error in the real world [11, 10]. Some other works use human demos to guide RI. [20] and others directly use demos to learn policies [32].
Dexterous grasping: While in-hand reorientation is an important task most of the uses of dexterous hand involve grasping objects in different poses. Because of the large degrees of freedom, grasp synthesis is significantly more challenging. The classical approach is to use optimization [21, 33, 34]. This approach is still used today with the form or force closure objective [35, 36, 37]. Some methods use the contact between the object and the hand as a way to learn proper grasping [38, 39, 40, 41J. A VAE can be trained on these generated poses to learn a function that maps from object to grasp pose [36, 42]. Recent works leverage differentiable simulation to synthesize stable grasp poses [43]. Other works don't decouple this problem into a grasp synthesis phase and learn end-to-end in simulation [44], from demonstrations [45, 46, 32] teleoperation [47, 48).
Functional Grasping: While simulation can be powerful tool to optimize grasp metrics, func- tional affordances are usually human data since there may be more than one physically valid grasp pose but only one functionally valid one that allows one to use the object properly. Some ap- proaches rely on clean annotations or motion capture datasets [49, 50, 51, 52] for hand object contact (53, 54, 55, 56, 57]. Some papers learn affordances from human images or video [58, 59] directly or through retargeting. These can however be noisy since they rely on hand pose detectors such as [60, 61] which are often noisy and difficult to learn from directly [45]. Some recent work in this area begin target functional grasping using large scale datasets as prior [62, 63, 64].
6 Limitations and Conclusion
We show that combining semantic information from models trained internet data with the robust- ness of low-level control trained in simulation can yield functional grasps for large range of objects. We show that using eigengrasps to restrict the action space of RL leads to policies that transfer better and are physically realistic. This leads to policies that are better to deploy in the real world on robot hand hardware.
The main failure case of our policy in the real world is due to incorrect pre-grasps from the affordance model. In particular, if the pre-grasp is such that the knuckle of the thumb joint lies over the object then the grasp fails since the hand cannot get the thumb around the object. One way to address this limitation is to equip the robot with local field view around the wrist such that it can finetune its grasp even if the affordance model is incorrect.
Our method currently does not leverage joint pose information from the affordance model. While found this to not be necessary in the set of objects we have, it might be useful in the case of more fine-grained manipulation such as picking up very thin objects like coins or credit cards.

---

7 Acknowledgements
We would like to thank Russell Mendonca, Shikhar Bahl and Murtaza Dalal for fruitful discussions. KS is supported by NSF Graduate Research Fellowship under Grant No. DGE2140739. This work is supported by ONR N00014-2211-29% and the DARPA Machine Common Sense grant.
References
[1] K. Libertus, A. S. Joh, and A. W. Needham. Motor training at 3 months affects object exploration
12 months later. Developmental Science, 196))1058-1066, 2016.
[2] E. J. Gibson. Exploratory behavior in the development of perceiving, acting, and the acquiring of knowledge. Annual review of psychology, 39(1):1-42, 1988.
[3] K. E. Adolph and S. E. Berger. Motor Development, chapter 4. John Wiley & Sons, Ltd, 2007. ISBN 9780470147638. www.h//pwwbbuoo////../////////0.
URL
https:// httpp:://onlimelbraayy wiley. com cat.//aa/aaao/0..0000//777111160 chpey0
[4] T. Bruce. Learning through play, for babies, toddlers and young children. Hachette UK, 2012.
[5] R. A. Cortes, A. E. Green, R. F. Barr, and R. M. Ryan. Fine motor skills during early childhood predict visuospatial deductive reasoning in adolescence. Developmental Psychology, 2022.
[6] O. M. Andrychowicz, B. Baker, M. Chociej. R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron,
M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The Interna- tional Journal of Robotics Research, 39(1);3-20, 2020.
[7] T. Chen, M. Tippur, S. Wu, V. Kumar, E. Adelson, and P. Agrawal. Visual dexterity: In-hand dexterous manipulation from depth. arXiv preprint arXiv:2211. 2022.
[8] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk,
K. Van Wyk, A. Zhurkevich, B. Sundaralingam, al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. arXiv preprint arXiv:2210.13700 2022.
[9] Z. Yin, B. Huang, Y. Qin, Q. Chen, and X. Wang. Rotating without seeing: Towards in-hand dexterity through touch. Robotics: Science and Systems, 2023.
[10] A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.093:: 2020.
[11] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMLR, 2020.
[12] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[13] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. CoRI, 2022.
[14] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7662):ebb22822, 2022.
[15] G. B. Margolis and P. Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity behavior. In Liu, Kulic, and Ichnowski, editors, Proceedings The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 22-31. PMLR, 14-18 Dec 2023. URL https://proceeeee margolis23a. html.
mlr press/v205/

---

[16] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. RSS, 2021.
[17] D. Hadjivelichkov, S. Zwane, M. P. Deisenroth, L. de Agapito, and D. Kanoulas. One-shot transfer affordance regions? affcorrs! Conference on Robot Learning, 2022.
[18] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and Misra. Detecting twenty- thousand classes using image-level supervision. In ECCV, 2022.
[19] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning, 2020.
[20] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 10111709000000 2017.
[21] M. Ciocarlie, C. Goldfeder, and P. Allen. Dexterous grasping via cigengrasps: A low- dimensional approach high-complexity problem. Roboties: Science and systems manipulation workshop-sensing and adapting the real world, 2007.
[22] M. Shridhar, I.. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipula- tion. In Conference on Robot Learning, pages 894-906. PMLR, 2022.
[23] T. Lüddecke and A. Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7086-7099 2022.
[24] K. Shaw, A. Agarwal, and D. Pathak. Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning. 2023.
[25] V. Makoviychuk, Wawrzyniak, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 1044000,0 2021.
[26] R. R. Ma and A. M. Dollar. On dexterity and dexterous manipulation. In 201 15th International Conference on Advanced Robotics (ICAR), pages 1-7. IEEE, 2011.
[27] N. Kamakura, M. Matsuo, H. Ishii, F. Mitsuboshi, and Y. Miura. Patterns of static prehension in normal hands. The American journal of occupational therapy, 34(7):437-445, 1980.
[28] C. MacKenzie and T. Iberall. The grasping hand. Elsevier, 1994.
[29] H. Qi, Kumar, R. Calandra, Ma, and J. Malik. In-Hand Object Rotation via Rapid Motor Adaptation. In Conference on Robot Learning (CoRL), 2022.
[30] I. Akkaya, M. Andrychowicz, M. Chociej. M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXiv: 110.:011, 2019.
[31] Y. Qin, B. Huang, Z.H. Yin, H. Su, and X. Wang. Dexpoint: Generalizable point cloud rein- forcement learning for sim-to-real dexterous manipulation. In Conference on Robot Learning, 2022.
[32] Arunachalam, S. Silwal, B. Evans, and L. Pinto. Dexterous imitation made easy: learning- based framework for efficient dexterous manipulation. arXiv preprint arXiv:2203.13255 2022.
[33] A. Miller and P. Allen. Graspit! a versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, 1144):110-22,, 2004. doi: 10011000//A.22....11...11
10

---

[34] D. Berenson and S. S. Srinivasa. Grasp synthesis in cluttered environments for dexterous hands. In Humanoids 2008-8th IEEE-RAS International Conference on Humanoid Robots, pages 189-196. IEEE, 2008.
[35] R. Wang, Zhang, J. Chen, Y. Xu, P. Li, T. Liu, and H. Wang. Dexgraspner: A large- scale robotic dexterous grasp dataset for general objects based simulation. arXiv preprini 2022.
[36] P. Li, T. Liu, Y. Li, Y. Geng, Y. Zhu, Y. Yang, and S. Huang. Gendexgrasp: Generalizable dexterous grasping. arXiv preprint (.Xix:2210000722 2022.
[37] K. M. Lynch and C. Park. Modern robotics. Cambridge University Press, 2017.
[38] P. Grady, C. Tang, C. D. Twigg, M. Vo, S. Brahmbhatt, and C. C. Kemp. ContactOpt: Optimizing contact improve grasps. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[39] Mandikal and K. Grauman. Learning dexterous grasping with object-centric visual affor- dances. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6169-6176, 2021. doi: 10: (00//CCRA48506.221....3 2021.9561802.
[40] S. Brahmmbatt, C. Ham, C. C. Kemp, and J. Hays. Contactdb: Analyzing and predicting grasp contact via thermal imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[41] S. Brahmbhatt, Handa, Hays, and D. Fox. Contactgrasp: Functional multi multi-finger grasp synthesis from contact. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2386-226622 2393, 2019.
[42] Y. Xu, W. Wan, J. Zhang, H. Liu, Z. Shan, H. Shen, R. Wang. H. Geng, Y. Weng, J. Chen, T. Liu, L. Yi, and H. Wang. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal goal-conditioned policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4737-4746 June 2023.
[43] D. Turpin, L. Wang, E. Heiden, Y.-.. Chen, M. Macklin, S. Tsogkas, S. Dickinson, and
A. Garg. Grasp'd: d: Differentiable contact-rich grasp synthesis for multi-fingered hands. In Computer ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI, pages 201-221. Springer, 2022.
[44] Y. Qin, B. Huang, Z. Yin, H. Su, and X. Wang. Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation. In Deep Reinforcement Learning Workshop NeurIPS 2022.
[45] K. Shaw, S. Bahl, and D. Pathak. VideoDex: Learning Dexterity from Internet Videos. In Conference on Robot Learning (CoRL), 2022.
[46] Y. Qin, Y.-H. Wu, S. Liu, Jiang, R. Yang, Y. Fu, and X. Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXIX, pages 570-587. Springer, 2022.
[47] A. Sivakumar, K. Shaw, and D. Pathak. Robotic telekinesis: Learning robotic hand imitator by watching humans on youtube, 2022.
[48] A. Handa, K. Van Wyk, W. Yang, Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and
D. Fox. Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164-9170. IEEE, 2020.

---

[49] Z. Fan, O. Taheri, D. Tzionas, M. Kocabas, M. Kaufmann, M. J. Black, and O. Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[50] R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitad et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842-5850, 2017.
[51] C. Zimmermann, D. Ceylan, J. Yang, B. Russell, M. Argus, and T. Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813-822, 2019.
[52] O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas. GRAB: A dataset of whole-body human grasping of objects. In European Conference on Computer Vision (ECCV), 2020. URL https://geet is. tue. mpg. de.
[53] S. Brahmbbatt, C. Ham, C. C. Kemp, and Hays. Contactdb: Analyzing and predicting grasp contact via thermal imaging. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8709-87 97199,771,00099 2019.
[54] Y. Liu, Y. Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and L. Yi. Hoi4d: 4d egocentric dataset for category- human-object interaction, 2022.
[55] O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas. Grab: A dataset of whole-body human grasping of objects. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 581-600. Springer, 2020.
[56] S. Dasari, A. Gupta, and V. Kumar. Learning dexterous manipulation from exemplar object trajectories and pre-grasps. In IEEE International Conference on Robotics and Automation 2023, 2023.
[57] A. Patel, A. Wang, I. Radosavovic, and J. Malik. Learning imitate object interactions from internet videos, 2022.
[58] Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak. Affordances from human videos as versatile representation for robotics. 2023.
[59] Y. Ye, X. Li, A. Gupta, S. D. Mello, S. Birchfield, J. Song, S. Tulsiani, and S. Liu. Affordance diffusion:
Synthesizing hand-object interactions. In CVPR, 2023.
[60] Y. Rong, T. Shiratori, and H. Joo. Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1749-1759, 2021.
[61] A. Mittal, A. Zisserman, and H. Torr Hand detection using multiple proposals. In Bmvc, volume 2, page 5, 2011.
[62] Z. Q. Chen, Van Wyk, Y.W. Chao, W. Yang, A. Mousavian, A. Gupta, and D. Fox. Learning robust real-world dexterous grasping policies via implicit shape augmentation. arXiv preprint arXiv:2210.1 2022.
[63] J. Ye, J. Wang, B. Huang, Y. Qin, and X. Wang. Learning continuous grasping function with a dexterous hand from human demonstrations. IEEE Robotics and Automation Letters, 8(5): 2889, 2023.
[64] S. Brahmbhatt, A. Handa, J. Hays, and D. Fox. Contactgrasp: Functional multi-finger grasp synthesis from contact. In 20/9 JEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pages 2386-2392 IEEE, 2019.
12

---

A Grasping along multiple axes
In some cases, an object may be kept upright and a top-down angle of approach does not work. To deal with these cases, we setup three cameras along each axis (Fig. 7) and run affordance matching for each one. We finally pick the axis that has the highest score and move the hand along that axis to the pre-grasp pose. See Fig. 3, for vizualization. Empirically, find that the confidence score is indeed always highest for the correct direction of approach.
Figure 3: Affordance prediction an upright drill from multiple angles. The best angle approach is from the side and that is also the angle with highest affordance score. Our system picks this angle and executes grasp.
Confidence
Figure Affordance prediction an upright mug from multiple angles. Our system picks the angle with highest affordance score and executes grasp.
13

---

B Training curves in simulation
Baselines (simulation)
350
300
rewards/step
250
150
100
unconstrained
ours vae feedforward
Number of envsteps
1e8
Figure 5: Training curves for baselines in simulation. Each baseline is run over 5 seeds. We see that ours outperforms the other baselines and also is stable with respect the seed. This because of the lower dimensional action space.
C
Qualitative results for affordance prediction
Ground Truth
CLIP-Seg
CLIPort
Ours
Hammer
Saucepan
Spatula
Figure 6: Qualitative comparisons of the affordance prediction from method and CLIP- CLIPort. Overall, our method produces predictions that more functionally aligned. CLIP-Seg zero-shot method and fails to localize the object correctly in many cases. CLIPort is able to localize the object but predicts grasp points that are not functional, for instance it predicts a bounding box around the head of the saucepan in addition the handlc.
14

---

D Hardware Setup
Top View
Side View
Front View
Figure 7: Hardware setup with LEAP hand mounted on xarmó with one D435 along each axis.
MNV
Figure 8: (left) the Manus VR glove we use to teleoperate our hand (right) the hand in the retargeted pose.
E
Domain Randomization
For robustness, domain randomize physics parameters as shown in Tab. 4.
Name
Range
object scale
[0.8, 1.2
object mass scaling
00.5, 1.5
Friction coefficient
[0.7, 1.3]
stiffness scaling
[0.75, 1.5)
damping scaling
[0.3,
Table 4: domain randomization in simulation
15


Reviews:
[{"review_id": "xhH_FhESkW", "reviewer": "Reviewer_TMM4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses the functional grasping of objects with a dexterous hand. The process of dexterously manipulating an object is broken down into 3 stages including approaching the object with a pre-grasp that enables subsequent manipulation, stably picking up the object, and finally executing a post-grasp trajectory for tool use. This paper mainly investigates the second stage where a RL policy is learned to pick up objects in simulation and then transferred to the real world.", "summary_of_recommendation": "This paper mainly introduces a method to stably pick up objects with large handles with a dexterous hand. The policy is trained in RL and has shown reasonable transfer to the real world. This grasping module can potentially be combined with task-oriented pre-grasp and post-grasp adjustment to achieve dexterous tool use. However, only the grasping module is evaluated and the whole pipeline is left to be validated.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. Please discuss whether the method can generalize to a wider range of objects.\n2. Please discuss and compare to existing vision-based policies.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. Several design choices are made to improve the performance of the grasping policy including using a recurrent neural network to encode the history of joint angles during the grasping motion and transforming the action space from the raw joint angles to linear weights for 9 eigengrasps computed from mocap data. These two strategies are shown to improve the success rate in simulation. \n2. The problem this paper studies is relevant. Transitioning task-oriented grasping and tool use from parallel jaw grippers to dexterous hands is a logical and necessary step.\n\nWeakness:\n1. The paper claims to achieve functional grasping for objects in the wild. However, the diversity of the objects is rather limited. Most of the functional grasps seem to be on objects with large cylindrical handles. It would be necessary to show whether the proposed method can generalize to objects with more complex shapes such as scissors, cups, glasses, and knives.\n2. The main algorithmic novelty seems to be limited to the use of the eigengrasps. \n3. It’s unclear whether other vision-based policies can outperform the blind policy and whether visual observation is needed to generalize to more diverse objects. Comparisons with vision-based methods are needed to verify the benefit of the blind policy.\n4. Even though this paper presents a complete pipeline including determining pre-grasp, picking up the object, and finally executing the manipulation trajectory. The first and last part of the pipeline is rather limited and lack experiment validation. There are many details that need to be considered to build a robust pipeline. For example, how to map pre-grasp poses to objects in random 6-DoF poses and how to recognize the poses of the object in hand after the grasping motion such that the post-grasp trajectory can be reliably executed.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "tilnzf8jBGD", "reviewer": "Reviewer_Kobm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes a method for learning to grasp and execute functional grasping motions on objects with a dexterous hand by using a combination of learning from demonstrations and reinforcement learning. Their key contributions are a) using eigengrasps to reduce the action space of the hand from demonstrations and reduce the difficulty of exploration for reinforcement learning based methods, b) a sim to real approach that leverages a hierarchical policy which breaks the task into a pre-grasp, grasp, and post-grasp trajectory. Their method experimentally validates their contribution by demonstrating their method's superior success rate and performance in simulated and realistic robotic grasping and control environments.", "summary_of_recommendation": "I recommend this paper be accepted given it's strong demonstrated results on real hardware. It also showcases a novel use of eigengrasps as a way of simplifying the action space for dexterous grasping and manipulation. However, this work would benefit from clearly contextual comparisons to related work, which go largely unaddressed. Lastly, this paper includes a fairly convincing set of demonstrations, with results that seem to outperform the human demonstrator on specific objects, which should be built on further in future work in terms of scaling the approach.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "The main limitations I see from this work stem from the experimental setup and which methods are compared with in their approach. This is where some details on which features provide what sort of benefit to the overall system, and which ones are extraneous are in question. For instance, were there any concrete advantages to using DINO-ViT features when training the model, or would some form of CLIP embeddings with perform just as well? These are the kinds of comparisons and evaluations that could help Table 1 clearly demonstrate the key advantages to the approach taken. A mention of competing/similar features used for training similar kinds of systems would be useful to include in the related work and/or Limitation section. For example, see methods like [1-3].\n\nSimilarly, what are the limitations of existing works that attempt works in approaching dexterous grasping with a similar approach, such as Dasari et al. '23, Handa et al. '22, and others? These kinds of comparisons to what is presented can be simple without requiring additional evaluation, but the related works section currently does not reference these methods very clearly in the context of this work, besides the topical relevance. It would be more convincing if they are able to demonstrate the clear advantages that are posed by this method (which are there given the real world demonstrations that many of these past methods fail to show).\n\nLastly, It would be good for this work to highlight how this approach may benefit from larger scale datasets, and showing an additional experiment comparing the performance using fewer oracle demonstrations would be convincing in showing how this approach could be built on further by more data-collection.\n\n[1] Wen, M., Kuba, J., Lin, R., Zhang, W., Wen, Y., Wang, J., & Yang, Y. (2022). Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems, 35, 16509-16521.\n[2] Xie, Z., Lin, Z., Li, J., Li, S., & Ye, D. (2022). Pretraining in Deep Reinforcement Learning: A Survey. arXiv preprint arXiv:2211.03959.\n[3] Turpin, D., Zhong, T., Zhang, S., Zhu, G., Liu, J., Singh, R., ... & Garg, A. (2023). Fast-Grasp'D: Dexterous Multi-finger Grasp Generation Through Differentiable Simulation. arXiv preprint arXiv:2306.08132.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, the clarity of this paper is very well presented, and clearly delivers an original approach to improving dexterous grasping using a combination of human demonstration and reinforcement learning, which they demonstrate on real hardware. \n\nThis paper shows a fairly well-developed set of comparisons across competing metods, including policies implemented as a VAE, feed-forward, and without the eigengrasp action space. It would be worth showing in the supplemental how different combinations of these other methods, for instance using the VAE with the unconstrained actions, could potentially improve the performance of either, since the VAE would implicitly learn a compressed representation of the actions, without relying on the eigengrasps as yet another lower-dimensional representation of the hand poses. \n\nThe experimental setup is also clearly described and demonstrated, although it is also very surprising that the tele-op \"oracle\" performs worse than the trained policy on lifting some of the objects, and it is certainly a strong positive result from this proposed method.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "X7ZEN7pNSw", "reviewer": "Reviewer_qb6f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The presented work deals with the problem of generating functional grasps for arbitrary objects  with dexterous robot hands. The paper proposes a two-phase method in which first an affordance point in the image is found and the robot is moved to a pre-grasp pose. Then, a learned RL policy is applied to properly grasp the object. To learn the RL policy, the authors propose a sim2real approach.\n\nTo obtain the pre-grasp pose in the image, the authors leverage DINO-ViT features. The authors propose matching the features of a previously annotated object with the object of interest. This feature matching will provide a (x,y) pixel in the image that is the one that best matches the features of the annotated image. \n\nOnce in the pregrasp pose, the grasping policy is learned by RL in a reduced dimensional action space that is linear to the original action space (Eigengrasps).", "summary_of_recommendation": "I suggest a rejection in the current stage of the work. Nevertheless, I believe that the paper could be an accepted paper if they properly address the current weaknesses.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The questions for rebuttal are in line with the highlighted weaknesses.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well written and it is easy to follow. The authors motivate their problem properly with an example that helps the reader understand the complexity of the problem to be tackled. \n\nOne limitation of their introduction is the lack of proper introduction of previous approaches to the same problem. It is important to frame the proposed method in comparison with other works. This allows to frame the papers contribution in contrast with previous approaches. Is there any work aiming to solve dexterous grasping? How do they tackle their problem? What are their limitations? How does your proposed method improve over those limitations?\n\nWhile the paper is not introducing novel methods in the algorithmic side, the proposed model might be interesting from a systems perspective. Nevertheless, the author fail to cite previous works on similar approaches in which the robot skill is represented as a two-phase problem of choosing a place to go and executing the skills. These methods are usually known as Coars-to-Fine approaches [1,2].\n\nThe paper presents a good evaluation of the grasp policy and compare their proposed EigenGrasp with other approaches that reduces the action space. Nevertheless, there is no any evaluation of the pre-grasp selection phase. Given, the pre-grasp selection is claimed as a contribution, it is mandatory to have an evaluation of that part.\n\n\n**Strenghts**\n\n1. The proposed problem is challenging and the proposed solution is interesting and makes sense.\n\n2. The real-robot experiments for such a challenging task are highly appreciated.\n\n**Weaknesses**\n\n1. The lack of visual/tactile input in the RL policy limits the scope of the solution. The robot might need a visual signal to properly execute the grasping.\n\n2. While the authors provide an interesting comparison of different approaches in the RL part, the paper is composed of two distinguished parts. The authos should provide a proper comparison of the affordance based approach with other affordance based methods, such as CLIP-Port [3] or Transported-Networks [4]. \n\n3. The paper is missing proper citations to Coarse-to-fine works that frame the problem in similar way.\n\n\n[1] Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration, Johns\n\n[2] Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learning, Valassakis et al.\n\n[3] Shridhar, Mohit, Lucas Manuelli, and Dieter Fox. \"Cliport: What and where pathways for robotic manipulation.\" Conference on Robot Learning. PMLR, 2022.\n\n[4] zeng, Andy, et al. \"Transporter networks: Rearranging the visual world for robotic manipulation.\" Conference on Robot Learning. PMLR, 2021", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "V-VsefQyxpz", "reviewer": "Reviewer_6xz6", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a system for functional grasping of household objects with a multifingered robot hand. The objects is placed without clutter on a tabletop and its image is matched to a canonical image of an object of the same class. Functional grasp affordance analysis on that image object is thus transferred to the real object, and the robot and is moved to the vicinity of the grasp location. Next, an RL policy trained in simulation is used to pick up and grasp the object. Finally, the arm is moved away using a keyframed motion.\n\nExperiments are conducted on a real robot setup and the proposed algorithm is compared to a teleoperation policy that is recorded using cybergloves, and a hardcoded grasping primitive.", "summary_of_recommendation": "I am voting to \"weak accept\" this paper because of strong experimental performance on a real robotic system for a relevant and non-trivial problem - functional grasping with multi-fingered hand. The paper clearly mentions the research problem and shows that the proposed method addresses it.\n\n### Post-rebuttal\nThe authors addressed both my concerns, so I am upgrading my rating to \"strong accept\".", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please include some information about evaluation of the affordance transfer method. For example, annotate the ground truth affordance on a couple of objects, place them in various poses and evaluate whether the proposed method is able to predict correct affordance. This is important because the downstream grasp step depends completely on affordance prediction. Do DINO-ViT features track correspondence across viewpoints? For example, if the prototypical Internet image used shows the drill lying down, but in the real world the drill is standing up resting on its handle.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Strengths\n- The paper clearly formulates the research problem, describes a method to solve it, and evaluates it with experiments. The paper is well written and easy to understand.\n- Strong real robot performance.\n- Teleoperation as a baseline is a nice idea, because it also upper bounds the imitation learning baseline.\n\n### Weaknesses\n- The paper does not evaluate the accuracy of the affordance prediction method it uses.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a method to perform functional grasping of objects with a dexterous hand. They do this by breaking down the problem into multiple components. They perform affordance prediction from image observations, followed by a scripted move to a pre-grasp pose. They then use an RL policy in simulation to pick up and manipulate the object and transfer this to the real world. The authors present experiments on a real world system and break down evaluation of both the pre-grasp performance (added during rebuttal) and the post-grasp RL system. The experiments and ablations are quite comprehensive.

Reviewers highlighted that the rebuttal responses were compelling and this AC agrees that the rebuttal is particularly strong. Early feedback was primarily around experimental setup and missing ablations. In fact this was consistent with all reviewers. The authors addressed this feedback almost entirely by providing new experimental results and comparing (both experimentally and via updated manuscript content) with existing works..

While this did not impact the final recommendation, one AC comment would be to refrain from highlighting and summarizing reviewer comments that are not necessarily consistent with all reviewer feedback. As an example, “novel approach” is highlighted yet this is misleading as not all reviewers considered the system components novel, yet the author's response would suggest it is.
Decision: Accept (Poster)