Get a Grip: Multi-Finger Grasp Evaluation at Scale Enables Robust Sim-to-Real Transfer
Authors: Tyler Ga Wei Lum
Albert H. Li
Preston Culbertson
Krishnan Srinivasan
Aaron Ames
Mac Schwager
Jeannette Bohg
Venue: CoRL 2024 (2024)
Keywords: Multi-Fingered Grasping
Large-Scale Grasp Dataset
Sim-to-Real
TL;DR: We show that learned grasp evaluators enable robust real-world dexterous grasping when trained at sufficient scale with perceptual data. We release a dataset of 3.5M labeled grasps and show evaluators trained on it achieve SOTA hardware performance.

Abstract: This work explores conditions under which multi-finger grasping algorithms can attain robust sim-to-real transfer. While numerous large datasets facilitate learning *generative* models for multi-finger grasping at scale, reliable real-world dexterous grasping remains challenging, with most methods degrading when deployed on hardware. An alternate strategy is to use *discriminative* grasp evaluation models for grasp selection and refinement, conditioned on real-world sensor measurements. This paradigm has produced state-of-the-art results for vision-based parallel-jaw grasping, but remains unproven in the multi-finger setting. In this work, we find that existing datasets and methods have been insufficient for training discriminitive models for multi-finger grasping. To train grasp evaluators at scale, datasets must provide on the order of millions of grasps, including both positive *and negative examples*, with corresponding visual data resembling measurements at inference time. To that end, we release a new, open-source dataset of 3.5M grasps on 4.3K objects annotated with RGB images, point clouds, and trained NeRFs. Leveraging this dataset, we train vision-based grasp evaluators that outperform both analytic and generative modeling-based baselines on extensive simulated and real-world trials across a diverse range of objects. We show via numerous ablations that the key factor for performance is indeed the evaluator, and that its quality degrades as the dataset shrinks, demonstrating the importance of our new dataset. Project website at: https://sites.google.com/view/get-a-grip-dataset.

Get a Grip: Multi-Finger Grasp Evaluation at Scale Enables Robust Sim-to-Real Transfer
Tyler Ga Wei Lum" " Albert H. Li Preston Culbertson', Krishnan Srinivasan', Jeannette
Aaron D. Ames", Mac Jeannett Bohg" Equal Contribution, f Stanford University, Caltech ftylerlum, krshna, schwager, falberthii, pculbert, ames)@caltechh edu
edu
Abstract: This work explores conditions under which multi-finger grasping al- gorithms can attain robust sim-to-real transfer. While numerous large datasets facilitate learning generative models for multi-finger grasping at scale, reliable real-world dexterous grasping remains challenging, with most methods degrading when deployed on hardware. An alternate strategy is to use discriminative grasp evaluation models for grasp selection and refinement, conditioned on real-world sensor measurements. This paradigm has produced state-of-the-art results for vision-based parallel-jaw grasping, but remains underutilized in the multi-finger setting. In this work, we contend that existing datasets and methods have been in- sufficient for training performant discriminative models for multi-finger grasping. To train grasp evaluators at scale, datasets must provide on the order of millions of grasps, including both positive and negative examples, with corresponding per- ceptual data resembling measurements at inference time. To that end, we release new, open-source dataset of 3.5M grasps 4.3K objects annotated with RGB images, point clouds, and trained NeRFs. Leveraging this dataset, we train mul- tiple vision-based grasp evaluators that outperform both analytic and generative modeling- brased baselines without evaluators on extensive simulated and real-world trials across a diverse range of objects. show via numerous ablations that the key factor for performance is indeed the evaluators, and that their quality degrades as the dataset shrinks, demonstrating the importance of our new dataset. Project web- site at: https: /sites......///s..////////////e/ google. com/vient //tp//gtaa a a-grip- datt-aate aset.
Keywords: Multi-Fingered Grasping, Large-Scale Grasp Dataset, Sim-to-Real
1 Introduction
Dexterous, multi-finger grasping is a longstanding challenge in robotics, with applications in tool use, human-robot collaboration, and robot surgery [1, 2, 33. A common approach for grasp synthesis samples grasp candidates from a generative sampler and refines them with a discriminative evaluator using some grasp metric. Table summarizes prior works employing this "coarse-to to-fine" approach.
In terms of grasp evaluation, classical methods [4, 6] analytic metrics [7] to optimize grasps based grasp mechanics, which require ground-truth object geometry and are sensitive sen- sor noise [8, 9]. In contrast, data-driven grasp evaluators [10, 11, 12] estimate grasp quality by conditioning on realistic perceptual data, thus better predicting empirical robustness. While this dis- criminative approach to grasp synthesis has proven effective for parallel-jan grasping [13, 14, 15, 16), its application to the multi-finger setting has seen considerably less success. Consequently, many works study an alternate generative approach of learning and sampling from a distribution over high-quality grasps using large-scale multi-finger grasp datasets [17, 18, 19, 20, 21]. However, the real-world uncertainties in modeling and perception introduce significant sim-to-real gap that com- plicates reliable grasp generation on hardware. Therefore, this work's 's main goal is the synthesis of
8th Conference Robot Learning (CoRL 2024), Munich, Germany.

---

Table I: Comparison of dexterous grasp synthesis methods. SDF Signed Distance Field, BPS Basis Point Set, CVAE Conditional Variational Autoencoder, EBM Energy-Baan -aagee Model.
Method
Sampler
Evaluator
Obj. Data
Graspit! [4]
Heuristic
Analytic (Ferrari Canny)
Shape primitives Mesh, SDF Point cloud
FRoGGeR [22]
Heuristic
Analytic (Min-weight)
Wu et al. [23]
Learned (CVAE)
Analytic (Force closure)
Nex [17]
Hcuristic
Analytic (Force closure proxy)
Mesh
et al. [10]
Heuristic
Learned (CNN)
RGB-D
FFHINet [24]
Learned (CVAE)
Learned (Classifier)
BPS
DexDiffuser [25]
Learned (Diffusion)
Learned (Classifier)
BPS
UniDexGrasp [18]
Learned (EBM)
Learned (RL)
Point cloud
Ours
Learned (Success prob.)
BPS, NeRF
robust, real-world dexterous grasps on novel objects, a challenge that has persisted despite numerous previous efforts.
Upon re-examining the success of discriminative evaluator- based approaches for parallel-jaw grasp synthesis, we observed key criteria in the associated training data: (i) positive and negative grasp examples with high-quality labels, (ii) corresponding perceptual data (e.g., point clouds) used to train meassueement-conditioned models, and (iii) large enough scale to allow generalization in the real world. For example, the evaluator in the seminal work DexNet 2.0 [14] was trained on over 6.7M triples of simulated parallel-jaw grasps, top-down images of the workspace, and expected grasp quality labels (aggregated over several perturbations of each grasp).
In comparison, for multi-finger grasping, no dataset satisfies all three criteria. Almost all recent large datasets only satisfy criteria (i) or (ii), and never simultaneously, because they were designed with generative models in mind (Table 2). Conversely, prior works on evaluator-based methods typically only employ (closed-source) datasets with tens of thousands of grasps, failing to satisfy criterion (iii). We show the historical trend of evaluator dataset sizes in Fig. 18 of Appendix E.
In light of this, we hypothesize that like parallel-jaw grasping. grasp evaluator trained on a large-scale dataset satisfying the above criteria would enable robust, real- world, multi-finger grasp synthesis. We test this in controlled setting where we first train various grasp evaluators dataset of purely simulated grasps and perceptual data, then deploy them zero-shot in the real world on a robotic manipulator over a diverse set of objects. Our many experiments and ablations show that evaluators improve performance (compared to purely generative methods [18, 24, 25, 23]), the resulting performance boosts are uniquely attributable to the evaluators, and dataset size is indeed crucial for performance. In summary, our contributions are follows.
large
An open-source, large-scale dataset. Our dataset consists of 3.5M grasps (for each of the Allegro and LEAP [26] hands) on 4.3K unique objects at multiple scales, which is larger (per-hand) than all existing multi-finger grasp datasets [17, 20, 27). We release synthetic images of these objects from multiple views, corresponding point clouds, and NeRFs trained over these images. We include over 250 successful and failed grasps for each object and soft labels indicating quantities such as the probability of grasp success (PGS).
Comprehensive simulated and real-world evaluations. We test our evaluators trained on our dataset on thousands of simulated and hundreds of real-world grasps on the Allegro hand, demon- strating state-of-the--rt performance. We show that grasp evaluators provide a clear, uniquely attributable performance improvement over evaluator-free methods. Further, our work is the first to quantify the positive effect that dataset scale has on grasp success rates in the real world.
2 Related Work
Traditional approaches to (precision) dexterous grasp synthesis optimize analytic metrics for grasp quality. These methods focus contacts between robot hand and object, typically optimizing for robustness to external disturbances. For review of this large body of work, we refer to [28, 29]. In

---

Table 2: Dexterous Grasp Dataset Comparison. PGS probability of grasp success.
(*) denotes total number of grasps divided by number of hands in the dataset.
Dataset
Grasps
Labels
Objects
Code
Data Observations Has Table
MultiDex [27]
~87k"
+ve only
58
Yes
Possible
Grasp' D-IM [45]
-333k*
+ve only
2.3k
No
Visual
Yes
DexGraspNet [17]
1.32M
+ve only
5.3k
Yes
No
MultiGripperGrasp [46]
~2.8M*
Ranked
345
Yes
No
Get Grip (Ours)
3.5M
PGS
4.3k
Yes
Visual
Yes
general, these approaches [6, 4, 5] are initialized using a heuristic sampler and use sampling-based optimization to optimize an analytic grasp metric. More recent approaches [23, 22, 30] leverage
bilevel optimization to generate provably force closure grasps. These approaches suffer when accurate
models of object geometry are unavailable, which must either be recorded with specialized scanning rigs [31, 32] or reconstructed from visual information into low-quality meshes [30].
When generating grasps in real world, grasp planners must use real-world object representations like RGB images, depth images, or point clouds. A more recent promising representation is the basis point set (BPS) [33], a fixed-length feature vector containing the minimal distances to fixed set of basis points. Neural Radiance Fields (NeRFs) [34] are another representation used for model view synthesis that has proven useful for grasp generation, representing affordances, and semantic grasping [35, 36, 37, 38, 39,40, 40, 41]. Importantly, all of these works study only parallel-jaw grippers, which have limited use for complex manipulation tasks. One reason for this emphasis is that they can be parameterized with single element of SE(3), and it is clear that the relevant NeRF features lie between the jaws. In contrast, the parameterization of multi-finger grasps is not straightforward: we address this challenge below.
Data-driven grasp synthesis methods, as extensively reviewed in [42, 8), leverage grasp datasets to train grasp planners that take real-world perceptual information as input. Many works have used simulation to generate large parallel-jaw grasping datasets [43, 14], which have shown to be highly effective [44, 14]. However, generating large-scale datasets for multi- multi-fingered hands is much more complex due to the high number of degrees of freedom and ambiguities in grasp parameterization. Despite this, recent works have released large-scale dexterous grasping datasets, as shown in Table 2, such as MultiDex [27], DexGraspNet [17], and Grasp' IM [45]. One key limitation of these datasets that they only contain positive examples, so they cannot be used to train evaluator models that can distinguish between successful and unsuccessful grasps. One counterexample MultiGripperGrasp [46), which uses Grasplt! [4] generate grasps that then evaluated in parallel simulation and provides a ranking of grasp goodness. However, it is currently limited to small set of 345 objects, does not contain visual observations, and does not account for table collisions, which are important for most grasping applications. Many methods use combination of sampling and evaluator models, such as Mayer et al. [24], Weng et al. [25], who propose learning both to sample and evaluate multi- fingered grasps using basis point sets as input. However, they needed to generate their own dataset of 180K grasps to train these models. Another line of works focuses on learning differentiable grasp evaluators that can optimize grasps through backpropagation [47, 48, 49, 50]. These works also created their own small datasets on the order of 10k grasps across about 100 objects. Our work seeks to aid in future investigations of evaluator-basee grasping by supplying large-scale dataset which mitigates the substantial effort required to generate a new one.
3 Get a Grip: Dataset Construction and Grasp Synthesis
3.1 Dataset Generation
Our dataset contains objects with associated grasps, RGB images, point clouds, NeRFs, and labels (illustrated in Figure 1). Letn, and ny denote the number of hand joints and fingers. We parameterize
precision grasps as G = (Toh, 0, di...., dny)), where Toh E SE(() is the hand pose relative to
3

---

Images
Point Clouds
NeRFs
Grasps
Compute Smooth Labels
Yool &
Dataset
YPGS
Simulate
Data and label generation. Our dataset supplies the components for robust sim-to-real transfer. For each object in the dataset, provide RGB images of from several random views, a full point cloud, and trained NeRF weights, which allows our data to be compatible with most vision-based methods. We also supply hundreds of grasps per object, each of which is simulated in Isaac Gym multiple times with slight wrist pose perturbations. Averaging, this yields three smooth regression targets indicating the probability of (I) unwanted collisions, (2) simulated pick success, and (3) grasp success, the logical conjunction of (2) and (3).
Figure 1:
the object, R" is the pre-grasp joint configuration, and d; S2 is the direction in which the 7th fingertip moves during the grasp. The grasp data are stored tuples [(* y y y pick ras) where each y E [0, is distinet smooth label for grasp G generated in simulation: Ycoll denotes grasp with unwanted collisions, Ypick that the simulated pick was successful, and YPGS the probability of grasp success, defined as the conjunction of the two.
Our grasp generation pipeline heavily motivated by Wang et [17], but requires key modifications, including (1) placing objects on table with consistent gravity (instead of floating with different gravity directions) to better match the real world, (2) switching from binary success label continuous success probability label evaluated over multiple trials,
(3) modifying evaluation and simulation procedures to improve sim-to-real transfer, and (4) restricting contact points to fingertips.
To procure this dataset, we first generate hundreds of grasps on each object from a diverse object set given by DexGraspNet [17], yielding an initial dataset of about 2M unlabeled grasps. Second, these grasps are simulated in Isaac Gym [51] to compute the associated grasp labels that denote probability of grasp success. Finally, for each of the grasps that more likely to pass than fail, the dataset augmented by perturbing each of these grasps several times and re-evaluating them. Because the initial dataset of grasps has many more failed grasps than successful ones, this augmentation gives new set of grasps with more balanced labels, which, when combined with the initial dataset, yields the final dataset of over 3.5M examples.
Step 1: Grasp Generation. We follow the sampling strategy from Wang et al. [17] by defining an energy function E(G) that is the sum of various penalty terms grasp and sampling low-energy grasps from the distribution P()) exp(-E(G)) using Langevin sampling. For details, see [17].
The samples are initialized by spawning the hand on some inflated level set the object mesh such that the palm faces the object with canonical open-hand joint configuration and then perturbing the configuration with noise. Our weighted energy function (dependence on G omitted) is
E = Ex: + Edis + Wjjoins + Wpen Epn +
W Wspen
+ W Wrpen Expen.
where Ege is the differentiable force closure estimator introduced in [17] that encourages force closure grasps; Edis encourages hand tondttt-object proximity; Ejoints penalizes joint violations; Epen and Expen penalize hand-object and self penetration respectively; Expen is a new energy term that penalizes hand-table penetration that was added keep the grasps above the table. Without this term, many optimized grasps would converge grasps that fully cage the object from above and below in way that would not be possible in the real world due to table collisions.

---

Compared to the energy function in DexGraspNet [17]. we only specify desired contacts on the fingertips to sample precision grasps, we recover pre-grasp poses with fingertips 1 1.5em off the surface instead of on the surface (more clearance aids collision-free motion planning), and we penalize hand-table collisions for kinematic feasibility. Let Prablee)) return the closest point on the table. Then, the table- pebeeeeentttttonn energy term given by
Σ max (p Ptahle g. 8.0).
E Eipen
PEP(H)
where P(H) is a set of predefined points on the hand H and ğ is the gravity vector. Lastly, for each grasp, the directions {d,} are the directions of the fingertips to the closest points on the mesh. Specifying these directions eliminates ambiguity in grasp execution, aiding reproducibility.
Step 2: Grasp Evaluation. We simulate grasps for the above samples in Isaac Gym. The object is first spawned and allowed settle, and then the hand is spawned at the pre-grasp pose relative the object. We assume fingertip i moves by 5cm along d,, and compute corresponding closed-hand configuration Oelie using inverse kinematics. We then use PD control to drive O - Oclose executing the grasp. Lastly, the object is lifted above the table by 20cm.
For each grasp, we check whether (1) the pre-grasp had no collisions, and (2) it was successfully picked while the hand-object pose did not significantly deviate, yielding labels ycall and ypick- We need both, as Isaac Gym's physics may exploit nonphysical hand-object penetration to achieve pick, which we forbid. We then construct the label yPGS Ycoll ypick to indicate the grasp is true, physically- achievable success. Further, we simulate each grasp 5 times with additional small perturbations, averaging the results to recover non-binary labels in [O, I] for regression. This smoothing process has been found to help learn robust grasps [52, 53]; in particular, YPGS is supervised to match an empirical success probability, rather than being a measure of classifier confidence. We emphasize that Yooll. Ypick, and yPGS are smooth labels, which are computed by averaging over all perturbations.
In contrast with [17], we check for feasibility in a tabletop setting rather than spawning the scene in floating environment. We found this assisted sim-to-real transfer by accounting for table penetrations. Moreover, spawning the hand in pre-grasp (rather than in-contact) position helped eliminate non- physical contact modeling behavior in Isaac Gym that led to instabilities in the simulation labels.
Step 3: Grasp Augmentation. For each object, 2-20% of sampled grasps were successes. Small amounts of noise are added to these grasps and fingertip directions 5 times and re-evaluated in Isaac Gym, which yields smoothed, balanced labels after averaging the results. These perturbations are larger than those from Step 2 to promote data diversity. See Appendix B.2 for details.
Object NeRF Dataset. In total, we use 4.3K unique meshes, each of which are nominally centered and randomly scaled such that its maximum extent is between 0.1 and 0.3m. To introduce more scale diversity while retaining dataset size under 2.5TB (see Appendix B.4), a subset of 1700 of the objects are instead duplicated at three distinct scales with max extents drawn uniformly from the distributions Usmanl 0. Umea(0.1, 0.2), Ulagee 0.3) Lastly, we train NeRFs of various quality by uniformly sampling 100 images over a spherical cap above the object with radius 0.45m and polar angle sampled from U(0, 4 and training for 400 iterations with ne frr udio [54, 55].
3.2 Grasp Synthesis with Learned Evaluators
Let the choice of object representation, e.g., point cloud, be denoted O Then, the grasp evaluator is parameterized as a neural network Spen- fpick. fPGS = (G, O) trained by minimizing an equally- weighted 1.2 loss over all 3 labels. At inference time, grasp sampling proceeds in 3 steps: (1) a large batch of candidate grasps T is drawn from a sampler, (2) an initial evaluation culls all but the top K grasps by probability grasp success fPGS into a smaller set S c T, and (3) we iteratively refine all grasps in S and finally compute G arg maxges fPGS (9) afterwards.
We use simple sampling-based update where at iterate we draw some noisy perturbation AG Otherwise, we set ((i-l) =g At inference time, we do not maximize Ppen and Spick. which are only used to guide evaluator training.
frr)) (A and let g(i+1) g(i) + (i if (C()( +AG"))
5

---

(A) Simulation Success Distributions
(B) Dataset Size Ablation
Success Rate
1.00
Success Rate
1.00
0.75
0.50
0.50
0.25
0:25
0.00
Diffusion
Diffusion
BPS-5%
BPS-100%
Method (Sampler/Evaluaton)
Method (Sampler/Evaluator)
Violin plots showing sim results over hundreds of unseen test objects. The short lines mark the median and IQR. "Diffusion" vs. "Fixed" refers to sampling initial grasp g from a diffusion model or fixed set of grasps. "NeRF" vs. "BPS" refers to whether the object O is given by NeRF or basis point set features. (A) Independent of the choice of sampler or object representation, evaluator refinement (right 4 columns) drastically improves planned grasp quality compared to sampling without refinement (left columns). (B) Ablations on dataset size used to train the evaluator, which demonstrates strong correlation between dataset size and grasp planner performance. The percentages indicate the fraction of the dataset used to train the evaluator.
Figure 2:
We choose to study representations O that can be computed with an eye-in-hand setup. We seek to use the most favorable possible visual features to disentangle the effect of the evaluator and dataset from low-quality measurements in the real world. We move the eye-in-hang camera along a spiral- shaped trajectory that gives it diverse views of the object from many angles, yielding high-quality object reconstructions (see the right side of Fig. 10 for visualization).
Two representations compatible with the eye-in-hand are basis point sets (BPSs) [33] and NeRFs [34]. BPSs are fixed sets of points in R labeled with the closest distance to an object mesh or point cloud, essentially acting as fixed-length discretized distance fields for objects, allowing them to be inputs networks like MIPs. NeRFs are rendering-basee models that take in a set of images and their poses, producing a volumetric density field that allows novel-view synthesis. Since parallel- jaw grasping works have shown they are promising representation, for multi-finger grasping. This gives us two distinct evaluators to compare below. We sample point cloud from the NeRF and clean it for usage in training, simulation evaluation, and hardware evaluation (see Appendix D for details on network architecture and this pre-processing).
decide also study them
4 Experiments
We hypothesize that learned grasp evaluators could help achieve robust sim-to-real transfer for multi-finger grasp planners, but only if the associated data are large-scale and account perceptual data modalities that can be measured in the real world. now test this hypothesis on the dataset introduced above. We also ablate other modeling choices like object representation or sampler choice to measure the performance increase uniquely attributable to the evaluator. Lastly, measure the effect of dataset scale evaluator performance (see Appendix A for additional analysis).
4.1 Simulation Results
Though our goal is robust grasping on hardware, evaluation in simulation useful since we can test grasps on a larger set of objects. Further, we can execute multiple perturbed grasps test grasp robustness the identical object pose and perceptual inputs.
First, we study the importance of learned grasp evaluators by comparing with methods that do not use them. We perform an ablation study which we vary the sampling strategy and the evaluation method. For sampling, we either use a diffusion model-based sampler (motivated by Weng et al. [25]) or a simple baseline in which we store fixed set of ~4000 grasps from the training set (details in Appendix D.2). For evaluator refinement, we either use a BPS evaluator, NeRF evaluator, or no evaluator, which randomly chooses grasps (can be viewed untrained evaluator). Figure 2A shows that all methods that use learned grasp evaluators outperform the methods that do not by

---

Robot
Easy
Medium
Hard
Successes
Failures
Figure 3: Hardware setup. Our robot consists of an Allegro hand and ZED 2i camera mounted onto a Franka Research 3. We select objects categorized as easy, medium, or hard based on the complexity of their geometry. the presence of "distractor" geometry, and/or resemblance to objects in the training dataset (see App. E.2 for details). We selected these objects maximize size/shape diversity while providing a clear upper bound on expected performance via the hard objects. The bottom row shows representative successes and failures for the "Fixed/BPS" configuration.
Hardware Success Rates
1.0
0.33
Method (Sampler/Evaluator)
0.9
0.87 0.87
0.83
FRoGGeR
0.8
Diffusion/No Evaluator
0.7
0.6
0.63
0.55
0.62
Diffusion/BPS
0.5
0.43
Diffusion/NeRF
0.4
Fixed/PS
0.3
0.2
0.1
0.0
Easy
Medium Object Difficulty
Hard
Figure 4: Average grasping success rates on hardware across object difficulties. We find that methods that leverage evaluators (hatched) outperform both analytic and generative-modelng-oon baselines. Detailed per-object success rates are shown in Appendix E.3.
large margin, independent the choice of sampler or object representation. The methods that use the fixed set of grasps perform similarly with the methods that use the diffusion sampler when used with learned evaluator, but unsurprisingly, the method that uses the fixed set of grasps performs significantly worse than the diffusion sampler.
Next, we evaluate the effect of training dataset size on evaluator performance. We perform an ablation study in which we train 4 BPS evaluator models with identical architectures and training parameters on different fractional sizes of the training dataset (1%, 5%, 25% 50%, 100%), all of which were trained convergence (see Fig. 19 in Appendix E). After training, we compare the performance of simulation setup with the same fixed diffusion sampler model. Figure 2B
these models in the shows that there is strong correlation between the dataset size and grasp planner performance. It is plausible that this trend will continue with increasing dataset size, but we leave this for future work.
We evaluate all methods on a held-out set of 212 objects. We first use each method to generate grasps per object. To evaluate the robustness of these grasps, we simulate each of these grasps times in simulation, each perturbed with small wrist translation and rotation noise, resulting in 25 grasps per object, which used to compute per-object success rate.
4.2 Hardware Results
In this section, evaluate whether grasp evaluators trained dataset improve grasping performance in the real world. We compare samplers with/without the evaluator, with/without

---

diffusion- based sampler inspired by Weng et al. [25], and also vary the object representation in order to identify the factors driving performance. We also compare to FRoGGieR [22], a fast analytic method that reports strong real-world results [30] as an additional gauge for performance.
For each method, we execute 5 grasps on each of the 20 objects of varying geometry and appearance for total of 100 grasps per method. Our dataset includes simple objects (like boxes), and adversarial ones (like reflective ski goggles), which we organize into easy, medium, and hard categories as shown in Figure 3. We remark that the baselines' reported hardware results were largely limited to convex or nearly-convex objects at low quantity [30, 25]. For a more challenging evaluation, we included more objects with non-trivial visual or geometric properties.
Our hardware setup consists of an Allegro hand mounted onto a Franka Research 3 arm with an eye-in-hand ZED 2i camera. We only monocular RGB because the objects are too close to the camera for reliable depth readings. For repeatability, we choose a canonical pose for each object that we use for all trials. The arm follows a fixed trajectory to collect 100 NeRF training images for every grasp. For each method, we request 40 grasps from the planner (~GPU memory upper bound), then execute the highest-ranking one according to the method's metric (Diffusion No Evaluator has no metric, so the first grasp is selected). Because FroGGeR generates grasps serially, we request only 10 grasps or many as it can generate in 20 seconds. A grasp is considered failure if the planner fails to return any grasps or if it does not pick the object up 20em without dropping it. If feasible grasps are generated but the motion generation fails either during planning or execution (i.e., before the grasp), we discard the trial as the failure is not due to the grasp planner itself.
Figure 4 shows the hardware results. We find that all evaluator methods outperform the baselines by a considerable margin independent of the sampler or object representation, with the worst model in each category achieving 93%, 83%, and 55% success rates respectively. We also observe that all evaluator- based methods perform similarly, which suggests the model performance is not sensitive to the object representation. This may help disentangle representations useful for different downstream tasks from potential negative effects on grasping performance present in other learning-based methods. We discuss qualitative analysis in the Appendices A and E.3. Table 4 also shows that independent of object difficulty categorization, evaluators clearly improve performance.
Limitations
Limitations of our work are as follows. First, our hardware experiments required a time-consuming data collection trajectory and NeRF training to create good quality object models. Future work could explore training models on lower-fidelity object representations from our dataset (e.g., point cloud from depth camera). Second, our methods achieve low success rates on challenging objects such as the mallet and the strainer. The performance of these models is limited by the diversity of their training data, which could be improved by training on a larger set of unique objects, including non- rigid or articulated structures. Third, our grasp generation methods are designed for tabletop settings without clutter or occluding obstacles. Future work could follow the approach used by Sundermeyer al. [44] for parallel-jaw grasping, which trains a clutter-aware model using a no-clutter dataset.
Conclusion
We introduce new dataset of 3.5M grasps on 4.3K unique object with RGB images, point clouds, and NeRFs integrated into dataset, which addresses key challenges for achieving robust sim-to- real transfer of multi-finger grasping. Using this dataset, we perform comprehensive ablation studies that highlight the crucial role of learned grasp evaluators, which significantly outperform existing analytic and generative modeling-based baselines in both simulation and the real world. Our work underscores the critical importance of large-scale grasping datasets and the scale of data for required to train robust multi-finger grasp evaluators. Our contributions represent significant step forward in the development of reliable and generalizable grasping algorithms, demonstrating the potential for large-scale data and learned evaluators drive progress in real-world robotic manipulation.
8

---

Acknowledgments
We thank the reviewers for their helpful suggestions and feedback. This work is supported by the National Science Foundation under Grant Number 2342246 and by the Natural Sciences and Engineering Research Council of Canada (NSERC) under Award Number 526541680.
References
[1] M. Toussaint, K. R. Allen, K. A. Smith, and J. B. Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. Robotics: Science and Systems, 2018.
[2] X. Shu, Ni, X. Fan, Yang, C. Liu, B. Tu, Y. Liu, and H. Liu. A versatile hu- manoid robot platform for dexterous manipulation and human-robot collaboration. CAAI Transactions Intelligence Technology, 9(2):526-544 2024. doi.htts::/doi.oor/10... 1221.. UtLhttps: htts://eetreeeeeeee www.eeeeniniaaannneeeeeeeanniait wiley com/ doi/ /bs/ 1..1049/ /tt49//c/t.... 12214.
[3] F. Khadivar, V. Mendez, C. Correia, I. Batzianoulis, A. Billard, S. Micera. Emg-driven shared human-robee compliant control for in-hand object manipulation in hand prostheses. Journal of Neural Engineering, 19, 2022. 11...110///7111555222222
[4] A. Miller and Allen. Grasplt! IEEE Robotics Automation Magazine, 1104):100-12,, Dec. 2004. ISSN 1070-9932. ieee wtwpg//dcueett//eemmm 11t.tt/11311111
URL http://ieeexplee
[5] M. Ciocarlie, C. Goldfeder, and Allen. Dexterous grasping via eigeenggr:ss:: Alow-dimensional approach to high-complexity problem. In Robotics: Science and systems manipulation workshop-sensing and adapting to the real world, 2007.
[6] C. Borst, M. Fischer, and G. Hirzinger. Grasping the dice by dicing the grasp. In Proceedings 2003 IEEE/RSJ international conference on intelligent robots and systems (IROS 2003) (cat. No.03CH37453), volume 4, pages 3692-3697 vol.3, 2003. doi:10.1 http:/RRO...0..124999999
[7] C. Ferrari and Canny. Planning optimal grasps. ICRA, volume 3, page 6, 1992. Number:
4.
[8] J. Bohg, A. Morales, T. Asfour, and D. Kragic. Data-driven grasp Synthesis A survey. IEEE Transactions on Robotics, 30(2):289-309 2014. 10::10.1199/7R0.20132229::
[9] C. Rubert, D. Kappler, J. Bohg, and A. Morales. Predicting grasp success in the real world study of quality metrics and human assessment. Robotics Autonomous Systems, 121:103274, 2019. ISSN 0921-8890. dechttps//bbiogg/////000l...ooo0 www. www.sscienceedcccc rect. htt.://ac....ctccc.c//.ooooo/eea
URL https: //
[10] Q Lu, K. Chenna, B. Sundaralingam, and Hermans. Planning Multi-Fingered Grasps as Probabilistic Inference in Learned Deep Network. In International Symposium of Roboties Research, 2018.
(11) Q. Lu, M. Van der Merwe, B. Sundaralingam, and T. Hermans. Multifingered Grasp Planning via Inference in Deep Neural Networks: Outperforming Sampling by Learning Differentiable Models. IEEE Robotics and Automation Magazine, 27(2):55 65, 2020. doi: 10t:00.119/MRA.. 2020.2976322.
[12] L. Shao, Ferreira, M. Jorda, V. Nambiar, J. Luo, Solowjow, J. A. Ojea, O. Khatib, and Bohg. Unigrasp: Learning a unified model grasp with multifingered robotic hands. IEEE Robotics and Automation Letters, 5(2):2286-2DD April 2020. ISSN 2377-3774, doi:

---

[13] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eye coordi- nation for robotic grasping with deep learning and large-scale data collection. The Interna- tional Journal of Robotics Research, 37(4-5):421-4 -36, Apr. 2018. ISSN 0278-3649, 1741- URL http: //tr//oourssssss ss. sagepub com/dot /ori/ publisher: SAGE Publications Sage UK: London, England
3176. 16110001777022775541777706
.C..177/02788 http://222664917710000 tex.ids= levine201 8a, dennneeeermingggnnessoooooooonnn
[14] J. Mahler, J. Liang. S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint arXiv:
2017. tex.ids= mahler201 7b, mahler2017des arXiv: 1703.09312 TublicationTitle:
mahler2017dexneta. aatherrDeneetDeeppleaannnggg.. Robotics: Science and systems (RSS).
[15] C. Wu, Chen, Q. Cao, Zhang, Y. Tai, L. Sun, and K. Jia. Grasp proposal networks: An end-to-end solution for visual learning of robotic grasps, 2020. URL htttt:///axxxv abs /2009. 62606.
[16] S. Wang, Z. Zhangli, and Z. Kan. When transformer meets robotic grasping: Exploits context for efficient grasp detection. IEEE Robotics and Automation Letters, 7:1-8, 07 2022. doi: 10.1 20.1000....222.111777
[17] R. Wang, J. Zhang, J. Chen, Y. Xu, P. Li, T. Liu, and H. Wang. DexGraspNet: A large- scale robotic dexterous grasp dataset for general objects based on simulation. arXiv preprint arXiv:2210.02697, 2022.
[18] Y. Xu, W. Wan, J. Zhang, H. Liu, Z. Shan, H. Shen, R. Wang, H. Geng, Y. Weng, J. Chen, T. Liu, L. Yi, and H. Wang- Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy, 2023.
[19] D. Turpin, L. Wang, E. Heiden, Y.-C. Chen, M. Macklin, S. Tsogkas, S. Dickinson, and A. Garg. Grasp'd d: Differentiable contact-rich grasp synthesis for multi nulti-fingered hands, 2022.
[20] D. Turpin, T. Zhong, Zhang, G. Zhu, Liu, R. Singh, E. Heiden, M. Macklin, S. Tsogkas, S. Dickinson, and Garg. Fast-grasp'd: Dexterous multi- lilt-ffingerre grasp generation through differentiable simulation, 2023.
[21] M. Attarian, M. A. Asif, J. Liu, R. Hari, A. Garg, I. Gilitschenski, and J. Tompson. Geometry matching for multi-embodiment grasping, 2023. URL https://arxiv. org/ absp//bbs/22......//// 03864.
[22] A. H. Li, P. Culbertson, J. W. Burdick, and A. D. Ames. Frogger: Fast robust grasp generation via the min-weight metric. arxiv:2302. 13687, February 2023.
[23] A. Wu, M. Guo, and K. Liu. Learning diverse and physically feasible dexterous grasps with generative model and bilevel optimization. In 6th annual conference on robot learning, 2022. URL www.hhttts//ppeeeeeeeee
ew, net. forum? i Andd1 05kFr.
[24] V. Mayer, Q. Feng, Deng, Shi, Z. Chen, and A. Knoll. Ffhnet: Generating multi-fing calti-finggred robotic grasps for unknown objects in real-time. In 2022 International Conference Roboties and Automation (ICRA), pages 762-769, 2022.
[25] Z. Weng, H. Lu, D. Kragic, and J. Lundell. Dexdiffuser: Generating dexterous grasps with diffusion models, 2024.
[26] K. Shaw, A. Agarwal, and D. Pathak. Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning, 2023. URL https:////// /aaxai.v wwwp:i..org/ab//233332 /2009.2300..0 06440.
[27] P. Li, T. Liu, Y. Li, Y. Zhu, Y. Yang, and S. Huang. Gendexgrasp: Generalizable dexterous grasping. arXiv preprint ar arXiv:2210.0077 2022.
10

---

[28] S. El-Khoury and A. Sahbani. Handling objects by their handles. In IROS-2008 workshop on grasp and task learning by imitation, 2008.
[29] R. N. Murray, Z. Li, and S. Sastry. mathematical introduction to robotics manipulation. CRC Press, 1994.
[30] A. H. Li, P. Culbertson, and A. D. Ames. Toward an analytic theory of intrinsic robustness for dexterous grasping, 2024.
[3]] A. Singh, J. Sha, K. S. Narayan, T. Achim, and Abbeel. BigBIRD: A large-scale 3D database of object instances. In 2014 IEEE international conference robotics and automation (ICRA), pages 509-516, 2014. 010.10.1110//RA22114.6000000
[32] L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann, B. McHugh, and V. Vanhoucke. Google scanned objects: A high-quality dataset of 3D scanned household items. In 2022 international conference on robotics and automation (ICRA), pages 2553-2560, 2022. 2022.9981 1809.
doi: 10.1
[33] S. Prokudin, C. Lassner, and J. Romero. Efficient learning on point clouds with basis point sets, 2019.
[34] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[35] J. Ichnowski Y. Avigal # J. Kerr, and K. Goldberg, Dex-NeRF: Using neural radiance field to grasp transparent objects. Conference on robot learning (CoRL), 2020.
[36] Kerr, L. Fu, H. Huang, Y. Avigal, M. Tancik, J. Ichnowski, A. Kanazawa, and K. Gold- berg. Evo-NeRF: Evolving NeRF for sequential robot grasping of transparent objects. In 6th annual conference on robot learning, 2022. URL https: opw.revien..reeeeeeeeeeeee net www.:frrtffffi forum? id=Bxr45keYrf.
https://openzeview.
[37] Q. Dai, Y. Zhu, Y. Geng, C. Ruan, Zhang, and H. Wang. GraspNeRF: Multiview-baser 6-DoF grasp detection for transparent and specular objects using generalizable NeRF. 2023. Publication Title: IEEE international conference roboties and automation (ICRA).
[38] S. Jauhri, I. Lunawat, and G. Chalvatzaki. Learning any-view 6dof robotic grasping in cluttered scenes via neural surface rendering, 2024.
[39] A. Rashid, S. Sharma, C. M. J. Kerr, L. Y. Chen, Kanazawa, and K. Goldberg. Language embedded radiance fields for zero-shot task-oriented grasping. In 7th Annual Conference Robot Learning, 2023. URL https::/p//o opwpreeeeeevveeeeeeeeeeeeee eew net. forum? F
[40] L. Yen-Chen, Florence, A. Zeng. J. T. Barron, Y. Du, W.-C. Ma, A. Simeonov, A. R. Garcia, and Isola. MIRA: Mental imagery for robotic affordances. Conference on Robot Learning (CoRL), 2022.
[41] W. Shen, G. Yang, A. Yu, J. Wong, L. P. Kaelbling, and P. Isola. Distilled feature fields enable few-shot language-guice manipulation. In 7th Annual Conference on Robot Learning, 2023.
[42] R. Newbury, M. Gu, L. Chumbley, A. Mousavian, Eppner, Leitner, Bohg, Morales, T. Asfour, D. Kragic, D. Fox, and A. Cosgun. Deep learning approaches to grasp synthesis: A review, 2022, arXiv: 2207.02556 [cs.RO].
[43] C. Eppner, A. Mousavian, and D. Fox. ACRONYM: A large-scale grasp dataset based on simulation. 2021 IEEE Int. Conf. on Robotics and Automation, ICRA, 2020.
[44] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspnet Efficient 6-dof grasp generation in cluttered scenes. 2021.

---

[45] D. Turpin, T. Zhong, S. Zhang, G. Zhu, E. Heiden, M. Macklin, S. Tsogkas, S. Dickinson, and A. Garg. Fast-grasp'd: d: Dexterous multi-finger grasp generation through differentiable simulation. In ICRA, 2023.
[46] L. F. C. Murrilo, N. Khargonkar, B. Prabhakaran, and Y. Xiang. Multigrippergrasp: A dataset for robotic grasping from parallel jaw grippers to dexterous hands, 2024.
[47] M. Liu, Z. Pan, Xu, K. Ganguly, and D. Manocha. Deep differentiable grasp planner for high-dof grippers. 2020.
[48] M. V. der Merwe, Q. Lu, B. Sundaralingam, M. Matak, and T. Hermans. Learning continuous reconstructions for geometrically aware grasping, 2020.
[49] Q. Lu, K. Chenna, B. Sundaralingam, and T. Hermans. Planning multi-fingered grasps as probabilistic inference in learned deep network, 2018.
[50] Q. Lu, M. Van der Merwe, B. Sundaralingam, and T. Hermans. Multifingered grasp plamning via inference in deep neural networks: Outperforming sampling by learning dif- ferentiable models. IEEE Robotics and Automation Magazine, 27(2):55-65, 2020. doi: 10.1 N0.100///A.2220.29766..
[51] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance GPU-Based physies simula- tion for robot learning, 2021.
[52] Weisz and P. K. Allen. Pose error robust grasping from contact wrench space metrics. In 2012 IEEE International Conference on Robotics and Automation, pages 557-562, 2012. doi: 10.1 111/111...0....222222
[53] D. Kappler, J. Bohg, and S. Schaal. Leveraging big data for grasp planning. 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 4304-4311, 2015. doi: 10.
[54] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a mul- tiresolution hash encoding. ACM Transactions on Graphics, 41(4: July 2022. ISSN 1557- 7368. doi: 145/3528223.3550000 URL httt:://ax.bbi..rr/10..1111. 111o:/orr//011115552222222 3530127.
[55] M. Tancik, E. Weber, Ng, R. Li, B. Yi, T. Wang, A. Kristoffersen, Austin, K. Salahi, A. Ahuja, D. Mcallister, J. Kerr, and A. Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings, SIGGRAPH '23. ACM, July 2023. doi: 10. 1145/558443235515516 URL http: /xx doi. www.wooo..ooooooooooooooooooeeee /14445/5/332222 3591516
12

---

Appendix
A Additional Insights and Discussion of Results
In this section, we provide further discussion that may aid the interpretation of our results.
How "fair" is the diffusion sampler? To ensure reasonable performance, we train our generative model exclusively on high-quality grasps with probability of grasp success yrgs 1.0. This approximately 300K grasps (8.45% of the training data), roughly 10 times the amount used train existing models [1, 2]. Figure 21 shows real-world grasps generated by the diffusion sampler, and shows that many failed grasps reasonably "close" successful grasp.
In general, grasp evaluators have superior data efficiency in the sense that they can train on "bad"" grasps (in our case, there are ~3M) which are byproducts generating "good" ones. We stress that we do not claim that grasp evaluation is superior grasp generation as paradigm only that evaluators can achieve robust sim-to-real transfer by leveraging large volume of negative examples.
We emphasize that there are no suitable datasets for dexterous grasping larger than ours on which we can train our diffusion sampler. To our knowledge, the only large dataset with visual observations closed-source [3]. Further, no existing datasets parameterize the grasp execution motion, few consider a tabletop setting, and many their grasps intersect with objects
non-physical way.
Are the simulation results "reasonable"" Our simulation success rates at first appear much lower than our hardware success rates and simulation results of similar studies [1, 2]. This discrepancy arises from subtleties in labeling. In the real world, we only consider collision-free grasps, and thus report the empirically estimated conditional probability pssuccess no collisions). However, in simulation we evaluate all planned grasps, including those collision (which count as failures), and instead report p( psuucees)). other words, the real-works experiments answer the question "how many collision-free generated grasps succeed?", while the simulations answer the question "how many generated grasps (collision-free or otherwise) succeed?"
Figure 5 shows histograms for all simulated evaluator-based methods, which compares unconditional success rates and those conditioned on Ycoll 2 0.8. The median success rates are 37% and 80% respectively. Note the similarity between this 80% mark and our evaluator-based hardware results, which achieve 76-81% across all objects (see Table 4). Lastly, other works differ from ours in three major ways: they often do not consider tabletop settings (allowing non-physical fully-caging grasps), they have less object diversity, and they plan power grasps while plan precision grasps. These factors make our learning problem more challenging than those in other studies.
Simulated Success Rates
Unconditioned
Conditioned on Ycoll 2 0.8
Number of Grasps
2000
2000
1500
1500
1000
1000
500
500
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Success Rate
Success Rate
Figure 5: Success rates for all simulated evaluator-based methods. When only considering grasps with Ycol 2 0.8, the median success rate increases from 37% to 80%. 2100/4240 grasps satisfied Fcoll 0.8. On the right, the unconditioned distribution plotted with dashed lines for comparison.

---

Interpreting the Fixed vs. Diffusson Sampler Performance. There are a few unintuitive results regarding the fixed vs. diffusion samplers in both simulation and hardware. First, the Evaluator methods seem to outperform the Diffusion+ Evaluator methods in simulation, but they perform nearly identically in hardware. We believe that since the train and test objects in simulation are sampled from the same distribution, some test objects will resemble training objects, so some grasps from the fixed dataset may have high quality due to slight overfitting. Since there real-world objects in the training set, this effect disappears hardware.
Second, is interesting that the diffusion sampler alone far outperforms the fixed sampler, but they perform nearly identically after evaluator refinement. We believe that this is because the best samples from the fixed and diffusion samplers have similar quality, but the average far better than the average fixed grasp. Because only the best sampled grasps are further refined, and only the best of the refined grasps executed on hardware, if the fixed sampling distribution is reasonable prior for good grasp, then it is likely that at least one candidate would be good initial guess for refinement.
cettuuioo--samlll. grasp
B
Grasp Dataset
B.1
Grasp Generation
Parameter
Value
Parameter
Value
Parameter
Value
__c_per n_o_per_finger
5
switch_prot
0.5
jitter
0.1
w_fc
0.5
mu
0.98
dist lower
dist_lower
0.2
w __dis
500
step_size
0.005
0.3
w w_pen
300.0
stepsize
stepsize_period
50
theta_lower
-I /6
w_spen
100.0
starting_temp
18
theta_upper π 766
w_ joints
1.0
annealing_perioo
30
w_ff
3.0
tempdecay
0.95
w_fp
w_tpen
100.0
Table 3: Grasp generation parameters. Variables names taken from DexGraspNet [4]. Left:
Energy function parameters. Middle: Optimization parameters. Right: Initialization parameters.
Our grasp generation pipeline inspired by Wang al. [4], but required key modifications. Further details about these modifications not discussed in the main text are described here.
As mentioned, data generation pipeline yields pre- grasp poses with the fingertips 1.5cm off of the surface of the object, while DexGraspNet places the fingers on or very near the surface. Due to this, when planning grasp trajectory, we compute a new configuration corresponding to fingertip locations 3.5cm below the object surface, which we call the post-grasp pose.
We choose to generate precision grusps rather than power grasps, since this synergizes with the grasp motion parameterization described in the main text. To implement this, we only specify contact candidates on the fingertip of the hand, which function as attractors to the object surface during grasp generation. However, the surface points used to penalize hand- object or hand-table collision still cover the whole hand. See Figure 6 for visualization.
We sample 500 surface points uniformly from the mesh of each link's collision geometry, distributed proportionally based on mesh surface area. 128 contact candidates are densely sampled over the "front" each Allegro fingertip. which is parameterized by a spherical section centered on the (spherical) fingertip center. This spherical section has radius of 1. 1.2cm, polar angular sweep of =54", and azimuthal angular sweep of 54" 126". The local fingertip coordinate frame has the axis pointing in the "palm out" direction and the axis aligned with the fingers.

---

i
Contact Candidates
Surface Points
Figure 6: Visualization of the Allegro hand contact candidates and 500 surface points. Contact candidates are used to compute Edis. encouraging hand-object proximity. Surface points are used to compute Eipen: which penalizes hand-table penetration to keep the grasps above the table.
We now describe the dataset augmentation procedure in detail. Recall that first, large set of nominal grasps are generated using the modified DexGraspNet pipeline. In this step, the fingertip grasp directions d, are generated by computing the direction of each fingertip to the closest point on the mesh. Because there are a large proportion of failures after this step, we augment the dataset with more positive examples by taking all nominal grasps satisfying yPGs 2 0.9, perturbing them slightly, re-evaluating them, and adding them the dataset.
These perturbations are sampled using Halton sequence sampling [5], which generates quasi-random, low-discrepancy sequences that sample more uniformly across the input space compared to standard random sampling and reducing clustering. We sample perturbations from the following ranges:
wrist translation: each spatial coordinate draws a perturbation from [-5mm, 5mm];
wrist orientation: each roll, pitch, and yaw draw a perturbation from , 2.5%];
finger joints: each angle draws a perturbation from [-0.05rad, 0.05rad]:
fingertip grasp directions: each of two axes orthogonal the nominal direction draw perturbations from [-10", 10%).
Each high-success grasp is perturbed five times and re-evaluated on the same object. Additionally, these grasps are perturbed more times and evaluated on different, randomly sampled objects from the dataset. These additional perturbations typically yield unsuccessful grasps, but we believe this protects the model from overfitting and provides useful training signal regarding object geometry. Table 3 summarizes all parameters used for grasp generation.
B.2 Grasp Label Generation in Simulation
To generate the labels for the grasps in the dataset as well as corresponding rendered visual data, we use Isaac Gym [6]. Our simulated evaluation proceeds as follows. We spawn the hand in the pre-grasp pose, execute the grasp motion parameterized by fingertip directions, then lift the object 20cm off of the table (with gravity enabled).
A pick is considered a simulation success (''pick = 1) only if the following conditions are met: (1) the hand contacts the object on at least links; (2) the palm and proximal links of the hand do not touch the object; (3) the change in the object's position relative to the hand does not exceed 10cm throughout the lift; (4) the change the object's orientation relative to the hand does not exceed 45 degrees about each Euler angle throughout the lift. A pick is considered collision-free (Ycoll 1) only if the hand does not contact the table object during the pre-grasp pose. Recall that JPGS Ypick A Ycoll-

---

Simulated Grasp Evaluation
Snowman
settle
pregrasp
lift
I ggrasp
Figurine
settle
I ppregrapp
lift
Egrasp
time
Figure Visualization of grasp evaluation in simulation on a snowman object (top) and figurine object (bottom). After the object settles on the table (Nsette), the Allegro hand is moved to the pre-grasp position ((pregrasp). executes the grasp (/gras)), and lifts the object (fnn). fpregrasps we compute Ycoll by checking for hand collisions with the object and table. At fjit- we compute ypick by checking if the object pose relative to the hand significantly changed from fpregrusp.
To generate non-binary labels, each grasp is corrupted with small noise and re-simulated times. Note that this is a different perturbation than the one used for dataset augmention described in Appendix B.1. Only the wrist pose is perturbed. The wrist translations corrupted with uniform noise drawn from [-5mm, 5mm] along each axis, and the wrist orientation is corrupted about the roll, pitch, and yaw axes with uniform noise drawn from [-2.5%, 2.5%].
The simulation timestep is set to 1/60 seconds, but the integrator solves at rate of 1/120 seconds for numerical stability using the Truncated Gauss-Seidel (TGS) solver. The TGS solver runs position and 8 velocity iterations per simulation timestep. "Force at distance" (i.e., the contact offset parameter) applied starting from 1mm of separation between collision geometries, substantially lower than the lcm distance used by [4] in DexGraspNet. This helps reduce unwanted non-physical collisions. Geometries are processed by a convex decomposition using the default settings in Isaac Gym. Each object is spawned 2cm above the table and dropped. The simulation starts when the object settles. The simulation pipeline visualized Figure 7.
B.3 Simulated Label Distribution
Figure shows the distribution of grasp labels across the dataset of 3.5M grasps. The median and IQR of each label is follows:
YrGs 0.0 (0.0, 0.48)
Ypick: 0.2 (0.0, 0.8)
Yool:: 0.08 (0.6, 1.0)
B.4 Object Dataset and Generation
The objects considered in this work strict subset of those in DexGraspNet [4]. DexGraspNet contains 5.3K unique objects, while only consider 4.3K of them. The main reason for this is

---

x110
Distribution of Yrgs Labels
# Grasps
0.0
0.2
0.4
0.6
0.8
1.0
YPGS
x10
Distribution of ypick Labels
# Grasps
0.2
0.4
0.6
0.8
1.0
Ypck
Distribution of Ycoll Labels
Grasps
*
0.0
0.2
0.4
0.6
0.8
1.0
Coll
Figure 8: Dataset label distribution across 3.5M grasps.
that assume tabletop setting while DexGraspNet does not, which necessitates canonical "up" direction. Thus, many objects were manually processed to be oriented in a reasonable manner. We only use an object if after this reorientation, the object successfully settles when dropped from a height of 2em. If the object tips over or does not settle after 200 timesteps the Isaac Gym simulation, then is excluded. its relative quaternion remains greater than 0.95 throughout the simulation. We consider it have settled if the translational components stay within mmm along each axis and roll, pitch, and yaw stay with le-2 radians for 15 consecutive timesteps. For all objects, used uniform friction coefficient of 0.9 and a constant object density of 500 kg/m² (as in [4]). Thus, object masses were determined entirely by volume, as in other works [7, 4, 8].
consider an object have remained upright if the real part of
Figure visualizes subset of our object dataset. To introduce more scale diversity while retaining dataset size under 2.5TB, subset of approximately 1.7K of the objects duplicated three distinct scales, while the remaining objects used at scale each, resulting in approximately 7.7K objects. The large dataset storage requirement comes from the storage of multiple 3D NeRF density grids used to represent each grasp for the NeRF evaluator (more details about the NeRF density grid representation in Appendix D.3.2).
B.5 Train, Validation, and Test Split
We create train, validation, and test splits of the datasets very carefully. We make that each unique object mesh only exists in one of these three sets.
We split the 4305 unique object meshes into 3981 (92.5%) train objects, 216 (5%) validation objects, and 108 (2.5%) test objects. Each object may appear at more than one scale, so this results in 7695 different objects after all scaling operations, split into 7108 (92.5%) train objects, 375 (5%) validation objects, and 212 (2.5%) test objects. This results in total of 3,531,098 grasps, with 3,261,228 (92.5%) for training, 170, 128 (5%) for validation, and 99,742 (2.5%) for testing.
B.6 Compute and Timing
We used 4 Nvidia A100 GPUs generate our dataset. took ~4 days generate the full dataset: ~1 day for grasp generation, ~1 day for grasp label generation, and ~2 days for image generation, NeRF training, and point cloud generation.
5

---

Figure 9: Visualization of a random subset of 100 objects out of the 212 test objects. The Allegro hand is shown at the bottom for scale. The object dataset is subset of the DexGraspNet dataset's meshes [4]. See their work for more details.
C Object Representations
C.1 Neural Radiance Fields
We train NeRFs using both simulated data and real- world data and compare them here.
For NeRFs trained on simulated data, the object is first spawned and allowed to settle on the table. Then, we uniformly sample 100 images over spherical cap above the object with radius of 0.45m and polar angle sampled from U(0, 2), and then train for 400 iterations with nerfstudi [9, 10].
For NeRFs trained on real-world data, we first collect 100 images while moving the wrist-mounted camera along a hard-coded trajectory encircling the object and then train for 400 iterations with nerfstudio [9, 10]. This trajectory consists of 3 spirals around the object, with the object placed roughly in the center of the spirals.
Figure 10 shows a qualitative comparison between the RGB images and NeRFs trained on simulated data versus real-workd data. Both types of NeRFs exhibit floater artifacts [11], which must either be processed away ignored by downstream models.
We train without scene contraction, auto-scaling poses, centering method, orientation method, and use a scale factor of 1.0, which ensures that the coordinate space in the NeRF is not modified from the given data. The remainder of the parameters follow the default ner facto settings in ner nerfstudio.
C.2 Basis Point Sets
Methods that represent the object basis point sets (BPSs) use the following procedure.
1. Train NeRF following the procedure described in Appendix C.I.
2. Sample point cloud with 5000 points using nerfstudio [10] by rendering out depth images with opacity exceeding 0.5 in the axis-aligned bounding box parameterized by the lower and upper bounds [-0.2m, -0.2m, 0.0m] [0.2m, 0.2m, 0.3m] using z-up convention.
3. Preprocess the point cloud remove most outliers and floaters. First, we use open [12] remove statistical outliers (nb_ neighbors=20, std ratio=2. 0) and radius out- liers sab_points 16, radius-0, 05). Next, construct undirected graph of the

---

Simulated Snowman
Real-World Dragon
Figure 10: Comparison between NeRFs trained on simulated snowman data and real-world dragon data. This demonstrates the qualitative similarity between the two NeRFs. First row: two of the RGB images used for NeRF training. Second row: Camera poses used for NeRF training. Third row: NeRF-rendered RGB image and accumulation image.
remaining points, where points are nodes and edge exists between two nodes if the points are within Imm. Floaters removed by keeping the largest connected component.
4. Generate a random set of 4096 basis points within a sphere of radius 0.3m, centered O. 0.15m above the table. This set basis points remains fixed for all experiments. We utilize bps [13] for basis point set operations.
5. Compute basis point set values by calculating the distance from each basis point to its closest point in the point cloud.
Figure 11 shows an example of a point cloud and BPS for both simulated and real-world object.
C.3 Meshes
We triangle meshes for purposes: the analytic grasp planning method (FRoGGeR) that as bascline and collision-free motion planning from the start pose to the pre-grasp pose (all methods).
For both applications, the mesh is generated by the following procedure.
1. Training NeRF following the procedure described in Appendix C.I.
2. Using scikit-inne image [14] perform marching cubes, which extracts 2D surface mesh from 3D volume with NeRF density level set of 15 within bounding box of [-0.2m, -0.2m, 0.0m] x [0.2m, 0.2m, 0.3m] (with z being the up-direction).

---

Basis Point
Simulated Snowman
Real-World Dragon
Distances
0.2
0.15
0.1
Figure 11: Visualization of a point cloud (5000 black points) and a basis point set (4096 points colored by distance to the point cloud). These are generated by NeRF for both a simulated snowman object (left) and real-world dragon object (right).
Simulated Snowman
Real-World Dragon
Ground-Truth Mesh
NeRF-Generated Mesh
RGB Image
Genertted Mesh
We use NeRF's to generate meshes for two purposes: analytic grasp planning methods (FRoGGeR) and collision-free motion planning from the start pose the pre-grasp pose (all methods). Left: Comparison between the ground-truth mesh and NeRF-generated mesh of a simulated snowman object, where the extra vertices at the bottom are due to artifacts from shadows/lighting effects. Right: Comparison between an RGB image and the MeRF-generated mesh of weal-world dragon object. We do not have ground-truu truth mesh for the real-world dragon object.
Figure 12:
3. Using trimesh [15] to remove floaters. This is done by only keeping connected compo- nents with at least 31 edges.
Figure 12 shows examples of NeRF-generated meshes. We found the parameters above to be reasonable for all test objects.
D Grasp Planning
As explained in the main text, during grasp planning, a batch of candidate grasps is sampled, and only the top K of these retained for the refinement phase. In simulation experiments, all refined and executed in simulation. In our real-world experiments, we let
let K = 5, which K 40, which are all refined and the best is executed on hardware. See Figure 13 for schematic of the sample/refine process.
Recall that our refinement is sampling -ampeing-based, where the previous iterate is perturbed by some number of samples and the best is chosen be the next iterate. For all perturbations, we use zero-mean Gaussian noise as follows.
Wrist translation: standard deviation of 5mm per spatial coordinate.

---

Grasp Candidates
Ranked Grasps
YPGS = 0.8
Top K Refined Grasps
Sampler
Evaluator (Rank)
Evaluator (Refine)
YPGS = 0.6
0.95
0.2
YPGS 0.1
Figure 13: Our grasp planners consist of sampler and evaluator. First, the sampler generates a batch of grasp candidates. The evaluator ranks them, and the top K grasps are refined using sampling-based optimization, where the objective is given the evaluator.
Wrist and grasp orientations: the noises are sampled using standard deviation of 0.05m from so(3), which are then converted to elements of SO(3) via the exponential map.
Joint angles: standard deviation of 0.01 radians.
The refinement run for 50 iterations. Other refinement strategies could work as well, such as gradient-based methods or other sampling-based methods.
To allow grasp parameters to be represented vector inputs or outputs, we parameterize them as g = (x, r, 6, dj.,.... day) E R94n,+3n, where x & the wrist position, R6 the wrist orientation represented by a continuous 6-D rotation vector [16], 0 E R" the pre-grasp joint configuration, and d, e I the direction which the 7h fingertip moves during the grasp.
D.1 Diffusion Sampler
Figure 14 shows the diffusion sampler architecture inspired by Weng et al. [1]. As their implemen- tation is not publicly available, we re-implemented their architecture to the best our ability via the textual description. We use an embedding dimension of 128 and a sequence length of 4 all attention modules. The sequence length is created by reshaping the outputs of the fully-connected layers (to (4, 128)). We use Denoising Diffusion Implicit Models (DDIM) [17] with linear sched- uler on the noise variance & that moves from 0.0001 to 0.02 over total of 1000 diffusion timesteps. Figure 15 shows the train and validation loss curves for this model.
Although the diffusion sampler is trained to only generate successful grasps, can still produce unsuccessful grasps as seen in previous works [1, 2]. While it could typically generate feasible- looking grasps diverse range of objects, its grasps that were "close" to successful but with fingers slightly too close far successfully execute, and (2) generating grasp that was substantially too far from the object.
common failure modes were (1) generating
D.2 Sampling from Fixed Dataset
To disentangle the importance the learned evaluator from the impact of the diffusion sampler, perform additional experiments in which replace the diffusion sampler with simple baseline in which we store fixed set of 4349 grasps from the training set. More specifically, this was

---

Cross Attention
BPS
FC
FC
E
FC
Noised Grasp
Self Attention
FC ResBlock
concat
Figure 14: Diffusion Sampler architecture. Our model takes in basis point set, noised grasp, and diffusion timestep. The noised grasp and the diffusion timestep are processed into query using self-attention block. The basis point set is processed into key-value pair. These are embedded together using a cross-attention block and used to compute the predicted noise. This is similar architecture the one used by Weng et [1].
Loss Curves: Diffusion Sampler
Train Loss
Validation Loss
Loss
2000
4000
6000
2000
4000
6000
Epochs
Epochs
Figure 15: Train and validation loss curves for the diffusion sampler.
generated by storing grasp per training set object that exhibited high probability of success (YrGs 2 0.9). For objects that did not have any such grasps, did not store any. We emphasize that these grasps were from training set objects, so they were not designed for the unseen test objects used for simulation evaluation or the real-world objects used for hardware evaluation.
D.3 Grasp Evaluators
Recall that the evaluators are trained by regressing on three distinct soft labels: yrGs- Yeolls and ypick- Although we only use the fras prediction at inference time, choose to regress the other two labels fooll and Spick train time, found that these labels provide additional signal about why given grasp may be failing.
10

---

BPS
FC ResBlock
FC ResBlock
softmax
Yooll
FC ResBlocd
concat
FC
softmax
+ S
Grasp
softmax
+YPGS
Figure 16: BPS Evaluator architecture. Our model takes in basis point set and a grasp, which are concatenated together and then passed through three fully-connected resblocks and final fully- connected layer, which returns logits for 3 labels: whether (i) there are unwanted collisions in the scene, (ii) the pick succeeded the simulator, and (iii) both (i) and (ii) are true. See the official implementation from Mayer et al. [2] [here]. This is the same architecture used by Weng et al. [1] and our work.
D.3.1 BPS Evaluator Details
Figure shows the architecture used for the BPS evaluator. We used the official implementation from Mayer et al. [2] [here], which is the same architecture used by Weng et al. [1].
D.3.2 NeRF Evaluator Details
Figure 17 shows the architecture used for the NeRF evaluator. We use a novel grasp representation that leverages NeRF features directly.
We will motivate this representation and then describe detail. A key factor for grasp success the surface geometry the contact points. Modeling this from images is difficult, and few (if any) fast, reliable surface reconstruction methods are adequate for grasp planning. Here, features that we hypothesize capture accurate estimates of the object surface. Centered at finger i's position x,, square of side length 0.06m is swept along d; by 0.08m to recover rectangular prism which is discretized into 4D tensor N E R 44331x31x11 where the first channel dimension is the NeRF density and the last 3 are spatial coordinates. These grid dimensions overapproximate both the fingertip size and the grasp depth to ensure the geometric information captured not too local. Further, to capture the global object geometry, we generate grid centered on the estimated object centroid with side lengths 0.4m Ng R4x41x41x11 The centroid is estimated by assuming uniform mass density and integrating over spatial regions with NeRF density exceeding 15.0.
NeRF
In summary, our architecture uses local NeRF densities and global NeRF densities as inputs. The local NeRF densities are sampled grids approximating the fingertip swept volumes when moving along the grasp directions. The global NeRF densities are a sampled grid of fixed size to capture object's global geometric features.
D.4 FRoGGeR Details
For all experiments that use FRoGGeR [18, 19), we use the open-source implementation provided at https:/ /ttp:s:://ttti thub. com foom/all http/babbetthhee i frogger. The procedure for acquiring the meshes used for planning is described in Appendix C.3.
Out of the 100 real-world pick attempts using FRoGGeR, there were total of 49 failures. 16 failures caused by planned grasps that failed to lift the object, and 33 failures caused by planning issues in which no grasp could be found within the timeout period.

---

N
softmax
&coll
Fingertip CoordConv
N
concat
MLP
softmax
pick
n
N
Global CoordConv
softmax yPGS
Figure 17: NeRF Evaluator architecture. Our model takes in local and global NeRF feature grids, passing them through 3D CoordConv networks and an MLP, which returns logits for labels: whether (i) there are unwanted collisions in the scene, (ii) the pick succeeded in the simulator, and (iii) both (i) and (ii) are true. The Fingertip CoordCony weights are shared for all fingertips.
Learned Grasp Evaluator Dataset Sizes
Ablation
*100%
A: Lu et al., 2018
* Ours 100%
50%
die Merwe et al., 2020
Ours 50%
Num Grasps
25%
2020
Ours
5%
20a
E: Liuu et al., 2020
Ours 1%
E
F: Corsare et al., 2021
CO
8
Mayer et al., 2022
10a
Weng al.,
10a
2018
2019
2020
2021
2022
2023
2024
2025
Date
Comparison between learned grasp evaluator dataset sizes in prior works over time. Our dataset and our ablations shown on the right blue. The full training set contains 3.26M grasps.
Figure 18:
FRoGGeR allows to either plan with floating hand the full hand-arm system. We opt use the full hand-arm system that the generated grasps account for kinematic reachability of grasp poses when using given arm, resulting in fewer downstream motion planning failures.
E Experiment Details
E.1 Simulation: Ablation Study Dataset Size Details
We hypothesize that learned grasp evaluators could help achieve robust sim-to-real transfer for multi-finger grasp planners, but only if the associated data are large-scale and account for realistic perceptual inputs.
Looking at prior works, we see clear trend toward larger multi-fingered grasp datasets used for training grasp evaluators (see Figure 18). Recognizing this trend, our goal with this ablation is to study how much the increased scale of dataset (3.5M grasps across 4.3K objects) impacts performance, which measure by examining the improvement in the simulated probability of grasp success gained by evaluator refinement.
Figure shows the train and validation loss curves each model trained on different fraction of the training dataset and validated on the same validation dataset. We see that all models are trained convergence and achieve similar train losses, but their corresponding validation losses show consistent correlation between dataset scale and validation performance.
E.2
Hardware: Object Selection
Figure 20 shows the real-world objects labeled with their corresponding names. Easy objects are those that have high-quality grasps when approached from any most directions. For example, tall, cylindrical objects can be grasped overhead or from the side. Medium objects are those with more
12

---

Loss Curves: Ablation Study on Evaluator Training Set Size
Train Loss
Validation Loss
0.30
0.30
0.25
5%
0.25
5%
25%
25%
0.20
0.20
50%
50%
Loss
0.15
100%
0.15
100%
0.10
0.10
0.05
0.05
0.00
0.00
200
400
600
800
1000
200
400
600
800
1000
Epochs
Epochs
Figure 19: Comparison loss curves across each model trained on a different fraction of the training dataset and validated on the same validation dataset. These show clear correlation between dataset scale and validation performance. The full training set contains 3.26M grasps.
Easy
Medium
Hard
Apple
Cube
Redmug
Goblet
Spray
Dragon
Goggles
Minion
Strainer
Conditioner
Greentea
Purplecup
Milk
Redbox
Bunny
Squirrel
Dino
Mallet
Figure 20: Real-world objects labeled with their names corresponding Table 4.
unusual geometry, including "distractors," or objects which are harder to grasp when the hand is arbitrarily oriented. For example, the mug has a handle and the lunchbox is more easily pinched in the thin direction. Lastly, hard objects contain very unusual, non-convex geometry/visual features, or are difficult to grasp without affordances. For example, the goggles are both reflective and transparent, which poses challenge for large distractor for grasp planners. The bunny, squí rrel, and d no custom printed hard test objects. The strainer and ma let were chosen to be out-of-distribution objects, providing a ceiling on the performance of our grasping algorithms. No method obtained any successful grasps on the strainer.
algorithms, and the bunny's 's ears are
E.3 Hardware: Detailed Results
Table 4 presents the detailed quantitative results of all grasp planning methods on various objects in the real world.
Qualitatively, observed that evaluator- based methods very rarely exhibited edge-seeking behavior, wherein the fingers are placed on corners or edges of objects, and has been noted as a common failure mode by other works [18, 19]. We suspect this may be result of our label smoothing, though we do not rigorously test this. However, "distractor" geometries like the bunny ears or spray nozzle frequently caused grasp planning failures, attracting the fingers towards them. Methods like FRoGGeR rely on online mesh reconstruction, so objects like the goggles le caused issues due transparencyyieeieetiittyy Lastly, many of the diffusion sampler's unrefined grasps appeared "close" to success when failing, which suggests that generative modeling may be viable strategy with more data. Some examples shown in Figure 21.
13

---

Methods
No Evaluator
Evaluator
Difficulty
Objects
Diffusion
Diffusion
Diffuston
Fixed
FRoGGeR
No Evaluator
BPS
NeRF
BPS
Apple
4/5
3/5
5/5
5/5
5/5
Cube
3/5
1/5
5/5
4/5
5/5
Yelloweup
5/5
4/5
5/5
5/5
5/5
Conditioner
4/5
5/5
575
4/5
5/5
Easy
Greentea
3/5
4/5
4/5
5/5
5/5
Purplecup
5/5
5/5
5/5
5/5
5/5
24/30
22/30
29/30
28/30
30/30
TOTAL
(80%)
(73%)
(97%)
(93%)
(100%)
Redmug
2/5
4/5
5/5
5/5
5/5
Goblet
5/5
3/5
5/5
5/5
5/5
Spray
4/5
2/5
3/5
Milk
1/5
5/5
5/5
5/5
5/5
Medium
Redbox
1/5
4/5
3/5
Lunchbox
2/5
0/5
5/5
2/5
4/5
19/30
13/30
26/30
26/30
25/30
TOTAL
(63%)
(43%)
(87%)
(87%)
(83%)
Dragon
0/5
0/5
2/5
2/5
4/5
Goggles
0/5
0/5
4/5
2/5
4/5
Minion
3/5
5/5
5/5
5/5
Strainer
0/5
0/5
0/5
0/5
0/5
Bunny
175
3/5
Hard
Squirrel
175
2/5
4/5
3/5
4/5
Dino
2/5
575
5/5
5/5
Mallet
5/5
075
175
175
075
8/40
8/40
26/40
22/40
25/40
TOTAL
(20%)
(20%)
(65%)
(55%)
(62%)
51/100
43/100
81/100
76/100
80/100
All
TOTAL
(51%)
(43%)
(81%)
(76%)
(80%)
Table 4: Results of different grasp planning methods on various objects in the real world.
Images of these objects can be found in Figure 20.
Figure 21:
Some examples failed grasps generated by the diffuusion sampler. Though they failed, the grasps appear qualitatively reasonable.
E.4 Hardware: Motion Planning
To evaluate a given grasp in the real world, we need to use a motion planner to move the hand to the corresponding pre-grasp pose. We use cuRobo [20], which performs parallelized collision-fres motion generation. In particular, use cuRobo's graph planner, which Probabilistic Road Map (PRM) to find collision-free path. Object collision avoidance requires object mesh, which is acquired through process explained in Appendix C.3. We use collision buffer of Imm.
14

---

When executing a grasp on hardware, we decompose the motion into three stages: (1) start pose to pre-grasp pose, (2) pre-grasp pose to grasp pose, (3) grasp pose to lift pose.
The first stage the most challenging motion planning problem because the hand needs to find collision-free path to get very close the object. To simplify this problem, utilize inverse kinematics to adjust the pre-grasp finger joints, moving them an additional 3cm backward along the fingertip grasp directions.
In addition to ensuring kinematic feasibility, we also need to avoid damaging the cabling of the wrist-mounted camera during grasp execution. To achieve this, we filter out grasp samples if either (I) the normal direction of the palm is within 60 degrees from the upward direction of the world frame or (2) the direction from the palm to the middle finger deviates by more than 60 degrees from the forward direction of the world frame.
To account for these checks, we request 40 grasps from the grasp planner, which are then sorted by the grasp planner's metric. no metric is available, such as case of a diffusion sampler without an evaluator, the grasps remain unsorted. Next, we utilize cuRobo solve all 40 motion planning problems in parallel and then execute the first grasp for which motion plan is successfully found.
15

---

References
[1] Z. Weng, H. Lu, D. Kragic, and J. Lundell. Dexdiffuser: Generating dexterous grasps with diffusion models, 2024.
[2] V. Mayer, Q. Feng, J. Deng. Y. Shi, Z. Chen, and A. Knoll. Ffhnet: Generating multi-fingered robotic grasps for unknown objects in real-time. In 2022 International Conference on Robotics and Automation (ICRA), pages 762-769, 2022. 2022. doi:10. 109//CCAA46339902222222 1666.
[3] D. Turpin, T. Zhong, S. Zhang, G. Zhu, J. Liu, R. Singh, E. Heiden, M. Macklin, S. Tsogkas, S. Dickinson, and A. Garg. Fast-grasp'd: Dexterous multi-finger grasp generation through differentiable simulation, 2023.
[4] R. Wang, J. Zhang, J. Chen, Y. Xu, P. Li, T. Liu, and H. Wang. DexGraspNet: A large- scale robotic dexterous grasp dataset for general objects based on simulation. arXiv preprint arXiv:2210.026%/ 2022.
[5] J. H. Halton. On the efficiency of certain quasi-random sequences of points in evaluating multi- dimensional integrals. Numerische Mathematik, 2(1): 1960. doi: 100100000//FF01186611 URL https::t/////m / doi. org.
[6] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin,
A. Allshire, A. Handa, and G. State. Isaac gym: High performance GPU-Based physics simulation for robot learning, 2021.
[7] C. Eppner, A. Mousavian, and D. Fox. ACRONYM: Large-Scale Grasp Dataset Based on Simulation. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 0222-6227, 2021. doi:
[8] P. Li, T. Liu, Y. Li, Y. Zhu, Y. Yang, and S. Huang. Gendexgrasp: Generalizable dexterous grasping. arXiv preprint arXiv:2210.00722 2022.
[9] Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphies primitives with mul- tiresolution hash encoding. ACM Transactions Graphics, 41(4):1-15, July 2022. ISSN 1557-7368.
doi: 06::10.145//528233350012 01188223.330112.
URL http://dxx doi. org /trrg/10.11455
[10] M. Tancik, E. Weber, E. Ng, R. B. Yi, T. Wang, A. Kristoffersen, Austin, K. Salahi, A. Ahuja, D. Mcallister. J. Kerr, and A. Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings, SIGGRAPH '23. ACM, July 2023. doi: 10. 1145/3588432.3591511 URL http: dxx doi. org. 10. 114. 3588432...... 3591516.
[11] F. Warburg*, E. Weber*, M. Tancik, A. Ho lyński, and A. Kanazawa. Nerfbusters: Removing ghostly artifacts from casually captured nerfs. In International Conference on Computer Vision (ICCV), 2023.
[12] Q.-Y. Zhou, J. Park, and V. Koltun. Open3D: A modern library for 3D data processing. w.Xiv:1801.08//4 2018.
[13] S. Prokudin, C. Lassner, and J. Romero. Efficient learning point clouds with basis point sets. In Proceedings of the IEEE International Conference on Computer Vision, pages 4332-433 4341, 2019.
[14] S. der Walt, J. L. Schönberger, J. Nunez-Iglesias, F. Boulogne, J. D. Warner, N. Yager, E. Gouillart, T. Yu, and the scikit-image contributors. scikit-image: image processing in Python. PeerJ, 2xe453, 6 2014. ISSN 2167-8359, doi:100.7117ppeerr455.. URL https:
//doi. 7717 http:/0007777/perr.4.5..
[15] Haggeryy et al. trimesh. URL httpt:httttrnniti
16

---

[16] Y. Zhou, C. Barnes, L. Jingwan, Y. Jimei, and L. Hao. On the continuity of rotation representa- tions neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[17] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv:2010.02502, October 2020. URL https: //apiviv//xxii...///a/pap org/ wttggag////laaa/o/ 2010 02502.
[18] A. H. Li, P. Culbertson, W. Burdick, and D. Ames. Frogger: Fast robust grasp generation via the min-weight metric. arxiv:2302. February 2023.
[19] A. H. Li, P. Culbertson, and A. D. Ames. Toward an analytic theory of intrinsic robustness for dexterous grasping. 2024.
[20] B. Sundaralingam, S. K. S. Hari, A. Fishman, C. Garrett, K. Van Wyk, V. Blukis, A. Millane, H. Oleynikova, A. Handa, F. Ramos, N. Ratliff, and D. Fox. Curobo: Parallelized collision- free robot motion generation. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 12-81119 2023. doi: 100.1119//RAA48880000 2023. 10160765.
17


Reviews:
[{"review_id": "go7cnaEact", "reviewer": "Reviewer_JEWj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces \"Get a Grip,\" a large-scale simulation dataset for multi-finger grasp evaluation, featuring 3.5 million grasps on 4.3 thousand diverse objects annotated with RGB images, point clouds, and NeRFs. This dataset enables the training of robust discriminative grasp evaluators. The approach demonstrates significant improvements in both simulation and real-world environments, showcasing effective sim-to-real transfer. The method outperforms baseline models, highlighting the importance of large-scale datasets and the integration of learned evaluators for robust multi-finger grasping.", "summary_of_recommendation": "The paper introduces a large-scale multi-finger grasp evaluation dataset, demonstrating improvements in both simulation and real-world environments. The dataset, featuring 3.5 million grasps on 4.3 thousand diverse objects, is well-annotated with RGB images, point clouds, success labels, and pre-trained NeRFs. This rich data source enables effective training of robust grasp evaluators, facilitating sim-to-real transfer. While the method shows promise, it is primarily focused on simulated rigid objects, assumes non-occluded single objects, and is specific to the Allegro hand, limiting its generalizability to more complex and cluttered real-world scenarios. Given these strengths and limitations, I recommend a weak accept.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- In Figure 2 (a), why does fixed methods with learned evaluators perform than that using diffusion? Since the massive improvement comes from the usage of evaluators, could the implementation of evaluators affect the results of fixed vs diffusion?\n- What was the runtime and compute resources for collecting this dataset?\n- Could the choice of heuristics bias toward certain type of grasps for success grasps?\n- What strategies are in place to ensure the robustness of the method when applied to objects with different physical properties (e.g., weight, friction) in real-world scenarios?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2gZ1Z3Rh9u", "reviewer": "Reviewer_DT3i", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper released a new, open-source dataset of 3.5M grasps on 4.3K objects annotated with RGB images, point clouds, and trained NeRFs.  Using this dataset, this paper trained vision-based grasp evaluators that outperform existing analytic and generative modeling-based baselines in both simulated and real-world.", "summary_of_recommendation": "This paper released a large scale grasping dataset and generation pipeline, the paper can be recommended as weak accept.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. The dataset contains 4.3K objects, but details are missing, how were these items selected?\n2. the dataset link in the project website is not available now, when will it be available?\n3. The 20 objects in the hardware test are categorized as easy, medium, or hard based on the complexity of their geometry, Is there any quantitative metrics?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "04NoM0AIQd", "reviewer": "Reviewer_k59v", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a new large scale dataset of simulated grasps that comes with RGB images, point clouds and trained NeRF.", "summary_of_recommendation": "Thank you to the authors for this work. The community can definitely benefit from a new large multi-finger grasp dataset that has a wider variety of input modalities. However, further clarity should be provided on how the grasp evaluator argument and experiments connect to the dataset itself. If the argument is that grasp evaluators are generally better than analytic and generative methods on large scale data regimes such as the new dataset, then more comparisons with more baselines from both categories should be added to support this argument. I’m willing to increase my score if more such comparisons or added or if authors explain more clearly the connection of the dataset and the grasp evaluators hypothesis in case I misunderstood.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. How do you label borderline grasps? Edit: reading later now, it sounds like labels are 0-1 probabilities for each grasp succeeding?\n\n2. Dataset scale being necessary for training effective grasp evaluators, re: Why do you distinguish grasp evaluators here? This should generally hold with generative approaches without evaluators as well. In general, it should hold with any grasping approach that trains a neural network.\n\n3. Ln. 119-120: Therefore, y_coll and y_pick are strictly 0 or 1 where y_pgs is a continuous value between 0 and 1?\n\n4. Ln. 129-130: How many trials do you do in IsaacGym per grasp? And is the label computed by a simple <attempts_passed>/<total_attempts> formula? \n\n5. Could you explain what perturbation exactly do you do on grasps likely to pass to solve the label imbalance issue?\n\n6. How are the predefined points on the hand chosen and when? Also how are they defined/represented?\n\n7. Was 5cm chosen as a hyperparam? Was there a reason behind the choice of the number?\n\n8. Ln. 161: “while the hand-object pose did not significantly deviate”. What does this mean? What is hand-object pose in this context? The pose of the hand in reference to the object? Or? Also significantly deviate from what? I imagine a successful grasp where each finger moved by 5cm and the picked object and hand both moved up by 20cm, to have deviated quite a bit from their respective initial configurations. \n\n9. Y_pgs is a logical and between y_coll and y_pick? Therefore still a discrete 0 or 1? Then you add a separate continuous label after averaging 5 trials? Is this label saved separately than y_pgs? \n\n10. Ln. 208-209: \"and account for realistic perceptual inputs\", re: what does this mean exactly in this context? The dataset is all SIM which vastly differs from real. \n\n11. Could you provide a hypothesis about why the diffusion and fixed samplers, both with an evaluator, perform on par? I’d expect any kind of refinement to perform better if starting from a better initial pose and I assume the fixed set of grasps is strictly a worse initial pose sampler. \n\n12. Why is there no hardware result corresponding to diffusion/point cloud? If a point cloud coming out of e.g. 2-4 views performs on par with NeRF, wouldn’t that just eliminate the need for a tedious data collection as stated in the limitations section?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors have provided thorough answers to the questions and concerns of the reviewers. The initial scores ( 2 weak accepts and 1 weak reject) have been turned into 3 weak accepts.
Decision: Accept