Get a Grip: Multi-Finger Grasp Evaluation at Scale Enables Robust Sim-to-Real Transfer

Authors: tyler  ga
wei  lum
albert h li
preston  culbertson
krishnan  srinivasan
aaron d ames
mac  schwager
jeannette  bohg
equal  contribution
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-16 03:54:15

Abstract: This work explores conditions under which multi-finger grasping algorithms can attain robust sim-to-real transfer. While numerous large datasets facilitate learning *generative* models for multi-finger grasping at scale, reliable real-world dexterous grasping remains challenging, with most methods degrading when deployed on hardware. An alternate strategy is to use *discriminative* grasp evaluation models for grasp selection and refinement, conditioned on real-world sensor measurements. This paradigm has produced state-of-the-art results for vision-based parallel-jaw grasping, but remains unproven in the multi-finger setting. In this work, we find that existing datasets and methods have been insufficient for training discriminitive models for multi-finger grasping. To train grasp evaluators at scale, datasets must provide on the order of millions of grasps, including both positive *and negative examples*, with corresponding visual data resembling measurements at inference time. To that end, we release a new, open-source dataset of 3.5M grasps on 4.3K objects annotated with RGB images, point clouds, and trained NeRFs. Leveraging this dataset, we train vision-based grasp evaluators that outperform both analytic and generative modeling-based baselines on extensive simulated and real-world trials across a diverse range of objects. We show via numerous ablations that the key factor for performance is indeed the evaluator, and that its quality degrades as the dataset shrinks, demonstrating the importance of our new dataset. Project website at: https://sites.google.com/view/get-a-grip-dataset.

Main paper: 
 introduction: 
 Dexterous, multi-finger grasping is a longstanding challenge in robotics, with applications in tool use, human-robot collaboration, and robot surgery [1, 2, 3] . A common approach for grasp synthesis samples grasp candidates from a generative sampler and refines them with a discriminative evaluator using some grasp metric. Table 1 summarizes prior works employing this "coarse-to-fine" approach.In terms of grasp evaluation, classical methods [4, 5, 6] use analytic metrics [7] to optimize grasps based on grasp mechanics, which require ground-truth object geometry and are sensitive to sensor noise [8, 9] . In contrast, data-driven grasp evaluators [10, 11, 12] estimate grasp quality by conditioning on realistic perceptual data, thus better predicting empirical robustness. While this discriminative approach to grasp synthesis has proven effective for parallel-jaw grasping [13, 14, 15, 16] , its application to the multi-finger setting has seen considerably less success. Consequently, many works study an alternate generative approach of learning and sampling from a distribution over high-quality grasps using large-scale multi-finger grasp datasets [17, 18, 19, 20, 21] . However, the real-world uncertainties in modeling and perception introduce a significant sim-to-real gap that complicates reliable grasp generation on hardware. Therefore, this work's main goal is the synthesis of general, these approaches [6, 4, 5] are initialized using a heuristic sampler and use sampling-based optimization to optimize an analytic grasp metric. More recent approaches [23, 22, 30] leverage bilevel optimization to generate provably force closure grasps. These approaches suffer when accurate models of object geometry are unavailable, which must either be recorded with specialized scanning rigs [31, 32] or reconstructed from visual information into low-quality meshes [30] .When generating grasps in the real world, grasp planners must use real-world object representations like RGB images, depth images, or point clouds. A more recent promising representation is the basis point set (BPS) [33] , a fixed-length feature vector containing the minimal distances to a fixed set of basis points. Neural Radiance Fields (NeRFs) [34] are another representation used for model view synthesis that has proven useful for grasp generation, representing affordances, and semantic grasping [35, 36, 37, 38, 39, 40, 41] . Importantly, all of these works study only parallel-jaw grippers, which have limited use for complex manipulation tasks. One reason for this emphasis is that they can be parameterized with a single element of SE (3) , and it is clear that the relevant NeRF features lie between the jaws. In contrast, the parameterization of multi-finger grasps is not as straightforward; we address this challenge below.Data-driven grasp synthesis methods, as extensively reviewed in [42, 8] , leverage grasp datasets to train grasp planners that take real-world perceptual information as input. Many works have used simulation to generate large parallel-jaw grasping datasets [43, 14] , which have shown to be highly effective [44, 14] . However, generating large-scale datasets for multi-fingered hands is much more complex due to the high number of degrees of freedom and ambiguities in grasp parameterization. Despite this, recent works have released large-scale dexterous grasping datasets, as shown in Table 2 , such as MultiDex [27] , DexGraspNet [17] , and Grasp'D-1M [45] . One key limitation of these datasets is that they only contain positive examples, so they cannot be used to train evaluator models that can distinguish between successful and unsuccessful grasps. One counterexample is MultiGripperGrasp [46] , which uses GraspIt! [4] to generate grasps that are then evaluated in parallel simulation and provides a ranking of grasp goodness. However, it is currently limited to a small set of 345 objects, does not contain visual observations, and does not account for table collisions, which are important for most grasping applications. Many methods use a combination of sampling and evaluator models, such as Mayer et al. [24] , Weng et al. [25] , who propose learning both to sample and evaluate multifingered grasps using basis point sets as input. However, they needed to generate their own dataset of 180K grasps to train these models. Another line of works focuses on learning differentiable grasp evaluators that can optimize grasps through backpropagation [47, 48, 49, 50] . These works also created their own small datasets on the order of 10k grasps across about 100 objects. Our work seeks to aid in future investigations of evaluator-based grasping by supplying a large-scale dataset which mitigates the substantial effort required to generate a new one.3 Get a Grip: Dataset Construction and Grasp Synthesis 
dataset generation: 
 Our dataset contains objects with associated grasps, RGB images, point clouds, NeRFs, and labels (illustrated in Figure 1 ). Let 𝑛 𝑗 and 𝑛 𝑓 denote the number of hand joints and fingers. We parameterize precision grasps as G = (T 𝑂𝐻 , 𝜃, d 1 , . Figure 1 : Data and label generation. Our dataset supplies the components for robust sim-to-real transfer. For each object in the dataset, we provide RGB images of it from several random views, a full point cloud, and trained NeRF weights, which allows our data to be compatible with most vision-based methods. We also supply hundreds of grasps per object, each of which is simulated in Isaac Gym multiple times with slight wrist pose perturbations. Averaging, this yields three smooth regression targets indicating the probability of (1) unwanted collisions, (2) simulated pick success, and (3) grasp success, the logical conjunction of ( 2) and (3). the object, 𝜃 ∈ R 𝑛 𝑗 is the pre-grasp joint configuration, and d 𝑖 ∈ 𝑆 2 is the direction in which the 𝑖 th fingertip moves during the grasp. The grasp data are stored as tuples G 𝑘 , 𝑦 𝑘 coll , 𝑦 𝑘 pick , 𝑦 𝑘 PGS , where each 𝑦 𝑘 * ∈ [0, 1] is a distinct smooth label for grasp G 𝑘 generated in simulation: 𝑦 coll denotes a grasp with unwanted collisions, 𝑦 pick that the simulated pick was successful, and 𝑦 PGS the probability of grasp success, defined as the conjunction of the two.Our grasp generation pipeline is heavily motivated by Wang et al. [17] , but requires key modifications, including (1) placing objects on a table with consistent gravity (instead of floating with different gravity directions) to better match the real world, (2) switching from a binary success label to a continuous success probability label evaluated over multiple trials, (3) modifying evaluation and simulation procedures to improve sim-to-real transfer, and (4) restricting contact points to fingertips.To procure this dataset, we first generate hundreds of grasps on each object from a diverse object set given by DexGraspNet [17] , yielding an initial dataset of about 2M unlabeled grasps. Second, these grasps are simulated in Isaac Gym [51] to compute the associated grasp labels that denote probability of grasp success. Finally, for each of the grasps that are more likely to pass than fail, the dataset is augmented by perturbing each of these grasps several times and re-evaluating them. Because the initial dataset of grasps has many more failed grasps than successful ones, this augmentation gives a new set of grasps with more balanced labels, which, when combined with the initial dataset, yields the final dataset of over 3.5M examples.Step 1: Grasp Generation. We follow the sampling strategy from Wang et al. [17] by defining an energy function 𝐸 (G) that is the sum of various penalty terms on a grasp and sampling low-energy grasps from the distribution 𝑝(G) ∝ exp(-𝐸 (G)) using Langevin sampling. For details, see [17] .The samples are initialized by spawning the hand on some inflated level set of the object mesh such that the palm faces the object with a canonical open-hand joint configuration and then perturbing the configuration with noise. Our weighted energy function (dependence on G omitted) is𝐸 = 𝐸 fc + 𝑤 dis 𝐸 dis + 𝑤 joints 𝐸 joints + 𝑤 pen 𝐸 pen + 𝑤 spen 𝐸 spen + 𝑤 tpen 𝐸 tpen ,where 𝐸 fc is the differentiable force closure estimator introduced in [17] that encourages force closure grasps; 𝐸 dis encourages hand-to-object proximity; 𝐸 joints penalizes joint violations; 𝐸 pen and 𝐸 spen penalize hand-object and self penetration respectively; 𝐸 tpen is a new energy term that penalizes hand-table penetration that was added to keep the grasps above the table. Without this term, many optimized grasps would converge to grasps that fully cage the object from above and below in a way that would not be possible in the real world due to table collisions.Compared to the energy function in DexGraspNet [17] , we only specify desired contacts on the fingertips to sample precision grasps, we recover pre-grasp poses with fingertips 1.5cm off the surface instead of on the surface (more clearance aids collision-free motion planning), and we penalize hand-table collisions for kinematic feasibility. Let p table (•) return the closest point on the table. Then, the table-penetration energy term is given by𝐸 tpen = ∑︁ p∈ 𝑃 (H) max (p -p table (p)) • ĝ, 0 ,where 𝑃(H) is a set of predefined points on the hand H and ĝ is the gravity vector. Lastly, for each grasp, the directions {d 𝑖 } are the directions of the fingertips to the closest points on the mesh. Specifying these directions eliminates ambiguity in grasp execution, aiding reproducibility.Step 2: Grasp Evaluation. We simulate grasps for the above samples in Isaac Gym. The object is first spawned and allowed to settle, and then the hand is spawned at the pre-grasp pose relative to the object. We assume fingertip 𝑖 moves by 5cm along d 𝑖 , and compute a corresponding closed-hand configuration 𝜃 close using inverse kinematics. We then use PD control to drive 𝜃 → 𝜃 close , executing the grasp. Lastly, the object is lifted above the table by 20cm.For each grasp, we check whether (1) the pre-grasp had no collisions, and (2) it was successfully picked while the hand-object pose did not significantly deviate, yielding labels 𝑦 coll and 𝑦 pick . We need both, as Isaac Gym's physics may exploit nonphysical hand-object penetration to achieve a pick, which we forbid. We then construct the label 𝑦 PGS = 𝑦 coll ∧ 𝑦 pick to indicate the grasp is a true, physicallyachievable success. Further, we simulate each grasp 5 times with additional small perturbations, averaging the results to recover non-binary labels in [0, 1] for regression. This smoothing process has been found to help learn robust grasps [52, 53] ; in particular, 𝑦 PGS is supervised to match an empirical success probability, rather than being a measure of classifier confidence. We emphasize that 𝑦 coll , 𝑦 pick , and 𝑦 PGS are smooth labels, which are computed by averaging over all perturbations.In contrast with [17] , we check for feasibility in a tabletop setting rather than spawning the scene in a floating environment. We found this assisted sim-to-real transfer by accounting for table penetrations. Moreover, spawning the hand in a pre-grasp (rather than in-contact) position helped eliminate nonphysical contact modeling behavior in Isaac Gym that led to instabilities in the simulation labels.Step 3: Grasp Augmentation. For each object, 2-20% of sampled grasps were successes. Small amounts of noise are added to these grasps and fingertip directions 5 times and re-evaluated in Isaac Gym, which yields smoothed, balanced labels after averaging the results. These perturbations are larger than those from Step 2 to promote data diversity. See Appendix B.2 for details.Object NeRF Dataset. In total, we use 4.3K unique meshes, each of which are nominally centered and randomly scaled such that its maximum extent is between 0.1 and 0.3m. To introduce more scale diversity while retaining a dataset size under 2.5TB (see Appendix B.4), a subset of 1700 of the objects are instead duplicated at three distinct scales with max extents drawn uniformly from the distributions 𝑈 small (0.05, 0.1), 𝑈 med (0.1, 0.2), 𝑈 large (0.2, 0.3). Lastly, we train NeRFs of various quality by uniformly sampling 100 images over a spherical cap above the object with radius 0.45m and a polar angle sampled from 𝑈 (0, 𝜋 4 ) and training for 400 iterations with nerfstudio [54, 55] .Let the choice of object representation, e.g., a point cloud, be denoted O. Then, the grasp evaluator is parameterized as a neural network ŷpen , ŷpick , ŷPGS = 𝑓 𝜑 (G, O) trained by minimizing an equallyweighted L2 loss over all 3 labels. At inference time, grasp sampling proceeds in 3 steps: (1) a large batch of candidate grasps T is drawn from a sampler, (2) an initial evaluation culls all but the top 𝐾 grasps by probability of grasp success ŷPGS into a smaller set S ⊆ T , and (3) we iteratively refine all grasps in S and finally compute G * = arg max G ∈S ŷPGS (G) afterwards.We use a simple sampling-based update where at iterate 𝑖, we draw some noisy perturbation ΔG (𝑖) , and let ) ). Otherwise, we set G (𝑖+1) = G (𝑖) . At inference time, we do not maximize ŷpen and ŷpick , which are only used to guide evaluator training. We choose to study representations O that can be computed with an eye-in-hand setup. We seek to use the most favorable possible visual features to disentangle the effect of the evaluator and dataset from low-quality measurements in the real world. We move the eye-in-hand camera along a spiralshaped trajectory that gives it diverse views of the object from many angles, yielding high-quality object reconstructions (see the right side of Fig. 10 for visualization).G (𝑖+1) = G (𝑖) + ΔG (𝑖) if ŷPGS (G (𝑖) + ΔG (𝑖) ) > ŷPGS (G (𝑖Two representations compatible with the eye-in-hand are basis point sets (BPSs) [33] and NeRFs [34] . BPSs are fixed sets of points in R 3 labeled with the closest distance to an object mesh or point cloud, essentially acting as fixed-length discretized distance fields for objects, allowing them to be inputs to networks like MLPs. NeRFs are rendering-based models that take in a set of images and their poses, producing a volumetric density field that allows novel-view synthesis. Since paralleljaw grasping works have shown they are a promising representation, we decide to also study them for multi-finger grasping. This gives us two distinct evaluators to compare below. We sample a point cloud from the NeRF and clean it for usage in training, simulation evaluation, and hardware evaluation (see Appendix D for details on network architecture and this pre-processing). 
experiments: 
 We hypothesize that learned grasp evaluators could help achieve robust sim-to-real transfer for multi-finger grasp planners, but only if the associated data are large-scale and account for perceptual data modalities that can be measured in the real world. We now test this hypothesis on the dataset introduced above. We also ablate other modeling choices like object representation or sampler choice to measure the performance increase uniquely attributable to the evaluator. Lastly, we measure the effect of dataset scale on evaluator performance (see Appendix A for additional analysis).Though our goal is robust grasping on hardware, evaluation in simulation is useful since we can test grasps on a larger set of objects. Further, we can execute multiple perturbed grasps to test grasp robustness on the identical object pose and perceptual inputs.First, we study the importance of learned grasp evaluators by comparing with methods that do not use them. We perform an ablation study in which we vary the sampling strategy and the evaluation method. For sampling, we either use a diffusion model-based sampler (motivated by Weng et al. [25] ) or a simple baseline in which we store a fixed set of ∼4000 grasps from the training set (details in Appendix D.2). For evaluator refinement, we either use a BPS evaluator, a NeRF evaluator, or no evaluator, which randomly chooses grasps (can be viewed as an untrained evaluator). Figure 2A shows that all methods that use learned grasp evaluators outperform the methods that do not by a We select 20 objects categorized as easy, medium, or hard based on the complexity of their geometry, the presence of "distractor" geometry, and/or resemblance to objects in the training dataset (see App. E.2 for details). We selected these objects to maximize size/shape diversity while providing a clear upper bound on expected performance via the hard objects. The bottom row shows representative successes and failures for the "Fixed/BPS" configuration.Medium Hard Successes FailuresFigure 4 : Average grasping success rates on hardware across object difficulties. We find that the methods that leverage evaluators (hatched) outperform both analytic and generative-modeling-only baselines. Detailed per-object success rates are shown in Appendix E.3. large margin, independent of the choice of sampler or object representation. The methods that use the fixed set of grasps perform similarly with the methods that use the diffusion sampler when used with a learned evaluator, but unsurprisingly, the method that uses the fixed set of grasps performs significantly worse than the diffusion sampler.Next, we evaluate the effect of training dataset size on evaluator performance. We perform an ablation study in which we train 4 BPS evaluator models with identical architectures and training parameters on different fractional sizes of the training dataset (1%, 5%, 25%, 50%, 100%), all of which were trained to convergence (see Fig. 19 in Appendix E). After training, we compare the performance of these models in the same simulation setup with the same fixed diffusion sampler model. Figure 2B shows that there is a strong correlation between the dataset size and grasp planner performance. It is plausible that this trend will continue with increasing dataset size, but we leave this for future work.We evaluate all methods on a held-out set of 212 objects. We first use each method to generate 5 grasps per object. To evaluate the robustness of these grasps, we simulate each of these grasps 5 times in simulation, each perturbed with small wrist translation and rotation noise, resulting in 25 grasps per object, which is used to compute a per-object success rate.In this section, we evaluate whether grasp evaluators trained on our dataset improve grasping performance in the real world. We compare samplers with/without the evaluator, with/without a diffusion-based sampler inspired by Weng et al. [25] , and also vary the object representation in order to identify the factors driving performance. We also compare to FRoGGeR [22] , a fast analytic method that reports strong real-world results [30] as an additional gauge for performance.For each method, we execute 5 grasps on each of the 20 objects of varying geometry and appearance for a total of 100 grasps per method. Our dataset includes simple objects (like boxes), and adversarial ones (like reflective ski goggles), which we organize into easy, medium, and hard categories as shown in Figure 3 . We remark that the baselines' reported hardware results were largely limited to convex or nearly-convex objects at low quantity [30, 25] . For a more challenging evaluation, we included more objects with non-trivial visual or geometric properties.Our hardware setup consists of an Allegro hand mounted onto a Franka Research 3 arm with an eye-in-hand ZED 2i camera. We only use monocular RGB because the objects are too close to the camera for reliable depth readings. For repeatability, we choose a canonical pose for each object that we use for all trials. The arm follows a fixed trajectory to collect 100 NeRF training images for every grasp. For each method, we request 40 grasps from the planner (∼GPU memory upper bound), then execute the highest-ranking one according to the method's metric (Diffusion No Evaluator has no metric, so the first grasp is selected). Because FroGGeR generates grasps serially, we request only 10 grasps or as many as it can generate in 20 seconds. A grasp is considered a failure if the planner fails to return any grasps or if it does not pick the object up 20cm without dropping it. If feasible grasps are generated but the motion generation fails either during planning or execution (i.e., before the grasp), we discard the trial as the failure is not due to the grasp planner itself.Figure 4 shows the hardware results. We find that all evaluator methods outperform the baselines by a considerable margin independent of the sampler or object representation, with the worst model in each category achieving 93%, 83%, and 55% success rates respectively. We also observe that all evaluatorbased methods perform similarly, which suggests the model performance is not sensitive to the object representation. This may help disentangle representations useful for different downstream tasks from potential negative effects on grasping performance present in other learning-based methods. We discuss qualitative analysis in the Appendices A and E.3. Table 4 also shows that independent of object difficulty categorization, evaluators clearly improve performance. 
limitations: 
 Limitations of our work are as follows. First, our hardware experiments required a time-consuming data collection trajectory and NeRF training to create good quality object models. Future work could explore training models on lower-fidelity object representations from our dataset (e.g., point cloud from 1 depth camera). Second, our methods achieve low success rates on challenging objects such as the mallet and the strainer. The performance of these models is limited by the diversity of their training data, which could be improved by training on a larger set of unique objects, including nonrigid or articulated structures. Third, our grasp generation methods are designed for tabletop settings without clutter or occluding obstacles. Future work could follow the approach used by Sundermeyer et al. [44] for parallel-jaw grasping, which trains a clutter-aware model using a no-clutter dataset. 
conclusion: 
 We introduce a new dataset of 3.5M grasps on 4.3K unique object with RGB images, point clouds, and NeRFs integrated into our dataset, which addresses key challenges for achieving robust sim-toreal transfer of multi-finger grasping. Using this dataset, we perform comprehensive ablation studies that highlight the crucial role of learned grasp evaluators, which significantly outperform existing analytic and generative modeling-based baselines in both simulation and the real world. Our work underscores the critical importance of large-scale grasping datasets and the scale of data for required to train robust multi-finger grasp evaluators. Our contributions represent a significant step forward in the development of reliable and generalizable grasping algorithms, demonstrating the potential for large-scale data and learned evaluators to drive progress in real-world robotic manipulation.Interpreting the Fixed vs. Diffusion Sampler Performance. There are a few unintuitive results regarding the fixed vs. diffusion samplers in both simulation and hardware. First, the Fixed+Evaluator methods seem to outperform the Diffusion+Evaluator methods in simulation, but they perform nearly identically in hardware. We believe that since the train and test objects in simulation are sampled from the same distribution, some test objects will resemble training objects, so some grasps from the fixed dataset may have high quality due to slight overfitting. Since there are no real-world objects in the training set, this effect disappears on hardware.Second, it is interesting that the diffusion sampler alone far outperforms the fixed sampler, but they perform nearly identically after evaluator refinement. We believe that this is because the best samples from the fixed and diffusion samplers have similar quality, but the average diffusion-sampled grasp is far better than the average fixed grasp. Because only the best sampled grasps are further refined, and only the best of the refined grasps is executed on hardware, if the fixed sampling distribution is a reasonable prior for a good grasp, then it is likely that at least one candidate would be a good initial guess for refinement. Our grasp generation pipeline is inspired by Wang et al. [4] , but required key modifications. Further details about these modifications not discussed in the main text are described here. 
 
 Appendix: • As mentioned, our data generation pipeline yields pre-grasp poses with the fingertips 1.5cm off of the surface of the object, while DexGraspNet places the fingers on or very near the surface. Due to this, when planning a grasp trajectory, we compute a new configuration corresponding to fingertip locations 3.5cm below the object surface, which we call the post-grasp pose.• We choose to generate precision grasps rather than power grasps, since this synergizes with the grasp motion parameterization described in the main text. To implement this, we only specify contact candidates on the fingertip of the hand, which function as attractors to the object surface during grasp generation. However, the surface points used to penalize handobject or hand-table collision still cover the whole hand. See Figure 6 for a visualization.• We sample 500 surface points uniformly from the mesh of each link's collision geometry, distributed proportionally based on mesh surface area. 128 contact candidates are densely sampled over the "front" of each Allegro fingertip, which is parameterized by a spherical section centered on the (spherical) fingertip center. This spherical section has a radius of 1.2cm, a polar angular sweep of ±54 • , and an azimuthal angular sweep of 54 After the object settles on the table (𝑡 settle ), the Allegro hand is moved to the pre-grasp position (𝑡 pregrasp ), executes the grasp (𝑡 grasp ), and lifts the object (𝑡 lift ). At 𝑡 pregrasp , we compute 𝑦 coll by checking for hand collisions with the object and table. At 𝑡 lift , we compute 𝑦 pick by checking if the object pose relative to the hand significantly changed from 𝑡 pregrasp .To generate non-binary labels, each grasp is corrupted with small noise and re-simulated 5 times. Note that this is a different perturbation than the one used for dataset augmention described in Appendix B.1. Only the wrist pose is perturbed. The wrist translations are corrupted with uniform noise drawn from [-5mm, 5mm] along each axis, and the wrist orientation is corrupted about the roll, pitch, and yaw axes with uniform noise drawn from [-2.5 • , 2.5 • ].The simulation timestep is set to 1/60 seconds, but the integrator solves at a rate of 1/120 seconds for numerical stability using the Truncated Gauss-Seidel (TGS) solver. The TGS solver runs 8 position and 8 velocity iterations per simulation timestep. "Force at a distance" (i.e., the contact offset parameter) is applied starting from 1mm of separation between collision geometries, substantially lower than the 1cm distance used by [4] in DexGraspNet. This helps reduce unwanted non-physical collisions. Geometries are processed by a convex decomposition using the default settings in Isaac Gym. Each object is spawned 2cm above the table and dropped. The simulation starts when the object settles. The simulation pipeline is visualized in Figure 7 .Figure 8 shows the distribution of grasp labels across the dataset of 3.5M grasps. The median and IQR of each label is as follows:• 𝑦 PGS : 0.0 (0.0, 0.48)• 𝑦 pick : 0.2 (0.0, 0.8)• 𝑦 coll : 0.08 (0.6, 1.0)The objects considered in this work are a strict subset of those in DexGraspNet [4] y coll 0 1 2 # Grasps ×10 6Distribution of y coll Labels that we assume a tabletop setting while DexGraspNet does not, which necessitates a canonical "up" direction. Thus, many objects were manually processed to be oriented in a reasonable manner. We only use an object if after this reorientation, the object successfully settles when dropped from a height of 2cm. If the object tips over or does not settle after 200 timesteps in the Isaac Gym simulation, then it is excluded. We consider an object to have remained upright if the real part of its relative quaternion remains greater than 0.95 throughout the simulation. We consider it to have settled if the translational components stay within 1mm along each axis and roll, pitch, and yaw stay with 1e-2 radians for 15 consecutive timesteps. For all objects, we used a uniform friction coefficient of 0.9 and a constant object density of 500 kg/m 3 (as in [4] ). Thus, object masses were determined entirely by volume, as in other works [7, 4, 8] .Figure 9 visualizes a subset of our object dataset. To introduce more scale diversity while retaining a dataset size under 2.5TB, a subset of approximately 1.7K of the objects is duplicated at three distinct scales, while the remaining objects are used at one scale each, resulting in approximately 7.7K objects. The large dataset storage requirement comes from the storage of multiple 3D NeRF density grids used to represent each grasp for the NeRF evaluator (more details about the NeRF density grid representation in Appendix D.3.2).We create our train, validation, and test splits of the datasets very carefully. We make sure that each unique object mesh only exists in one of these three sets.We We used 4 Nvidia A100 GPUs to generate our dataset. It took ∼4 days to generate the full dataset: ∼1 day for grasp generation, ∼1 day for grasp label generation, and ∼2 days for image generation, NeRF training, and point cloud generation. We train NeRFs using both simulated data and real-world data and compare them here.For NeRFs trained on simulated data, the object is first spawned and allowed to settle on the table. Then, we uniformly sample 100 images over a spherical cap above the object with a radius of 0.45m and a polar angle sampled from 𝑈 (0, 𝜋 4 ), and then train for 400 iterations with nerfstudio [9, 10] . For NeRFs trained on real-world data, we first collect 100 images while moving the wrist-mounted camera along a hard-coded trajectory encircling the object and then train for 400 iterations with nerfstudio [9, 10] . This trajectory consists of 3 spirals around the object, with the object placed roughly in the center of the spirals.Figure 10 shows a qualitative comparison between the RGB images and NeRFs trained on simulated data versus real-world data. Both types of NeRFs exhibit floater artifacts [11] , which must either be processed away or ignored by downstream models.We train without scene contraction, auto-scaling poses, centering method, or orientation method, and use a scale factor of 1.0, which ensures that the coordinate space in the NeRF is not modified from the given data. The remainder of the parameters follow the default nerfacto settings in nerfstudio.Methods that represent the object as basis point sets (BPSs) use the following procedure.1. Train a NeRF following the procedure described in Appendix C.1.2. Sample a point cloud with 5000 points using nerfstudio [10] by rendering out depth images with opacity exceeding 0.5 in the axis-aligned bounding box parameterized by the lower and upper bounds [-0.2𝑚, -0.2𝑚, 0.0𝑚] × [0.2𝑚, 0.2𝑚, 0.3𝑚] using a z-up convention.3. Preprocess the point cloud to remove most outliers and floaters. First, we use open3d [12] to remove statistical outliers (nb_neighbors=20, std_ratio=2.0) and radius outliers (nb_points=16, radius=0.05). Next, we construct an undirected graph of theReal-World Dragon remaining points, where points are nodes and an edge exists between two nodes if the points are within 1mm. Floaters are removed by keeping the largest connected component.4. Generate a random set of 4096 basis points within a sphere of radius 0.3m, centered 0.15m above the table. This set of basis points remains fixed for all experiments. We utilize bps [13] for basis point set operations.5. Compute basis point set values by calculating the distance from each basis point to its closest point in the point cloud.Figure 11 shows an example of a point cloud and BPS for both a simulated and real-world object.We use triangle meshes for two purposes: the analytic grasp planning method (FRoGGeR) that we use as a baseline and collision-free motion planning from the start pose to the pre-grasp pose (all methods).For both applications, the mesh is generated by the following procedure.1. Training a NeRF following the procedure described in Appendix C.1.2. Using scikit-image [14] to perform marching cubes, which extracts a 2D surface mesh from a 3D volume with a NeRF density level set of 15 within a bounding box of [-0.2𝑚, -0.2𝑚, 0.0𝑚] × [0.2𝑚, 0.2𝑚, 0.3𝑚] (with z being the up-direction).Real-World Dragon NeRF-Generated Mesh NeRF-Generated MeshGround-Truth Mesh RGB Image Comparison between an RGB image and the NeRF-generated mesh of a real-world dragon object. We do not have a ground-truth mesh for the real-world dragon object.3. Using trimesh [15] to remove floaters. This is done by only keeping connected components with at least 31 edges.Figure 12 shows examples of NeRF-generated meshes. We found the parameters above to be reasonable for all test objects.As explained in the main text, during grasp planning, a batch of candidate grasps is sampled, and only the top 𝐾 of these are retained for the refinement phase. In our simulation experiments, we let 𝐾 = 5, which are all refined and executed in simulation. In our real-world experiments, we let 𝐾 = 40, which are all refined and the best is executed on hardware. See Figure 13 for a schematic of the sample/refine process.Recall that our refinement is sampling-based, where the previous iterate is perturbed by some number of samples and the best one is chosen to be the next iterate. For all perturbations, we use zero-mean Gaussian noise as follows.• Wrist translation: standard deviation of 5mm per spatial coordinate.Evaluator (Rank) Evaluator (Refine) Grasp Candidates Ranked Grasps Top K Refined Grasps Figure 13 : Our grasp planners consist of a sampler and an evaluator. First, the sampler generates a batch of grasp candidates. The evaluator ranks them, and the top 𝐾 grasps are refined using sampling-based optimization, where the objective is given by the evaluator.• Wrist and grasp orientations: the noises are sampled using a standard deviation of 0.05m from 𝔰𝔬(3), which are then converted to elements of 𝑆𝑂 (3) via the exponential map.• Joint angles: standard deviation of 0.01 radians.The refinement is run for 50 iterations. Other refinement strategies could work as well, such as gradient-based methods or other sampling-based methods.To allow grasp parameters to be represented as vector inputs or outputs, we parameterize them as g = (x, r, 𝜃, d 1 , ..., d 𝑛 𝑓 ) ∈ R 9+𝑛 𝑗 +3𝑛 𝑓 , where x ∈ R 3 is the wrist position, r ∈ R 6 is the wrist orientation represented by a continuous 6-D rotation vector [16] , 𝜃 ∈ R 𝑛 𝑗 is the pre-grasp joint configuration, and d 𝑖 ∈ R 3 is the direction in which the 𝑖 th fingertip moves during the grasp.Figure 14 shows the diffusion sampler architecture inspired by Weng et al. [1] . As their implementation is not publicly available, we re-implemented their architecture to the best of our ability via the textual description. We use an embedding dimension of 128 and a sequence length of 4 for all attention modules. The sequence length is created by reshaping the outputs of the fully-connected layers (to (4, 128)). We use Denoising Diffusion Implicit Models (DDIM) [17] with a linear scheduler on the noise variance 𝛽 that moves from 0.0001 to 0.02 over a total of 1000 diffusion timesteps.Figure 15 shows the train and validation loss curves for this model.Although the diffusion sampler is trained to only generate successful grasps, it can still produce unsuccessful grasps as seen in previous works [1, 2] . While it could typically generate feasiblelooking grasps on a diverse range of objects, its most common failure modes were (1) generating grasps that were "close" to successful but with fingers slightly too close or far to successfully execute, and (2) generating a grasp that was substantially too far from the object.To disentangle the importance of the learned evaluator from the impact of the diffusion sampler, we perform additional experiments in which we replace the diffusion sampler with a simple baseline in which we store a fixed set of 4349 grasps from the training set. More specifically, this was generated by storing one grasp per training set object that exhibited a high probability of success (𝑦 PGS ≥ 0.9). For objects that did not have any such grasps, we did not store any. We emphasize that these grasps were from training set objects, so they were not designed for the unseen test objects used for simulation evaluation or the real-world objects used for hardware evaluation.Recall that the evaluators are trained by regressing on three distinct soft labels: 𝑦 PGS , 𝑦 coll , and 𝑦 pick .Although we only use the ŷPGS prediction at inference time, we choose to regress the other two labels ŷcoll and ŷpick at train time, as we found that these labels provide an additional signal about why a given grasp may be failing. Figure 16 shows the architecture used for the BPS evaluator. We used the official implementation from Mayer et al. [2] [here], which is the same architecture used by Weng et al. [1] .Figure 17 shows the architecture used for the NeRF evaluator. We use a novel grasp representation that leverages NeRF features directly.We will motivate this representation and then describe it in detail. A key factor for grasp success is the surface geometry at the contact points. Modeling this from images is difficult, and few (if any) fast, reliable surface reconstruction methods are adequate for grasp planning. Here, we use NeRF features that we hypothesize capture accurate estimates of the object surface. Centered at finger 𝑖's position x 𝑖 , a square of side length 0.06m is swept along d 𝑖 by 0.08m to recover a rectangular prism which is discretized into a 4D tensor N 𝑖 ∈ R 4×31×31×41 , where the first channel dimension is the NeRF density and the last 3 are spatial coordinates. These grid dimensions overapproximate both the fingertip size and the grasp depth to ensure the geometric information captured is not too local. Further, to capture the global object geometry, we generate a grid centered on the estimated object centroid with side lengths 0.4m N 𝑔 ∈ R 4×41×41×41 . The centroid is estimated by assuming uniform mass density and integrating over spatial regions with NeRF density exceeding 15.0.In summary, our architecture uses local NeRF densities and global NeRF densities as inputs. The local NeRF densities are sampled grids approximating the fingertip swept volumes when moving along the grasp directions. The global NeRF densities are a sampled grid of fixed size to capture the object's global geometric features.For all experiments that use FRoGGeR [18, 19] , we use the open-source implementation provided at https://github.com/alberthli/frogger. The procedure for acquiring the meshes used for planning is described in Appendix C.3.Out of the 100 real-world pick attempts using FRoGGeR, there were a total of 49 failures. 16 failures were caused by planned grasps that failed to lift the object, and 33 failures were caused by planning issues in which no grasp could be found within the timeout period. FRoGGeR allows users to either plan with a floating hand or the full hand-arm system. We opt to use the full hand-arm system so that the generated grasps account for kinematic reachability of grasp poses when using a given arm, resulting in fewer downstream motion planning failures.We hypothesize that learned grasp evaluators could help achieve robust sim-to-real transfer for multi-finger grasp planners, but only if the associated data are large-scale and account for realistic perceptual inputs.Looking at prior works, we see a clear trend toward larger multi-fingered grasp datasets used for training grasp evaluators (see Figure 18 ). Recognizing this trend, our goal with this ablation is to study how much the increased scale of our dataset (3.5M grasps across 4.3K objects) impacts performance, which we measure by examining the improvement in the simulated probability of grasp success gained by evaluator refinement.Figure 19 shows the train and validation loss curves of each model trained on a different fraction of the training dataset and validated on the same validation dataset. We see that all models are trained to convergence and achieve similar train losses, but their corresponding validation losses show a consistent correlation between dataset scale and validation performance.Figure 20 shows the real-world objects labeled with their corresponding names. Easy objects are those that have high-quality grasps when approached from any or most directions. For example, tall, cylindrical objects can be grasped overhead or from the side. unusual geometry, including "distractors," or objects which are harder to grasp when the hand is arbitrarily oriented. For example, the mug has a handle and the lunchbox is more easily pinched in the thin direction. Lastly, hard objects contain very unusual, non-convex geometry/visual features, or are difficult to grasp without affordances. For example, the goggles are both reflective and transparent, which poses a challenge for object reconstruction algorithms, and the bunny's ears are a large distractor for grasp planners. The bunny, squirrel, and dino were custom printed as hard test objects. The strainer and mallet were chosen to be out-of-distribution objects, providing a ceiling on the performance of our grasping algorithms. No method obtained any successful grasps on the strainer.Table 4 presents the detailed quantitative results of all grasp planning methods on various objects in the real world.Qualitatively, we observed that our evaluator-based methods very rarely exhibited edge-seeking behavior, wherein the fingers are placed on corners or edges of objects, and has been noted as a common failure mode by other works [18, 19] . We suspect this may be a result of our label smoothing, though we do not rigorously test this. However, "distractor" geometries like the bunny ears or spray nozzle frequently caused grasp planning failures, attracting the fingers towards them. Methods like FRoGGeR rely on online mesh reconstruction, so objects like the goggles caused issues due to transparency/reflectivity. Lastly, many of the diffusion sampler's unrefined grasps appeared "close" to success even when failing, which suggests that generative modeling may be a viable strategy with more data. Some examples are shown in Figure 21 . MethodsHardTo evaluate a given grasp in the real world, we need to use a motion planner to move the hand to the corresponding pre-grasp pose. We use cuRobo [20] , which performs parallelized collision-free motion generation. In particular, we use cuRobo's graph planner, which uses Probabilistic Road Map (PRM) to find a collision-free path. Object collision avoidance requires an object mesh, which is acquired through a process explained in Appendix C.3. We use a collision buffer of 1mm.When executing a grasp on hardware, we decompose the motion into three stages: (1) start pose to pre-grasp pose, (2) pre-grasp pose to grasp pose, (3) grasp pose to lift pose.The first stage is the most challenging motion planning problem because the hand needs to find a collision-free path to get very close to the object. To simplify this problem, we utilize inverse kinematics to adjust the pre-grasp finger joints, moving them an additional 3cm backward along the fingertip grasp directions.In addition to ensuring kinematic feasibility, we also need to avoid damaging the cabling of the wrist-mounted camera during grasp execution. To achieve this, we filter out grasp samples if either (1) the normal direction of the palm is within 60 degrees from the upward direction of the world frame or (2) the direction from the palm to the middle finger deviates by more than 60 degrees from the forward direction of the world frame.To account for these checks, we request 40 grasps from the grasp planner, which are then sorted by the grasp planner's metric. If no metric is available, such as in the case of a diffusion sampler without an evaluator, the grasps remain unsorted. Next, we utilize cuRobo to solve all 40 motion planning problems in parallel and then execute the first grasp for which a motion plan is successfully found. 


Decision: The authors have provided thorough answers to the questions and concerns of the reviewers. The initial scores ( 2 weak accepts and 1 weak reject) have been turned into 3 weak accepts.

Reviews: review: title: title: Simulated Multi-Finger Dataset and Evaluators Enabling Sim2Real Transfer for Grasping
paper_summary: paper_summary: The paper introduces "Get a Grip," a large-scale simulation dataset for multi-finger grasp evaluation, featuring 3.5 million grasps on 4.3 thousand diverse objects annotated with RGB images, point clouds, and NeRFs. This dataset enables the training of robust discriminative grasp evaluators. The approach demonstrates significant improvements in both simulation and real-world environments, showcasing effective sim-to-real transfer. The method outperforms baseline models, highlighting the importance of large-scale datasets and the integration of learned evaluators for robust multi-finger grasping.
main_review: main_review: ### Strengths
- **Large-Scale Multi-Finger Dataset**: The paper introduces a comprehensive dataset for multi-finger grasp evaluation, featuring 3.5 million grasps on 4.3 thousand diverse objects. This extensive dataset includes RGB images, point clouds, success labels, and pre-trained NeRFs, providing a rich source of data for training robust grasp evaluators.
- **Effectiveness in Simulation and Real-World Environments**: The approach demonstrates the effectiveness of the learned evaluators in both simulation and real-world settings, showcasing significant improvements over baseline models and successfully translating to practical applications in robot grasping tasks.
- **Strong Experimental Validation**: The extensive experimental validation, including thorough ablation studies, underscores the robustness and effectiveness of the proposed method. The experiments cover a wide range of scenarios, proving the versatility and practical applicability of the approach in diverse environments.
- **Clear Articulation**: The paper is clearly articulated, with well-structured sections and detailed explanations of the methods and experiments. For example, the authors provide a step-by-step description of the dataset creation process and the implementation of the grasp evaluators, making it easy for readers to understand and replicate the work.

### Weaknesses
- **Limited to Simulated Rigid Objects**: The method and dataset are primarily focused on simulated rigid objects, which limits their applicability to real-world scenarios involving non-rigid or articulated objects.
- **Non-Occluded Single Object Assumption**: The approach assumes the presence of non-occluded single objects, which may not hold true in more complex and cluttered real-world environments.
- **Specific to Allegro Hand**: The dataset generation and evaluation pipeline are specifically designed for the Allegro hand, which may limit the generalizability and applicability of the method to other robotic hands and setups.
questions: questions: - In Figure 2 (a), why does fixed methods with learned evaluators perform than that using diffusion? Since the massive improvement comes from the usage of evaluators, could the implementation of evaluators affect the results of fixed vs diffusion?
- What was the runtime and compute resources for collecting this dataset?
- Could the choice of heuristics bias toward certain type of grasps for success grasps?
- What strategies are in place to ensure the robustness of the method when applied to objects with different physical properties (e.g., weight, friction) in real-world scenarios?
limitations: limitations: 3
review_summary: review_summary: The paper introduces a large-scale multi-finger grasp evaluation dataset, demonstrating improvements in both simulation and real-world environments. The dataset, featuring 3.5 million grasps on 4.3 thousand diverse objects, is well-annotated with RGB images, point clouds, success labels, and pre-trained NeRFs. This rich data source enables effective training of robust grasp evaluators, facilitating sim-to-real transfer. While the method shows promise, it is primarily focused on simulated rigid objects, assumes non-occluded single objects, and is specific to the Allegro hand, limiting its generalizability to more complex and cluttered real-world scenarios. Given these strengths and limitations, I recommend a weak accept.
title: title: data paper, weak accept
paper_summary: paper_summary: This paper released a new, open-source dataset of 3.5M grasps on 4.3K objects annotated with RGB images, point clouds, and trained NeRFs.  Using this dataset, this paper trained vision-based grasp evaluators that outperform existing analytic and generative modeling-based baselines in both simulated and real-world.
main_review: main_review: The introduction outlines the problem through an extensive literature review. The methodology and calculation models are presented clearly and in sufficient detail. The discussion section effectively explains the results obtained. The graphic material is of good quality and self-explanatory.

Strengths:
1. the dataset generation pipeline is clear and easy to follow.
2. The data scale is large and types are rich and can be used in different algorithms.
3. Simulation and real-world tests are implemented in the dataset.
4. The comprehensive ablation studies provide valuable insights into the critical role of learned grasp evaluators,  demonstrating the importance of large-scale datasets for training robust algorithms.

Weaknesses:
1. The dataset generation pipeline is currently designed specifically for the Allegro hand, which may limit its immediate applicability to other dexterous hands.
2. The hardware experiments require a time-consuming data collection process and NeRF training, which may hinder the practical application of the methods.
3. The methods achieve low success rates on certain challenging objects, indicating a need for more diverse training data and improved algorithms.
4. The grasp generation methods are designed for uncluttered tabletop settings, which may not fully represent real-world environments with obstacles and clutter.
questions: questions: 1. The dataset contains 4.3K objects, but details are missing, how were these items selected?
2. the dataset link in the project website is not available now, when will it be available?
3. The 20 objects in the hardware test are categorized as easy, medium, or hard based on the complexity of their geometry, Is there any quantitative metrics?
limitations: limitations: 3
review_summary: review_summary: This paper released a large scale grasping dataset and generation pipeline, the paper can be recommended as weak accept.
title: title: Very useful dataset contribution, unsure about how the dataset connects to grasp evaluator hypothesis and subsequent experiments ran
paper_summary: paper_summary: The paper introduces a new large scale dataset of simulated grasps that comes with RGB images, point clouds and trained NeRF.
main_review: main_review: 1. Missing references (non exhaustive list):

When Transformer Meets Robotic Grasping: Exploits Context for Efficient Grasp Detection. S. Wang, Z. Zhou, and Z. Kan. 

Geometry Matching for Multi-Embodiment Grasping. M. Attarian, M. A. Asif, J. Liu, R. Hari, A. Garg, I. Gilitschenski, J. Tompson

Grasp Proposal Networks: An End-to-End Solution for Visual Learning of Robotic Grasps. C. Wu, J. Chen, Q. Cao, J. Zhang, Y. Tai, L. Sun, K. Jia

2. Ln. 56-57: “condition on images taken from a trajectory encircling the object”, re: unclear sentence. What does this mean exactly? 

3. Ln. 59-60: “We find that dataset scale is necessary for training effective grasp evaluators, which significantly degrade in performance when trained on smaller subsets of the data.”. I would expect this to also hold for generative methods without an evaluator. 

4. Ln. 70-71: “We test our evaluators trained on our dataset on thousands of simulated and hundreds of real-world grasps, and demonstrate state-of-the-art performance.”. Surprised to see a SoTA claim. Is there a new evaluator method introduced in this paper? Impression left from the introduction was that multiple existing evaluators are used. If there is a new evaluator method introduced that achieves SoTA, please introduce in introduction section. 

5. Ln. 130-134: It’s not completely obvious or intuitive that perturbing grasps that are more likely to pass, and reevaluating, yields more grasps that are likelier to pass. 

6. Ln. 154: Unclear why directions of fingers are needed for ambiguity of grasp execution. Typically each joint has an angle range that it can take that corresponds to a configuration and the finger will move in whatever direction it needs to to get to that configuration.  

7. Fig. 3: Sounds like the categorization of objects in easy, medium, hard is subjective. Complexity of geometry: how is that defined/measured? Resemblance with objects in the training set: how is that measured?

8. Only one baseline included for hardware results. I’d argue that diffusion/no evaluator is not a baseline, it’s an ablation. 

9. The claim that evaluators are strictly outperforming analytic and generative methods without an evaluator, is a bit of an overclaim. There was no comparison provided to any generative methods.
questions: questions: 1. How do you label borderline grasps? Edit: reading later now, it sounds like labels are 0-1 probabilities for each grasp succeeding?

2. Dataset scale being necessary for training effective grasp evaluators, re: Why do you distinguish grasp evaluators here? This should generally hold with generative approaches without evaluators as well. In general, it should hold with any grasping approach that trains a neural network.

3. Ln. 119-120: Therefore, y_coll and y_pick are strictly 0 or 1 where y_pgs is a continuous value between 0 and 1?

4. Ln. 129-130: How many trials do you do in IsaacGym per grasp? And is the label computed by a simple <attempts_passed>/<total_attempts> formula? 

5. Could you explain what perturbation exactly do you do on grasps likely to pass to solve the label imbalance issue?

6. How are the predefined points on the hand chosen and when? Also how are they defined/represented?

7. Was 5cm chosen as a hyperparam? Was there a reason behind the choice of the number?

8. Ln. 161: “while the hand-object pose did not significantly deviate”. What does this mean? What is hand-object pose in this context? The pose of the hand in reference to the object? Or? Also significantly deviate from what? I imagine a successful grasp where each finger moved by 5cm and the picked object and hand both moved up by 20cm, to have deviated quite a bit from their respective initial configurations. 

9. Y_pgs is a logical and between y_coll and y_pick? Therefore still a discrete 0 or 1? Then you add a separate continuous label after averaging 5 trials? Is this label saved separately than y_pgs? 

10. Ln. 208-209: "and account for realistic perceptual inputs", re: what does this mean exactly in this context? The dataset is all SIM which vastly differs from real. 

11. Could you provide a hypothesis about why the diffusion and fixed samplers, both with an evaluator, perform on par? I’d expect any kind of refinement to perform better if starting from a better initial pose and I assume the fixed set of grasps is strictly a worse initial pose sampler. 

12. Why is there no hardware result corresponding to diffusion/point cloud? If a point cloud coming out of e.g. 2-4 views performs on par with NeRF, wouldn’t that just eliminate the need for a tedious data collection as stated in the limitations section?
limitations: limitations: 3
review_summary: review_summary: Thank you to the authors for this work. The community can definitely benefit from a new large multi-finger grasp dataset that has a wider variety of input modalities. However, further clarity should be provided on how the grasp evaluator argument and experiments connect to the dataset itself. If the argument is that grasp evaluators are generally better than analytic and generative methods on large scale data regimes such as the new dataset, then more comparisons with more baselines from both categories should be added to support this argument. I’m willing to increase my score if more such comparisons or added or if authors explain more clearly the connection of the dataset and the grasp evaluators hypothesis in case I misunderstood.