Geometry Matching for Multi-Embodiment Grasping
Authors: Maria Attarian
Muhammad Adil Asif
Jingzhou Liu
Ruthrash Hari
Animesh Garg
Igor Gilitschenski
Jonathan Tompson
Venue: CoRL 2023 Poster (2023)
Keywords: Multi-Embodiment
Dexterous Grasping
Graph Neural Networks

Abstract: While significant progress has been made on the problem of generating grasps, many existing learning-based approaches still concentrate on a single embodiment, provide limited generalization to higher DoF end-effectors and cannot capture a diverse set of grasp modes. In this paper, we tackle the problem of grasping multi-embodiments through the viewpoint of learning rich geometric representations for both objects and end-effectors using Graph Neural Networks (GNN). Our novel method - GeoMatch - applies supervised learning on grasping data from multiple embodiments, learning end-to-end contact point likelihood maps as well as conditional autoregressive prediction of grasps keypoint-by-keypoint. We compare our method against 3 baselines that provide multi-embodiment support. Our approach performs better across 3 end-effectors, while also providing competitive diversity of grasps. Examples can be found at geomatch.github.io.

Geometry Matching for Multi-Embodiment Grasping
Maria Attarian Muhammad Adil Asife Jingzhou Liu?, Ruthrash Hari?, Animesh Garg2 Igor Gilitschenski", Jonathan Tompson'
'Google DeepMind, UUniversity of Toronto, 'Georgia Institute of Technology
Figure 1: GeoMatch: Our method enables multi-embodiment grasping by conditioning the grasp selection on end-effector and object geometry.
Many existing learning-based grasping approaches concentrate single embodiment, provide limited generalization to higher DoF end-effectors and cannot capture a diverse set of grasp modes. We tackle the problem of grasp- ing using multiple embodiments by learning rich geometric representations for both objects and end-effectors using Graph Neural Networks. Our novel method GeoMatch applies supervised learning on grasping data from multiple em- bodiments, learning end-to-end contact point likelihood maps as well as condi- tional autoregressive predictions of grasps cerpointby--kyyoinn our method against baselines that support multiple embodiments. Our approach performs better across three end-effectors, while also producing diverse grasps. Examples, including real robot demos, can be found at geo-match, gi thtth..... io.
Abstract:
We compare
Keywords: Muboo--Emb,dient,, Dexterous Grasping, Graph Neural Networks
1 Introduction
Dexterous grasping remains an open and important problem for robotics manipulation. Many tasks where robots are involved boil down to some form of interaction with objects in their environment. This requires grasping objects with all kinds of different geometries. In addition, the large variety of robot and end-effector types necessitates that grasping should also be achievable with new and arbitrary end-effector geometries. However, the cross-embodinent gap between grippers does not permit simply applying grasping policies from one end-effector to another, while domain adaptation i.. "translating" actions from embodiment to another, is also not straightforward. In compari- son, humans are extremely versatile: they can adapt the way they grasp objects based on what they know about object geometry even if the object is new them, and they can do this in multiple ways.
There has been much research in grasping thus far, with many works focusing on one embodiment at time [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] and fewer looking at the nulti-embodiment problem [12, 13, 14]. Methods are divided between hand-agnostic or hand-aware, and experiment with different representations for grasping, such as contact maps [12], contact points [13] even root pose and
Correspondence emails: jmattarian@ggoole.comm adil.asif@mailutoornnee mail.utorooto.ca
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

z-offset [14]. Existing multi- multi-embodiment approaches either require explicit representation of joint limits that becomes exponentially harder higher DoF end-effeetors, or expect heavy manual work to adapt to new end-effectors, or showcase mixed rates of success across different embodiments with some gripper/hand morphologies performing significantly better than others.
Drawing inspiration from how humans adapt their grasps easily and successfully based on priors they have learned about 3D geometry of both objects space and their own hands, we propose endowing robotic agents with a similar sense of geometry via a generalized geometry embedding that is able to represent both objects and -ffeccorss This embedding can be used to predict grasps that demonstrate stability, diversity. generalizability, and robustness. More specifically, we propose to rely on Graph Neural Networks (GNN) to encode meaningful geometry representations and use them to predict keypoint contacts on the object surface in an autoregressive manner.
In summary, our contributions are as follows:
a) We propose formulating robot grasp learning geometry matching problem through learning correspondences between geometry features, where both, end-effector and object geometries, are encoded in rich embedding spaces.
b) To solve the above problem, we introduce GeoMatch, a novel method for learning expressive geometric embeddings and keypoint contacts.
We demonstrate that our method is competitive against baselines without any extra requirements to support higher DoF end-effectors and while also showcasing high performance across multiple embodiments. Finally, we provide demos of our method on real Franka Emika with an Allegro hand.
2 Related Work
Dexterous Grasping. Many dexterous grasping works do not look into multi-embodiment and in- stead focus on diversity of objects using a single end-effector. Many grasping methods support 2-finger parallel grippers [2, 3, 4, 5, 6, 7, 8, 15] with several others looking into high-DoF dexterous manipulation [9, 10, 1, 16, 17. 18]. Some work has also been conducted towards multi-emmboimmen grasping. Several of those address the problem from the differentiable simulation grasp synthesis point of view [19, 11, 20). GenDexGrasp [12] advocate for hand-agnostic contact maps generated by trained e VAE [21] with a newly introduced align distance, and optimize matching of end-effectors against produced contact maps via a specialized Adam optimization method. This is the most recent work to our knowledge, attempting tackle cessing to support higher DoF grippers. In contrast, we choose to operate on hand-specific contact maps as we are interested in learning both object and embodiment geometry conditioned grasps, and empirically found our method to perform more evenly well multi-embodiients.
grasping without significant prepro-
Intuitively, our work is closest to UniGrasp [13]. UniGrasp operates on object and end-effector point clouds to extract features and ultimately output contact points which are then fed into an Inverse Kinematics (IK) solver, similarly to us. Their encoder of choice is PointNet++ and the contact prediction is done through a Point Set Selection Network (PSSN) [22]. Their proposed architecture adds one stage per finger, which means supporting more than 3 finger grippers requires manually adding another stage. As a result, adapting the method to more than 2-finger and 3- finger grippers requires significant work. Moreover, the method requires explicit representation of boundary configurations through combination of minimum and maximum values for each joint angle which explodes exponentially on higher DoF end-effectors. In contrast, we rely on learned geometry features to identify viable configurations as opposed to explicitly encoding them through joint limit representation. Our method only requires small number of user-selected keypoints per end-effector, same number for all end-effeectos. This disentangles the dependency between number of fingers and applicability of our method. Similarly to UniGrasp, EfficientGrasp
[23] also uses PointNet++ and PSSN model for contact point prediction. TAX-Pose [24] is another recent work that shares some high level concepts. Instead of encoding the end-effector, authors consider tasks involving objects that interact with each other in a particular way, and what the relative pose of interacting objects should be for the task to be considered successful. They proceed with encoder objects or object parts using DGCNN and learn cross- cross-attention model that predicts relative poses of objects that accomplish a task. AdaGrasp [14] uses 3D Convolutional Neural Networks to learn scoring function for possible generated grasps, and finally executes the best one.

---

Finally, many of the methods mentioned, rely on deterministic solvers which can result in decreased diversity of generated grasps. While we also rely on a deterministic solver, we address this issue by leveraging the scoring we obtain by the learned full unnormalized distribution of contacts. The score is used select a first keypoint that will guide the remaining contact point prediction. This permits higher diversity without having to sample a large number of grasps.
Graph Neural Networks. Graph Neural Networks were first introduced by Scarselli al. [25] as a proposed framework to operate on structured graph data. Since then, many advancements have been made towards extending their capabillities and expressivity [26]. Specifically in the grasping literature, there have been multiple applications of GNN. E.g. Huang et al. [27] propose learning a GNN predict 3D stress and deformation fields based on using the finite element method for grasp simulations. The use of GNNs for end end-effector parameterization has been proposed before in [28] where tactile sensor data is fed into a GNN to represent the end-effector as part of grasp stability prediction. Lou et al. [29] leverage GNN to represent the spacial relation between objects in a scene and suggest optimal 6-DoF grasping poses. Unlike previous methods, we propose applying GNN as general geometry representation for any rigid body, both objects and end-effectors jointly. For the purposes of this work, we leverage the GNN implementation by [30] due to the readily available and easily adaptable code base.
Geometry-Aware Grasping. Several works have emphasized the role of geometry in the grasping problem. Some works study geometry-awan grasping under the light of shape completion [31, 32]. Yan et al. [33] encodes RGBD input via generative 3D shape modeling and 3D reconstruction. Subsequently, based on this learned geometry-awan representation grasping outcomes are predicted with solutions coming out of an analysis-by-synn bysstthesis optimization. In the same vein, Van al. [34] proposed leveraging learned 3D reconstruction as means of understanding geometry, and further rely on this for grasp success classification as an auxiliary objective function for grasp optimization and boundary condition checking. Bohg et al. [35] introduced classifier trained on labeled images to predict grasps via shape context. Finally, Jiang et al. [6] propose to learn grasp affordances and 3D reconstruction as auxiliary task, using implicit functions. Unlike these works, we suggest looking at geometry itself directly from 3D as a feature representation without imposing any 3D reconstruction constraints.
3 Method
In this work, we aim to learn robust and performant grasping prediction from the geometries of both the object Go and the end-effector GG. Our approach models the contact between pre-defined keypoints on the end-effector and contact points on the object surface. That is, we aim to learn a model c CN GeoMatch (Go, 9c, (k.) where & E R3 are the predicted contact points on the object surface each of which corresponds to one of N pre-defined keypoints Kę E on the end-effector.
Method Overview. We represent object and end-effector geometries using graphs Go (Vo. Eo), and GG = (Va, Eg) which can easily be generated from object point clouds and a URDF descrip- tion respectively (Sec. 3.1). This formulation allows us to utilize GNNs to learn geometry- aware emheddings which are subsequently fed into an autoregressive grasp generation module (Sec. 3.2). The losses are used to supervise the graph embeddings in addition to supervising the output of the autoregressive module (Sec. 3.3). These losses require likelihood maps that can directly be obtained from the dataset (Sec. 3.4). At test time sample the first keypoint randomly and execute the grasp using inverse kinematics (Sec. 3.5).
3.1 Object and End-effector Representations
To create the graphs, we generate point cloud representing each end-effector in canonical pose. As result, we have point cloud representations of both, the objects and the end-effectors. Those can directly be converted into graphs (described below). For the end-effector, we use the resulting graphs to manually select the canonical keypoints.
End-effector representation. We set the end-effector to canonical rest pose grea. We chose the rest pose to be vector with all joint angles in middle range of their respective joint limits, zero root
3

---

Figure 2: Object and end-effector inputs. Objects are initially represented as regularly sampled point clouds which are converted into graph for further processing. End-effector geometries are given as meshes and converted to coarser graphs by randomly sampling points from the mesh as an intermediate step. User-selected keypoints are highlighted in red.
translation, and identity root rotation. In this pose we convert the end-effector mesh into a point cloud by sampling Sa 1000 points from the mesh surface.
The vertices of the graphs are obtained by simply considering each point in the point cloud a vertex. Similarly to the end-effectors, the number of points per object is the same for each object, this work it is So 2048. The edges are meant to capture local surface geometry and are created by connecting each point to its K closest neighbours, a hyper-parameter that depends on point cloud density and object structure (we empirically chose K 8).
Graph Creation.
Canonical Contact Points. The canonical contact points k; E Va (i E (1...N)) on the surface of each end-effector selected visually. This needs only to be done once for each end-effector in a dataset ensuring that the selected keypoints have good coverage of each gripper with respect to its morphology and its grasping behavior. Please note that the order in which keypoints and might impact model performance. In our case, keypoint order was chosen to be as semantically consistent as much as possible across multiple embodiments (e.g. fingertips are ordered left to right, etc).
selected is
a
Dataset. For the purposes of this work, and used by them to train the CMap- VAE model their approach. The dataset is comprised end-effectors one 2-finger, two 3-finger, 4-niger, and one 5-finger. 58 common objects from YCB [36] and ContactDB [37] are used. The dataset finally contains 50,802 diverse grasps over the set of hands and objects, each represented by object name, end-effector name, and the end-effector pre-grasp pose in the form:
subset of the MultiDex dataset introduced by [12]
Grest (t, R, 0, - -1)
where 3 is the root translation, R is the root rotation in continuous 6D representation as introduced in [38], and fg...., ON are the joint angles.
3.2 Architecture
Ideally, we would like to compute the full joint distribution of vertex-to-vertex contact, i.e. P(c., Vy) for all object & end-effector vertices v. E Vo. Vy VG. This is typically intractable. We reduce the complexity by only focusing on N landmark keypoints kg...., ky on the end-effector. We, thus, try to approximate P(v.. Co. .... "o.....) CN) where c, [0, .N) are the vertices on the object where the k; keypoint makes contact, through learning a set of factorizations by applying the Bayes rule resulting
P(uo: Co. CN) II PM, (Vo, cslco...,
II PM (Vor G)(cz)),
where u, C: E Vo. (kg...,, C Va and PM. (V,, Ca cp.... .-..) are the factorized marginals to be learned in autoregressive manner.
We seek an architecture that can embed local geometry information well. From the architecture choices that demonstrate such properties, we chose Graph Neural Networks (GNN) [30] to create

---

Vo
to
Autoregressive Matching
M2
M1g
Mx
Na
Lo
gutber
Vo. Ro. Ra
V4
Vag
V4...
Va
VaN
RaN
Autoregrasaive Matching
(a) Full overview of GeoMatch.
(b) Autoregressive modules.
Figure 3: GeoMatch architecture. The object and gripper graphs are passed through the two encoders followed by linear layers. The gripper keypoint embeddings are gathered and are passed as input along with the object embeddings in the autoregressive modules.
object and end-effector embeddings. We, subsequently, gather the embeddings corresponding the keypoints on the end-effector resulting in
Do (9())
VG = Ea
Va,N Gather (Ve. (M:)
The gathered embeddings passed into autoregressive matching module jointly with the full object and end-effector embeddings. i.e.
Co...., CN AutoregressvveaatchingCCCC VG.N).
Geometry Processing. For our experiments, we used the Graph Convolutional Networks (GCN) implementation by Kipf et al. [30] with 3 hidden layers of size 256 and 512 output embedding dimension, one for objects and one for end-effectors. The subsequent linear projections for each encoder Lo and LG respectively, were of size 64 without bias.
Autoregressive Matching. The downprojected geometric embeddings passed into autore- gressive matching module, consisting of layers, M,. Each layer is responsible for predicting the index of the object vertex on where keypoint k, makes contact, given outputs of previous keypoints kn. - This is done by concatenating the full geometric embeddings with vertex-keypoin distances. These used as input vertex classifier that selects the resulting contact points G, on the object. Details are given in appendix A.l.
3.3 Losses
Our full training objective contains two components. One focusing on the geometric embeddings and another on the predicted contact points resulting in the overall loss
Liotal = a CP + 3 LPM
(3)
with each component having equal weight in our experiments (a 3=
Geometric Embedding Loss L The unnormalized likelihood map of object-keypoint con- tacts intuitively represents a score that given object vertex is in contact with a given gripper key- point and given by
(v, c2) = (v.)) (As))))A)))
(4)
This is optimized against the dot product the hand-specific object contact map Co ki) which will be explained further below, via binary cross-entropy loss
C M BCEx, (Pr, (Vor 43), Co k;)),
(5)
where X, is the positive weight used address the class imbalance (we A, 500).
Predicted Contact Loss L.M This loss models the joint distribution of contacts via set of factorizations PM,(vo cilCo...., 1-11 Vi [o, N) each of which are represented by the outputs of the autoregressive layers in our network. The outputs are optimized against the ground truth binary contact map label of the i-th gripper keypoint, contributing to second binary cross-entropy loss term
Lrme
E BCE,BEE (PM, (No: Ci -o, Co(vo,
6
5

---

where, similarly, A is the positive weight hyperparameter used to address the class imbalance (we use >,- 200). A visual representation of the autoregressive layers can be seen Fig. 3b. Note that for = 0, Pro (No, co) constitutes the first marginal for kn and thus: Pry (vo: co) = PMa (Vos Co).
3.4 Likelihood Maps
In order to learn the above, we assumed access to ground truth likelihood maps used for supervised learning which we obtain as follows. For each grasp in our dataset, instead of an object contact map. generate (2048, N) per-gripper-keyponnt proximity map of M nearest neighbors (NN) each of the contact vertices for the canonical keypoints:
{
1, vo (NN)
Proxo(Vor kt)
=
0,
otherwise.
lly z||H < M). We also generate a gripper contact map for the selected keypoints where the contacts are defined as the keypoints closer than given threshold, to the object point cloud:
where (N)V((k), = fv E Vo H E Vo (o(k)l <
1, 0,
For IIVo (k:))) < threshold,
Ck,)k) =
otherwise.
where Prozo (vor kg) is the object proximity map, and C3 (k:) is the gripper contact map. For this work, we empirically assumed M = 20 and threshold of 0.04, Finally, the hand-specific object contact map can be obtained as Colu., kz) (r, k2) (k()).
3.5 Grasp Execution at Inference
At test time, the independent unnormalized distribution for O, Pro (Voy Co). is leveraged to sample keypoint O which will commence the autoregressive inference. Details of this process are provided in Appendix A.2. Moreover, it should be noted that this autoregressive representation does present some limitations. More specifically, the ordering with which the keypoint contacts are being learned and ultimately selected could change the result. However, we refrain from experimenting with all possible combinations of keypoint ordering in the context of this work.
The end-effector joint angles then inferred by feeding the predicted contact points into an In- verse Kinematics (IK) solver. For our purposes, we used SciPy's Trust Region Reflective algo- rithm (TRF) [39]. The initial pose given to IK is a heuristic pose calculated by applying rota- tion/translation that aligns the palm with the closest object vertex while keeping all non-root joints their rest pose configuration.
4 Experiments
We evaluate our method through the lens of a number of research questions.
Q1: How successful is the model at producing stable and diverse grasps for various embodi- ments? We train our method with a training set containing samples of 5 end-effectors and 38 objects. We then generate grasps on each of the end-effectors and 10 new unseen objects. We evaluate these using the evaluation protocol introduced by [12] which considers 3 out of the 5 end-effectors and tests for grasp stability in Isaac Gym [40]. Similarly to [12], we apply consistent 0.5ms- accel- eration on the object from all zyz directions sequentially, for 1 second each. If the object moves more than 2em after every such application, the grasp is declared as failure. We also follow the same contact-aware refinement procedure, which applies force closure via a single step of Adam with step size 0.05. In addition, we provide calculated diversity as the standard deviation of the joint angles of all successful grasps, comparably to [12]. There is limited number of methods that tackle grasping across multiple embodiments. For our comparisons, for the recent methods, we chose GenDexGrasp [12], which assumes hand-agnostic contact maps, AdaGrasp (initOnly as it is the closest setup to our task) [14] which assumes table top grasping only and parameterizes grasps as pick location and z-axis rotation, and finally DFC [19] which is differentiable force closure synthesis method. In summary, we selected a set of methods that look at the cross-embent cros-eembodiment grasping problem through various different lenses.

---

Figure 4: Qualitative results. Generated grasps using GeoMatch on unseen objects with ezgripper, barrett, robotiq-3finger, allegro and shadowhand. For each grasp, another perspective is included where the GeoMatch predicted keypoints on each object are marked with purple and the gripper user selected keypoints matching these, are marked with yellow.
Results can be found in Tab. 1. In addition, provide number of qualitative results in Fig. 4. More qualitative results in the form of rendered grasps can be seen in Fig. 1.
Success (%)
Diversity (rad) 1
Method
ezgripper
barrett
shudowhand
Mean
ezgripper
barrett
shadowhand
DFC [19]
58.81
85.48
72.86
72.38
0,3095
0.3770
0.3472
AdaGrasp [14]
60.0
80.0
70.0
0.0003
0.0002
GenDexGrasp [12]
77.03
64.01
0.238
0.248
0.211
GeoMatch (Ours)
75.0
90.0
72.5
79.17
0.188
0.249
0.205
Table 1: Success and diversity comparisons. GeoMatch performs more consistently across end- effectors with a varied DoF number while maintaining diversity of grasp configurations. Success rates are provided per end-effector where the mean calculated across the end-effectors presented, for ease of review.
In our experiments, we observed that GeoMatch is performing slightly worse (-2%) on the 5-finger gripper Shadowhand than the best performing baseline, however performance for the 2-finger and 3-finger grippers increases by 5-30% compared other methods. Diversity remains competitive to other methods. Overall, the minimum performance observed for GeoMatch is significantly higher than baselines and the average performance multi-embodinents beats all baselines we compared against.
Q2: Is the multi-embodiment model performing better than a model trained on individual em- bodiments? We hypothesize that training our method on data containing variety of end-effectors will result in learning better geometry representations. To investigate this, we train our method on each single embodiment separately by filtering our dataset for each given end-effector. We then compare against the multi-embodiment model. Each of the single end-effector models is trained only on grasp instances of that gripper while the multi-embodimen effectors and objects the training set. The validation set cases contains 10 unseen objects. We provide results in Tab. 2. The model trained multi-embodiment data is performing 20 %-35%% % better than single end-effector models. This advocates the value multi-embodimentt grasping policies opposed to single model policies trained on more data.
model is trained on all 5 end-
Success (%)*
Diversity (rad) t
Method
ezgripper
barreu
shadowhand
ezgripper
barrett
shadowhand
Single embodiment
40.0
70.0
40.0
0.157
0.175
0.154
Multi embodiment
75.0
90.0
72.5
0.188
0.249
0.205
Table Comparisons between the Multi-emboodment model and models trained on individual grip- pers.
Q3: How robust is the learned model under relaxed assumptions? While our method demon- strates compelling results, it has been trained on full point clouds. Acknowledging that this is often
7

---

a strict assumption, especially when considering real-world environments, we evaluate robustness of the approach under conditions more similar real-world robotic data. We experiment with grasp generation using: a) noisy point clouds, b) partial point clouds, and c) partial point clouds including noise. For each of these, we perturbed the object point clouds accordingly, and collected grasps using our method zero-shot. Success rate was 77.5%, 66.7% %, and 67.5 % across end-effectors for each type of augmentation respectively. As demonstrated, our method shows reasonable robustness. Experiment details and breakdown of numbers can be found at Appendix A.
Q4: How important are various components of the design? Finally, we investigate the design decisions of our approach and how they affect performance. More specifically, we perform two ablations:
PointNet++ as the encoder of choice instead of GNN. We evaluate our choice towards GNN by swapping out the two GNN encoders with PointNet+++(2],. a popular encoder architecture for point clouds. Our results show that GNN was indeed a good choice as it performs better than the Point- Net++ ablation, by 10% averaging across end-effectors. In addition, we empirically observed 12x slow down when using PointNet++ due to the difference in the number of model parameters, which also makes GNN more light weight and fast. breakdown per end-effector can be found in Appendix A.
Non-shared weights between keypoint encoders. We hypothesize that shared encoder among all end-effectors is beneficial for learning features that represent local geometry and this subsequently, informs autoregressive prediction of keypoints. To validate this hypothesis, we conducted an abla- tion where we separated the end-effector encoder to 6 separate identical encoders, per keypoint. Our main model with shared weights across all end-effectors and keypoints outperforms the split encoders by 9%. Further analysis per end-effector can be found in Appendix A.
5 Limitations
Our method showcased that grasp learning can benefit from multi-embodiment data in terms of gen- eralization to new objects as well as robustness. Obtaining large amounts of such multi-mmbodiment grasping data, especially in real world setups can be challenging, time consuming and expensive. However, given that single embodiment grasping policy was shown to require more data to per- form comparably, we argue that spending resources on a multi-embodiment dataset to yield a policy that performs well across variety of grippers is better choice. On another note, on the notion of cross-embodiment and if a unified grasping policy is achievable, the point clouds used for this work were complete slightly noisy/partial models. This does not reflect the distribu- tion of noise encountered real world point clouds derived from depth camera data. Grasps being directed by small set of keypoints could be viewed another limitation. This design choice may indeed limit the areas of a gripper that make contact with an object or maybe prevent prediction of power grasps. This could perhaps be mitigated by choosing larger set of keypoints for better coverage of the end-effector surface. Lastly, our method relies on the robustness of the IK solution. We empirically observed cases where there was a reasonable grasp solution for set of predicted keypoints, however the chosen IK solution reached the maximum iteration steps and terminated in some suboptimal configuration.
focus was
6 Conclusion
This work presented novel tulti-embbdimment grasping method that leverages GNN to learn pow- erful geometry features for object and embodiment representation. Our approach demonstrates that a joint encoder trained on multiple embodiments can better embed geometry in generalizable fashion and ultimately result higher grasping success rate on unseen objects. The proposed framework also showcased robustness to more realistic point cloud inputs. Diversity of generated grasps re- mains competitive while producing such diverse grasps as simple as conditioning with a different high likelihood starting contact point for the first keypoint. Code and models will be released.
8

---

Acknowledgments
Authors would like thank Claas Voelcker for valuable feedback and discussion, well Silvia Sellán and Alec Jacobson for providing invaluable resources on Blender visualizations. Finally, we'd like to thank reviewers for constructive feedback that resulted in improving our work.
References
[1] A. Nguyen, D. Kanoulas, D. G. Caldwell, and N. G. Tsagarakis. Detecting Object Affordances with Convolutional Neural Networks. In International Conference on Intelligent Robots and Systems (IROS), 2016.
[2] H.-S. Fang. C. Wang, M. Gou, and C. Lu. GraspNet- IBillion: Large-Scart Seale- Benchmark for General Object Grasping. In Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[3] A. Mousavian, C. Eppner, and D. Fox. 6-DOF GraspNet: Variational Grasp Generation for Object Manipulation. International Conference on Computer Vision (ICCV), 2019.
[4] A. ten Pas, M. Gualtieri, K. Saenko, and R. Platr. Grasp Pose Detection in Point Clouds. In The International Journal of Robotics Research, 2017.
[5] M. Gualtieri, A. ten Pas, K. Saenko, and R. Platt. High Precision Grasp Pose Detection in Dense Clutter. International Conference on Intelligent Robots and Systems (IROS), 2016.
[6] Z. Jiang, Y. Zhu, M. Svetlik, K. Fang, and Y. Zhu. Synergies Between Affordance and Geom- etry: 6-DoF Grasp Detection via Implicit Representations. In Robotics: Science and Systems (RSS), 2021.
[7] S. Wang. Z. Zhou, and Kan. When Transformer Meets Robotic Grasping: Exploits Context for Efficient Grasp Detection. In Robotics and Automation Letters, 2022.
[8] C. Wu, J. Chen, Q. Cao, J. Zhang, Y. Tai, L. Sun, and K. Jia. Grasp Proposal Networks: An End-to-End Solution for Visual Learning of Robotic Grasps. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
[9] Y. Xu, W. Wan, J. Zhang, H. Liu, Z. Shan, H. Shen, R. Wang, H. Geng, Y. Weng, J. Chen, al. UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Con--Conditioned Policy. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[10] W. Wan, H. Geng, Y. Liu, Z. Shan, Y. Yang, L. Yi, and H. Wang. UniDexGrasp+4: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist- Specialist Learning. arXiv preprint arXiv:2304.00460 2023.
[11] D. Turpin, L. Wang, E. Heiden, Y.-. Chen, M. Macklin, S. Tsogkas, S. Dickinson, and A. Garg. Grasp'D: D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands. In Proceedings of the European Conference Computer Vision (ECCV), 2022.
[12] Li, T. Liu, Y. Li, Y. Geng, Y. Zhu, Y. Yang, and S. Huang. GenDexGrasp: Generalizable Dexterous Grasping. In International Conference Robotics and Automation (ICRA), 2023.
[13] L. Shao, F. Ferreira, M. Jorda, V. Nambiar, J. Luo, E. Solowjow, J. A. Ojea, O. Khatib, and
J. Bohg. UniGrasp: Learning Unified Model to Grasp with Multifingered Robotic Hands. In Robotics and Automation Letters, 2020.
[14] 7. Xu, B. Qi, S. Agrawal, and S. Song. AdaGrasp: Learning an Adaptive Gripper-Aware Grasping Policy. International Conference on Robotics and Automation (ICRA), 2021.
[15] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Grasp Generation Cluttered Scenes. In International Conference on Robotics and Automa- tion (ICRA), 2021.
Efficient 6-DoF

---

[16] S. Brahmbhatt, A. Handa, J. Hays, and D. Fox. ContactGrasp: Functional Multi-finger Grasp Synthesis from Contact. In International Conference on Intelligent Robots and Systems (IROS), 2019.
[17] J. Varley, J. Weisz, J. Weiss, and P. Allen. Generating Multi-Fingered Robotic Grasps via Deep Learning. In International Conference on Intelligent Robots and Systems (IROS), 2015.
[18] J. Lundell, E. Corona, T. N. Le, F. Verdoja, P. Weinzaepfel, G. Rogez, F. Moreno-Noguer, and V. Kyrki. Multi-FinGANN: Generative Coarse-To-Fine Sampling of Multi-Finger Grasps. In International Conference on Robotics and Automation (ICRA), 2021.
[19] T. Liu, Z. Liu, Z. Jiao, Y. Zhu, and S.-C. Zhu. Synthesizing Diverse and Physically Sta- ble Grasps with Arbitrary Hand Structures using Differentiable Force Closure Estimator. In Robotics and Automation Letters, 2021.
[20] D. Turpin, T. Zhong, S. Zhang, G. Zhu, E. Heiden, M. Macklin, S. Tsogkas, S. Dickinson, and A. Garg. Fast-Grasp' D: Dexterous Multi-finger Grasp Generation Through Diffcrentiable Simulation. In International Conference on Robotics and Automation (ICRA), 2023.
[21] K. Sohn, H. Lee, and X. Yan. Learning Structured Output Representation using Deep Condi- tional Generative Models. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
[22] C. R. Q. Yi, H. Su, and J. Guibas. PointNet++: Deep Hierarchical Feature Learn ing on Point Sets in a Metric Space. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[23] K. Li, Baron, X. Zhang, and N. Rojas. EfficientGrasp: A Unified Data-Efficient Learning to Grasp Method for Multi-fingered Robot Hands. In Robotics and Automation Letters, 2022.
[24] C. Pan, B. Okorn, H. Zhang, B. Eisner, and D. Held. TAX-Pose: Task-Specific Cross-Post Estimation for Robot Manipulation. In Conference on Robot Learning (CoRL), 2022.
[25] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The Graph Neural Network Model. In Transactions on Neural Networks, 2009.
[26] Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Liu, L. Wang, C. Li, and M. Sun. Graph Neural Networks: A Review of Methods and Applications. In A/ Open, 2020.
[27] I. Huang, Y. Narang, R. Bajesy, F. Ramos, T. Hermans, and D. Fox. DefGraspNets: Grasp Planning on 3D Fields with Graph Neural Nets. In International Conference on Robotics and Automation (ICRA), 2023.
[28) A. Garcia-Garcia, B. S. Zapata-Impata, S. Orts-Escolano, Gil, and J. Garcia-Rodriguezz. Tac- tileGCN: A Graph Convolutional Network for Predicting Grasp Stability with Tactile Sensors. In International Joint Conference on Neural Networks (IJCNN), 2019.
[29] X. Lou, Y. Yang, and C. Choi. Learning Object Relations with Graph Neural Networks for Target-Driven Grasping in Dense Clutter. In International Conference on Robotics and Au- tomation (ICRA). 2022.
[30] T. N. Kipf and M. Welling. Semi-Supervised Classification with Graph Convolutional Net- works. In International Conference on Learning Representations (ICLR), 2017.
[31] J. Varley, C. DeChant, A. Richardson, J. Ruales, and P. Allen. Shape Completion Enabled Robotic Grasping. In International Conference on Intelligent Robots and Systems (IROS), 2017.
[32] Lundell, F. Verdoja, and V. Kyrki. Robust Grasp Planning Over Uncertain Shape Comple- tions. In International Conference on Intelligent Robots and Systems (IROS), 2019.
[33] X. Yan, J. Hsu, M. Khansari, Y. Bai, A. Pathak, A. Gupta, J. Davidson, and H. Lee. Learning 6-DOF Grasping Interaction via Deep Geoaetry-Aware 3D Representations, In International Conference Robotics and Automation (ICRA), 2018.
10

---

[34] M. Van der Merwe, Q. Lu, B. Sundaralingam, M. Matak, and T. Hermans. Learning Contin- uous 3D Reconstructions for Geometrically Aware Grasping. In International Conference on Robotics and Automation (ICRA), 2020.
[35] J. Bohg and D. Kragic. Learning Grasping Points with Shape Context. In Robotics and Au- tonomous Systems, 2010.
[36] B. Calli, A. Singh, J. Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M. Dollar. Yale-CMU-Berkelle Dataset for Robotic Manipulation Research. In The International Journal of Robotics Research, 2017.
[37] S. Brahmbhatt, C. Ham, C. C. Kemp, and J. Hays. ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging. In Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[38] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the Continuity of Rotation Representations in Neural Networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[39] J. Moré and D. C. Sorensen, Computing a Trust Region Step. In SIAM Journal on Scientific and Statistical Computing, 1983.
[40] V. Makoviychuk, L. Wawrzyniak, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac Gym: High Performance GPU Based Physics Sim- ulation For Robot Learning- In Advances in Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS D&B), 2021.
[41] A. T. Miller and P. K. Allen, GraspIt! A Versatile Simulator for Robotic Grasping. In Robotics & Automation Magazine, 2004.

---

A Method Details
A.1 Autoregressive Matching Module Details
The downprojected geometric embeddings are passed autoregressive matching module, Mi. consisting of 5 layers, each responsible for predicting the index of the object vertex C₁ where key- point kn makes contact, given outputs of keypoints ko, ... hn 1- Each layer n concatenates the embedding of the 1--th keypoint of the end-effector along with the object embedding. Then, it cal- culates the relative distance map of each object vertex each of the 1 object vertices where the previous n 1 keypoints make contact. Note that is done via teacher forcing: instead of using the predictions of each π layer, we use the previous n ground truth contact points. This avoids error propagation during training. The relative distance maps are stacked and concatenated with the object and n-th keypoint embeddings. This constitutes the input to an MLP that predicts binary classification prediction over the object vertices that indicates the predicted n-th contact point Oz.
A.2 Keypoint Sampling at Inference Time
As noted above, the independent unnormalized distribution for k = 0 is leveraged to sample key- point 0 which will commence the autoregressive inference. We use the 0-th dimension as scoring mechanism for sampling high likelihood points where keypoint 0 makes contact. This is then passed into the model as the previous contact of keypoint 1. At inference time, at the n-th step, teacher fore- ing is substituted with passing in the (n - 1) predicted contact vertices. Finally, the end result is a tensor of 6 coordinates of the object graph. As previously mentioned, grasping is a multi-modal distribution and our model should be able to sample from the various modes. In our method, this can be achieved straightforwardly by sampling a variety of starting top-K points for keypoint O. The intuition behind this is that diverse, yet likely starting points for keypoint O will condition subse- quent predicted points differently, and ultimately yield different grasp modes. For our experiments, sampled 4 such top-K points, namely top-0, 20, 50, 100, in order to explore the capacity of our method to generate diverse grasps. more sophisticated sampling algorithm such as Beam search, could be applied here, however we empirically achieved sufficient diversity through multimodal sampling of keypoint 0.
B Additional Experimental Details
B.1 Robustness Experiments
To investigate the robustness of the learned representations, we conducted set of experiments where we tested out our trained model with noisy object point clouds, partial point clouds and partial point clouds with noise added as well:
Noisy point clouds. For this experiment, we processed the 10 object point clouds of our evaluation set by adding Gaussian noise with standard deviation 0.001 and clipping to a one standard deviation interval, to each of the points. Our evaluation process was then repeated with these as input zero- shot, i.e. grasps were generated and evaluated with the same process.
Partial point clouds. For this experiment, we emulated a table top scenario where objects placed on table would be missing the bottom of their surface. To achieve this, for each object point cloud, we defined a z-plane
Zmax
Zynén
Ethres
6
where Zyins Zmax are the minimum and maximum valul found in each object point cloud respec- tively. We then remove all points with 2 < >thhrss in order to emulate such a table effect. The resulting point clouds are again used zero-shot on our model to predict grasps.
Noisy partial point clouds. For this experiments, the table top emulating partial point clouds gen- erated for the previous experiment are augmented with Gaussian noise of standard deviation 0.001 and clipping to a one standard deviation interval. Grasping generation occurs again zero-shot on our model, and the evaluation process remains the same as all other experiments.
Comparative results for all 3 experiments against noiseless inputs can be viewed in Tab. 3.
12

---

Success (%)
Diversity (rad) f
Augmentation
ezgripper
barrett
shadowhand
ezgripper
barrett
shadowhand
noiseless
72.5
90.0
75.0
0.188
0.249
0.205
noisy
75
95.0
62.5
0.183
0.245
0.196
partial
67.5
67.5
65.0
0.181
0.207
0.197
noisy partial
65
75.0
62.5
0.143
0.227
0.212
Table 3: Comparisons between noiseless, noisy, partial, and noisy partial object point cloud inputs.
We observe that our model generally demonstrates robustness to noise with performance actually increasing in two out of three evaluated end-effectors, Partial point clouds cause the performance to drop as expected, however the model is still performing at good level multi-embodiments.
B.2 PointNet++ Ablation
Our choice of GCN as a geometry encoder is, of course, not the single architectural option avail- able for representing 3D geometry features, with PointNet++ [22] being popular choice in the literature. In this ablation, we investigate the efficacy of GCN in the multi-embodiment grasping setup compared to PointNet++ by replacing both GCN object and end-effector encoders with PointNet++ architecture".
Success (%)
Diversity (rad) f
Encoder
ezgripper
barrett
shadowhand
ezgripper
barrett
shadowhand
GCN [30]
75.0
90.0
72.5
0.188
0.249
0.205
PointNet++ [22)
75.0
70.0
65.0
0.154
0.223
0.151
Table 4: Comparison between GCN and PointNet++ encoder choices.
Results in Tab. 4 show that the GCN encoder variant outperforms the PointNet++ one for the 3- finger and 5-finger gripper while performs on par with for the 2-finger gripper. The GCN variant is also showing higher diversity of grasps for all 3 end-effectors.
B.3 Non-Shared Weights Ablation
For our main method, assumed shared weights between the representations used in the autore- gressive modules predicting each keypoint contact. However, of interest to investigate how performance gets impacted if each autoregressive module is free to influence geometry representa- tions for the keypoint it is responsible for. We thus, disentangled encoding weights for each of the autoregressive modules by passsing in a separate end-effector encoder each.
Success (%)
Diversity (rad)
Ablation
ezgripper
barrett
shadowhand
ezgripper
barrett
shadowhand
Shared weights
75.0
90.0
72.5
0.188
0.249
0.205
Non-shared weights
70.0
82.5
60.0
0.165
0.259
0.163
Table 5: Comparison between shared and non-shared weights of the end-effector encoder for au- toregressive learning.
The comparison is provided in Tab. 5 and indicate that training end-to-end with shared end-effector encoder for all keypoint predictions, is still significantly better performant choice. The shared weights variant performs 5 -1.5%% better among the 3 sample embodiments than the non-shared weights ablation.
We used the implementation from
13

---

B.4 Comparison to Grasplt!
Additionally to baselines presented in the main paper. we include comparison of our method to the simulated annealing planner in Grasplt! [41] for the Barrett end-effector. We produced grasps with the highest quality as per Graspit!'s quality metric, and further converted them to the format expected by the IsaacGymm evaluation protocol we used for our method. Results can be found in Tab. below.
Success (%)
Diversity (rad) 1
Method
barrett
barrett
Grasplt! [41]
89.99
0.00347
GeoMatch (ours)
90.0
0.24900
Table 6: Comparison of GeoMatch and Grasplt!'s simulated annealing planner for barrett.
B.5 Impact of the number of embodiments included at training
In order to better understand the impact of the number of embodiments seen by the unified grasp- ing policy and investigate potential diminishing returns, we trained variants of the policy with an increasing number of end-effectors and compared all on Shadowhand. The results presented in Tab. 7 suggest that indeed training on multi-embodiment data and adding more end-effectors during training increase performance significantly.
Training set contains
Success (%)
ezgripper
barrett
robotiq
allegro
shadowhand
40.0
47.5
55.0
55.0
72.5
Table 7: Comparison between increasing number of end-effectors seen at training time. Evalua-
tion is performed on an nn-distribution end-effector (Shadowhand) but on our unseen set of objects.
C Implementation Details
Implementation of all experiments was done using an Adam optimizer with learning rate of le-4 for 200 epochs. An assortment of GPU was used, namely RTX3090, V100, T4. Other hyperparameters used were provided in the main paper but for completeness, we include all hyperparameters here. The GNN used had hidden layers of size 256. The output feature size of the GNN encoder was 512. The two parts of the loss were weighed by 0.5 each while the two positive weights used for two BCE losses were 500 and 200 for the independent distributions and marginals respectively. The dataset used was the subset of MultiDex used by [12] to train the CMap-CVAB model of their approach, which contains 50,802 diverse grasping poses for hands and objects from YCB and ContactDB. The training set contained 38 objects and the validation set the remaining 10. The projection layer was a Linear layer without bias with an output dimension of 64 and each of the MLP autoregressive modules had 3 hidden layers of size 256.
For the IK, SciPy's 's TRF algorithm was used where each resulting set predicted keypoints was moved 5mm away from the surface of the object on the direction of the normal in order to form pre-grasp pose. The initial pose guess provided, was a heuristic calculated by orienting the palm of the gripper to align with the negative of the normal on the object surface at the closest surface point. For evaluation, grasps per object-gripper pair were sampled by selected the top-[0, 20, 50, 100] most likely keypoint O.
14

---

The Isaac Gym based evaluation scripts from [12] were used as is, aside from the one Adam step of force closure where the step size used was 0.05 in order make the force closure smoother and less abrupt.
15


Reviews:
[{"review_id": "r7ASCAq-FX", "reviewer": "Reviewer_vXSr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper studies multi-embodiment in dexterous grasping. It presents a method based on a Graph Neural Network (GNN) that estimates the contact likelihood for each of the keypoints in a high-degree-of-freedom end-effector. This facilitates the prediction of grasps across various end-effector models. Experimental evaluations show that the proposed method surpasses a number of baselines in terms of grasp success rate in simulations.", "summary_of_recommendation": "The paper offers a novel method for dexterous grasping that can generalize across different gripper geometries. I believe the paper contributes valuable insight to the field, thus I recommend accepting the paper. However, I believe it is essential to demonstrate the generalization ability of the proposed method to novel gripper geometry, which will make the paper much more convincing. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Fig. 3b is not referenced or explained in the paper. What does the minus sign in the figure represent?\n2. In Section 3.4, the authors state `More specifically, the ordering with which the keypoint contacts are being learning and ultimately selected could vastly change the result. However, we refrain from experimenting with all possible combinations of keypoint ordering in the scope of this work.` However, the method of `experimenting with all possible combinations of keypoint` is unclear. Do you train the model with all possible combinations? \n3. Following the previous point, it it is not clear how the sequence of the keypoints are decided. For example, will the finger keypoints be estimated before the palm, or vice versa? \n4. Why does Table 1 only show the result for three grippers instead of all five? Is the `Mean` averaged over five grippers?\n5. There are some typo in the paper. It would be ideal to do a proof-reading or run the paper through a grammar checker.\n    1. Line 126, `It is recommended to selected keypoints having …` -> ‘It is recommended to select keypoints having …’\n    2. Line 181, there is a comma after `Both`.\n    3. Line 231, `being learning` should be ‘being learned’.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. The paper addresses an essential issue in dexterous grasping. The suggested method is novel and demonstrates impressive experimental results.\n2. The concept of an autoregressive model to predict the contact of each keypoint sequentially is appealing.\n\nWeaknesses:\n1. While the paper emphasizes the need for transferability to new end-effector geometries, the experiments do not demonstrate the model's ability to generalize to new end-effectors.\n2. Although the paper is generally understandable, the method section seems a bit wordy.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "fFlM9L_eJ9x", "reviewer": "Reviewer_AGep", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose equipping robotic agents with a sense of geometry via a generalized geometry embedding. This embedding represents both objects and end-effectors, helping predict stable grasps. They propose using Graph Neural Networks (GNN) to encode meaningful geometry representations and use them to predict keypoint contacts on the object surface in an autoregressive manner. The authors' contributions include formulating robot grasp learning as a geometry matching problem, introducing a novel method named GeoMatch that learns expressive geometric embeddings, and comparing GeoMatch against baselines across multiple embodiments.", "summary_of_recommendation": "The reviewer believes that the proposed method has considerable potential for enhancing multi-embodiment grasping and even dexterous manipulation. However, the current state of the paper could be improved with a more coherent presentation in Section 3 and a more comprehensive experimental section.\n\n\n-----------------------------------\nPost-Rebuttal:\nI appreciate the authors' response to the comments. The clarity of the manuscript has improved, so I have raised my rating to a weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Would the proposed method generalize to unseen end-effectors if the training dataset is generated using a larger variety of grippers? \n- Line 243 mentions that five end-effectors were used in the evaluation. Is there a specific reason why Tables 1 and 2 only demonstrate the data for three end-effectors?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength:\n- The authors have provided a comprehensive literature review on grasping and graph neural networks, along with a discussion on these existing works. This paper describes an end-to-end learning framework that proposes grasp points for various end-effectors and the trained model generalizes to unseen objects. It presents simulated experiments that demonstrate how the proposed method improves the robustness of generated grasps against disturbances and enhances grasp diversity compared to three baseline methods: DFC, AdaGrasp, and GenDexGrasp.\n\nWeakness:\n- The presentation of Section 3 could benefit from improvements in clarity and readability. While this section provides abundant technical details, the connections and explanations between them could be made clearer. A possible improvement could be incorporating an algorithmic presentation, which would provide a step-by-step illustration of the processes discussed. Additionally, augmenting or supplementing Figure 3 with a larger, more detailed visualization could help in conveying the information more effectively. This would allow readers to better comprehend the intricate technical details and the connections among them.\n- Section 4 could use a more thorough discussion concerning the selection of the baseline methods. It's important to justify why these specific methods were chosen and how they technically compare to the proposed method. The authors should highlight the unique characteristics of each selected method and elucidate why they are relevant for a side-by-side evaluation. This could include discussing the similarities and differences in methodology, the principles they are based upon, or how they approach the problem differently. By providing such context, readers will gain a better understanding of the strengths and weaknesses of the proposed method in comparison to established techniques in the field.\n- While the term \"dexterous grasp\" is frequently mentioned throughout the article, the experimental evaluation focuses predominantly on a simpler grasping task involving resistance to disturbances. The learned geometry representations can be useful for most manipulation tasks. It would be considerably more compelling to see how the proposed method influences more complex tasks. Activities such as object reorientation or object transfer, which demand a higher degree of dexterity and precision from the robotic end-effectors, would provide a more rigorous testing ground for the method.\n- The current experimental results focus on evaluating the stability and diversity of grasps for various robotic grippers in a simulated environment. However, it would significantly enhance the relevance of this work to the community if the authors could demonstrate how effectively the learned model transfers to real robot grippers. Even a demonstration with just two different types of robot grippers would provide valuable insights into the practical utility of the proposed method. The experiment in Q3 (line 273) simulates only perception noise, neglecting many other physical and control uncertainties present in real-world robotic systems.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "V6wXExildY", "reviewer": "Reviewer_dMoq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper focuses on the problem of generalized grasp synthesis across different objects and gripper models. It addresses the research gap of having a single, general method for grasping that works with different grippers without requiring re-training for each specific gripper. The method utilizes geometry information from the object and gripper point clouds in GNN setting\tand predicts contact likelihood. This works along with a sequential, auto-regressive prediction for 6 keypoints corresponding to the gripper palm and a maximum of 5 fingers. Once the correspondence on the object points is established, the grasp is recovered using an existing I.K solver. Diverse grasps are generated by considering top-K likely points for initial contact and subsequent iterations. Experiments are conducted entirely in simulation and show promising performance w.r.t existing baselines.", "summary_of_recommendation": "The proposed work focused on the resolving the gap left by prior works (UniGrasp [13]) in terms of predicting contact points for high-DoF and > 3 finger grippers. The method is able to scale well to such configurations and shows promising results in simulation, although the real world experiments and more discussion on probable limitations/pitfalls could help the readers since some assumptions might make it hard to deploy to real world use case!  \n\n\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Does the proposed way to encode the gripper geometry also somehow handles the gripper kinematics (i.e how the joints will move). Is there any intuition as to how the model resolves this or adapts to this ambiguity just from the geometry + training data?\n\n- Some details about the runtime for the model, even if the numbers are from simulation. How many proposals actually pass the I.K stage on average?\n\n- How is the 6D gripper pose inferred from the predicted contacts? Are the contacts predicted in a pre-defined order (with some correspondence to gripper fingers?)", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper models the grasp prediction problem for multi-embodiment (gripper model) scenarios by focusing on the likely object contact points given the gripper model. In the discussion of related work, the authors bring out the current research gap, and how they seek to address some parts of it. I feel that the most close work to the overall idea is UniGrasp [13] as it also relies on a sequential point prediction conditioned upon previous predictions. The proposed method based on GNN of selected gripper points is able to scale well to high-DoF grippers and it looks to be a simple but effective design choice. The intuition for each part is brought out clearly and writing/figures seem clear from the context. \n\nI don't believe that this is exactly a solved problem (nor do the authors claim so) and hence readers would like some more insight in the limitations section about what parts of the entire pipeline are the bottleneck in terms of giving good results. Overall, this is a good extension of existing works tackling the problem and seems to be an effective method and modeling design to tackle highly different geometries of grippers.\n\n\n\nStrengths:\n\n- Clear motivation for the probelem and design choice for the modeling strategy. Object centric viewpoint is a natural way to keep the procedure somewhat gripper agnostic. Key related works are sufficiently covered.\n\n- The key limitations of prior works seem to be sufficiently handled in terms of scaling to high-DoF grippers and unseen objects. The gripper encoding scheme and use of GNN instead of PointNet++ helps to bridge prior limitations.\n\n- The authors perform adequete ablation studies for their design choices and the proposed method shows strong improvement when compared to existing related work. Results from simulation seem promising and such a method can perhaps future works tackling a generalized, multi-emboidment settings.\n\n\nWeakeness:\n\n- The proposed way to handle the different geometries does not really explicitly (or even implicitly?) encode the gripper kinematics. Although simply using the geometry information (via the canonical gripper point cloud) still shows good results, the contribution then is a bit limited as the gripper encoding is simply a somewhat manual process. \n\n- Lack of real world experiments and inference runtime discussion? On average how many grasp candidates have a valid I.K solution? So as whole, how does the proposed pipeline's runtime look like?\n\n- In RQ-2 about comparing to single embodiment focused models, the ideal comparison should be to methods which target a single gripper and whose output is a pose + joint angle vector since those are some strong baselines to compare against.\n\n- The experiment with partial point cloud assumes that only the bottom part of the object is not visible with the point cloud. But this is a strong assumption to make as in real world settings, due to the camera pose, major parts of the object can be missing. Its not a strong limitation as the paper does not claim to work with partial point clouds. The experiment results show a high performance with partial inputs which is not strictly correct in some sense. Maybe a clarifying remark in the main paper will help readers (instead of appendix).\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "BTqkNfAhjy0", "reviewer": "Reviewer_U6zR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses a timely problem in data-driven dexterous grasping: how to train a single grasp sampler that generalizes over gripper morphologies? To solve this problem, a graph neural network named GeoMatch is introduced to learn rich geometric representations for both objects and end-effectors. GeoMatch takes as input both the object and the gripper morphologies as graphs and outputs, autoregressively, the contact region on the object for one finger at a time. The final gripper configuration can then be extracted from the contact regions using an inverse kinematics algorithm. The experimental evaluation in simulation shows that GeoMatch outperforms three other learning-based methods that do not explicitly model both objects and end-effectors geometries jointly.", "summary_of_recommendation": "The proposed GeoMatch is a data-driven method to enable multi-embodiment grasping. The method is novel and non-incremental and the results are good with better grasp success rates than the baseline. The lack of comparison to GraspIt!'s simulated annealing planner and no real-world experimental evaluation is the main limitations. Nevertheless, this reviewer recommends a weak acceptance due to the novelty of the work, but if the paper would include a comparison to GraspIt! and some real-world experiments, then it would most probably be a strong acceptance. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- What is $M_i$ in Equation 2?\n- I have problems understanding the meaning of Equation 3 and the loss in Equation 4. I understood the loss in Equation 6, but I do not understand why you need the loss in Equation 4.\n- On lines 184-185, what is mean by: \"Each layer n concatenates the embedding of the n-th keypoint of the end-effector along with the object embedding\"\n- It would be interesting to see some results of Q2 in Section 4 but with models trained on 1, 2, 3, 4, and 5 end-effectors respectively. How much the success rates improve with the number of end-effectors could perhaps indicate if there are diminishing returns the number of grippers to be used. \n- Could you do the experiments but with point-clouds that remove self occluded regions? That would be the closest to a real-world experiment you could probably come. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Quality\n\nThe quality of this work is high. The authors provide a clear and detailed description of their method, including the mathematical formulation and implementation details. The simulation experiments are well-designed and the results are presented in a clear and concise manner. The authors also provide a thorough analysis of their results, including a comparison to existing methods and an ablation study to evaluate the contribution of different components of their method. Nevertheless, the authors should also compare their method to the simulated annealing planner in GraspIt! as that is still a relevant baseline for dexterous grasping. Also, some real-world experiments on two to three grippers would defiantly improve the quality of this work substantially.\n\n### Clarity\n\nThe paper is structured and oftentimes quite well written. Nevertheless, at times, the paper is hard to understand and requires multiple reads to comprehend fully. For instance, many long paragraphs make reading taxing, and quite some grammatical errors are still in the text.\n\nI have the following suggestions for clarity:\n- Consider splitting section 2 on dexterous grasping into more paragraphs. In its current state, that section is just a huge one-paragraph text which is very cognitively heavy to read.\n- Figure 1 is not referred to in the text.\n- Read through and fix some of the grammatical errors. A tool such as Grammarly would probably catch these grammatical errors. \n\n### Originality\n \nThe authors present a novel method for multi-embodiment grasping that uses Graph Neural Networks to learn rich geometric representations of objects and end-effectors. While there have been previous works on multi-embodiment grasping, the authors demonstrate that their method outperforms existing methods and provides competitive diversity of grasps. Using Graph Neural Networks to learn geometric representations is also a novel contribution. \n\nThe authors do miss some of the earlier works on dexterous grasping, such as [1,2] and on Geometry-Aware Grasping [3,4]. Please add these to your manuscript.\n\n### Significance\n\nThe work addresses the challenging problem of data-driven multi-embodiment grasping. While there have been previous works on multi-embodiment grasping, the authors demonstrate that their non-incremental method outperforms existing methods. The method is also sound when the complete model is known, which is quite a big assumption in real-world grasping. The authors do miss some of the earlier works on dexterous grasping, such as [1,2] and on Geometry-Aware Grasping [3,4]. Please add these to your manuscript.\n\n### Limitations\n\nThe limitation points out relevant shortcomings with the method. However, it completely ignores that the method is only evaluated in simulation and that nothing can really be said about real-world performance. Also, it should highlight that the method requires a complete object model or at least one that is close to complete.  \n\n### Strengths\n\n- Clear and detailed description of the method\n- Well-designed experiments with a thorough analysis of the results\n- Novel contribution of using Graph Neural Networks to learn joint object and gripper geometric representations\n- Outperforms existing methods and provides competitive diversity of grasps\n\n### Weaknesses\n\n- The method may not generalize well to real-world scenarios with more complex objects and end-effectors\n- The keypoints ensure that only those parts of the gipper are in contact with the object, which might prevent generating specific multi-contact grasps such as power grasps.\n- The method should also be compared against the non-learning-based simulated annealing method in GraspIt!\n- No real-world experiments. I understand that it is unlikely that the authors have all the grippers used in the simulation. Thus, for me, it would be enough just to use two to three grippers.\n\n[1] Varley, Jacob, et al. \"Generating multi-fingered robotic grasps via deep learning.\" _2015 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2015.\n\n[2] Lundell, Jens, et al. \"Multi-fingan: Generative coarse-to-fine sampling of multi-finger grasps.\" _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021.\n\n[3] Varley, Jacob, et al. \"Shape completion enabled robotic grasping.\" _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2017.\n\n[4] Lundell, Jens, Francesco Verdoja, and Ville Kyrki. \"Robust grasp planning over uncertain shape completions.\" _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors suggest enhancing robots' grasp capabilities using generalized geometry embeddings, which predict stable grasps by representing objects and end-effectors. They propose Graph Neural Networks (GNN) for encoding meaningful geometry representations to predict keypoint contacts on objects in an autoregressive manner. Their contributions involve framing grasp learning as a geometry matching task, introducing the GeoMatch method for learning rich geometric embeddings, and evaluating GeoMatch against baselines in various scenarios in Isaac Gym. **After rebuttal**: The authors have managed to demo on a real robot demo with Allegro and baselines on GraspIt!, so it's encouraged to add this demo to the final version.

**Strengths**:

- A comprehensive literature review on grasping and graph neural networks, along with a discussion on these existing works

- An end-to-end learning framework that proposes grasp points for various end-effectors and the trained model generalizes to unseen objects

- Adequate ablation studies for their design choices and the proposed method shows strong improvement when compared to existing related work

**Weaknesses**:

- The experiment with partial point cloud assumptions is not too realistic

- While the paper emphasizes the need for transferability to new end-effector geometries, the experiments do not demonstrate the model's ability to generalize to new end-effectors.

Post-rebuttal: Please revise the paper in line with the rebuttal discussion for the camera-ready submission.
Decision: Accept (Poster)