Grasp as You Say: Language-guided Dexterous Grasp Generation
Authors: Yi-Lin Wei
Jian-Jian Jiang
Chengyi Xing
Xiantuo Tan
Xiao-Ming Wu
Hao Li
Mark Cutkosky
Wei-Shi Zheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Dexterous Grasp Generation
Robotics
Primary area: robotics

Abstract: This paper explores a novel task "Dexterous Grasp as You Say'' (DexGYS), enabling robots to perform dexterous grasping based on human commands expressed in natural language. However, the development of this field is hindered by the lack of datasets with natural human guidance; thus, we propose a language-guided dexterous grasp dataset, named DexGYSNet, offering high-quality dexterous grasp annotations along with flexible and fine-grained human language guidance. Our dataset construction is cost-efficient, with the carefully-design hand-object interaction retargeting strategy, and the LLM-assisted language guidance annotation system. Equipped with this dataset, we introduce the DexGYSGrasp framework for generating dexterous grasps based on human language instructions, with the capability of producing grasps that are intent-aligned, high quality and diversity. To achieve this capability, our framework decomposes the complex learning process into two manageable progressive objectives and introduce two components to realize them. The first component learns the grasp distribution focusing on intention alignment and generation diversity. And the second component refines the grasp quality while maintaining intention consistency. Extensive experiments are conducted on DexGYSNet and real world environments for validation.

Grasp as You Say:
Ganguage-guideds Dexterous Grasp Generation
Yi-Lin Wei', Jian-Jian Jiang', Chengyi Xing',
Xian-Tuo Tan', Xiao-Ming Wu', Hao Li,, Mark Cutkosky?, Wei-Shi Zheng!.f
1 School of Computer Science and Engineering, Sun Yat-sen University, China
2 Stanford University, USA
Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China {weiylin5, jiangjj3s tanxt23, wuxm65) httlmmmaall12.... wys.l.sssss.... edu. cn {chengyix, 1i2053, cutkosky)@stannfork edu wazhengdieee. org
Abstract
This paper explores a novel task "Dexterous Grasp as You Say" (DexGYS), en- abling robots to perform dexterous grasping based on human commands expressed in natural language. However, the development of this field is hindered by the lack of datasets with natural human guidance; thus, we propose a language-guidee dexterous grasp dataset, named DexGYSNet, offering high-quality dexterous grasp annotations along with flexible and fine-grained human language guidance. Our dataset construction is cost-efficient, with the carefully-design hand-object inter- action retargeting strategy, and the L.L.-assisted language guidance annotation system. Equipped with this dataset, we introduce the DexGYSGrasp framework for generating dexterous grasps based on human language instructions, with the capability of producing grasps that are intent-an aatent-aligned, high quality and diversity. framework decomposes the complex learning pro-
To achieve this capability, cess into two manageable progressive objectives and introduce two components realize them. The first component learns the grasp distribution focusing intention alignment and generation diversity. And the second component refines the grasp quality while maintaining intention consistency. Extensive experiments are conducted on DexGY and real world environments for validation.
Introduction
Enabling robots perform dexterous grasping based on human language instructions is essential within the robotics and deep learning communities, offering promising applications industrial production and domestic collaboration scenarios.
With the advancements in data-driven deep learning and the availability of large-scale datasets, robot dexterous grasp methods achieve impressive performance 222 5 6 77. While previous approaches focus on the grasp stability, they have not fully utilized the potential of dexterous hands for intentional, human-like grasping. Recent studies, known as task-oriented 8 and functional dexterous grasping ( 10. aim to generate grasps based on specific tasks or functionality of objects. However, these approaches often depend on predefined, fixed and limited tasks or functions, restricting their flexibility and hindering natural human-robot interaction.
In this paper, we explore a novel task, "Dexterous Grasp as You Say" (DexGYS), as shown in Figure T We can see that natural human guidance is provided in this task, and can be utilized to
CCoresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1. Stable Grasp.
3. Our Language-uided Dexterous Grasp Generation.
To sse mag, graspit securely using your thumb. Index and middle finger on the handle.
Lift the power by grosping with all five Eingess arvund drill
Use a trigger sproyer gress the tigger with your loeefinger and hold the bottle body with other flagers.
To hold wineglass, use all five fingers support bowil securely.
Functiona Gresp.
To power place your forefinger on drill trigger and other tingars the drill
Hold the mag using Live fingess gresp the body securely.
Hold trgger spraye by wragping all five fingers around the bottle body.
lift up a wineglass, use your thuma, fondlinger, and middle finger hold the stem.
Use
Figure 1: Our Language-guidee Task vs. Traditional Dexterous Grasp Tasks. Traditional methods focus either solely on grasp quality or on fixed and limited functionalities. Our approach enables the generation of dexterous grasps based on human language, enhancing natural human-robot interactions.
drive dexterous grasping generation, thereby facilitating more user-friendly human-robot interactions. However, the new task also brings in new challenges. First, the high costs of annotating dexterous pose and the corresponding language guidance, present barrier for developing and scaling dexterous datasets. Second. the demands of generating dexterous grasps that ensure intention alignment, high quality and diversity, present considerable challenges to the model learning.
To address the first challenge, we propose a large-scale DexGYSNet. YSXG is constructed in cost-effective manner by exploiting human grasp behavior and the extensive capabilities of Large Language Models (LLM). Specially, we introduce the Hand-Objes Interaction Retargeting (HOIR) strategy to transfer easily-obtained human hand-object interactions to robotic dexterous hand, to maintain contact consistency and high-quality grasp posture. Subsequently, we develop the LLM-assisted Language Guidance Annotation system produce flexible and fine-grained language guidance for dexterous grasp data with the support of LLM. DexGYSNet dataset comprises 50,000 pairs of high-quality dexterous grasps and their corresponding language guidance, on 1,800 common household objects.
dexterous grasping dataset
With the support of the dataset, now our way overcome the second challenge. We pro- pose the DexG DSGaSp framework for dexterous grasp generation, which aligns with intentions, ensures high quality, and maintains diversity. At the beginning. we find the difficulty of mastering all objectives simultaneously results from the commonly used penetration loss [7] which used to avoid hand-object penetration. As shown in Figure 2 penetration loss substantially hinders the learning of grasp distribution, causing intention misalignment and reduced diversity. Conversely, despite the high diversity and aligned intention, the removal of penetration loss leads to unacceptable object penetra- tion, making the grasp infeasible. Based on this finding, we design our DexGY Srrssp framework in progressive strategy, decomposing the complex learning task into two sequential objectives managed by progressive components, Initially, the first component Iearns a grasp distribution, which focuses on intention consistency and diversity, optimizing effectively without the constraints of penetration loss. Subsequently, the second component refines the initial coarse grasps to high-quality ones with the same intentions and diversity. Our framework allows each component to focus on specific and manageable optimization objective, enhancing the overall performance of the generated grasps.
Extensive experiments are conducted the DSxe dataset and real-world scenarios. The results demonstrate that our methods are capable of generating intention-consisten,, high diversity and high quality grasp poses for wide range of objects.
2 Related work
2.1 Dexterous Grasp Generation
Dexterous hand endows robots with the capability to manipulate objects in a human-like man- ner. Previous methods have achieved impressive results in ensuring grasp stability by analytical
approaches 12 13 [5) and deep learning methods [ 16 17 [8]. However, the full po-
tential of dexterous hands for intentional and human-like grasping has not been completely exploited

---

use trigger spraver, place your forefinger trigger and wrap other fingers around the bottle's body
lift up mug, five fingers firmly grasp the body the mug.
(1) Pen.
(2) w/o Pen.
(3) (1) Pen.
(2) Pen.
(3)
(a) Comparison for Intention Consistency and Quality
Comparison Grasp Diversity Quality.
Figure 2: Visualization of the impact of penetration loss (Pen. in the figure) grasp performance: intention aligmment, quality, and diversity. (a) illustrates penetration loss causes intention misalign- ment and its absence results in severe object penetration. (b) shows three sampling results under the same conditions, and demonstrates that penetration loss leads to reduced diversity.
in these methods. Recently, some works have focused on functional dexterous grasping 8 8 10 19 aiming achieve human-like capabilities that extend beyond grasp stability alone, but still lack of flexibility and generalization. In this work, we explore novel task, Language-guided Dexterous Grasp Generation, which fully leverages the dexterity of robotic hands and enable robot execute dexterous grasp based on human natural language.
2.2 Grasp Datasets
The development of large-scale datasets has significantly improve the advancement of data-driven grasp methods, including parallel grasp (20
21 22 23 24], human grasp [25] 26 27 28 29 30),
and dexterous grasp approaches [1 2 14 [6]. Despite these advancements, the high cost of data
collection remains a significant challenge, particularly in the domain of dexterous hands, Previous
datasets for dexterous grasping primarily rely physical analysis approaches [12 31 mitigate this issue. However, these approaches often lack the specific semantic context or corresponding language guidance necessary for constructing our language-gui guided dexterous task. In this paper, present DexGYSNet dataset, with a cost-effective construction, providing high-quality dexterous grasp annotation along with flexible and fine-grained human language guidance.
2.3 Language-guided Robot Grasp
Language- guided robot grasp is important robotics. Previous works focusing on parallel grippers have made strides in achieving task-oriented grasping [32 23 33) language-guided grasping [34 35
39]. In contrast to parallel grippers, dexterous hand boast higher number of DOF (e.g., 28 for the Shadow Hand [40),, enabling broader dexterity. However, this high freedom also presents challenges for model learning. In this paper, we propose the DexGYSGrasp framework, capable of generating intention-aligee -aligned dexterous grasps with high-quality and diversity.
and manipulation [36
37
38
3 DexGYSNet Dataset
3.1 Dataset Overview
The DexGYSNet dataset is constructed with cost-effective strategy, as shown in Figure 3 We first collect object meshes and human grasps data from existing datasets [27]. Subsequently, we develop the Hand-Object Interaction Retargeting (HOIR) strategy to transform human grasps into dexterous grasps with high quality and hand-object interaction consistency. Finally, we implement an LLM--asisted Language Guidance Annotation system, which leverages the knowledge of Large Language Models (LLM)) to produce flexible and fine-grained annotations for language guidance.
3.2 Hand-Object Interaction Retargeting
Our Hand-Object Interaction Retargeting (HOIR) aims to transfer human hand-object interaction to dexterous hand-object interaction as shown in Figure The source MANO Al hand parameters denoted 9Th e R81, And the target dexterous hand parameters are denoted as Gdez (r, 4).

---

Initalize
fingertip align
refine
(a) Hand-Objed Interaction
Interaattiinnnnaiiiinnnn Retargeting
Brie guidance
System: Please generate natural instruction
Use lotion pump.
guiding rubot grasp based on the given information.
Contact Information:
The guidance and information as following.
User:
Index finger touches the pump head, Thumb, middle finger, ring finger, and little finger touch the bottle body.
pump, place your forefinger the pump head and rest other fingurs on the bottle body.
(b) LLM LLs-sssisted Languagg Guidance A Anootation
The construction process of the DexGYSNet dataset. (a) The HOIR strategy retargets the human hand to the dexterous hand by three step, maintaining hand-object interaction consistency and avoiding physical infeasibility (shown in black circle). (b) The annotation system automatically annotates language guidance for hand-object pairs with the help of LLM.
Figure 3:
where (3) represents the global rotation, E IR3 is the translation in world coordinates, and Q E R is the joint angles for a J-DoF dexterous hand, for example J 22 for Shadow Hand
Three steps are within the HOIR: pose initialization, fingertip alignment, and interaction refinement. In the first step, the dexterous poses are initialized by copying parameters from similar structures of human poses to establish better initial values. In the second step, the dexterous poses are optimized in the parameter space to align the fingertip positions Pk dea. with those of the human pk me This achieves retargeting consistency, and the optimization objective can be formulated as follows:
min gdex=(r.t.0)
Σ llrk
dex,ft
Pk
mano.
To improve the physical interaction feasibility while maintaining the consistency, the dexterous hand poses are further optimized in the third step by hand-object interaction and physical constraints losses [14]. Two key points designed to maintain the consistency: preserving the contact area of the optimized pose consistent with the output from the second step, and keeping the translation fixed during this step. The optimization objective can be formulated as follows:
min Spen + Copen Njont Nemap (r.a)
Here, the object penetration loss Eyen penalizes the depth of hand-object penetration. The self- penetration loss Lepen penalizes the self self-penetration. The joint angle loss Ljoint penalizes the out-of-limit joint angles. The contact map loss Lemap ensures the contact map on the object remains consistent with the output from the second stage. The details of losses can be found in Appendix(A.1.
3.3 L.M--assisted Language Guidance Annotation
To annotate flexible and fine-grained language guidance for dexterous hand-object pairs with low-cost, we design a coarse-to- tone automated language guidance annotation system with the assistance of the LLM, inspired by [42) [29], as shown in Figure 3 Specially, we initially generate brief guidance based on the object category and the brief human intention (c.g.. "using lotion pump"), which are collected by the human dataset [27] Subsequently, we compile the contact information for each finger by calculating the distances from the contact anchors on the hand to different parts of the object. We then organize the contact information into language descriptors (e.g. "forefinger touches pump head and other fingers touch the bottle body."), Finally, we input both brief guidance and the detailed contact information into the GPT3.5 to produce natural annotated guidance (e.g. "To use a lotion pump, press down on the pump head with your forefinger while holding the bottle with your other fingers.") "). More details about DexGYSNet construction can be found in Appendix A.2

---

Intention
CD
Diversity
Object Panebration
Pen
0.6
0.4
2
0.2
2
Apen
Apes
Apes
Apcn
Ours
Apex 0
Apex
Apen
Apm
Ouus
Apen 0
Apon
Apen
pm
OurS
50
100
100
50
= 100
100
50
=
100
100
Figure 4: Quantitative experimental results with different object penetration loss weights Xpen- Intention is quantified by the Chamfer distance (CD) between predictions and targets. Diversity is assessed by the standard deviation of hand translation 6G. Object penetration is evaluated by the penetration depth (Pen.) from the object point cloud to the hand mesh. Our method uniquely achieves high performance in terms of intention consistency, diversity, and penetration avoidance.
4 DexGYSGrasp framework
Given full object point clouds O and language guidance L as inputs, goal is generate dexterous grasps Gder with intention aligmment, high diversity and high quality.
4.1 Progressive Grasp Objectives.
Learning Challenge in DexGYS. The DexGYS places high demands on intention alignment (e.g.. accurately pressing your forefinger on trigger to use the sprayer), high diversity (e.g.. holding the bottle using various postures), and high quality (e.g., ensuring stable grasp and avoiding object penetration). However, we find that a single model struggles to meet these requirements simultaneously, due to the optimization challenge caused by the commonly used object penetration loss 45 14 16),
which is used to prevent hand-object penetration. As shown in Figure and Figure increasing the weight the penetration loss reduces object penetration but adversely affects intention alignment and generation diversity.
Grasp Objectives. To address these challenges, we propose decompose the complex learning objective into two more manageable objectives. The first objective is generative: it focuses on learning the grasp distribution, which does not prioritize quality but focuses on learning the grasp distribution with intention alignment and generation diversity. The second objective is regressive: it aims refine the coarse grasp to a specific high-quality grasp with same intention. By decomposing the complex objectives, we reduce the learning difficulty of the generative objective it does not concentrate on quality and avoids using penetration loss which could interfere the learning process. Additionally, the learning of regression is less complex than distributions, as it merely requires adjusting the pose to a specific target within small space. Hence, we can employ penetration loss to ensure that the refined dexterous hand avoids penetrating the object and with high quality.
Progressive
4.2 Progressive Grasp Components
Benefiting from our progressive grasp objectives in Section 4.1 progressive grasp components, which can achieve intention alignment, high diversity and high quality language-guided dexterous generation.
we design the following two simple
Intention and Diversity Grasp Component. We introduce intention and diversity grasp component to learn a grasp distribution efficiently, achieve intention aligned and diverse generation. Due to the distribution modeling objective, IDGC is build upon the conditional diffusion model [44 4 to predict the dexterous pose Giezz from noised Gdez The input object point clouds O encoded by Pointnet++ [45] and language L is encoded by pretrained CLIP model [46] as the condition. And
we employ DDPM 47 as sampling process, which can be formalized by the following equation:
Pa (gdezt 10, = =p (gden) II (gter gdez ,0,C).
5

---

(c) Prognessive Less Striegy
To usea tnager grighe place your Ranfinger on Dwitige other Eagars bodly
Regresion
Add Neiie
Dee Teee
objeett
Trensfemmer Decoder
object:
Point
Encoder
Enceder
[al Entention and Divenily Crasp Compununt
(b) Quality Graup Component
Stop gradient
Model forward process
Penetratonn barkwward
Rog Rogression
Figure 5: Overview of our framework. (a) With only the regression loss, intention and diversity grasp component is trained to reconstruct the original hand pose from the noise poses, based on language and object condition. (b) With both regression and penetration losses, Quality Grasp Component is trained to refine the coarse pose improve the grasp quality while maintain intension consistency.
Quality Grasp Component. The generated grasps of the first component possess well-aligned intentions and high diversity, but suffer from poor grasp quality due to significant object penetration. Therefore, we introduce Quality Grasp Component refine the grasp quality while maintaining intention consistency in regressive manner. Specially, it takes the coarse pose Òdex, coarse hand point clouds H(Gdde)) and object point clouds O as input, and outputs the pose Agder, The refined grasp is obtained by Gder Gder AGder The training pairs of this component are constructed by collecting coarse grasps generated by the first component alongside the most similar ground-trute grasps that share the similar intentions. This ensures the training targets are aligned with the language intention, thereby guaranteeing that the refined grasps maintain consistency with the intended actions.
4.3 Progressive Grasp Loss
Grasp Loss. We strategically employ regression losses and exclude object penetration loss to enhance the training efficacy of intention and diversity grasp component. By fo- cusing exclusively on the regression learning, this component facilitates more effective optimization process, achieving enhancements of intention consistency and grasp diversity. Concretely, we utilize L2 loss for pose parameter regression and incorporate the hand chamfer loss [48 assist by explicit hand shape. The loss function of intention and diversity grasp component.is defined as:
Intention and Diversity
LiDG = Spara (gAez _dex)) + X2 fer for (H(G) H(Gdez)),
where are dexterous hand point clouds of corresponding pose.
Quality Grasp Loss. Benefiting from the simplified training objectives, the quality grasp component focuses solely on refining coarse grasp to a specific target within relatively constrained space, thereby reducing the negative impact of object penetration. Therefore, we employ the well-designed loss including object penetration. The loss function of quality grasp component can be formulated as:
LQg Spar fer
X Epen + XeemapFemap Xxpen Expen-
More details about loss function and model structure can be found in Appendix A.I
5 Experiments
5.1 Datasets and Evaluation Metrics
We split the DexDYSNet dataset at the object instance level, using 80% of the objects within each category for training and 20% for evaluation. Notably, none of the objects in the test set appear in the training set, ensuring that all experimental results evaluated on unseen objects.
Three types of metrics are employed for evaluation from the perspective of intention consistency, grasp quality and grasp diversity. 1) For intention consistency, we employ Fréchet Inception Distance (FID), using sampling point cloud features extracted from [49 to calculate P-FTD and rendering image features extracted from (50) to calculate FID. Additionally, Chamfer distance

---

Intention P-FID
Quality
Diversity
Method
FID
CD
Con. 4
Success t
Q1 f
Pen. 4
8, t
6r 7
8,
GraspCVAH 51
31.26
29.02
3.138
0.096
29.12%
0.054
0.551
0.179
1.762
0.179
GraspTTA 43
35.41
33.15
12.19
0.111
43.46%
0.071
0.188
2.111
6.150
3.869
SceneDiffuser 4
20.44
7.932
1.679
0.045
62.24%
0.083
0.253
0.346
3.455
0.387
D
23.31
15.77
2.895
0.078
51.91%
0.078
0.163
2.037
14.01
4.299
Ours
6.538
5.595
1.198
0.036
63.31 %
0.083
0.223
6.118
55.68
6.118
Table Results on DexGYSNet compared with the SOTA methods.
ladd Lilinoe:
180 pez.
octini the pane with your Blazk Sandiagec (ala) Geguo sappor.
To tee a pitow, zni the pearlingens
To grag kitnng ser caeyor Prgrringentt hole siza accurty.
Hirat overthe biee uskag tagor leas.
Baxlove le pepwee groptng arrend
your Grapen
To asea tugper sprgee. pour tometagee lagaor bollottee body.
To bhoociass ploceal al tagers nt
To soe a lebon penp pann pourtordagero on da anp botda bodi.
To hold lobon peet. Erages la grip searely.
To BR scoweddaer gig lawilingess
Tespuso
reocnd the walletiieee bereyy
Figure 6: Visualization of generated dexterous grasp. The top visualizes one sample for each object and guidance pair. The bottom visualizes four samples, the bottom left shows that the generated grasp are consistent with clear and specific guidance, while the bottom right shows that the diversity achieved under relatively ambiguous instructions.
(CD), used to measure the distance between predicted hand point clouds and targets; Contact distance (Con.) is used to measure the L2 distance of object contact map between the prediction and targets. 2) For grasp quality, Success rate in Issac gym and Q1 33 measure grasp stability. We set the contact threshold to cm and set the penetration threshold to 5 mm following [14]. Maximal penetration depth (cm), denoted as Pen., reflects the maximal penetration depth from the object point cloud to hand meshes. 3) For diversity, we employ the Standard deviation of translation &
rotation 6, and joint angle 8, of eight samples within same condition, following 77. More details can be found in Appendix A.3.2
5.2 Implementation Details
For the construction of DexGYSNet, the step 2 and 3 are optimized for 20 and 300 iterations with learning rates of 0.01 and 0.0001 respectively. We set Apen = 100 and set Aspen. joine- framework, the training epochs set to 100 for intention and diversity grasp component and 20 for Quality Grasp Component. The loss weights are configured as follows: Xpara = X 10, Xehamfer
Xemap, each 10. For training
Xchammfer 1, Xamap 10. Xpm. 100, A2pee 10. Throughout all training processes, the model is optimized with batch size of 64 using the Adam optimizer, with a weight decay rate of 5.0 x 10-6 The initial learning rate is 2.0 x and decay to 2.0 x 10 using cosine learning rate [52] scheduler. All experiment implemented with
PyTorch on a single RTX 4090 GPU.
5.3 Comparison with SOTA methods
The comparison results are presented in Table I We reproduce the SOTA methods to suit our task by concatenating the language condition with the point cloud features, the details can be found
in Appendix A.3.3 As seen in the Table, our framework significantly outperforms all previous
7

---

Intention
Quality
Diversity
CD
Con.
Q1 f
Pen. 4
8, f
o, 1
6g 1
IDGC (gnen = 0)
1.276
0.028
0.024
0.534
5.710
54.75
7.741
IDGC (A 50)
2.980
0.061
0.074
0,271
2,421
33.27
3.391
IDGC (A 2
100)
4.009
0.067
0.072
0.175
2.701
38.27
3.785
IDGC (Apen 500)
4.185
0.072
0.107
0.037
0.547
8.807
0,481
IDGC (A gen
100)
3.181
0.056
0.093
0.302
1.341
16.53
2.211
IDGC (Apen 0) + TTA
20.09
0.102
0.057
0.178
4.849
51.91
8,479
IDGC (Apen = 100) + QGC
2.009
0.042
0.099
0.143
3,414
40.35
2.844
IDGC (22
QGC
1.198
0.036
0.083
0.223
6.118
55.68
6.118
Table 2: Ablation study for our framework. Intention and diversity grasp component is abbreviated as IDGC, Quality Grasp Component is abbreviated as QGC. Xgen the penetration loss weight the training of IDGC. Ours is colored in gray.
Intention Con. +
Quality
Intention
Quality Q1
Diversity & t
step1
step2
step3
Q f
Pen 4
CD 4 Con.
Pen 4
0.048
0.037
0,572
GraspCVAE w/o Lpen
3.138
0.096
0.054
0.551
0.179
0.101
0.833
0,516
2.638
0.090
0.005
0.921
0.266
0.012
0.029
0.477
QGC
2.425
0.056
0.074
0.261
0.315
0.015
0.063
0.369
SceneDiffuser
1.679
0.045
0.083
0.225
0.346
all in stage
0.075
0.090
0,271
w/o Eyre
1.511
0,041
0.019
0.488
6.323
w/o fix translation
0.051
0.074
0.332
QGC
1.495
0.040
0.082
0.241
6.015
Table 3: Ablation study for HOIR.
Table 4: Plug-and-plas Experiments.
methods in terms of intention consistency and grasp diversity, while also achieving comparable performance grasp quality. Previous methods struggle with learning robust language conditional grasp distribution due to the optimization challenges outlined in Section 4.1 They often yield misaligned yet high quality grasps, resulting in comparable grasp quality, but less aligned intention and limited diversity compared to our framework. Overall, these results confirm that our framework achieves SOTA performance in generating iieettio-allgned, high-quality and diverse grasps.
In Figure 6 we visualize the generated grasp to qualitatively demonstrate the grasp generation capabilities of our framework. The bottom figure visualizes the results of four samples, the bottom left highlights our framework's 's ability to produce precise and consistent grasps under deterministic guidance (e.g., the way use trigger sprayer deterministic). In the other hand, the bottom right illustrates our framework's diversity in generating grasps when provided with ambiguous guidance (e.g., the way to hold a bottle is diverse).
5.4 Necessity of Progressive Components and Losses
The results presented Table validate the core insight of our framework: decomposing the complex task into progressive objectives, employing progressive components, and learning with progressive losses. The initial four lines of results demonstrate that single component, without progressive objectives, fails balance all objectives. Moreover, single component, progressive objectives, that adjusts Aen from to 100 after several training epochs, does not enhance performance. The similar result occurs when using progressive components without corresponding progressive losses, DDG( pen pen = 100) + QGC. Moreover, the commonly used quality refinement strategy test-time adaptation (TTA) 43). though improves grasp quality but results in extremely poor intention consistency. Overall, only the progressive designs of our DexGYSGrasp framework ensures excellence in intention alignment, high quality and diversity.
with
5.5 Plug-and-play Experiments
We conducted experiments to evaluate the applicability of our insights to other state-of-the-art (SOTA) methods. Specifically, we trained GraspCAVE and SceneDiffuser without the object penetration

---

power peess dril leiguer wilh your fondinger and unip it wiith your other flegers
To hand over the hammer, with ali fingers the hammer
hold
fingers
To use hammer, gropp it with all Enngers secure the hammer grip.
grip the drill securdl..
To use trgger
the
mager fondinger and hold your otler frngess.
waneglass, Eingres to grip the bowl securely.
Hold the ingger spray with
your fingers conract
Use fiogers geop the stem the wizeglass securely.
with the bottle budl.
Hard over game controllee by
Hold thee frying pan with two fanpers supporting the
graeping it with fingers on the
cnizzller pazel and lianiles.
a ganee controller, grip fingers touching contmiler panel.
Grasp the frying pan by contering the landle with all Etngees.
Figure 7: Visualization of real world experiments.
constraint and trained the quality grasp component (QGC) to refine the coarse outcomes. As depicted in Table 4 removing the object penetration loss leads to improved intention consistency, which corroborates our findings discussed in Section 4. Moreover, our quality grasp component can significantly enhance grasp quality while maintaining the intention consistency.
5.6 Effectiveness of Hand-Object Interaction Retargeting
We conducted ablation studies to evaluate our Hand-Object Interaction Retargeting (HOIR) strategy in constructing DexGYSNet dataset. As shown in Table our three-step HOIR significantly improves both the quality and the intention consistency progressively. We observed that optimizing all losses in Equations and 2 in one step (all in one stage), results in worse contact consistency and better grasp quality. Similar outcomes occur when the root translation is not fixed in step 3 (w/o fix translation). We believe this trade-off arises from inherent noise in the hand-object interaction data and the structural differences between human grasps and dexterous hands, making it challenging to excel in all aspects. Overall, think that three-step HOIR strategy achieves more comprehensive outcomes, especially in the most important aspect of hand object contact consistency.
5.7 Experiments in Real World
We conducted real-world grasp experiments to verify the practical application of our methods, as shown in Figure 7 The experiments conducted on Allegro hand, a Flexiv Rizon
and an Intel Realsense D415 camera. Although our framework is designed for full object point clouds, we integrate several te--theshell methods to enhance its practicality. Specifically, partial object point clouds are obtained through visual grounding [53) and SAM [54], which are then fed into a point cloud completion network (55) obtaín full point clouds. execution, we first move the arm to the 6-DOF pose of the dexterous hand root node, and then control the dexterous hand joint angles to the predicted poses. Real world experiments further validate the effectiveness of our method. More implementation details can be found in Appendix A.5
6 Conclusions
We believe that enabling robots to perform high quality dexterous grasps aligned with human language is crucial within the deep learning and robotics communities. In this paper, we explore this novel task, "Dexterous Grasp You Say" (DexGYS). This task is non-trival, we propose DexGYSNet dataset and DexGYSGrasp framework to accomplish it. DexGYSNet dataset is constructed cost-effectively using the object-hand interaction retargeting strategy and the language guidance annotation system assisted by LLMs. Building on DexGY DexGYSSrrasp framework, comprised of two progressive components, which can achieve intention-aligned, high diversity, and high quality dexterous grasp generation. Extensive experiments in DexGYSNer and real-world settings demonstrate that our framework significantly outperforms SOTA methods, confirming the potential and effectiveness of our approach.

---

Acknowledgements
This work was supported partially by NSFC(92470202, U21A20471), Guangdong NSF Project (No. 2023B15 15 1514400255) Additionally, sincerely thank the help of Guo-Hao Xu and Dian Zheng for the valuable suggestions for the paper.
References
[1] Min Liu, Zherong Pan, Kai Xu, Kanishka Ganguly, and Dinesh Manocha. Deep differentiable grasp planner for high-dof grippers. arXiv preprint arXiv:2002.01530 2020.
[2] Puhao Li, Tengyu Liu, Yuyang Li, Yiran Geng, Yixin Zhu, Yaodong Yang, and Siyuan Huang. Gendexgrasp: Generalizable dexterous grasping. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023.
[3] Lin Shao, Fabio Ferreira, Mikael Jorda, Varun Nambiar, Jianlan Luo, Eugen Solowjow, Juan Aparicio Ojea, Oussama Khatib, and Jeannette Bohg. Unigrasp: Learning a unified model to grasp with multifingered robotic hands. IEEE Robotics and Automation Letters, 2020.
[4] Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang. and Song-Chun Zhu. Diflusion-based generation, optimization, and planning in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[5] Jiaxin Lu, Hao Kang, Haoxiang Li, Bo Liu, Yiding Yang. Qixing Huang, and Gang Hua. Ugg: Unified generative grasping. arXiv preprint arXiv:2311. 166917, 2023.
[6] Zehang Weng, Haofei Lu, Danica Kragie, and Jens Lundell. Dexdiffuser: Generating dexterous grasps with diffusion models. arXiv preprint 2402:0:4000029899 2024.
[7] Guo Hao Xu, Yi-Lin Wei. Dian Zheng, Xiao-Ming Wa, and Wei-Shi Zheng. Dexterous grasp transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024.
[8] Jiayi Chen, Yuxing Chen, Jialiang Zhang, and He Wang. Task-oriented dexterous grasp synthesis via diffcrentiable grasp wrench boundary estimator. arXiv preprint 2rXiv:2309. 2023.
[9] Tianqiang Zhu, Rina Wu, Jinglue Hang. Xiangbo Lin, and Yi Sun. Toward human-like grasp: Functional grasp by dexterous robotic hand via object- object-hand hand semantic representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[10] Wei Wei, Peng Wang, and Sizhe Wang. Generalized anthropomorphic functional grasping with minimal demonstrations. arXiv preprint arXiv:2303. /7808, 2023.
[11] Jean Ponce. Steve Sullivan, J-D Boissonnat, and J-P Merlet. On characterizing and computing three- and four-finger force-closure grasps of polyhedral objects. In [1993] Proceedings IEEE International Conference on Robotics and Automation, pages 821 827. IEEE, 1993.
[12] Tengyu Liu, Zeyu Liu, Ziyuan Jiao, Yixin Zhu, and Song-Chun Zhu. Synthesizing diverse and physically stable grasps with arbitrary hand structures using differentiable force closure estimator. IEEE Robotics and Automation Letters, 7(1):470-47) 2021.
[13] Carlo Ferrari, Canny, et al. Planning optimal grasps. In Proceedings, 1992 IEEE International Conference on Robotics and Automation, 1992., 1992.
[14] Ruicheng Wang, Jialiang Zhang, Jiayi Chen, Yinzhen Xu, Puhao Li, Tengyu Liu, and He Wang. Dexgrasp- net: A large-scale robotic dexterous grasp dataset for general objects based on simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023.
[15] Dylan Turpin, Liquan Wang. Eric Heiden, Yun-Chun Chen, Miles Macklin, Stavros Tsogkas, Sven Dickinson, and Animesh Garg. Grasp'd: d: Differentiable contact-rich grasp synthesis for multi-fingered hands. In European Conference on Computer Vision, 2022.
[16] Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[17] Kelin Li. Nicholas Baron, Xian Zhang, and Nicolas Rojas. Efficientgrasp: A unified data-efficient learning to grasp method for multi-fingente robot hands. IEEE Robotics and Automation Letters, 2022.
10

---

[18] Jacob Varley, Jonathan Weisz, Jared Weiss, and Peter Allen. Generating multi- multi-fingered robotic grasps via deep learning. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015.
[19] Tianqiang Zhu, Rina Wu, Xiangbo Lin, and Yi Sun. Toward human-like grasp: Dexterous grasping via semantic representation of object-hand. In Proceedings of the IEEF/CVP International Conference on Computer Vision, pages 15741-15751, 2021.
[20] Hao-Shu Fang. Chenxi Wang. Minghao Gou, and Cewu Lu, Graspnct-bbiliion: Ibillion: large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11444-1145, 2020.
[21] Clemens Eppner, Arsalan Mousavian, and Dieter Fox. Acronym: A large-scale grasp dataset based on simulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6222-62277 JEEE, 2021.
[22] Xiao-Ming Wu, Jiafeng Cai, Jian-Jian Jiang, Dian Zheng, Yi-Lin Wei, and Wei-Shi Zheng. An economic framework for 6-dof grasp detection. In European Conference on Computer Vision, 2024.
[23] Chao Tang, Dehao Huang, Lingxiao Meng, Weiyu Liu, and Hong Zhang. Task-oriented grasp prediction with visual-language inputs. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4881-4888. IEEE, 2023.
[24] Jia-Feng Cai, Zibo Chen, Xiao-Ming Wu, Jian-Jian Jiang, Yi-Lin Wei, and Wei-Shi Shi-Shi Zheng. Real-to-sim grasp: Rethinking the gap between simulation and real world in grasp detection. In Conference on Robot Learning, 2024.
[25] Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11807-1188 11816, 2019.
[26] Yu-Wei Chao, Wei Yang. Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar lqbal, Stan Birchfield, et al. Dexycb; benchmark for capturing hand grasping of objects. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9044-9053, 2021.
[27] Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu La. Oakink: A large- scale knowledge repository for understanding hand-object interaction. In Proceedings of the IEFF/CVF Conference Computer Vision and Patlern Recognition, pages 20953-20962. 2022.
[28] Juntao Jian, Xiuping Liu, Manyi Li, Ruizhen Hu, and Jian Liu. Affordpose: A large-scale dataset of hand-object interactions with affordance-driven hand pose. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14713-14724, 2023.
[29] Kailin Li, Jingbo Wang, Lixin Yang. Cewu Lu, and Bo Dai. Semgrasp: Semantic grasp generation via language aligned discretization. arXiv preprint arXiv:2404.08:: 03590, 2024.
[30] Yan-Kang Wang, Chengyi Xing, Yi-Lin Wei, Xiao-Ming Wu, Wei-Shi Zheng. Single-view scene point cloud human grasp generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[31] Andrew T Miller and Peter K Allen. Graspit! versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, 11(4): 10-2211,- 2004.
[32] Adithyavairavan Murali, Weiyu Liu, Kenneth Marino. Sonia Chernova, and Abhinav Gupta. Same object. differrnt grasps: Data and semantic knowledge for task-oriented grasping. In Conference on Robot Learning, pages 1557. PMLR. 2021.
[33] Chao Tang, Dehao Huang, Wenqi Ge, Weiyu Liu, and Hong Zhang. Graspgpt: Leveraging semantic knowledge from a large language model for task-oriented grasping. IEEE Robotics and Automation Letters, 2023.
[34] Kechun Xu, Shuqi Zhao, Zhongxiang Zhou, Zizhang Li, Huaijin Pi, Yifeng Zhu, Yue Wang, and Rong Xiong, A joint modeling of dision-language-actinn for target-oriented grasping in clutter. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1159--1160 IEEE, 2023.
[35] Shiyu Jin Jinxuan Xu, Yutian Lei, and Liangjun Zhang. Reasoning grasping via multimodal large language model. arXiy preprint 2rXiv:2402.06798 2024.

---

[36] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference Robot Learning, pages 990-. PMLR, 2022.
[37] Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEF Robotics and Automation Letters, (044):1100-11111 2022.
[38] Danny Driess, Fei Xia, Mehdi M. Sajjadi, Corey Lynch, and Aakanksha et al. Chowdhery. Palm-e: An embodied multimodal language model. In arXiv preprint arXii:2303.033330 2023.
[39] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipula- tion. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
[40]
Shadowrobot, https: / /ww. www.sswswwwwbo.ooomo http:/oooot.coo//eeeee com/dexterous hand-series/ series/ 2005.
[41] Javier Romero, Dimitrios Tzionas, and Michael J Black. Emhodied hands. ACM Transactions on Graphics, 36(6):1-17, 2017.
[42] Jieming Cui, Tengyu Liu, Nian Liu, Yaodong Yang, Yixin Zhu, and Siyuan Huang. Anyskill: Learning open- vpea-locayulary physical skill for interactive agents. In Conference on Computer Vision and Pattern Recognition(CVPRR 2024.
[43] Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-ohject contact consistency reasoning for human grasps generation. In Proceedings of the International Conference on Computer Vision, 2021.
[44] Julen Urain, Niklas Funk, Jan Peters, and Georgia Chalvatzaki. Se (iffdffsusionffssls: Learning smooth cost functions for joint grasp and motion optimization through diffusion. In 2023 IEEE International Conference on Robotics and Automation (ICRA). 2023.
[45] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnct++: Deep hierarchical feature learning point
metric space. Advances in Neural Information Processing Systems, 2017.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[47] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Syxtems, 33:6840-6851, 2020. 2020.
[48] Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from single image. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition. 2017.
[49] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint 12X.0222 2022.
[50] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Procensing Systems. 30, 2017.
[51] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learinng structured output representation using deep conditional generative models. Advances in Neural Information Processing Systems, 2015.
[52] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2016.
[53] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-ser pee object detection, arXiv preprint arXiv:2303.00 055999 2023.
[54] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, el al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026, 2023.
[55] Wentao Yuan. Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pen: Point completion network. In 2018 International Conference on 3D Vision (3DV), pages 728 73777 JEEE, 2018.
12

---

[56] Chenxi Wang, Hao-Shu Fang, Minghao Gou, Hongjie Fang, Jin Gao, and Cewu Lu. Graspness discovery in clutters for fast and accurate grasp detection. In Proceedings of the IEEE/CVF International Conference on Computer Vixion, 2021.
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin. Attention is all you need. Advances Neural Information Processing Systems, 2017.
[58]
Robotics. Allegro robot hand. http://wwwwwwalllloooooooooo//rr
[59] Kenneth Shaw. Ananye Agarwal, and Deepak Pathak. Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning. arXiv preprint arXiv:2309.0644 2023.
[60] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1152-1161, 2022.
13

---

A Appendix supplemental material
A.1
DexGYSGrasp Details
A.1.1 Diffusion Background
The diffusion model is used in our intention and diversity grasp component to generate grasp distribution with aligned intention and high diversity, which represents a class of generative models characterized by a forward process of noise addition and a reverse process denoising. The forward process entails Markov Chain that incrementally introduces Gaussian noise into the data across multiple time steps. Originating from the initial data To. this process transitions the data to conform with standard Gaussian distribution xy after T time steps. This transformation is mathematically formulated as follows:
3t aa + v1 0404-1,
where Og denotes a time-dependent noise coefficient, o₁ = II 04.
Therefore, 2y given To follows normal distribution,
(a = MM varso, (1 ar T..
The first equation delineates the stepwise diffusion, whereas the second equation offers a direct approximation of any intermediate state H; from ø0.
During the reverse process, the model is trained closely approximate the reverse conditional distribution p(zx-1/21), which is described as:
po( = (i t). o (1), t)),
where (o, 1) and o3 1) are the mean and variance parameters for the distribution of 1t-1, respectively, and indicates the parameters of the model used to predict from Tt.
The classical sampling strategy for the reverse process is exemplified by DDPM [47, where the model iteratively learns reverse the noise addition process to reconstruct the original data from noise. It estimates the distribution p(x+-1
(xxe) and predicts the noise e, represented by:
1bo (xx,t) já,
Xt:
(xe vI d:Ey (xx))
(9)
Var
3r
(x):())
(10)
var
āt
Tt-1 M4 (xt,)) () + ao( l)z,
(11)
where de consists non-trainable, time-dependent constants, and z represents Gaussian noise.
A.1.2 Intention and Diversity Grasp Component
Point Encoder We utilize three-layer PointNet++ as our point encoder, following recent works [56] 14] 7 in the field of robotic grasping. Specifically, each layer 4, i 1, 3. receives point clouds and corresponding features (initially the raw XYZ coordinates for the first layer) from the preceding layer. It then performs down-sampling and feature aggregation using the "set-aggregatiin" operation 45) The aggregated features are processed by a three-layer perceptron, which consists of three Linear BatchNorm ReLU blocks. The output of point encoder is Fobs IR Nat Cots
Language Encoder For the language encoder, we employ the CLIP model with the ViT-L/14 architecture [46]. The input text sequence is tokenized and converted into token embeddings with positional embeddings added. This sequence is processed through multiple Transformer encoder layers to obtain the language feature Flan E R Nian Clan,
Transformer Decoder We employ four layers of MLPs and Transformer decoder 57 The time embedding and pose feature are incorporated in MLPs to obtain Fde Subsequently, the Fa serves the query, and the concatenated features of language and object, Fian_obj. act as key and value in Transformer block. The corss attention process is formalized as:
[F fa obi))"
F Fout& softmax
FF(aaan_b)))
(12)
Vdy
where fnz fkr fo are MLPs, and dy is the channel of features. Finally, a MLP is adopted to regress the dexterous grasp parameters Gdez, Faees (Fdez) + fo (FFime)) where fo. fi, f2 are MLPs.
14

---

maggrr sprarst
Ensader
paze 57
Transformer Decoder
object.0
object:
Point
Pncradrr
Diversity Grasp Component
Quality Grasp Component
Figure 8: Inference pipeline of our DexGYSGrasp.
A.1.3 Quality Grasp Component
Quality grasp component tasks coarse pose gdez, coarse hand point clouds H(Gdez) and object point clouds O as input, outputs refined grasp Gdez, The object and hand are encoded by the PointNet++ same with intention and diversity grasp component. And in the transformer decoder, coarse pose features act as the query, object and hand features serve as key and value.
A.1.4 Inference Pipeline
We also demonstrate the inference pipeline of our DexGYSGrasp. as shown in Fig. 8 We sample random noise from Gaussian distribution the input, with the point cloud and language guidance as the conditions. We first generate the coarse grasp by the intention and diversity grasp component, and then refine it with the quality grasp component.
A.1.5 Loss Function
This section provides a detailed exposition of the loss functions utilized during the construction of datasets and the training of models.
Parameter Regression Loss. We utilize the mean squared error (MSE) to quantify the deviation between the generated dexterous hand pose Gdea and the ground truth gdez
Epara
1 N
Iggdex
gdenn.i12
(13)
Hand Chamfer Loss. The predicted hand point clouds H(Gdtez) and the ground truth H(dder)
are derived by sampling from the hand mesh. We then compute the chamfer distance assess the discrepancies between the predicted and ground-truth hand shapes.
Lehamfer
Σ
min 1
Σ (G
min (G
|| -yl
(14)
Contact Map Loss. The contact map loss Lemap ensures consistency between the predicted hand contact map booj on object and the target cohj_ The contact map is calculate by the distance from object point to the closest dexterous hand point.
Lemap =
Σ
|c obs
2o04
(15)
Object Penetration Loss. The object penetration loss Lpen penalizes the depth of hand-object penetration, where drdff denotes the signed distance from the object point to the hand mesh.
Epen
EI(a 0) a.df
(16)
Leff-enntrraion Loss. The self-penetration loss Lapen punishes the penetration among the different parts of the hand, where sep denotes predefined anchor spheres the hand [14).
Lspen Σ Ili + j) max(6 d(v; dex, sp dez, dez,op)).
P2
(17)
15

---

Language Guidence
butle, you fur Teggents anp snound
bod meuredd frgan rake contrt caneea
the wedeevtt flgges
rnb sa Frgens contedath courder
litupa all fragen wagss
Heal now mony your Eagami
Hunaa Hand
Shadow Hazd
Allegm
Rasd
Figure 9: The Extension of DexGYSNet to dexterous hands.
Joint Angle Loss. Given the physical structure limitations of the robotic hand, each joint has designated upper and lower limits. The joint angle loss penalizes deviations from these limits.
Ljoint
Σ
(max(( 4, ()aa)) + max(q, min 9:)).
(18)
A.2 DexGYSNet Datasets Details
A.2.1 Prompt LLM
We introduce the prompt for using GPT-3.5 this section.
"You are an assistant in creating language instruction, aimed at guiding robot on how to grasp objects. Given brief instruction and a fine-gained interaction information. Your task is generate natural and more informative instruction. The instruction should start with the given brief instruction, which limited sentence and about 10-15 words."
System Prompt:
User Prompt: "Brief instruction: To <brief intention> a <object category>. Hand-object interaction information: <contact informations."
The brief intention and object category are sourced from the hand-object dataset OakInk 277 The contact information is derived by calculating the distances from predefined contact anchors on each finger to the segmentation parts of the object. Details on predefined contact anchors are available in DexGraspNet []4], and segmentations annotated in OakInk (27).
example of user prompt is: "Brief instruction: To use trigger sprayer. Hand-object interaction information: forefinger touches the trigger. thumb, middle finger, ring finger and little finger touches the finger." " An example LLM output is: "To use trigger sprayer, press the trigger with your forefinger and hold the bottle with your other fingers."
A.2.2 DexGYSNet Extension
Our cost-effective dataset construction strategy can easily extended to various types of dexterous hands. As shown in Figure g besides the Shadow Hand[40], which features highly biomimetic design replicating most degrees of freedom human hands high cost of $100,000, we also expand to the Allegro Hand [58] and Leap Hand [59]. These latter models, while offering fewer
16

---

sampling
Point Encoder
P-FID
Rendering Views
Please mvilutt the contrnt the "nobee greep objas" inage and the descripiin text, and oursidency
point
from to besed wheter they shire the same intertion.
holdd e lottle, wapp your land anuuni Its body with Engees touching
rendering
Image Encoder
FID
images
Consstrocy scorp
(6) The evaluation pipeline of P- P-FID and FID.
(b) LLM-assisted evaluation.
Figure 10: (a) Evaluation of intention consistency using the Fréchet Inception Distance between the <generation hand and object> and the ground truth. (b) When the ground truth is not available (e.g., evaluation on 3D object dataset), we employ GPT4-0 for evaluation.
degrees of freedom, are significantly more affordable, costing $16,000 and $2,000, respectively, making them practical for promoting the of robotic arms in laboratory environments. We have trained our method on the DexGYSNet dataset using the Allegro Hand and implemented in real robot experiments.
A.3
Implementation Details
A.3.1 Dataset Split
We split the DexDYS dataset at the level of object instances. Specially, for all objects within each category, 80% of the objects instances are used for training and 20% for evaluation. Concretely, the training set includes approximately 1,200 objects with 40k grasps, while the evaluation set comprises about 300 objects with 10k grasps. Therefore, all objects in the test set DexGYSNet don't exist in the training set.
A.3.2 Metrics Detials
Target Assignment. For target assignment in the testing phase, the grasp targets of an object-guidance pair consist of all poses that share the same contact part and brief guidance. And the matrices of intention consistency are calculated by comparing the prediction to the most similar grasp target.
Fréchet Inception Distance, which is commonly used in generative task[60] by measuring the distance between the generated distribution and the ground truth distribution. We use sampling point cloud features extracted from|44 [49] to calculate P-FID and rendering image features extracted from (50)
calculate FID. The details are shown in Figure 10 (a).
Chamfer distance, denoted as CD, used to measure the distance between predicted hand point clouds and targets measure the consistency from the aspect of hand consistency. Please look Equation I4 for details.
Contact distance, denoted as Con. to measure the L2 distance of object contact map between the prediction and targets measure the consistency from the aspect of object contact consistency. Please look at Equation 11 details.
We evaluate the grasp success rate in Issac Gym simulation environment. To simulate the force exerted by dexterous hands grasping objects in real environments, we contract each finger the direction of the object. the grasp can withstand at least one of the six directions of gravity, is considered successful.
Success rate.
Mean Q1. Intuitively, the Q1 metric reflects the norm of the smallest wrench which can disrupt the stability of a grasp. We follow [14) to set the contact threshold to Icm and set the penetration threshold to 5mm. Any grasp with its maximal penetration depth greater than 5mm is considered invalid and we set the Q1 of it to 0 before taking the average.
Maximal penetration depth, which is the maximal penetration depth from the object point cloud to hand meshes.
Diversity. We use the standard deviation of translation 8, rotation 8, and joint angle ó, to measure the diversity of generated grasps. We perform eight samples the intention and diversity component under the same input conditions, and each sample is individually sent to the quality component
17

---

Figure 11:
Visualization grasps before and after quality grasp component. Our quality grasp component improves grasp quality and maintains intention consistency.
138
bold
138
hold
USB
hold
Figure 12: Visualization of our DexGYSGrasp framework with task-oriented simple input.
refinement. Before calculation, 8, and 6, are converted to Euler angles degrees, while ò, measured in centimeters.
A.3.3 Implementation Details of SOTA Methods
We replicate SOTA methods on our DexGYSNet dataset using the same encoder structure and the loss functions defined Equation 5 to ensure fair comparison. Specifically, we reimplement GraspCAVE based on [51]. GraspTTA [45], SceneDiffuser [4] and DGTR [7]. To introduce language information, we use an identical CLIP language encoder. For GraspCAVE, we concatenate the language feature, object feature, and latent feature to send to the decoder. Based on GraspCAVE, GraspTTA employs a test-time adaptation strategy for quality refinement. For SceneDiffuser, we concatenate the language and object features the model condition. For DGTR, the language and object features are concatenated to send to its transformer decoder.
A.4
Additional Experiments
A.4.1 Qualitative Experiments of Quality Grasp Component.
We provide additional qualitative results to verify the effectiveness of Quality Grasp Component. Figure IT shows the grasps before and after the application of the Quality Grasp Component, demonstrating that QGC can prevent object penetration and maintain consistency with the original intention.
A.4.2 Qualitative Experiments of Task-oriented Guidance.
We conduct qualitative experiments to demonstrate the generalization of our DexGYSGrasp frame- work to task-oriented or functional grasp task. Specifically, we input task-oriented guidance (e.g.. "use" or "hold") into our framework, which has been trained on DexGYSNet. As shown in Figure 12 our GGGrapp framework exhibits good compatibility with these inputs. This further confirms that our approach enables more flexible and natural human-robot interactions.
18

---

Objects
Realsetze
D415 Camesa
Flexiv Rizon Am
System sutugi
Object
Figure 13: The illustration of our real world experiments settings.
Grounding DINO
SAX
RGB
Rounding
Segmestation
Point Cloud Completion
Dopth Input
Partisl Point Clouds
Full Point Clouds
Figure 14: Real world experiments pipeline.
A.5 Real World Experiments Details
Experimental Environment Figure [3] shows the settings our real world experiments. The experiments conducted on Allegro hand, Flexiv Rizon4 arm and an Intel Realsense D415 camera. The experimental object 3D printed object from test set of DexGYSNet.
Experiment Pipeline Our DexGYSGrasp takes full point clouds as input following recent works in make our methods more practical, we employ three off-the-shed models in cascade to obtain a full point cloud from scene point cloud. As shown in Figure [4 we input the object category and the RGB image into an open-set detection model [53
dexterous grasping 11
4 7
detect the
bounding box of the object. This bounding box is then used as prompt for SAM segmentation of the object. Next, we crop the target depth image using the segmentation map and the depth input. Finally, we convert the partial depth image into point clouds and feed it into point completion network (55) to obtain the final full point clouds. Then, the full point clouds are fed into framework to obtain the dexterous grasp pose, which is then transformed into the real coordinate system. execution, we first move the arm to the 6-DOF pose of the dexterous hand root node, and then control the joint angles to achieve the target pose.
54
obtain the
Experiment Results The experiment results are presented in Table For each object, we command robot with different language instruction, and each instruction is tested five times, resulting in total of ten grasping trials per object. grasp is deemed successful if aligns with the intended instruction and maintains stability, preventing the object from falling. Our method demonstrates moderate success rate, indicating its effectiveness. Further research on enhance the robustness of our approach.
scenarios recommended to
19

---

To hold power drill, all fingers to grip the drill motor securely.
To use power drill, press the drill trigger with your forefinger and grip it with your other fingers.
To hammer, grasp with all five fingers to secure the hammer grip.
To hand over the hammer, grip with all your fingers around the hammer head.
Figure 15:
Real world experiments.
Power drill
Hammer
Trigger sprayer
Game controller
Pincer
Frying pan
Wineglass
Success
9/10
4/10
3/10
8/10
5/10
3/10
6/10
Table 5: Real word experiment results
A.6
Societal Impacts and Limitations
The core innovation of this paper has a significant positive impact on society. We propose a novel task: language-guided dexterous grasp generation, which can promote human-robot interaction and expedite the deployment of robots in real-world scenarios. Additionally, we introduce an innovative framework accomplish this task. Our approach can generate high-quality grasps while ensuring consistency of intent and diversity of grasps.
However, method still faces some challenges in real-world deployment. Due to limitations in the current development of robotic arm control and physical structures, we cannot guarantee success in every grasp execution in the real world. In future work, we will further enhance the quality of grasp generation to improve the success rate in real-world scenarios.
20

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work in the conclusion and supple- mentary material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well cell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
21

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper fully disclose all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
22

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer:[No
Justification:
We provide examples of our dataset supplementary materials. We promise release all code and the complete dataset after the publication of this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. htte//uublcggguueeeeee hoooo////uiddes/oeeeuubbii /at////ooeSSiiiseeeeeeeeeeaeeeee ississsiss ssionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because performing multiple runs for each experi- ment would be too computationally expensive.
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides sufficient information the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics ftctts://neuripeee http://ppw...pppcp/cc/cccccaaaaa
Question:
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]

---

Justification:
The paper discusses both potential positive societal impacts and negative societal impacts of the work in the conclusion and supplementary material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original owners of assets, used in the paper, are properly credited and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
25

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper are well documented and the documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
26

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "Uvsqodh6Tf", "reviewer": "Reviewer_MqTF", "summary": "This work proposes a language-guided dexterous grasp dataset based on OakInk, named DexGYSNet utilizing LLM and a corresponding two-component framework DexGYSGrasp which generates dexterous grasps based on human language instructions. Real-world experiments are conducted to validate DexGYSGrasp's performances.", "questions": "- The language descriptions generated via the large language model, even with the inclusion of rich contact information as shown in Figure 6, still appear to lack precision. In this case, it is questionable whether directly applying metrics like Chamfer distance and contact distance is the most appropriate approach to reflect the consistency of the grasping intention. The suitability of these particular evaluation metrics for the language-based grasping task deserves further examination.\n- A minor question: the naming of the dataset as \"DexGYSNet\" seems somewhat confusing, as it suggests a naming convention more suited for a pipeline or system, rather than a dataset. \"DexGYSGrasp\" would be a more intuitive name for the dataset itself, as it aligns better with the specific task of grasping being addressed.", "limitations": "See Weaknesses", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The construction of the DexGYSNet dataset is cost-effective with the help of a large language model-assisted annotation system. The use of an LLM-based approach allows for efficient and scalable data labeling, which is crucial for developing robust models in this domain.\n- Extensive experiments are conducted to verify the performance advantages of the DexGYSGrasp method, particularly in terms of grasping intention and diversity, as well as the design of the evaluation metrics. The thorough experimental evaluation provides a comprehensive assessment of the approach and its capabilities.", "weaknesses": "- While the DexGYSGrasp framework presents several notable contributions, such as the use of penetration loss and the Quality Grasp Component, the primary strength of the approach seems to be in the training methods rather than the overall framework design. \n- The Quality Grasp Component appears to function more as a post-processing step rather than a core component of the framework. This suggests that the key innovations of the work are centered around the training techniques employed, rather than the architectural design of the system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your reviews. Below are our responses to your questions.\n\n**Q1**: The main contribution.  \n\nA1: The main contribution of our paper is **NOT** just a dataset and a baseline. We believe our idea can contribute to robotics community. The main contribution can be summarized as:  \n1) This paper explores a **meaningful task**, \"Dexterous Grasp as You Say\" (DexGYS), enabling robots to perform dexterous grasping based on human language commands.\n2) This paper proposes the **DexGraspNet dataset** and a **cost-effective dataset construction pipeline** for language-guided dexterous grasp generation with the idea of retargeting from human grasps and LLM assistance. This pipeline can inspire more language-guided and dexterous robotics tasks.\n3) This paper proposes a progressive two-stage framework that decouples the challenging problem of dexterous grasp generation into two sub-problems. One-stage methods struggle to simultaneously achieve high quality, diversity, and intention consistency due to optimization challenges, while our framwork achieve these.\n\nThe framework surpasses all previous methods and only our framework can achieve high-quality, diversity and intention consistency generation.\n\n**Q2**: The framework is a baseline. \n\nA2: Our framework is **NOT** a baseline, but a novel framework that solves the identified optimization problems and achieves capabilities that previous works can’t.\n1. We benchmark several SOTA methods as baseline as shown in both Table 1 in the main paper and rebuttal PDF. The results show that **only** our framework can achieve the generative ability of intention consistency, diversity and high quality. \n2. We identify the **key optimization challenge** for dexterous grasp generation caused by penetration loss and propose to decouple this challenge into two easily solvable problems. And our framework effectively addresses both problems.\n3. What's more, the Quality Grasp Component is **not a post-processing method** like Test Time Adaptation [1], but a part of progressive two-stage framework. Quality Grasp Component needs to keep the intention consistency and improve the quality, which can’t be achieved by [1] as shown in Table 2 Line 6 ($IDGC(λ^2_{pen}=0)+TTA$) in the main paper.\n\n**Q3**: Evaluation metric.\n\nA3: To further evaluate, we employ Fréchet Inception Distance (**FID**) [2], which is commonly used in generative task [4]. We use sampling point cloud features extracted from [3] to calculate P-FID and rendering image features extracted from [2] to calculate FID. The results in Table 1 in global rebuttal PDF show that our method **surpasses** all previous methods. **More details can be found in global rebuttal (a), and Figure 1 (a) and Table 1 in rebuttal  PDF**.\n\n**Q4**: Why call the dataset DexGYSNet.\n\nA4: We follow the ImageNet [5] to name our dataset, other examples include DexGraspNet [6].\n  \n  \n\n[1] Jiang et al. Hand-Object Contact Consistency Reasoning for Human Grasps Generation. ICCV 2021\n\n[2] Heusel, et al. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NIPS 2017.\n\n[3] Nichol, et al. Point-E: A System for Generating 3D Point Clouds from Complex Prompts. OpenAI 2022.\n\n[4] Guo et al. Generating Diverse and Natural 3D Human Motions from Text. CVPR 2022\n\n[5] Deng et al. ImageNet: A Large-Scale Hierarchical Image Database. CVPR2009.\n\n[6] Wang, et al. DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation, ICRA 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0wC7kdvWg8", "reviewer": "Reviewer_dbTG", "summary": "The paper introduces \"Dexterous Grasp as You Say\" (DexGYS), a novel framework that integrates natural language instructions with robotic dexterous grasping. To support this integration, the authors present DexGYSNet, a dataset featuring 50,000 pairs of annotated dexterous grasps with corresponding human language instructions across 1,800 household objects, developed using innovative methods such as Hand-Object Interaction Retargeting and LLM-assisted annotation.\n\nBuilding on this dataset, the DexGYSGrasp framework is introduced, designed to generate robot grasps that are aligned with human intentions, ensuring diversity and quality. This two-phase framework initially learns a diverse grasp distribution aligned with intended commands and subsequently refines these grasps to enhance quality.\n\nExtensive testing both on DexGYSNet and in real environments confirms the framework's ability to produce practical, diverse, and high-quality grasps based on natural language commands, significantly advancing human-robot interaction capabilities for both industrial and domestic applications.", "questions": "My questions are primarily based on the Weaknesses:\n\n1. The authors need to further discuss their evaluation metrics. *CD* and *Con.* are too stringent for a generative task. Metrics based on collisions or the standard deviation of poses seem insufficient for evaluating language-based generative tasks.\n2. The authors might consider how to incorporate richer intents into DexGYSNet.\n3. The authors should include an evaluation of the quality of the DexGYSNet dataset.", "limitations": "The authors discuss the societal impacts and limitations of their method.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of the authors is reasonable: leveraging current large language models to provide fine-grained descriptions of grasping states. The DexGYSNet dataset contributes to the community.\n\n2. The authors validated their algorithm on a real robotic gripper, which is crucial.\n\n3. The authors showcased parts of the DexGYSNet data in the supplementary material. Providing visualization code would further enhance this contribution.", "weaknesses": "1. The evaluation metrics require further discussion. DexGYSGrasp is a generative framework, and the effectiveness of the proposed method based on Chamfer distance and Contact distance is not entirely convincing. Although the authors mention in Supp. (Line 521) that the target is \"the most similar grasp\", it is not always possible to find a closely matching grasp pose in the dataset. For instance, if the generated grasp is on the right temple of a pair of eyeglasses, but all dataset grasps are on the left temple, the Chamfer distance metric becomes ineffective.\n2. DexGYSNet only annotates which fingers contact the object and which parts of the object are touched. It lacks more complex intents, which might limit its applicability.\n3. The authors need to verify the quality of DexGYSNet. For example, what about the quality of the text annotations (e.g., could hallucinations from large models cause errors)? Additionally, the authors obtain dexterous hand parameters from MANO parameters through retargeting. They need to validate the grasp quality in physical simulations (e.g., Isaac Sim) to ensure that the objects are tightly grasped without dropping.\n4. In Line 170, the authors train the Quality Grasp Component based on the most similar ground truth grasp with the same intentions. Perhaps, adding noise directly to the generated grasp pose to create a large amount of training data would be more effective?\n5. The paper contains several typos. For example, Line 117 incorrectly uses $L_{spen}$. Lines 173 and 168 are repetitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your reviews. Below are our responses to your questions.\n\n**Q1**: Evaluation Metrics. \n\nA1: To further evaluate, we employ Fréchet Inception Distance (**FID**) [1], which is commonly used in generative task [6]. We use sampling point cloud features extracted from [2] to calculate P-FID and rendering image features extracted from [1] to calculate FID. The results in Table 1 in global rebuttal PDF show that our methods **surpass** all previous methods. **More details can be found in global rebuttal (a), and Figure 1 (a) and Table 1 in rebuttal  PDF**.\n\n**Q2**: Richer Intents.\n\nA2: \n1. The guidance texts about contacting in DexGYSNet are **better** aligned with the grasp action, thereby facilitating the generation of higher quality grasps. \n2. Our method can be naturally **extended** to enrich intentions. For the dataset construction, the high-level intention can be annotated by the LLM with proper prompts (e.g., “Please generate possible human intentions for grasping the mug handle.”). For methods, the high-level intention can be understood by the LLM and output as low-level guidance through chain-of-thought reasoning [3] (e.g., from \"I am thirsty\" to \"Grasp the handle of the mug\"). Then, the low-level guidance can be fed into our framework to generate grasp.\n\n**Q3**: Grasp Quality of DexGYSNet.\n\nA3: For grasp quality, we validate the grasp quality in Isaac Gym following [4], and the **success rate** is 85.1%. The overall grasp quality of the DexGYSNet is relatively high, compared with previous datsets [7]. And we find that the simulation errors, and small contact area in fine-grained operation (e.g., grasp the cap of a bottle) result in failure case. We will perform retargeting and verification again assisted with manual adjustment to ensure the high quality of DexGYSNet.\n\n**Q4**: Text Quality of DexGYSNet.\n\nA4: For text annotations quality, we use the LLM-assisted evaluation [5] (leveraging the powerful vision understanding ability of GPT4-o) to evaluate the consistency with the rendered grasp image as shown in Figure 1 (b) in global rebuttal PDF. The average **consistency score** is 4.38 (on a scale of 1-5), and the overall text quality of the DexGYSNet is relatively high, according to [5] . Some inconsistent samples are caused by fine-gained contact and specific modifiers (e.g., “tightly grasp the handle of the mug with all your fingers”, while the LLM thinks it's not tight enough or some fingers don’t contact the handle.). We will improve these with the assistance with LLM and manual check.\n\n**Q5**: Adding noise to train Quality Grasp Component.\n\nA5: We add the Gaussian noise to the normed grasp pose to augment the training set. There is no significant improvement, as shown in the following table. We think that the current training data is sufficient to train our model.\n\n|                | P-FID ↓ | Q1 ↑  |\n|----------------|----------|-------|\n| Ours           | 5.796    | 0.083 |\n| Add Noise      | 5.764    | 0.085 |\n\n[1] Heusel, et al. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NIPS 2017.\n\n[2] Nichol, et al. Point-E: A System for Generating 3D Point Clouds from Complex Prompts. OpenAI 2022.\n\n[3] Brohan, et al. Vision-language-action models transfer web knowledge to robotic control. CoRL 2023.\n\n[4] Wang, et al. DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation, ICRA 2023.\n\n[5] Zheng, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NIPS 2023.\n\n[6] Guo et al. Generating Diverse and Natural 3D Human Motions from Text. CVPR 2022.\n\n[7] Wei et al. DVGG: Deep Variational Grasp Generation for Dextrous Manipulation. RAL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9uRs1yqayO", "reviewer": "Reviewer_3CYr", "summary": "This paper tackles language-guided dexterous grasping by creating a large-scale dataset pairing grasps and language guidance and presenting a grasp generation pipeline based on language instructions. The dataset, DexGYSNet, leverages HOIR and LLMs to scale up the size of grasps and annotations while keeping the generation costs manageable. The DexGYSGrasp framework introduces a two-stage progressive generation to balance diversity, quality, and grasp-language alignment. The method is evaluated on the intention consistency, grasp quality, and diversity against SOTA methods. Experiments demonstrate the necessity of progressive components, effectiveness of HOIR, potential to transfer to real-world scenarios. Overall, the paper advances the field of dexterous grasp generation by natural language guidance, providing a valuable dataset, and proposing a robust framework for generating high-quality robotic grasps based on human instructions.", "questions": "1. The paper states that intention consistency is calculated by comparing the prediction to the most similar grasp target. While this approach addresses the issue of multiple valid grasp poses, could you provide more details on how the diversity of valid grasp poses is represented in your dataset and how it influences the evaluation metrics?\n2. The paper does not seem to explicitly address how the proposed method generalizes to different objects, unseen objects, or even unseen categories. The dataset is derived from OakInk, which has category information. Could you provide more details on the performance and adaptability of your method when applied to objects and categories not present in the training dataset? This generalizability is important for the network to understand the correspondence between the geometry and functional semantics. In-category and out-of-category generalization are both essential for a capable network.", "limitations": "1. The method is not yet ready for real-world applications due to several factors. It requires a full point cloud, which needs several adds-on modules like segmentation and completion for deployment. Additionally, there is a lack of discussion on how this method would benefit the subsequent grasping execution and how the grasped objects can be manipulated, which is crucial for practical use.\n2. The paper does not sufficiently address how the method generalizes to unseen objects or categories. Ensuring that the method can handle a wide range of objects, including those not present in the training dataset, is essential for practical applicability.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This problem of language-guided dexterous grasping is significant. This paper presents both a large-scale dataset and a method to generate diverse grasps. The dataset is a valuable resource for the community and the method can be integrated into a grasping policy and applied to real-world scenarios. The design of experiments justifies the choice to break the generation process into two stages and demonstrates the effectiveness of the method in improving grasp quality, diversity, and alignment with language instructions.", "weaknesses": "1. The method requires a full point cloud and is designed for static pose generation. In real-world applications, access to a complete point cloud is often unavailable, although this could be mitigated by using a point cloud completion network. Moreover, grasping is typically a precursor to a subsequent action, requiring an end-to-end pipeline that can handle both grasping and the intended use of the object. An integrated approach would likely be more effective than segregating these steps into different modules. For instance, the subsequent policy must not only use the generated pose but also comprehend instructions (again, since the generated pose alone is not enough) and object geometry to perform the correct actions, which an end-to-end system could potentially handle more seamlessly. Plus, there can be slight adjustments to gestures after grasping a tool from a table but before using the tool. In this sense, real world experiments on this synthesis method are not so legitimate in the first place.\n\n2. The paper does not sufficiently address how the method generalizes to unseen objects or categories. Given the diversity of real-world objects, it is crucial for the network to perform well on unseen items to be practically useful. Including more detailed experiments and analysis on in-category and out-of-category generalization would significantly strengthen the work. This could involve testing the method on a wider variety of objects and categories not present in the training dataset and assessing its adaptability given that OakInk has the category information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your reviews. Below are our responses to your questions.\n\n**Q1**: The full point cloud and end-to-end pipeline. \n\nA1: Our work is focused on language-guided dexterous grasp generation task and is the **base** of future work of partial observation and end-to-end pipeline. What’s more, our cascading pipeline works **well** in real robot experiment, where shows acceptable impact on the performance of our framework.\n\n**Q2**: Generalization on unseen objects.\n\nA2: \n1. All objects in the test set of DexGYSNet **DO NOT** exist in the training set. Therefore, the experiment results of generation performance are all evaluated on unseen objects in this paper. Our dataset consists of 34 categories, 1500 objects, and 50000 grasp annotations. We split it at the object level, with 80% for training and 20% for testing in each category. The training set contains about 1200 objects and 40,000 grasps, and the test set contains about 300 objects and 10,000 grasps.\n2. In addition, we conduct experiments on **cross-domain unseen objects**, collected from PartNet [3] (a 3D object understanding dataset). The results show the **good** generalization of our framework. The details and results of cross-domain unseen object experiment can be found in **global rebuttal (b), Table 3 and Figure 2 (b) in rebuttal PDF**. \n\n**Q3**: Generalization on unseen categories.\n\nA3: We conduct experiments on **unseen categories** by removing 15 categories in the training set. The results shows the **good** generalization of our method. Benefiting from our framework design and our large-scale datasets, model can learn from object geometric structure and language semantic information, not just overfitting certain category (e.g., learn how to use a mug by grasping the handle and generalize to how to use a teapot). The details and results can be found in **global rebuttal (b), Table 2 and Figure 2 (a) in rebuttal PDF.** \n\n**Q4**: Intention consistency and diversity calculation.\n\nA4: \n1. The ground trunth grasps of the object-guidance pair are all grasp poses sharing the same intention. The grasps with same simple intention and contact parts are considered as the same intention (e. g., all grasps that holding the bottle from different directions).\n2. In the inference time, the object and language are fed into our framework as condition, and multiple (specially 8 in our setting) samplings are performed to generate 8 grasps. \n3. Each generated grasp is matched with the nearest ground truth grasp by the hand chamfer distance. The **intention consistency metric** is calculated between the generated grasps and the matched ground truth. The **diversity metrics** calculate the standard deviation of these eight sampling grasps. \n4. We believe that a good model should be able to generate grasps that are both diverse and aligned with intention (e.g. hold the body of a bottle from different directions).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BkiaDG92To", "reviewer": "Reviewer_TfTv", "summary": "This paper proposes a novel robotic grasping task that utilizes natural language as guidance to generate grasping poses for dexterous hands that satisfy corresponding intentions. To accomplish this task, the paper first creates a large-scale language-guided dexterous grasping dataset. Moreover, the authors propose a progressive solution composed of a diffusion-based generative model and a regressive model to achieve generating intention-consistent, high diversity, and high-quality grasp poses for a wide range of objects.", "questions": "See weaknesses", "limitations": "See weaknesses", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Compared to existing work, this work advances further by posing a more general and challenging problem and offering a more comprehensive solution. The introduction of this problem promotes the direction of general robotic grasping.\n2. The paper contributes a grasping dataset that includes 18,000 objects with a total of 50,000 text-grasp pose pairs. This dataset fully utilizes the information provided by the Oakink dataset and transfers human hand poses to robotic hands in three steps. During the language annotation stage, the paper generates corresponding language descriptors by measuring the distance from the fingers to different regions of the object, then uses GPT to generate natural language. The entire process is technologically sound.\n3. The paper adopts a progressive framework to address this issue, utilizing a diffusion-based generative model and a regressive model to ensure that the generated poses are both diverse and high-quality.", "weaknesses": "Weaknesses\n1. When verifying intention consistency, the paper uses chamfer distance and contact distance to compare the error between the estimated poses and the target poses. However, since the intention is derived from natural language, which is inherently imprecise and non-unique, using a collected target pose as the sole standard is inappropriate. For instance, for \"to use a lotion pump, place your forefinger on the pump head and other fingers on the bottle body,\" there might be infinite ways to hold the lotion with the forefinger on the pump head, some of which might be different to the target one. Therefore, chamfer distance and contact distance may not be the best metrics.\n2. The dataset collected in the paper includes only 50,000 grasping poses. Compared to existing datasets for parallel-jaw grippers, which contain millions or even billions of poses, the diversity is limited. For a dexterous hand, which has a higher degree of freedom, collecting more data is inevitable to achieve more varied grasping methods. I believe the performance on unseen categories would degrade significantly.\n3. The authors mentioned that they use 20% of the data as a test set to evaluate the model's performance. However, they do not provide more specific information about the training and test sets. For example, whether the target poses and description texts of the same category of objects in the training and test sets are consistent, and what the results would be if given inconsistent descriptions, such as in Figure 6, \"To grasp a wineglass, use your fingers to hold the stem securely\" if changed to \"To grasp a wineglass, use your fingers to hold the GLASSES securely\" (I think this description will not exist in the dataset).\n4. Although the authors use the Q1 metric to measure grasp stability, this analytical approach still has limitations. The authors could try using a simulator (e.g., Isaac Sim) to test the grasp success rate, which would be more indicative.\n\nOther Comments\n1. The title of A.1.4 is \"inference time,\" but the content is unrelated.\n2. The real-world experiments in the video are not very promising. In the given demos, non-target areas of the objects are fixed by fixtures, leaving only the target areas available for grasping.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your reviews. Below are our responses to your questions.\n\n**Q1**: Intention consistency metric.\n\nA1: To further evaluate, we employ Fréchet Inception Distance (**FID**) [1], which is commonly used in generative task [3]. We use sampling point cloud features extracted from [2] to calculate P-FID and rendering image features extracted from [1] to calculate FID. The results in Table 1 in global rebuttal PDF show that our methods surpass all previous methods. **More details can be found in global rebuttal (a), and Figure 1 (a) and Table 1 in rebuttal PDF**.\n\n**Q2**: Generalization on unseen categories.\n\nA2: We conduct experiments on **unseen categories** by removing 15 categories in the training set. The results shows the **good** generalization of our method. Benefiting from our framework design and our large-scale datasets, model can learn from object geometric structure and language semantic information, not just overfitting certain category (e.g., learn how to use a mug by grasping the handle and generalize to how to use a teapot). **The details and results can be found in global rebuttal (b), Table 2 and Figure 2 (a) in rebuttal PDF.** \n\n**Q3**: Dataset splitting and description inconsistency. \n\nA3: \n1. The description texts for objects of the same category in the training and test sets are **NOT** the same. The description texts are generated **independently** for each sample by the LLM. Our dataset consists of 34 categories, 1500 objects, and 50000 grasp annotations. We split it at the object level, with 80% for training and 20% for testing in each category. The training set contains about 1200 objects and 40,000 grasps, and the test set contains about 300 objects and 10,000 grasps. \n2. Our framework are **robust** to the description noise, which we think is benefiting from our pre-trained CLIP language encoder [4].  For your example, if 'stem' is replaced with 'GLASSES,' the model tends grasps the stem or the body for different samples. And if you replace the \"stem\" to \"body\", the generated dexterous hand will hold the body of the wineglass. Due to the PDF is limited in one page in rebuttal, we will add more details to the supplementary materials.\n\n**Q4**: Grasp success rate.\n\nA4: We evaluate the grasp success rate in Issac Gym following [5], and the results show that our framework surpasses all previous works. The results can be found in **Table 1 in rebuttal PDF**.\n\n**Q5**: The title of A.1.4 is \"inference time\".\n\nA5: This section describes the pipeline in the inference time. We will change it to “Inference Time Pipeline”.  \n\n**Q6**: The fixtures in real-world experiments.\n\nA6: It is for the convenience of demonstration to fix the objects with fixtures. If the model generates inaccurate grasping, it will collide with the fixture. We also provide a new visualization that shows all areas are available for grasping without using fixtures in **Figure 2 (c) in rebuttal PDF**.\n\n[1] Heusel, et al. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NIPS 2017.\n\n[2] Nichol, et al. Point-E: A System for Generating 3D Point Clouds from Complex Prompts. OpenAI 2022.\n\n[3] Guo et al. Generating Diverse and Natural 3D Human Motions from Text. CVPR 2022.\n\n[4] Radford, et al. Learning Transferable Visual Models from Natural Language Supervision.\" ICML 2021.\n\n[5] Wang, et al. DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation, ICRA 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks for the constructive comments of all reviewers. We conduct additional experiments and analysis on the evaluation metrics and generalization. The figures and tables can be found in the **rebuttal PDF**.

(a) Evaluation metrics: 
1. To further evaluate, we employ Fréchet Inception Distance (**FID**) [1], which is commonly used in generative task [6] by measuring the distance between the generated distribution and the ground truth distribution. We use sampling point cloud features extracted from [2] to calculate P-FID and rendering image features extracted from [1] to calculate FID. The details are shown in Figure 1 (a). 
2. To further evaluate, we evaluate the grasp **success rate** in Issac gym following [4]. 
3. The results in Table 1 show that our method **surpasses** all previous methods in FID, P-FID and success rate.
4. The Chamfer Distance and Contact Distance **can** also evaluate the model performance. These metrics are calculated between the generation grasp and the most similar grasp in ground truth which share the same intention. The grasps with same simple intention and contact parts are considered as the same intention (e. g., all grasps that holding the bottle from different directions). As a whole, the generated grasp can find a proper ground truth with the same intention benefiting from the richness of DexGYSNet. The contact consistency is also used in previous work [7]. Therefore, these metrics can evaluate the model’s overall performance in the whole dataset.


(b) Generalization on unseen objects and categories.

1. All objects in the test set of DexGYSNet **DO NOT** exist in the training set. Therefore, the experiment results of model performance are all evaluated on unseen objects in this paper. 
2. To evaluate the generalization on **unseen categories**, we randomly sample 15 categories [8] and exclude these during training phase. The results in Table 2 and visualization in Figure 2 (a) show the good generalization of our method. Benefiting from our framework design and our large-scale dataset, model can learn from object geometric structures and language semantic information (e.g., learn how to use a mug by grasping the handle and generalize to how to use a teapot), not just overfitting the certain category. 
3. To further evaluate, we collect **cross-domain unseen objects** from PartNet [3] (a 3D object understanding dataset), containing 6 categories and 316 objects [9]. We collect language guidance generated by GPT4-o, where the prompt contains examples in DexGYSNet and the category of target object. We use GPT4-o to evaluate the consistency between rendering image and language guidance as shown in Figure 1 (b), due to the lack of ground truth. The results in Table 3 and visualization in Figure 2 (b) demonstrate that the models trained on DexGYSNet exhibit good generalization in cross-domain unseen objects and our framework **surpasses** the SOTA. 

[1] Heusel, et al. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, NIPS 2017.

[2] Nichol, et al. Point-E: A System for Generating 3D Point Clouds from Complex Prompts. OpenAI 2022.

[3] Mo, et al. PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding, CVPR 2019.

[4] Wang, et al. DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation, ICRA 2023.

[6] Guo et al. Generating Diverse and Natural 3D Human Motions from Text. CVPR 2022

[7] Agarwal, et al. Dexterous Functional Grasping. CoRL 2023

[8] "Unseen Categories": "cup", "cylinder_bottle", "teapot", "hammer", "screwdriver", "pincer", "fryingpan", "toothbrush", "wrench", "bowl", "lightbulb", "flashlight", "binoculars", "apple", "banana".

[9] "Cross-domain Unseen objects": "bottle", "mug", "headphones", "jar", "knife", "scissors".

Meta-review: The authors addressed all concerns raised by the reviewers with additional experiments and clear explanations. The paper's contributions, including the DexGYSNet dataset and the DexGYSGrasp framework, are significant for advancing the field of language-guided robotic grasping. The final evaluations from the reviewers were positive, recognizing the paper's technical merits and potential impact, leading to a recommendation for acceptance.
Decision: Accept (poster)