Title: Grasp as You Say: Language-guided Dexterous Grasp Generation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Dexterous Grasp Generation

Robotics
Abstract: This paper explores a novel task "Dexterous Grasp as You Say'' (DexGYS), enabling robots to perform dexterous grasping based on human commands expressed in natural language. However, the development of this field is hindered by the lack of datasets with natural human guidance; thus, we propose a language-guided dexterous grasp dataset, named DexGYSNet, offering high-quality dexterous grasp annotations along with flexible and fine-grained human language guidance. Our dataset construction is cost-efficient, with the carefully-design hand-object interaction retargeting strategy, and the LLM-assisted language guidance annotation system. Equipped with this dataset, we introduce the DexGYSGrasp framework for generating dexterous grasps based on human language instructions, with the capability of producing grasps that are intent-aligned, high quality and diversity. To achieve this capability, our framework decomposes the complex learning process into two manageable progressive objectives and introduce two components to realize them. The first component learns the grasp distribution focusing on intention alignment and generation diversity. And the second component refines the grasp quality while maintaining intention consistency. Extensive experiments are conducted on DexGYSNet and real world environments for validation.

Grasp as You Say:

Abstract

This paper explores a novel task 'Dexterous Grasp as You Say' (DexGYS), enabling robots to perform dexterous grasping based on human commands expressed in natural language. However, the development of this field is hindered by the lack of datasets with natural human guidance; thus, we propose a language-guided dexterous grasp dataset, named DexGYSNet , offering high-quality dexterous grasp annotations along with flexible and fine-grained human language guidance. Our dataset construction is cost-efficient, with the carefully-design hand-object interaction retargeting strategy, and the LLM-assisted language guidance annotation system. Equipped with this dataset, we introduce the DexGYSGrasp framework for generating dexterous grasps based on human language instructions, with the capability of producing grasps that are intent-aligned, high quality and diversity. To achieve this capability, our framework decomposes the complex learning process into two manageable progressive objectives and introduce two components to realize them. The first component learns the grasp distribution focusing on intention alignment and generation diversity. And the second component refines the grasp quality while maintaining intention consistency. Extensive experiments are conducted on DexGYSNet and real world environments for validation.

1 Introduction

Enabling robots to perform dexterous grasping based on human language instructions is essential within the robotics and deep learning communities, offering promising applications in industrial production and domestic collaboration scenarios.

With the advancements in data-driven deep learning and the availability of large-scale datasets, robot dexterous grasp methods achieve impressive performance [1, 2, 3, 4, 5, 6, 7]. While previous approaches focus on the grasp stability, they have not fully utilized the potential of dexterous hands for intentional, human-like grasping. Recent studies, known as task-oriented [8] and functional dexterous grasping [9, 10], aim to generate grasps based on specific tasks or functionality of objects. However, these approaches often depend on predefined, fixed and limited tasks or functions, restricting their flexibility and hindering natural human-robot interaction.

In this paper, we explore a novel task, 'Dexterous Grasp as You Say' (DexGYS), as shown in Figure 1. We can see that natural human guidance is provided in this task, and can be utilized to

1. Stable Grasp.

Hold the mug using

all five fingers to grasp

the body securely.

To use a mug, grasp it

securely using your thumb,

index and middle finger on

the handle.

Lift up the power drill

by grasping it with all five

fingers around

the drill motor.

To use a power drill,

place your forefinger on the

drill trigger  and other fingers

on the drill grip.

Use a trigger sprayer, press

the trigger with your

forefinger and hold the bottle

body with other fingers.

Hold the trigger sprayer by

wrapping all five fingers

around the bottle body.

To hold a wineglass, use all

five fingers to support

its bowl securely.

To lift up a wineglass, use

your thumb, forefinger, and

middle finger to hold the stem.

2. Functional Grasp.

Use

drive dexterous grasping generation, thereby facilitating more user-friendly human-robot interactions. However, the new task also brings in new challenges. First, the high costs of annotating dexterous pose and the corresponding language guidance, present a barrier for developing and scaling dexterous datasets. Second, the demands of generating dexterous grasps that ensure intention alignment, high quality and diversity, present considerable challenges to the model learning.

To address the first challenge, we propose a large-scale language-guided dexterous grasping dataset DexGYSNet . DexGYSNet is constructed in a cost-effective manner by exploiting human grasp behavior and the extensive capabilities of Large Language Models (LLM). Specially, we introduce the Hand-Object Interaction Retargeting (HOIR) strategy to transfer easily-obtained human hand-object interactions to robotic dexterous hand, to maintain contact consistency and high-quality grasp posture. Subsequently, we develop the LLM-assisted Language Guidance Annotation system to produce flexible and fine-grained language guidance for dexterous grasp data with the support of LLM. DexGYSNet dataset comprises 50,000 pairs of high-quality dexterous grasps and their corresponding language guidance, on 1,800 common household objects.

With the support of the dataset, we now turn our way to overcome the second challenge. We propose the DexGYSGrasp framework for dexterous grasp generation, which aligns with intentions, ensures high quality, and maintains diversity. At the beginning, we find the difficulty of mastering all objectives simultaneously results from the commonly used penetration loss [7] which used to avoid hand-object penetration. As shown in Figure 2, penetration loss substantially hinders the learning of grasp distribution, causing intention misalignment and reduced diversity. Conversely, despite the high diversity and aligned intention, the removal of penetration loss leads to unacceptable object penetration, making the grasp infeasible. Based on this finding, we design our DexGYSGrasp framework in a progressive strategy, decomposing the complex learning task into two sequential objectives managed by progressive components. Initially, the first component learns a grasp distribution, which focuses on intention consistency and diversity, optimizing effectively without the constraints of penetration loss. Subsequently, the second component refines the initial coarse grasps to high-quality ones with the same intentions and diversity. Our framework allows each component to focus on specific and manageable optimization objective, enhancing the overall performance of the generated grasps.

Extensive experiments are conducted on the DexGYSNet dataset and real-world scenarios. The results demonstrate that our methods are capable of generating intention-consistent, high diversity and high quality grasp poses for a wide range of objects.

2 Related work

2.1 Dexterous Grasp Generation

Dexterous hand endows robots with the capability to manipulate objects in a human-like manner. Previous methods have achieved impressive results in ensuring grasp stability by analytical approaches [11, 12, 13, 14, 15] and deep learning methods [5, 3, 7, 16, 17, 18]. However, the full potential of dexterous hands for intentional and human-like grasping has not been completely exploited

(1) w Pen.

(2) w/o Pen.

(3) ours

To use a trigger sprayer, place your forefinger on the

trigger and wrap other fingers around the bottle's body

(a) Comparison for Intention Consistency and Quality

(1) w Pen.

(2) w/o Pen.

(3) ours

To lift up a mug, use all five fingers

to firmly grasp the body of the mug.

(b) Comparison for Grasp Diversity and Quality.

in these methods. Recently, some works have focused on functional dexterous grasping [8, 9, 10, 19], aiming to achieve human-like capabilities that extend beyond grasp stability alone, but are still lack of flexibility and generalization. In this work, we explore a novel task, Language-guided Dexterous Grasp Generation, which fully leverages the dexterity of robotic hands and enable robot to execute dexterous grasp based on human natural language.

2.2 Grasp Datasets

The development of large-scale datasets has significantly improve the advancement of data-driven grasp methods, including parallel grasp [20, 21, 22, 23, 24], human grasp [25, 26, 27, 28, 29, 30], and dexterous grasp approaches [1, 2, 8, 14, 16]. Despite these advancements, the high cost of data collection remains a significant challenge, particularly in the domain of dexterous hands. Previous datasets for dexterous grasping primarily rely on physical analysis approaches [12, 31] to mitigate this issue. However, these approaches often lack the specific semantic context or corresponding language guidance necessary for constructing our language-guided dexterous task. In this paper, we present DexGYSNet dataset, with a cost-effective construction, providing high-quality dexterous grasp annotation along with flexible and fine-grained human language guidance.

2.3 Language-guided Robot Grasp

Language-guided robot grasp is important in robotics. Previous works focusing on parallel grippers have made strides in achieving task-oriented grasping [32, 23, 33], language-guided grasping [34, 35] and manipulation [36, 37, 38, 39]. In contrast to parallel grippers, dexterous hand boast a higher number of DOF (e.g., 28 for the Shadow Hand [40]), enabling a broader dexterity. However, this high freedom also presents challenges for model learning. In this paper, we propose the DexGYSGrasp framework, capable of generating intention-aligned dexterous grasps with high-quality and diversity.

3 DexGYSNet Dataset

3.1 Dataset Overview

The DexGYSNet dataset is constructed with a cost-effective strategy, as shown in Figure 3. We first collect object meshes and human grasps data from existing datasets [27]. Subsequently, we develop the Hand-Object Interaction Retargeting (HOIR) strategy to transform human grasps into dexterous grasps with high quality and hand-object interaction consistency. Finally, we implement an LLM-assisted Language Guidance Annotation system, which leverages the knowledge of Large Language Models (LLM) to produce flexible and fine-grained annotations for language guidance.

3.2 Hand-Object Interaction Retargeting

Our Hand-Object Interaction Retargeting (HOIR) aims to transfer human hand-object interaction to dexterous hand-object interaction as shown in Figure 3 . The source MANO [41] hand parameters are denoted as G m ∈ R 61 . And the target dexterous hand parameters are denoted as G dex = ( r, t, q ) ,

Contact information

：

Index  finger  touches  the  pump

head. Thumb, middle finger, ring

finger, and little finger touch the

bottle body.

Brief guidance

:

Use a lotion pump.

System:

Please

generate

a

natural

instruction

for

guiding robot grasp based on the given information.

User:

The  brief  guidance  and  contact  information  are

as following.

LLM:

To use a lotion pump, place your forefinger on

the pump head and rest other fingers on the bottle body.

initialize

fingertip

align

refine

(a) Hand-Object Interaction Retargeting

(b) LLM-assisted Language Guidance Annotation

一

S

where r ∈ SO ( 3 ) represents the global rotation, t ∈ R 3 is the translation in world coordinates, and q ∈ R J is the joint angles for a J -DoF dexterous hand, for example J = 22 for Shadow Hand[40].

Three steps are within the HOIR: pose initialization, fingertip alignment, and interaction refinement. In the first step, the dexterous poses are initialized by copying parameters from similar structures of human poses to establish better initial values. In the second step, the dexterous poses are optimized in the parameter space to align the fingertip positions p dex,ft k with those of the human p mano,ft k . This achieves retargeting consistency, and the optimization objective can be formulated as follows:

To improve the physical interaction feasibility while maintaining the consistency, the dexterous hand poses are further optimized in the third step by hand-object interaction and physical constraints losses [14]. Two key points are designed to maintain the consistency: preserving the contact area of the optimized pose consistent with the output from the second step, and keeping the translation fixed during this step. The optimization objective can be formulated as follows:

Here, the object penetration loss L pen penalizes the depth of hand-object penetration. The selfpenetration loss L spen penalizes the self-penetration. The joint angle loss L joint penalizes the out-of-limit joint angles. The contact map loss L cmap ensures the contact map on the object remains consistent with the output from the second stage. The details of losses can be found in Appendix A.1.5.

3.3 LLM-assisted Language Guidance Annotation

To annotate flexible and fine-grained language guidance for dexterous hand-object pairs with low-cost, we design a coarse-to-fine automated language guidance annotation system with the assistance of the LLM, inspired by [42, 29], as shown in Figure 3. Specially, we initially generate brief guidance based on the object category and the brief human intention (e.g., "using a lotion pump"), which are collected by the human dataset [27]. Subsequently, we compile the contact information for each finger by calculating the distances from the contact anchors on the hand to different parts of the object. We then organize the contact information into language descriptors (e.g. "forefinger touches pump head and other fingers touch the bottle body."). Finally, we input both the brief guidance and the detailed contact information into the GPT3.5 to produce natural annotated guidance (e.g. "To use a lotion pump, press down on the pump head with your forefinger while holding the bottle with your other fingers."). More details about DexGYSNet construction can be found in Appendix A.2.

0

1

2

3

4

5

Intention

CD ↓

↑

𝛿t

Pen. ↓

= 0

= 50

= 100

0 → 100

Ours

0

2

4

6

8

Diversity

= 0

= 50

= 100

Ours

0

0.2

0.4

0.6

Object Penetration

= 0

= 50

= 100

Ours

0 → 100

0 → 100

4 DexGYSGrasp framework

Given full object point clouds O and language guidance L as inputs, our goal is to generate dexterous grasps G dex with intention alignment, high diversity and high quality.

4.1 Progressive Grasp Objectives.

Learning Challenge in DexGYS. The DexGYS places high demands on intention alignment (e.g., accurately pressing your forefinger on trigger to use the sprayer), high diversity (e.g., holding the bottle using various postures), and high quality (e.g., ensuring stable grasp and avoiding object penetration). However, we find that a single model struggles to meet these requirements simultaneously, due to the optimization challenge caused by the commonly used object penetration loss [43, 14, 16], which is used to prevent hand-object penetration. As shown in Figure 2 and Figure 4, increasing the weight of the penetration loss reduces object penetration but adversely affects intention alignment and generation diversity.

Progressive Grasp Objectives. To address these challenges, we propose to decompose the complex learning objective into two more manageable objectives. The first objective is generative: it focuses on learning the grasp distribution, which does not prioritize quality but focuses on learning the grasp distribution with intention alignment and generation diversity. The second objective is regressive: it aims to refine the coarse grasp to a specific high-quality grasp with same intention. By decomposing the complex objectives, we reduce the learning difficulty of the generative objective as it does not concentrate on quality and avoids using penetration loss which could interfere the learning process. Additionally, the learning of regression is less complex than distributions, as it merely requires adjusting the pose to a specific target within a small space. Hence, we can employ penetration loss to ensure that the refined dexterous hand avoids penetrating the object and with high quality.

4.2 Progressive Grasp Components

Benefiting from our progressive grasp objectives in Section 4.1, we design the following two simple progressive grasp components, which can achieve intention alignment, high diversity and high quality language-guided dexterous generation.

Intention and Diversity Grasp Component. We introduce intention and diversity grasp component to learn a grasp distribution efficiently, achieve intention aligned and diverse generation. Due to the distribution modeling objective, IDGC is build upon the conditional diffusion model [44, 4] to predict the dexterous pose G dex 0 from noised G dex T . The input object point clouds O is encoded by Pointnet++ [45] and language L is encoded by a pretrained CLIP model [46] as the condition. And we employ DDPM [47] as sampling process, which can be formalized by the following equation:

Stop gradient

To use a trigger sprayer,

place your forefinger on

the trigger and other

fingers on bottle body.

Clip

Encoder

𝑜𝑏𝑗𝑒𝑐𝑡: 𝒪

𝑝𝑜𝑠𝑒: 𝒢

0

𝑑𝑒𝑥

Point

Encoder

Add Noise

𝒢𝑡

𝑑𝑒𝑥

𝑟

𝑡

𝑞

Penetration loss

Transformer

Decoder

Transformer

Decoder

Regression loss

Point

Encoder

𝑝𝑜𝑠𝑒:  መ

𝒢0

𝑑𝑒𝑥

𝑜𝑏𝑗𝑒𝑐𝑡: 𝒪

ℎ𝑎𝑛𝑑: ℋ

Model forward process

Penetration loss backward

Regression loss backward

(a) Intention and Diversity Grasp Component

(b) Quality Grasp Component

(c) Progressive Loss Strategy

Quality Grasp Component. The generated grasps of the first component possess well-aligned intentions and high diversity, but suffer from poor grasp quality due to significant object penetration. Therefore, we introduce Quality Grasp Component to refine the grasp quality while maintaining intention consistency in a regressive manner. Specially, it takes the coarse pose ˆ G dex , coarse hand point clouds H ( ˆ G dex ) and object point clouds O as input, and outputs the pose ∆ G dex . The refined grasp is obtained by ˜ G dex = ˆ G dex +∆ G dex . The training pairs of this component are constructed by collecting coarse grasps generated by the first component alongside the most similar ground-truth grasps that share the similar intentions. This ensures the training targets are aligned with the language intention, thereby guaranteeing that the refined grasps maintain consistency with the intended actions.

4.3 Progressive Grasp Loss

Intention and Diversity Grasp Loss. We strategically employ regression losses and exclude object penetration loss to enhance the training efficacy of intention and diversity grasp component. By focusing exclusively on the regression learning, this component facilitates a more effective optimization process, achieving enhancements of intention consistency and grasp diversity. Concretely, we utilize L2 loss for pose parameter regression and incorporate the hand chamfer loss [48] to assist by explicit hand shape. The loss function of intention and diversity grasp component.is defined as:

where H are dexterous hand point clouds of corresponding pose.

Quality Grasp Loss. Benefiting from the simplified training objectives, the quality grasp component focuses solely on refining coarse grasp to a specific target within a relatively constrained space, thereby reducing the negative impact of object penetration. Therefore, we employ the well-designed loss including object penetration. The loss function of quality grasp component can be formulated as:

More details about loss function and model structure can be found in Appendix A.1.

5 Experiments

5.1 Datasets and Evaluation Metrics

We split the DexDYSNet dataset at the object instance level, using 80% of the objects within each category for training and 20% for evaluation. Notably, none of the objects in the test set appear in the training set, ensuring that all experimental results are evaluated on unseen objects.

Three types of metrics are employed for evaluation from the perspective of intention consistency, grasp quality and grasp diversity. 1) For intention consistency, we employ Fréchet Inception Distance (FID), using sampling point cloud features extracted from [49] to calculate P -FID and rendering image features extracted from [50] to calculate FID . Additionally, Chamfer distance

To use a trigger sprayer,

pose your forefinger on

the trigger and the other

fingers on the bottle body.

To use binoculars,

place all fingers on

the objective lens for

stability and clarity.

To use a lotion pump,

place your forefinger on

the pump head and other

fingers on the bottle body.

To hold a lotion pump,

use all five fingers to grip

the bottle body securely.

To use a screwdriver,

grip the handle with all

five fingers for stability.

To grasp a cylinder bottle,

wrap your five fingers

around the bottle body.

To use a pen,

hold it between

your fingers to

write."

To grasp a wineglass,

use your fingers to hold

the stem securely.

Hand over the binoculars

using all fingers to gently

grasp the objective lens.

To use a pincer, grip the

handle with your fingers.

To use a camera, grip and

control the panel with your

thumb and forefinger, using

other finger to support.

Hand over the power drill by

grasping it with all five

fingers around the drill motor.

( CD ), is used to measure the distance between predicted hand point clouds and targets; Contact distance ( Con. ) is used to measure the L2 distance of object contact map between the prediction and targets. 2) For grasp quality, Success rate in Issac gym and Q 1 [13] measure grasp stability. We set the contact threshold to 1 cm and set the penetration threshold to 5 mm following [14]. Maximal penetration depth (cm), denoted as Pen. , reflects the maximal penetration depth from the object point cloud to hand meshes. 3) For diversity, we employ the Standard deviation of translation δ t , rotation δ r and joint angle δ q of eight samples within same condition, following [7]. More details can be found in Appendix A.3.2.

5.2 Implementation Details

For the construction of DexGYSNet, the step 2 and 3 are optimized for 20 and 300 iterations with learning rates of 0.01 and 0.0001 respectively. We set λ 1 pen = 100 and set λ 1 spen , λ 1 joint , λ 1 cmap each to 10. For training our framework, the training epochs are set to 100 for intention and diversity grasp component and 20 for Quality Grasp Component. The loss weights are configured as follows: λ 2 para = λ 3 para = 10 , λ 2 chamfer = λ 3 chamfer = 1 , λ 3 cmap = 10 , λ 3 pen = 100 , λ 3 spen = 10 . Throughout all training processes, the model is optimized with a batch size of 64 using the Adam optimizer, with a weight decay rate of 5 . 0 × 10 -6 . The initial learning rate is 2 . 0 × 10 -4 and decay to 2 . 0 × 10 -5 using a cosine learning rate [52] scheduler. All experiment are implemented with PyTorch on a single RTX 4090 GPU.

5.3 Comparison with SOTA methods

The comparison results are presented in Table 1. We reproduce the SOTA methods to suit our task by concatenating the language condition with the point cloud features, the details can be found in Appendix A.3.3. As seen in the Table, our framework significantly outperforms all previous

Table 3: Ablation study for HOIR.

Table 4: Plug-and-play Experiments.

methods in terms of intention consistency and grasp diversity, while also achieving comparable performance in grasp quality. Previous methods struggle with learning a robust language conditional grasp distribution due to the optimization challenges outlined in Section 4.1. They often yield misaligned yet high quality grasps, resulting in comparable grasp quality, but less aligned intention and limited diversity compared to our framework. Overall, these results confirm that our framework achieves SOTA performance in generating intention-aligned, high-quality and diverse grasps.

In Figure 6, we visualize the generated grasp to qualitatively demonstrate the grasp generation capabilities of our framework. The bottom figure visualizes the results of four samples, the bottom left highlights our framework's ability to produce precise and consistent grasps under deterministic guidance (e.g., the way to use a trigger sprayer is deterministic). In the other hand, the bottom right illustrates our framework's diversity in generating grasps when provided with ambiguous guidance (e.g., the way to hold a bottle is diverse).

5.4 Necessity of Progressive Components and Losses

The results presented in Table 2 validate the core insight of our framework: decomposing the complex task into progressive objectives, employing progressive components, and learning with progressive losses. The initial four lines of results demonstrate that a single component, without progressive objectives, fails to balance all objectives. Moreover, a single component, even with progressive objectives, that adjusts λ 2 pen from 0 to 100 after several training epochs, does not enhance performance. The similar result occurs when using progressive components without corresponding progressive losses, IDGC ( λ 2 pen = 100) + QGC . Moreover, the commonly used quality refinement strategy test-time adaptation (TTA) [43], though improves grasp quality but results in extremely poor intention consistency. Overall, only the progressive designs of our DexGYSGrasp framework ensures excellence in intention alignment, high quality and diversity.

5.5 Plug-and-play Experiments

We conducted experiments to evaluate the applicability of our insights to other state-of-the-art (SOTA) methods. Specifically, we trained GraspCAVE and SceneDiffuser without the object penetration

To hold a power drill, use all fingers

to grip the drill motor securely.

To use a power drill, press the drill

trigger with your forefinger and grip

it with your other fingers.

To use a hammer, grasp it with

all fingers to secure the hammer grip.

To hand over the hammer, grip it

with all your fingers around

the hammer head.

Hold the trigger sprayer with

your fingers in contact

with the bottle body.

To use a trigger sprayer, press the

trigger with your forefinger and hold

the bottle with your other fingers.

Use your fingers to grip the stem of

the wineglass securely.

To hold a wineglass, use all fingers

to grip the bowl securely.

To use a game controller, grip it with

fingers touching the controller panel.

Hand over the game controller by

grasping it with fingers on the

controller panel and handles.

Grasp the frying pan by contacting

the handle with all fingers.

Hold the frying pan with

two fingers supporting

the body of the pan.

A

89

constraint and trained the quality grasp component (QGC) to refine the coarse outcomes. As depicted in Table 4, removing the object penetration loss leads to improved intention consistency, which corroborates our findings discussed in Section 4.1. Moreover, our quality grasp component can significantly enhance grasp quality while maintaining the intention consistency.

5.6 Effectiveness of Hand-Object Interaction Retargeting

We conducted ablation studies to evaluate our Hand-Object Interaction Retargeting (HOIR) strategy in constructing DexGYSNet dataset. As shown in Table 3, our three-step HOIR significantly improves both the quality and the intention consistency progressively. We observed that optimizing all losses in Equations 1 and 2 in one step ( all in one stage ), results in worse contact consistency and better grasp quality. Similar outcomes occur when the root translation is not fixed in step 3 ( w/o fix translation ). We believe this trade-off arises from inherent noise in the hand-object interaction data and the structural differences between human grasps and dexterous hands, making it challenging to excel in all aspects. Overall, we think that three-step HOIR strategy achieves more comprehensive outcomes, especially in the most important aspect of hand object contact consistency.

5.7 Experiments in Real World

We conducted real-world grasp experiments to verify the practical application of our methods, as shown in Figure 7. The experiments are conducted on an Allegro hand, a Flexiv Rizon 4 arm and an Intel Realsense D415 camera. Although our framework is designed for full object point clouds, we integrate several off-the-shelf methods to enhance its practicality. Specifically, partial object point clouds are obtained through visual grounding [53] and SAM [54], which are then fed into a point cloud completion network [55] to obtain full point clouds. In execution, we first move the arm to the 6-DOF pose of the dexterous hand root node, and then control the dexterous hand joint angles to the predicted poses. Real world experiments further validate the effectiveness of our method. More implementation details can be found in Appendix A.5.

6 Conclusions

We believe that enabling robots to perform high quality dexterous grasps aligned with human language is crucial within the deep learning and robotics communities. In this paper, we explore this novel task, 'Dexterous Grasp as You Say' (DexGYS). This task is non-trival, we propose a DexGYSNet dataset and a DexGYSGrasp framework to accomplish it. DexGYSNet dataset is constructed cost-effectively using the object-hand interaction retargeting strategy and the language guidance annotation system assisted by LLMs. Building on DexGYSNet, DexGYSGrasp framework, comprised of two progressive components, which can achieve intention-aligned, high diversity, and high quality dexterous grasp generation. Extensive experiments in DexGYSNet and real-world settings demonstrate that our framework significantly outperforms all SOTA methods, confirming the potential and effectiveness of our approach.

References

Min Liu, Zherong Pan, Kai Xu, Kanishka Ganguly, and Dinesh Manocha. Deep differentiable grasp planner for high-dof grippers. arXiv preprint arXiv:2002.01530 , 2020.

Puhao Li, Tengyu Liu, Yuyang Li, Yiran Geng, Yixin Zhu, Yaodong Yang, and Siyuan Huang. Gendexgrasp: Generalizable dexterous grasping. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , 2023.

Lin Shao, Fabio Ferreira, Mikael Jorda, Varun Nambiar, Jianlan Luo, Eugen Solowjow, Juan Aparicio Ojea, Oussama Khatib, and Jeannette Bohg. Unigrasp: Learning a unified model to grasp with multifingered robotic hands. IEEE Robotics and Automation Letters , 2020.

Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song-Chun Zhu. Diffusion-based generation, optimization, and planning in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Jiaxin Lu, Hao Kang, Haoxiang Li, Bo Liu, Yiding Yang, Qixing Huang, and Gang Hua. Ugg: Unified generative grasping. arXiv preprint arXiv:2311.16917 , 2023.

Zehang Weng, Haofei Lu, Danica Kragic, and Jens Lundell. Dexdiffuser: Generating dexterous grasps with diffusion models. arXiv preprint arXiv:2402.02989 , 2024.

Guo-Hao Xu, Yi-Lin Wei, Dian Zheng, Xiao-Ming Wu, and Wei-Shi Zheng. Dexterous grasp transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024.

Jiayi Chen, Yuxing Chen, Jialiang Zhang, and He Wang. Task-oriented dexterous grasp synthesis via differentiable grasp wrench boundary estimator. arXiv preprint arXiv:2309.13586 , 2023.

Tianqiang Zhu, Rina Wu, Jinglue Hang, Xiangbo Lin, and Yi Sun. Toward human-like grasp: Functional grasp by dexterous robotic hand via object-hand semantic representation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Wei Wei, Peng Wang, and Sizhe Wang. Generalized anthropomorphic functional grasping with minimal demonstrations. arXiv preprint arXiv:2303.17808 , 2023.

Jean Ponce, Steve Sullivan, J-D Boissonnat, and J-P Merlet. On characterizing and computing threeand four-finger force-closure grasps of polyhedral objects. In [1993] Proceedings IEEE International Conference on Robotics and Automation , pages 821-827. IEEE, 1993.

Tengyu Liu, Zeyu Liu, Ziyuan Jiao, Yixin Zhu, and Song-Chun Zhu. Synthesizing diverse and physically stable grasps with arbitrary hand structures using differentiable force closure estimator. IEEE Robotics and Automation Letters , 7(1):470-477, 2021.

Carlo Ferrari, J Canny, et al. Planning optimal grasps. In Proceedings., 1992 IEEE International Conference on Robotics and Automation, 1992. , 1992.

Ruicheng Wang, Jialiang Zhang, Jiayi Chen, Yinzhen Xu, Puhao Li, Tengyu Liu, and He Wang. Dexgraspnet: A large-scale robotic dexterous grasp dataset for general objects based on simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , 2023.

Dylan Turpin, Liquan Wang, Eric Heiden, Yun-Chun Chen, Miles Macklin, Stavros Tsogkas, Sven Dickinson, and Animesh Garg. Grasp'd: Differentiable contact-rich grasp synthesis for multi-fingered hands. In European Conference on Computer Vision , 2022.

Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Kelin Li, Nicholas Baron, Xian Zhang, and Nicolas Rojas. Efficientgrasp: A unified data-efficient learning to grasp method for multi-fingered robot hands. IEEE Robotics and Automation Letters , 2022.

Jacob Varley, Jonathan Weisz, Jared Weiss, and Peter Allen. Generating multi-fingered robotic grasps via deep learning. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2015.

Tianqiang Zhu, Rina Wu, Xiangbo Lin, and Yi Sun. Toward human-like grasp: Dexterous grasping via semantic representation of object-hand. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15741-15751, 2021.

Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu. Graspnet-1billion: A large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11444-11453, 2020.

Clemens Eppner, Arsalan Mousavian, and Dieter Fox. Acronym: A large-scale grasp dataset based on simulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 6222-6227. IEEE, 2021.

Xiao-Ming Wu, Jiafeng Cai, Jian-Jian Jiang, Dian Zheng, Yi-Lin Wei, and Wei-Shi Zheng. An economic framework for 6-dof grasp detection. In European Conference on Computer Vision , 2024.

Chao Tang, Dehao Huang, Lingxiao Meng, Weiyu Liu, and Hong Zhang. Task-oriented grasp prediction with visual-language inputs. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 4881-4888. IEEE, 2023.

Jia-Feng Cai, Zibo Chen, Xiao-Ming Wu, Jian-Jian Jiang, Yi-Lin Wei, and Wei-Shi Zheng. Real-to-sim grasp: Rethinking the gap between simulation and real world in grasp detection. In Conference on Robot Learning , 2024.

Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11807-11816, 2019.

Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9044-9053, 2021.

Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A largescale knowledge repository for understanding hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20953-20962, 2022.

Juntao Jian, Xiuping Liu, Manyi Li, Ruizhen Hu, and Jian Liu. Affordpose: A large-scale dataset of hand-object interactions with affordance-driven hand pose. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14713-14724, 2023.

Kailin Li, Jingbo Wang, Lixin Yang, Cewu Lu, and Bo Dai. Semgrasp: Semantic grasp generation via language aligned discretization. arXiv preprint arXiv:2404.03590 , 2024.

Yan-Kang Wang, Chengyi Xing, Yi-Lin Wei, Xiao-Ming Wu, and Wei-Shi Zheng. Single-view scene point cloud human grasp generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024.

Andrew T Miller and Peter K Allen. Graspit! a versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine , 11(4):110-122, 2004.

Adithyavairavan Murali, Weiyu Liu, Kenneth Marino, Sonia Chernova, and Abhinav Gupta. Same object, different grasps: Data and semantic knowledge for task-oriented grasping. In Conference on Robot Learning , pages 1540-1557. PMLR, 2021.

Chao Tang, Dehao Huang, Wenqi Ge, Weiyu Liu, and Hong Zhang. Graspgpt: Leveraging semantic knowledge from a large language model for task-oriented grasping. IEEE Robotics and Automation Letters , 2023.

Kechun Xu, Shuqi Zhao, Zhongxiang Zhou, Zizhang Li, Huaijin Pi, Yifeng Zhu, Yue Wang, and Rong Xiong. A joint modeling of vision-language-action for target-oriented grasping in clutter. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 11597-11604. IEEE, 2023.

Shiyu Jin, Jinxuan Xu, Yutian Lei, and Liangjun Zhang. Reasoning grasping via multimodal large language model. arXiv preprint arXiv:2402.06798 , 2024.

Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning , pages 991-1002. PMLR, 2022.

Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters , 7(4):11205-11212, 2022.

Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, and Aakanksha et al. Chowdhery. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.03378 , 2023.

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL) , 2021.

Shadowrobot. https://www.shadowrobot.com/dexterous-hand-series/ , 2005.

Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands. ACM Transactions on Graphics , 36(6):1-17, 2017.

Jieming Cui, Tengyu Liu, Nian Liu, Yaodong Yang, Yixin Zhu, and Siyuan Huang. Anyskill: Learning open-vocabulary physical skill for interactive agents. In Conference on Computer Vision and Pattern Recognition(CVPR) , 2024.

Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. In Proceedings of the International Conference on Computer Vision , 2021.

Julen Urain, Niklas Funk, Jan Peters, and Georgia Chalvatzaki. Se (3)-diffusionfields: Learning smooth cost functions for joint grasp and motion optimization through diffusion. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , 2023.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in Neural Information Processing Systems , 2017.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017.

Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751 , 2022.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems , 30, 2017.

Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in Neural Information Processing Systems , 2015.

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations , 2016.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 , 2023.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015-4026, 2023.

Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pcn: Point completion network. In 2018 International Conference on 3D Vision (3DV) , pages 728-737. IEEE, 2018.

Chenxi Wang, Hao-Shu Fang, Minghao Gou, Hongjie Fang, Jin Gao, and Cewu Lu. Graspness discovery in clutters for fast and accurate grasp detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 2017.

W. Robotics. Allegro robot hand. https://www.wonikrobotics.com/research-robot-hand .

Kenneth Shaw, Ananye Agarwal, and Deepak Pathak. Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning. arXiv preprint arXiv:2309.06440 , 2023.

Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5152-5161, 2022.

A Appendix / supplemental material

A.1 DexGYSGrasp Details

A.1.1 Diffusion Background

The diffusion model is used in our intention and diversity grasp component to generate grasp distribution with aligned intention and high diversity, which represents a class of generative models characterized by a forward process of noise addition and a reverse process of denoising. The forward process entails a Markov Chain that incrementally introduces Gaussian noise into the data across multiple time steps. Originating from the initial data x 0 , this process transitions the data to conform with a standard Gaussian distribution x T after T time steps. This transformation is mathematically formulated as follows: √

where α t denotes a time-dependent noise coefficient, ¯ α t = ∏ t i =1 α i .

Therefore, x t given x 0 follows a normal distribution, √

The first equation delineates the stepwise diffusion, whereas the second equation offers a direct approximation of any intermediate state x t from x 0 .

During the reverse process, the model is trained to closely approximate the reverse conditional distribution p ( x t -1 | x t ) , which is described as:

where µ θ ( x t , t ) and σ 2 θ ( x t , t ) are the mean and variance parameters for the distribution of x t -1 , respectively, and θ indicates the parameters of the model used to predict ϵ from x t .

The classical sampling strategy for the reverse process is exemplified by DDPM [47], where the model iteratively learns to reverse the noise addition process to reconstruct the original data from noise. It estimates the distribution p ( x t -1 | x t ) and predicts the noise ϵ , represented by:

where σ θ consists of non-trainable, time-dependent constants, and z represents Gaussian noise.

A.1.2 Intention and Diversity Grasp Component

Point Encoder We utilize a three-layer PointNet++ [45] as our point encoder, following recent works [56, 14, 2, 7] in the field of robotic grasping. Specifically, each layer l i , i ∈ 1 , 2 , 3 , receives point clouds and corresponding features (initially the raw XYZ coordinates for the first layer) from the preceding layer. It then performs down-sampling and feature aggregation using the "set-aggregation" operation [45]. The aggregated features are processed by a three-layer perceptron, which consists of three Linear -BatchNorm -ReLU blocks. The output of point encoder is F obj ∈ R N obj × C obj .

Language Encoder For the language encoder, we employ the CLIP model with the ViT-L/14 architecture [46]. The input text sequence is tokenized and converted into token embeddings with positional embeddings added. This sequence is processed through multiple Transformer encoder layers to obtain the language feature F lan ∈ R N lan × C lan .

Transformer Decoder We employ four layers of MLPs and Transformer as decoder[57, 4]. The time embedding and pose feature are incorporated in MLPs to obtain F dex _ t . Subsequently, the F dex _ t serves as the query, and the concatenated features of language and object, F lan _ obj , act as key and value in Transformer block. The corss attention process is formalized as:

where f q , f k , f v are MLPs, and d k is the channel of features. Finally, a MLP is adopted to regress the dexterous grasp parameters ˆ G dex . F dex _ t = f 0 ( f 1 ( F dex ) + f 2 ( F time )) , where f 0 , f 1 , f 2 are MLPs.

To use a trigger sprayer,

place your forefinger on

the trigger and other

fingers on bottle body.

Clip

Encoder

𝑜𝑏𝑗𝑒𝑐𝑡: 𝒪

Point

Encoder

𝒢

!

"#$

𝑟

𝑡

𝑞

Transformer

Decoder

Transformer

Decoder

Point

Encoder

𝑝𝑜𝑠𝑒:	 -

𝒢

%

"#$

𝑜𝑏𝑗𝑒𝑐𝑡: 𝒪

ℎ𝑎𝑛𝑑:	ℋ

(a) Intention and Diversity Grasp Component

(b) Quality Grasp Component

A.1.3 Quality Grasp Component

Quality grasp component tasks coarse pose ˆ G dex , coarse hand point clouds H ( ˆ G dex ) and object point clouds O as input, and outputs refined grasp ˜ G dex . The object and hand are encoded by the PointNet++ same with intention and diversity grasp component. And in the transformer decoder, coarse pose features act as the query, object and hand features serve as key and value.

A.1.4 Inference Pipeline

We also demonstrate the inference pipeline of our DexGYSGrasp, as shown in Fig. 8. We sample a random noise from Gaussian distribution as the input, with the point cloud and language guidance as the conditions. We first generate the coarse grasp by the intention and diversity grasp component, and then refine it with the quality grasp component.

A.1.5 Loss Function

This section provides a detailed exposition of the loss functions utilized during the construction of datasets and the training of models.

Parameter Regression Loss. We utilize the mean squared error (MSE) to quantify the deviation between the generated dexterous hand pose ˆ G dex and the ground truth G dex .

Hand Chamfer Loss. The predicted hand point clouds H ( ˆ G dex ) and the ground truth H ( G dex ) are derived by sampling from the hand mesh. We then compute the chamfer distance to assess the discrepancies between the predicted and ground-truth hand shapes.

Contact Map Loss. The contact map loss L cmap ensures consistency between the predicted hand contact map ˆ c obj on object and the target c obj . The contact map is calculate by the distance from object point to the closest dexterous hand point.

Object Penetration Loss. The object penetration loss L pen penalizes the depth of hand-object penetration, where d sdf i denotes the signed distance from the object point to the hand mesh.

Self-Penetration Loss. The self-penetration loss L spen punishes the penetration among the different parts of the hand, where p dex,sp denotes predefined anchor spheres on the hand [14].

̸

To grasp a bottle, use

your four fingers to

wrap around the

body of the bottle.

To hold a camera,

ensure all fingers make

contact with the camera

body for a secure grip.

To use a frying pan,

grip the pan handle with

all five fingers for

better control.

To use a game controller,

make sure fingers are in

contact with  controller

panel and handle.

To lift up a mug, ensure

all five fingers make

contact with the mug's

body.

Hand over the pincer by

using your fingers to

grip the handle.

Shadow Hand

Allegro Hand

Language

Guidance

Human Hand

Joint Angle Loss. Given the physical structure limitations of the robotic hand, each joint has designated upper and lower limits. The joint angle loss penalizes deviations from these limits.

A.2 DexGYSNet Datasets Details

A.2.1 Prompt of LLM

We introduce the prompt for using GPT-3.5 in this section.

System Prompt : "You are an assistant in creating language instruction, aimed at guiding robot on how to grasp objects. Given a brief instruction and a fine-gained interaction information. Your task is generate a natural and more informative instruction. The instruction should start with the given brief instruction, which is limited in a sentence and about 10-15 words."

User Prompt : "Brief instruction: To < brief intention > a < object category >. Hand-object interaction information: < contact information >. "

The brief intention and object category are sourced from the hand-object dataset OakInk [27]. The contact information is derived by calculating the distances from predefined contact anchors on each finger to the segmentation parts of the object. Details on predefined contact anchors are available in DexGraspNet [14], and segmentations are annotated in OakInk [27].

An example of user prompt is: "Brief instruction: To use a trigger sprayer. Hand-object interaction information: forefinger touches the trigger. thumb, middle finger, ring finger and little finger touches the finger." An example of LLM output is: "To use a trigger sprayer, press the trigger with your forefinger and hold the bottle with your other fingers."

A.2.2 DexGYSNet Extension

Our cost-effective dataset construction strategy can be easily extended to various types of dexterous hands. As shown in Figure 9, besides the Shadow Hand[40], which features a highly biomimetic design replicating most degrees of freedom of human hands at a high cost of $100,000, we also expand to the Allegro Hand [58] and Leap Hand [59]. These latter models, while offering fewer

P-FID

FID

sampling

point clouds

rendering

images

To hold a bottle, wrap your hand around

its body with all fingers touching it.

Point

Encoder

Image

Encoder

Please evaluate the content of the 'robot grasp

object'  image  and  the  description  text,  and

output  a  consistency  score  from  1  to  5  based

on whether they share the same intention.

Consistency score is 5.

Rendering Views

(a) The evaluation pipeline of P-FID and FID.

(b) LLM-assisted evaluation.

degrees of freedom, are significantly more affordable, costing $16,000 and $2,000, respectively, making them practical for promoting the use of robotic arms in laboratory environments. We have trained our method on the DexGYSNet dataset using the Allegro Hand and implemented it in real robot experiments.

A.3 Implementation Details

A.3.1 Dataset Split

We split the DexDYS dataset at the level of object instances. Specially, for all objects within each category, 80% of the objects instances are used for training and 20% for evaluation. Concretely, the training set includes approximately 1,200 objects with 40k grasps, while the evaluation set comprises about 300 objects with 10k grasps. Therefore, all objects in the test set of DexGYSNet don't exist in the training set.

A.3.2 Metrics Detials

Target Assignment. For target assignment in the testing phase, the grasp targets of an object-guidance pair consist of all poses that share the same contact part and brief guidance. And the matrices of intention consistency are calculated by comparing the prediction to the most similar grasp target.

Fréchet Inception Distance , which is commonly used in generative task[60] by measuring the distance between the generated distribution and the ground truth distribution. We use sampling point cloud features extracted from[49] to calculate P-FID and rendering image features extracted from[50] to calculate FID. The details are shown in Figure 10 (a).

Chamfer distance, denoted as CD , is used to measure the distance between predicted hand point clouds and targets to measure the consistency from the aspect of hand consistency. Please look at Equation 14 for details.

Contact distance, denoted as Con. to measure the L2 distance of object contact map between the prediction and targets to measure the consistency from the aspect of object contact consistency. Please look at Equation 15 for details.

Success rate. We evaluate the grasp success rate in Issac Gym simulation environment. To simulate the force exerted by dexterous hands grasping objects in real environments, we contract each finger in the direction of the object. If the grasp can withstand at least one of the six directions of gravity, it is considered successful.

Mean Q 1 . Intuitively, the Q 1 metric reflects the norm of the smallest wrench which can disrupt the stability of a grasp. We follow [14] to set the contact threshold to 1cm and set the penetration threshold to 5mm. Any grasp with its maximal penetration depth greater than 5mm is considered invalid and we set the Q 1 of it to 0 before taking the average.

Maximal penetration depth, which is the maximal penetration depth from the object point cloud to hand meshes.

Diversity. We use the standard deviation of translation δ t , rotation δ r , and joint angle δ q to measure the diversity of generated grasps. We perform eight samples in the intention and diversity component under the same input conditions, and each sample is individually sent to the quality component

O

use

hold

use

hold

use

hold

for refinement. Before calculation, δ r and δ q are converted to Euler angles in degrees, while δ t is measured in centimeters.

A.3.3 Implementation Details of SOTA Methods

We replicate SOTA methods on our DexGYSNet dataset using the same encoder structure and the loss functions defined in Equation 5 to ensure fair comparison. Specifically, we reimplement GraspCAVE based on [51], GraspTTA [43], SceneDiffuser [4], and DGTR [7]. To introduce language information, we use an identical CLIP language encoder. For GraspCAVE, we concatenate the language feature, object feature, and latent feature to send to the decoder. Based on GraspCAVE, GraspTTA employs a test-time adaptation strategy for quality refinement. For SceneDiffuser, we concatenate the language and object features as the model condition. For DGTR, the language and object features are concatenated to send to its transformer decoder.

A.4 Additional Experiments

A.4.1 Qualitative Experiments of Quality Grasp Component.

We provide additional qualitative results to verify the effectiveness of Quality Grasp Component. Figure 11 shows the grasps before and after the application of the Quality Grasp Component, demonstrating that QGC can prevent object penetration and maintain consistency with the original intention.

A.4.2 Qualitative Experiments of Task-oriented Guidance.

We conduct qualitative experiments to demonstrate the generalization of our DexGYSGrasp framework to task-oriented or functional grasp task. Specifically, we input task-oriented guidance (e.g., "use" or "hold") into our framework, which has been trained on DexGYSNet. As shown in Figure 12, our DexGYSGrasp framework exhibits good compatibility with these inputs. This further confirms that our approach enables more flexible and natural human-robot interactions.

Realsense

D415 Camera

Flexiv

Allegro

Hand

Objects

System setup

Object set

SAM

Grounding

DINO

Bounding Box

Segmentation

Partial Point Clouds

Completion

A.5 Real World Experiments Details

Experimental Environment Figure 13 shows the settings of our real world experiments. The experiments are conducted on Allegro hand, a Flexiv Rizon4 arm and an Intel Realsense D415 camera. The experimental object is a 3D printed object from test set of DexGYSNet.

Experiment Pipeline Our DexGYSGrasp takes full point clouds as input following recent works in dexterous grasping [14, 4, 7]. To make our methods more practical, we employ three off-the-shelf models in a cascade to obtain a full point cloud from scene point cloud. As shown in Figure 14, we input the object category and the RGB image into an open-set detection model [53] to detect the bounding box of the object. This bounding box is then used as a prompt for SAM [54] to obtain the segmentation of the object. Next, we crop the target depth image using the segmentation map and the depth input. Finally, we convert the partial depth image into point clouds and feed it into a point completion network [55] to obtain the final full point clouds. Then, the full point clouds are fed into our framework to obtain the dexterous grasp pose, which is then transformed into the real coordinate system. In execution, we first move the arm to the 6-DOF pose of the dexterous hand root node, and then control the joint angles to achieve the target pose.

Experiment Results The experiment results are presented in Table 5. For each object, we command robot with different language instruction, and each instruction is tested five times, resulting in a total of ten grasping trials per object. A grasp is deemed successful if it aligns with the intended instruction and maintains stability, preventing the object from falling. Our method demonstrates a moderate success rate, indicating its effectiveness. Further research on real-world scenarios is recommended to enhance the robustness of our approach.

A.6 Societal Impacts and Limitations

The core innovation of this paper has a significant positive impact on society. We propose a novel task: language-guided dexterous grasp generation, which can promote human-robot interaction and expedite the deployment of robots in real-world scenarios. Additionally, we introduce an innovative framework to accomplish this task. Our approach can generate high-quality grasps while ensuring consistency of intent and diversity of grasps.

However, our method still faces some challenges in real-world deployment. Due to limitations in the current development of robotic arm control and physical structures, we cannot guarantee success in every grasp execution in the real world. In future work, we will further enhance the quality of grasp generation to improve the success rate in real-world scenarios.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations of the work in the conclusion and supplementary material.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully disclose all the information needed to reproduce the main experimental results of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer:[No]

Justification: We provide examples of our dataset in supplementary materials. We promise to release all code and the complete dataset after the publication of this paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all the training and test details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not reported because performing multiple runs for each experiment would be too computationally expensive.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides sufficient information on the computer resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses both potential positive societal impacts and negative societal impacts of the work in the conclusion and supplementary material.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The original owners of assets, used in the paper, are properly credited and the license and terms of use are explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: New assets introduced in the paper are well documented and the documentation is provided alongside the assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel robotic grasping task that utilizes natural language as guidance to generate grasping poses for dexterous hands that satisfy corresponding intentions. To accomplish this task, the paper first creates a large-scale language-guided dexterous grasping dataset. Moreover, the authors propose a progressive solution composed of a diffusion-based generative model and a regressive model to achieve generating intention-consistent, high diversity, and high-quality grasp poses for a wide range of objects.\n\nSTRENGTHS:\n1. Compared to existing work, this work advances further by posing a more general and challenging problem and offering a more comprehensive solution. The introduction of this problem promotes the direction of general robotic grasping.\n2. The paper contributes a grasping dataset that includes 18,000 objects with a total of 50,000 text-grasp pose pairs. This dataset fully utilizes the information provided by the Oakink dataset and transfers human hand poses to robotic hands in three steps. During the language annotation stage, the paper generates corresponding language descriptors by measuring the distance from the fingers to different regions of the object, then uses GPT to generate natural language. The entire process is technologically sound.\n3. The paper adopts a progressive framework to address this issue, utilizing a diffusion-based generative model and a regressive model to ensure that the generated poses are both diverse and high-quality.\n\nWEAKNESSES:\nWeaknesses\n1. When verifying intention consistency, the paper uses chamfer distance and contact distance to compare the error between the estimated poses and the target poses. However, since the intention is derived from natural language, which is inherently imprecise and non-unique, using a collected target pose as the sole standard is inappropriate. For instance, for \"to use a lotion pump, place your forefinger on the pump head and other fingers on the bottle body,\" there might be infinite ways to hold the lotion with the forefinger on the pump head, some of which might be different to the target one. Therefore, chamfer distance and contact distance may not be the best metrics.\n2. The dataset collected in the paper includes only 50,000 grasping poses. Compared to existing datasets for parallel-jaw grippers, which contain millions or even billions of poses, the diversity is limited. For a dexterous hand, which has a higher degree of freedom, collecting more data is inevitable to achieve more varied grasping methods. I believe the performance on unseen categories would degrade significantly.\n3. The authors mentioned that they use 20% of the data as a test set to evaluate the model's performance. However, they do not provide more specific information about the training and test sets. For example, whether the target poses and description texts of the same category of objects in the training and test sets are consistent, and what the results would be if given inconsistent descriptions, such as in Figure 6, \"To grasp a wineglass, use your fingers to hold the stem securely\" if changed to \"To grasp a wineglass, use your fingers to hold the GLASSES securely\" (I think this description will not exist in the dataset).\n4. Although the authors use the Q1 metric to measure grasp stability, this analytical approach still has limitations. The authors could try using a simulator (e.g., Isaac Sim) to test the grasp success rate, which would be more indicative.\n\nOther Comments\n1. The title of A.1.4 is \"inference time,\" but the content is unrelated.\n2. The real-world experiments in the video are not very promising. In the given demos, non-target areas of the objects are fixed by fixtures, leaving only the target areas available for grasping.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles language-guided dexterous grasping by creating a large-scale dataset pairing grasps and language guidance and presenting a grasp generation pipeline based on language instructions. The dataset, DexGYSNet, leverages HOIR and LLMs to scale up the size of grasps and annotations while keeping the generation costs manageable. The DexGYSGrasp framework introduces a two-stage progressive generation to balance diversity, quality, and grasp-language alignment. The method is evaluated on the intention consistency, grasp quality, and diversity against SOTA methods. Experiments demonstrate the necessity of progressive components, effectiveness of HOIR, potential to transfer to real-world scenarios. Overall, the paper advances the field of dexterous grasp generation by natural language guidance, providing a valuable dataset, and proposing a robust framework for generating high-quality robotic grasps based on human instructions.\n\nSTRENGTHS:\nThis problem of language-guided dexterous grasping is significant. This paper presents both a large-scale dataset and a method to generate diverse grasps. The dataset is a valuable resource for the community and the method can be integrated into a grasping policy and applied to real-world scenarios. The design of experiments justifies the choice to break the generation process into two stages and demonstrates the effectiveness of the method in improving grasp quality, diversity, and alignment with language instructions.\n\nWEAKNESSES:\n1. The method requires a full point cloud and is designed for static pose generation. In real-world applications, access to a complete point cloud is often unavailable, although this could be mitigated by using a point cloud completion network. Moreover, grasping is typically a precursor to a subsequent action, requiring an end-to-end pipeline that can handle both grasping and the intended use of the object. An integrated approach would likely be more effective than segregating these steps into different modules. For instance, the subsequent policy must not only use the generated pose but also comprehend instructions (again, since the generated pose alone is not enough) and object geometry to perform the correct actions, which an end-to-end system could potentially handle more seamlessly. Plus, there can be slight adjustments to gestures after grasping a tool from a table but before using the tool. In this sense, real world experiments on this synthesis method are not so legitimate in the first place.\n\n2. The paper does not sufficiently address how the method generalizes to unseen objects or categories. Given the diversity of real-world objects, it is crucial for the network to perform well on unseen items to be practically useful. Including more detailed experiments and analysis on in-category and out-of-category generalization would significantly strengthen the work. This could involve testing the method on a wider variety of objects and categories not present in the training dataset and assessing its adaptability given that OakInk has the category information.\n\nQUESTIONS:\n1. The paper states that intention consistency is calculated by comparing the prediction to the most similar grasp target. While this approach addresses the issue of multiple valid grasp poses, could you provide more details on how the diversity of valid grasp poses is represented in your dataset and how it influences the evaluation metrics?\n2. The paper does not seem to explicitly address how the proposed method generalizes to different objects, unseen objects, or even unseen categories. The dataset is derived from OakInk, which has category information. Could you provide more details on the performance and adaptability of your method when applied to objects and categories not present in the training dataset? This generalizability is important for the network to understand the correspondence between the geometry and functional semantics. In-category and out-of-category generalization are both essential for a capable network.\n\nLIMITATIONS:\n1. The method is not yet ready for real-world applications due to several factors. It requires a full point cloud, which needs several adds-on modules like segmentation and completion for deployment. Additionally, there is a lack of discussion on how this method would benefit the subsequent grasping execution and how the grasped objects can be manipulated, which is crucial for practical use.\n2. The paper does not sufficiently address how the method generalizes to unseen objects or categories. Ensuring that the method can handle a wide range of objects, including those not present in the training dataset, is essential for practical applicability.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces \"Dexterous Grasp as You Say\" (DexGYS), a novel framework that integrates natural language instructions with robotic dexterous grasping. To support this integration, the authors present DexGYSNet, a dataset featuring 50,000 pairs of annotated dexterous grasps with corresponding human language instructions across 1,800 household objects, developed using innovative methods such as Hand-Object Interaction Retargeting and LLM-assisted annotation.\n\nBuilding on this dataset, the DexGYSGrasp framework is introduced, designed to generate robot grasps that are aligned with human intentions, ensuring diversity and quality. This two-phase framework initially learns a diverse grasp distribution aligned with intended commands and subsequently refines these grasps to enhance quality.\n\nExtensive testing both on DexGYSNet and in real environments confirms the framework's ability to produce practical, diverse, and high-quality grasps based on natural language commands, significantly advancing human-robot interaction capabilities for both industrial and domestic applications.\n\nSTRENGTHS:\n1. The motivation of the authors is reasonable: leveraging current large language models to provide fine-grained descriptions of grasping states. The DexGYSNet dataset contributes to the community.\n\n2. The authors validated their algorithm on a real robotic gripper, which is crucial.\n\n3. The authors showcased parts of the DexGYSNet data in the supplementary material. Providing visualization code would further enhance this contribution.\n\nWEAKNESSES:\n1. The evaluation metrics require further discussion. DexGYSGrasp is a generative framework, and the effectiveness of the proposed method based on Chamfer distance and Contact distance is not entirely convincing. Although the authors mention in Supp. (Line 521) that the target is \"the most similar grasp\", it is not always possible to find a closely matching grasp pose in the dataset. For instance, if the generated grasp is on the right temple of a pair of eyeglasses, but all dataset grasps are on the left temple, the Chamfer distance metric becomes ineffective.\n2. DexGYSNet only annotates which fingers contact the object and which parts of the object are touched. It lacks more complex intents, which might limit its applicability.\n3. The authors need to verify the quality of DexGYSNet. For example, what about the quality of the text annotations (e.g., could hallucinations from large models cause errors)? Additionally, the authors obtain dexterous hand parameters from MANO parameters through retargeting. They need to validate the grasp quality in physical simulations (e.g., Isaac Sim) to ensure that the objects are tightly grasped without dropping.\n4. In Line 170, the authors train the Quality Grasp Component based on the most similar ground truth grasp with the same intentions. Perhaps, adding noise directly to the generated grasp pose to create a large amount of training data would be more effective?\n5. The paper contains several typos. For example, Line 117 incorrectly uses $L_{spen}$. Lines 173 and 168 are repetitive.\n\nQUESTIONS:\nMy questions are primarily based on the Weaknesses:\n\n1. The authors need to further discuss their evaluation metrics. *CD* and *Con.* are too stringent for a generative task. Metrics based on collisions or the standard deviation of poses seem insufficient for evaluating language-based generative tasks.\n2. The authors might consider how to incorporate richer intents into DexGYSNet.\n3. The authors should include an evaluation of the quality of the DexGYSNet dataset.\n\nLIMITATIONS:\nThe authors discuss the societal impacts and limitations of their method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a language-guided dexterous grasp dataset based on OakInk, named DexGYSNet utilizing LLM and a corresponding two-component framework DexGYSGrasp which generates dexterous grasps based on human language instructions. Real-world experiments are conducted to validate DexGYSGrasp's performances.\n\nSTRENGTHS:\n- The construction of the DexGYSNet dataset is cost-effective with the help of a large language model-assisted annotation system. The use of an LLM-based approach allows for efficient and scalable data labeling, which is crucial for developing robust models in this domain.\n- Extensive experiments are conducted to verify the performance advantages of the DexGYSGrasp method, particularly in terms of grasping intention and diversity, as well as the design of the evaluation metrics. The thorough experimental evaluation provides a comprehensive assessment of the approach and its capabilities.\n\nWEAKNESSES:\n- While the DexGYSGrasp framework presents several notable contributions, such as the use of penetration loss and the Quality Grasp Component, the primary strength of the approach seems to be in the training methods rather than the overall framework design. \n- The Quality Grasp Component appears to function more as a post-processing step rather than a core component of the framework. This suggests that the key innovations of the work are centered around the training techniques employed, rather than the architectural design of the system.\n\nQUESTIONS:\n- The language descriptions generated via the large language model, even with the inclusion of rich contact information as shown in Figure 6, still appear to lack precision. In this case, it is questionable whether directly applying metrics like Chamfer distance and contact distance is the most appropriate approach to reflect the consistency of the grasping intention. The suitability of these particular evaluation metrics for the language-based grasping task deserves further examination.\n- A minor question: the naming of the dataset as \"DexGYSNet\" seems somewhat confusing, as it suggests a naming convention more suited for a pipeline or system, rather than a dataset. \"DexGYSGrasp\" would be a more intuitive name for the dataset itself, as it aligns better with the specific task of grasping being addressed.\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}