SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Grasping pose generation: We created our success grasping pose based on the UniDexGrasp dataset [10]. We filtered the data to match our ShadowHand model's degrees of freedom. The dataset was split into three sets: training instances (3127 objects, 363,479 grasps), seen category unseen instances (519 objects, 2595 grasps), and unseen category instances (1298 objects, 6490 grasps).
Human grasping wrist patterns: To mimic real human grasping patterns, we resampled 200 real human grasping wrist trajectories from HandoverSim! [28], from which we extracted 200 wrist movement patterns. These patterns were split into 150 training patterns and 50 testing patterns.
The details are deferred to Appendix A.
Following the DexVIP [29], we report the following three metrics: 1) Success: The object can be lifted more than  $0.1\mathrm{m}$  off the table, while the change in distance between the object and hand position after lifting is smaller than  $0.05\mathrm{m}$ . 2) Posture: The distance between the target human hand pose and the agent hand pose. It tells us how human-like the learned grasps are. 3) Stability: The translation and rotation changes of the object during the agent's grasping process. Translation is measured by the Euclidean distance between the final object position and the initial object position. Rotation is measured by the cosine distance between the x-axis of the initial object and the final object.
We compare our method with the following RL-based methods. Note that all baselines are re-trained by taking the latest 5-frame wrist states as input. 1) IBS [27]: IBS explicitly compute rich contact information according to hand mesh and object mesh, which has good generalisation to objects and different contact situation. We modify the baseline to only output joint actions, and keep the reward the same as previous. 2) PPO [23]: We adopt PPO as ours pure RL baseline. For pure RL, we use the reward of fingertip distance (distance of fingertip to the closest point of object point cloud),  $r_{\mathrm{h}}$  and success reward. 3) PPO (Goal): We randomly sample the goal which corresponds to the current object as additional input to the agent. We add another goal pose matching reward compared to PPO baseline. 4) ILAD [30]: We choose ILAD as our RL+Imitation learning baseline, which will first generate sub-optimal grasping trajectories according to example grasp pose, then use these demonstrations for imitation learning and RL. We regenerate the trajectories for ILAD training based on our grasp data. We use the same reward as the PPO baseline. The details are deferred to Appendix C.
Figure 3: Qualitative results of comparison with baselines and different trajectories. a): final grasp poses of different methods. b): final grasp poses of our method under different human trajectories
Figure 4: Quantitative comparative results. Left: training curve of different methods. Note that IBS takes 144 hours on V100 to reach 3.5 million agent steps, while ours only takes 15 hours to reach 10 million agent steps. Middle: Success and Posture of different methods on seen category unseen instances. Right: Success and Posture of different methods on unseen category instances.
5.1 Comparative Results
As shown in Figure 4, Ours achieves comparable training efficiency to ILAD, even without the use of extra trajectory demonstrations. Ours can converge to a higher Success. Ours also surpasses other baselines of Success for both unseen and unseen category instances. This demonstrates the robust generalisation capabilities of our method. Ours also has the highest performance of Posture, which indicates that our method is better aware of the user's intentions. The qualitative results in Figure 3 b) demonstrate the successful grasping of objects across different user's intentions. As indicated in Table 1, Ours also causes the least disturbance to the object.
ILAD and IBS are considered as the strongest baseline. ILAD utilises additional trajectory demonstrations for RL training. However, Ours still surpasses this baseline because ILAD still relies on RL to generalise across diverse object categories. For IBS, we observed that IBS tends to fail when the human wrist trajectory has the potential to collide with the ground. This could be attributed to the reward design of IBS, which needs to balance between colliding with the scene and grasping the object. Additionally, computing the IBS representation is ten times more computationally expensive.
$PPO$  primarily fails because it attempts to learn a general policy that can grasp most objects, as shown in Figure 3 a), which is not suitable for diverse objects and grasp poses.  $PPO$  (Goal) faces challenges due to its inability to adjust the goal during the approach phase. Since the wrist is continuously moving during the process, the goal set at the initial state may not be suitable.
Seen Category	Unseen Category
Tran(cm) ↓	Rot (rad) ↓	Tran(cm) ↓	Rot (rad) ↓
PPO(Goal)	2.621±0.415	0.589±0.038	2.537±0.296	0.543±0.040
PPO	2.745±0.168	0.594±0.045	2.771±0.254	0.563±0.039
IBS	2.653±0.030	0.572±0.002	2.596±0.119	0.520±0.011
ILAD	2.443±0.042	0.548±0.027	2.534±0.101	0.515±0.022
Ours	2.131±0.138	0.449±0.020	2.127±0.165	0.428±0.029
Table 1: Results of Stability on seen category unseen instances and unseen category instances. Tran: translation of objects from the initial position to the final position. Rot: rotation of objects from initial orientation to final orientation.
5.2 Ablation Studies and Analysis
We conduct the ablation studies to investigate: 1) The effectiveness of decomposing the policy into primitive policy and residual policy. 2) The necessity of different action modules. 3) The impact of different action modules on final action.
The ablation results depicted in Figure 5 demonstrate the significance of our proposed approaches. Ours w/o GF experiences a significant performance decline, indicating the significance of combining RL policy with primitive action. Similarly, Ours w/o RL shows a substantial drop in performance by focusing solely on the 'how' of grasping, which results in collisions during the grasping process.
Although Ours w/o  $\mathbf{a}^r$  and Ours w/o  $\mathbf{a}^s$  exhibit higher training efficiency, both methods eventually experience training collapse. During this collapse, we observe that the agent tends to either follow the primitive action more closely or deviate from it to a greater extent. This suggests that the combination of both action modules allows for a better policy learning process that effectively utilises the primitive action.
The results shown in Table 2 indicate that the primitive action, which does not consider collisions due to physical constraints in the grasping procedure, can achieve similar performance to Ours. This suggests that the primitive action module has a good understanding of how to grasp, but it moves quickly towards the target without considering potential collisions, as shown in Figure 6. The inclusion of the  $\mathbf{a}^s$  module slows down the progress of  $\mathbf{a}^p$  to avoid collisions, as illustrated in Appendix D.1. However, this conservative approach leads to a drop in success. Nevertheless, after further addition of  $\mathbf{a}^r$ , the performance increases to  $56.50\%$ , which is comparable to  $\mathbf{a}^p$  w/o coll. By further combining the  $\mathbf{a}^r$  module, the final action is corrected
to achieve the 'how to grasp' knowledge learned by  $\mathbf{a}^p$  while also incorporating correct 'when to grasp' knowledge.
Figure 5: Ablation Study on decomposing policy and different action modules.
Figure 6: Qualitative results of grasping procedure governed by the primitive policy. The yellow circle highlights the collision between the finger and the object caused by premature closure.
Action Type	Success
a p	19.49 %
a p w/o coll	55.60 %
a p ⊙ a s	5.08 %
a p ⊙ a s + a r	56.50 %
5.3 Adaptability Results
To further demonstrate the adaptability of our method, we conduct both quantitative and qualitative experiments. Quantitatively, as shown in Table 3, we calculate the success rate for each object across five different grasp poses. For the objects that have been successfully grasped, Ours excels at grasping various parts of the objects. Qualitatively, Figure 7 demonstrates that the grasp pose generated by Ours closely resembles a human's intended grasp pose. Furthermore, we also show that GraspGF can adapt to wrist rotation; videos can be viewed at https://sites.google.com/view/graspgf.
Table 2: Success rate of different combinations of action modules.
Success Rate	1/5	2/5	3/5	4/5	5/5
Ours	13.67%	27.70%	31.41%	20.82%	6.40%
ILAD	25.44%	32.53%	26.61%	12.53%	2.89%
IBS	27.70%	34.73%	25.67%	10.29%	1.61%
PPO	48.20%	35.49%	13.10%	3.01%	0.20%
PPO (Goal)	77.04%	18.96%	3.62%	0.30%	0.08%
Table 3: Percentages of objects with varying success rates of grasping under different methods. The results are averaged over five different random seeds.
Figure 7: Qualitative results of grasping different parts of shoes with the same shape but different scales.
5.4 Real-world Results
In this section, we construct a real-world system to validate our method. As shown in Figure 8, the system consisted of calibrated multi-view RGB-D cameras (four Intel RealSense D415 sensors), a
UR10e robot arm equipped with a Shadow Hand as the end-effector, and an optical hand tracking sensor (Leap Motion Controller). To obtain the visual observation of the object, we reconstruct the
scene point clouds from the multiview RGB-D cameras. We only get the point cloud at the beginning of each grasping, assuming the point cloud observation does not change during the grasping. Then, we develop a teleoperation system inspired by the design of DexPilot [5] to enable the robot arm to track the trajectory of a human wrist, but with the difference of using a Leap Motion Controller as the hand tracking device to obtain human wrist and hand pose. It should be noted that we remove the estimated hand pose to emulate the human-assisting setting where only the wrist can be controlled by humans.
Figure 8: Real-world Experiment. Left: the real-world setup for real human-assisting dexterous grasping. Right: the qualitative result of real human-assisting dexterous grasping.
Following a similar setting in [16],
we evaluated our policy in the real world using ten objects from the YCB dataset [31]. For each object, we selected two grasp poses that were intended to be achieved by a human. Each pose was tested five times, with a human randomly randomised initial object pose and wrist movement trajectories. The details of evaluation in the real world are deferred to D.2. As shown in Table 4, our policy achieved an average success rate of  $66\%$  in real-world human-assisted grasping, demonstrating its generalisation ability in a real-world setting. The grasping policy tended to fail for certain large objects that were difficult for the dexterous hand to hold firmly, as well as for thin objects that were prone to colliding with the table. One of the reasons our policy still achieved a certain success rate in the real world is that when humans control the wrist of their hand, human will also try to adjust according to the grasping pose of the joint, which may help to grasp the object. Our real-world demonstrations can be viewed at https://sites.google.com/view/graspgf.
Object	Domino
TOMINO	Domino
TAMDAI	Tomato
TOMATO	JELLO	CHEEZIT	Average Success Rate
Pose 1	5/5	5/5	5/5	1/5	1/5	5/5	4/5	5/5	1/5
Pose 2	4/5	4/5	3/5	3/5	4/5	4/5	2/5	4/5	1/5
66%
Table 4: Results of success rate on 10 different YCB objects with two different poses in the real world.
6.1 Dexterous Hand Grasping
Existing studies on dexterous hand robotics mainly focus on dexterous grasping, assuming full control over the hand, including both the hand base and joint movements by the agent [27, 32, 14, 29, 30].
However, Reinforcement Learning (RL) based methods struggle with generalisation to different objects due to object diversity, high-dimensional state and action spaces, and sparse rewards [33]. To address this issue, most existing approaches rely on human-collected demonstrations [17, 16], which require engineering-heavy retargeting. The most relevant work, IBS [27], combines RL with a human-designed representation that explicitly incorporates contact information between the hand and objects to aid in grasping. However, this computationally costly representation limits the practicality. Furthermore, in our proposed approach, we address a different setting where the movement of the hand base is determined by the user, requiring the agent to be 'user-centric' rather than 'object-centric'. Additionally, our approach only requires a set of grasp examples and is free of human design.
In the medical AI field, there are studies with a similar setting to ours, known as prosthetic dexterous grasping [11, 12, 13]. These studies aim to control a prosthetic hand to assist users in grasping and explore open-looped approaches that initially predict the grasp type and then execute predefined primitives for control. In contrast, we explore a closed-loop control policy that adaptively adjusts the grasp pose based on the current user-object relationship and the user's movement trajectory history.
6.2 Score-based Generative Models
In the pursuit of estimating the gradient of the log-likelihood associated with given data distribution, the score-based generative model, originally introduced by [34], has garnered substantial attention in the research community [22, 34, 35, 36, 37, 19, 38]. The denoising score-matching (DSM), as proposed by [22], further introduces a tractable surrogate objective for score-matching. To enhance the scalability of score-based generative models, [35] introduces a sliced score-matching objective that projects the scores onto random vectors before comparing them. Song et al. also introduce annealed training for denoising score matching [36], along with corresponding improved training techniques [37]. They also extend the discrete levels of annealed score matching to a continuous diffusion process and demonstrate promising results in image generation [19]. Recent works further explore the design choices of the diffusion process [39], maximum likelihood training [38], and deployment on the Riemann manifold [40]. These recent advances show promising results when applying score-based generative models in high-dimensional domains and promote wide applications in various fields, such as object rearrangement [21], medical imaging [41], point cloud generation [42], scene graph generation [43], point cloud denoising [44], depth completion [45], and human pose estimation [18]. These works formulate perception problems into conditional generative modelling or inpainting, allowing the utilisation of score-based generative models to address these tasks.
In contrast, our focus lies in the application of score-based generative models for training low-level control policies. In this domain, [46] and [20] have proposed learning diffusion models from offline trajectories and leveraging these models for model predictive control. However, these approaches suffer from the drawback of requiring a substantial amount of offline data and inefficiency during test-time sampling. To the best of our knowledge, our method represents the first exploration of score-based generative models for learning closed-loop dexterous grasping policies.
In this work, we introduce human-assisting dexterous grasping, wherein a policy is trained to assist users in grasping objects by controlling the robotic hand's fingers. To search for a user-aware policy, we propose a novel two-stage framework that decomposes the task into learning a primitive policy via score-matching and training a residual policy to complement the primitive policy via RL. In experiments, we introduce a human-assisting dexterous grasping environment that consists of  $4900+$  on-table objects with up to 200 realistic human wrist movement patterns. Results demonstrate that our proposed method significantly outperforms the baselines across various metrics. Our analysis reveals that our trained policy is more tailored to the user's intentions. Our real-world experiments indicate that our learned policy can generalise to the real world to some degree without fine-tuning.
Limitations and Future works. Our method takes the full point cloud as the visual observation, which is not accessible in the wild. In the future, we may leverage teacher-student learning to generalise our method to a partial observation setting.
Ethics Statement. Our method has the potential to develop home-assistant robots and assist individuals with hand disabilities, thus contributing to social welfare. We evaluate our method in simulated environments, which may introduce data bias. However, similar studies also have such general concerns. We do not see any possible major harm in our study.
We would like to thank Qianxu Wang and Haoran Lu for supporting the real-world experiments. This work is supported by the National Natural Science Foundation of China - General Program (Project ID: 62376006), the National Youth Talent Support Program (Project ID: 820080081) and the Beijing Municipal Science & Technology Commission (Project ID: Z221100003422004).
[37] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438-12448, 2020.
[38] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415-1428, 2021.
[39] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364, 2022.
[40] Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet. Riemannian score-based generative modeling. arXiv preprint arXiv:2202.02763, 2022.
[41] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:2111.08005, 2021.
[42] Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In European Conference on Computer Vision, pages 364–381. Springer, 2020.
[43] Mohammed Suhail, Abhay Mittal, Behjat Siddique, Chris Broaddus, Jayan Eledath, Gerard Medioni, and Leonid Sigal. Energy-based learning for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13936-13945, 2021.
[44] Shitong Luo and Wei Hu. Score-based point cloud denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4583-4592, 2021.
[45] Ruizhi Shao, Zerong Zheng, Hongwen Zhang, Jingxiang Sun, and Yebin Liu. Diffustereo: High quality human reconstruction via diffusion-based stereo using sparse cameras. arXiv preprint arXiv:2207.08000, 2022.
[46] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991, 2022.
[47] Kai Chen and Qi Dou. Sgpa: Structure-guided prior adaptation for category-level 6d object pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2773-2782, 2021.
[48] Hansheng Chen, Pichao Wang, Fan Wang, Wei Tian, Lu Xiong, and Hao Li. Epro-pnp: Generalized end-to-end probabilistic perspective-n-points for monocular object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2781-2790, 2022.
A Task Details
A.1 Simulation
Human Wrist Trajectory Generation: We generate the human wrist trajectories using the following steps based on the human trajectory pattern dataset in A.2:
- Randomly select one object sample for each environment and add random noise to the x and y position change of the sampled object pose within the range of  $(-0.5, 0.5)$  meters.
- Sample a grasp pose that is different from the last sampled grasp pose for this object. We measure the distance between each grasp example of the object and the last sampled grasp example. If the sampled grasp example is already in the sample history, we randomly choose a new grasp example and clear the history.
- Based on the sampled grasp pose, we generate the initial wrist pose. First, we compute the centre of the fingertip pose and the wrist pose to obtain a vector pointing from the fingertip centre to the wrist. Then, we randomly sample the initial wrist position within a length range of 0.15 to 0.2 meters and randomly sample a deviation angle within the range of 0 to 20 degrees according to the vector.
- For the initial orientation, we randomly sample the delta angle change, denoted as  $\delta_{a}$ , from a distribution based on human rotation data. We set the initial angle as  $A_{target} * (1 - \delta_{a} / 2^{*}\pi)$ , where  $A_{target}$  represents the target angle.
- After obtaining the initial and final poses of the wrist, we aim to add noise to the trajectories during training. However, simply adding random noise would result in a shaking wrist trajectory that does not resemble human behaviour. To address this, we randomly sample two normalized trajectory patterns  $p1$  and  $p2$  during training. We then combine these two patterns using a random coefficient  $c$  sample from (0,1) to generate a new pattern:  $c * p1 + (1 - c) * p2$ . This generated pattern is then used to define the trajectory of the wrist, which serves as the human wrist trajectory, as shown in Figure 9.
By following the above generation process, we can obtain human trajectories with diverse human-like grasping patterns, diverse velocities, and diverse initial hand poses for diverse objects, while guaranteeing that the agent can grasp the object at the final wrist pose.
Figure 9: Visualizations of generated human trajectories.
State Space: For all joints  $\mathbf{J}$  and under-actuated joints  $\mathbf{J}^u$ , we normalize the joint states to the range of  $(-1, 1)$  based on their respective joint limits. As for the wrist pose  $\mathbf{b}$ , we directly utilize the absolute pose relative to the world frame. Regarding the object point cloud  $o$ , we sample 1024 points from the object mesh.
Action Space: The action corresponds to the relative change applied to the 18 joints  $\mathbf{J}$ . Each action is clipped to the range of  $(-1, 1)$  and then scaled by a factor of 0.05.
Grasp Example Dataset: The ShadowHand model used in IBS is equipped with 18 degrees of freedom, including four under-actuated joints that are not directly controlled by the agent. However, in the table-top dataset presented in [10], it is assumed that the ShadowHand model has direct control over these four under-actuated joints. Consequently, the grasp examples from the dataset may not be directly applicable to our ShadowHand model. To address this, we incorporate the grasp poses from
the dataset into our simulation environment and reevaluate the grasp examples. We assess each grasp example based on the successful lifting of the object, with each grasp pose being tested twice for reliable evaluation.
To ensure a fair comparison, we select five of the most diverse grasp poses for each object from the re-filtered dataset for both seen category unseen instances and unseen category instances.