SlowGuess/NeurIPS_2023_OCR huggingface dataset:
GraspGF: Learning Score-based Grasping Primitive for Human-assisting Dexterous Grasping
Tianhao Wu  $^{1,2,3*}$ , Mingdong Wu  $^{1,3*}$ , Jiyao Zhang  $^{1,2,3}$ , Yunchong Gan  $^{1}$ , Hao Dong  $^{1,3\dagger}$
<sup>1</sup> Center on Frontiers of Computing Studies, School of Computer Science, Peking University
2 Beijing Academy of Artificial Intelligence
$^{3}$  National Key Laboratory for Multimedia Information Processing,
{thwu,jiyaozhang}@stu.pku.edu.cn，{wmingd,yunchong,hao.dong}@pku.edu.cn
The use of anthropomorphic robotic hands for assisting individuals in situations where human hands may be unavailable or unsuitable has gained significant importance. In this paper, we propose a novel task called human-assisting dexterous grasping that aims to train a policy for controlling a robotic hand's fingers to assist users in grasping objects. Unlike conventional dexterous grasping, this task presents a more complex challenge as the policy needs to adapt to diverse user intentions, in addition to the object's geometry. We address this challenge by proposing an approach consisting of two sub-modules: a hand-object-conditional grasping primitive called Grasping Gradient Field (GraspGF), and a history-conditional residual policy. GraspGF learns 'how' to grasp by estimating the gradient from a success grasping example set, while the residual policy determines 'when' and at what speed the grasping action should be executed based on the trajectory history. Experimental results demonstrate the superiority of our proposed method compared to baselines, highlighting user awareness and practicality in real-world applications. The codes and demonstrations can be viewed at https://sites.google.com/view/graspgf.
The significance of human hands in everyday life cannot be overstated. However, there are situations where they may not always be available, especially in scenarios where an individual may have upper limb loss or need to interact with hazardous objects. In such instances, utilising an anthropomorphic dexterous robotic hand for assistance can be a viable option [1]. Such a dexterous hand possesses a high degree of freedom, allowing it to handle diverse daily tasks [2, 3, 4], given that many everyday objects are designed to match the structure of the human hand. This inspired us to propose a novel task called human-assisting dexterous grasping, in which a policy is trained to assist users with upper limb loss in grasping objects by controlling the robotic hand's fingers, as illustrated in Figure 1a.
Figure 1: a) Demonstration of human-assisting dexterous grasping. b) Challenges of our setting.
Traditional teleoperation methods [5, 6] are unsuitable for assisting upper limb amputees in grasping because no information about the human fingers can be accessed. Compared to conventional dexterous
grasping, human-assisting dexterous grasping poses a more complex challenge, as the policy must adapt to an exponentially growing number of pre-conditions. As shown in Figure 1b, human users may grasp an object with various intentions, such as grasping different parts for different purposes or moving the hand and wrist at different speeds due to the complexity and diversity of human behaviour. Consequently, the conditional policy must be tailored not only to the object's geometry, as required by conventional dexterous grasping, but also to the user's intentions, requiring the policy to be user-aware. In this context, open-looped methods such as grasp pose generation [7, 8, 9, 10] and classification-based methods [11, 12, 13] may fall short, as they do not factor in the user's intention. Reinforcement learning (RL) [2, 14, 15] presents a natural solution by enabling the training of a closed-loop human-object-conditioned policy. However, in human-assisting dexterous grasping, RL may encounter more severe generalisation issues, due to the need to generalise to diverse grasping pre-conditions. Prior RL-based approaches [16, 17] have explored leveraging human-collected and engineering-heavy demonstrations to address this issue. However, collecting a large volume of diverse demonstrations that encompass different objects, grasping timings, and locations may not be feasible.
To address the challenges associated with achieving dexterous grasping for assisting humans, an effective policy needs to tackle the following two crucial questions:
1) How should the robot grasp the object considering the current relative pose between the user and the object? 2) When and at what speed should the robot execute the grasping action based on the user movement trajectory history?
In this paper, we present a novel approach that consists of two sub-modules designed to address the aforementioned questions individually: 1) a hand-object-conditional grasping primitive, and 2) a history-conditional residual policy. The grasping primitive, which we call Grasping Gradient Field (GraspGF), is trained to learn 'how to grasp' by estimating the score function, i.e., the gradient of the log-density, of a success grasping example set. The GraspGF outputs a gradient that indicates the fastest direction to increase the 'grasping likelihood' conditioned on an object and the user's wrist. The gradient can be translated into primitive controls on each finger joint, enabling the fingers to reach an appropriate grasp pose iteratively. However, GraspGF is not capable of determining how fast the fingers should move along the gradient as it is history-agnostic. To determine 'when to grasp', we train a residual policy that outputs a 'scaling action' that determines how fast the finger joints should move along with the primitive action, based on the history of the wrist's trajectory. Besides, as the primitive policy is not aware of the environment dynamics due to the offline training, we further require the residual policy to output a 'residual action' to correct the primitive action.
Our proposed approach offers several conceptual advantages. GraspGF leverages the strong conditional generative modelling of score-based methods, as demonstrated in prior works [18, 19, 20], enabling it to output promising primitive actions conditioned on novel user's intentions. Additionally, the residual-learning design of GraspGF facilitates cold start exploration and enhances the efficiency of residual policy training. Compared to demonstration-based methods, our approach only requires a synthesised grasping example set and does not rely on exhaustive human labelling or extensive engineering effort, making it more practical for implementation in real-world applications.
In our experiments, we evaluate several methods on a dexterous grasping environment that assists humans in grasping over  $4900+$  on-table objects with up to 200 realistic human wrist movement patterns. Our comparative results demonstrate that our proposed method significantly outperforms the baselines across various metrics. Ablation studies further confirm the effectiveness of our proposed grasping gradient field and residual-learning design. Our analysis reveals that our method's superiority lies in its user-awareness, i.e., our trained policy is more tailored to the user's intentions. Additionally, we conduct real-world experiments to validate the practicality of our approach. Our results indicate that our trained model can generalise to the real world to some degree without fine-tuning.
- We introduce a novel and challenging human-assisting dexterous grasping task, which may potentially help social welfare.
- We propose a novel two-stage framework that decomposes the task into learning a primitive policy via score-matching and training a residual policy to complement the primitive policy via RL.
- We conduct experiments to demonstrate our method significantly outperforms the baselines and the effectiveness of our method deployed in the real world.
2 Human-assisting Dexterous Grasping
We study the human-assisting dexterous grasping, in which a policy is trained to assist users in grasping objects by controlling the robotic hand's fingers. We formulate the problem as follows:
State and Action Spaces: In this task, we consider a human-assisting grasping scenario involving a 28-DoF 5-fingered robotic hand. The 18-DoF joints of the fingers are denoted as  $\mathbf{J} \in \mathbb{R}^{18}$ , the 4-DoF under-actuated joints of the fingers are denoted as  $\mathbf{J}^u \in \mathbb{R}^4$ , and the 6-DoF pose of the wrist is represented by  $\mathbf{b} = [\mathbf{b}_p, \mathbf{b}_q]$ , where  $\mathbf{b}_p \in \mathbb{R}^3$  denotes the 3-D position and  $\mathbf{b}_q \in \mathbb{R}^4$  represents the 4-D quaternion. The action space  $\mathcal{A} \subseteq \mathbb{R}^{18}$  corresponds to the 18-D relative changes applied to the hand joints. Unlike traditional dexterous grasping tasks, the action space does not include the 6-D relative changes for the wrist, since the wrist pose is controlled by a human user.
Task Simulation: To simulate the movement of the human user's wrist, we sample a wrist trajectory at the start of each episode  $\tau_{\mathbf{b}} = \{\mathbf{b}_1,\mathbf{b}_2,\dots,\mathbf{b}_T\}$  ( $T$  denotes the horizon). At each time step  $t$ , the wrist's pose is set to  $\mathbf{b}_t$ . Specifically, we initially sample a target object  $O\sim p_O(O)$  from an object prior distribution. Then, the wrist trajectory is sampled  $\tau_{\mathbf{b}}\sim p_{\tau_{\mathbf{b}}}(\tau_{\mathbf{b}}|O)$  conditioned on the target object  $O$ . The terminal wrist state,  $\mathbf{b}_T$ , is designed to be 'graspable'. In other words, there exists a feasible hand joint  $\mathbf{J}^*$  such that the grasp pose  $[\mathbf{J}^*,\mathbf{b}_T]$  can successfully grasp the object  $O$ .
Observations: This task requires the agent to adapt to both the wrist trajectories  $\tau_{\mathrm{b}}$  and different objects  $O$ . Consequently, the policy  $\pi (\mathbf{a}|\cdot)$  should be conditioned on the finger joints  $\mathbf{J}$ , visual observations  $o$ , and the history of hand wrist poses  $H_{t} = [\mathbf{b}_{t - k},\mathbf{b}_{t - k + 1},\dots,\mathbf{b}_{t}]$ , where  $k$  is a hyperparameter. In this work, we use the full point cloud of the target object as the visual observation  $o$  and consider the last five wrist states as the history  $H_{t} = [\mathbf{b}_{t - 4},\dots,\mathbf{b}_{t}]$ . Note that the RL-based policies used in all the experiments also take under-actuated joints  $\mathbf{J}^u\in \mathbb{R}^4$  as input, for simplicity, we omit this input in the following notations.
Objective: The objective is to find a policy  $\pi (\mathbf{a}|\mathbf{J},o,H)$  that maximizes the expected grasping success rate over the initial distributions, i.e.  $O\sim p_O(O)$  and  $\tau_{\mathbf{b}}\sim p_{\tau_{\mathbf{b}}}(\tau_{\mathbf{b}}|O)$ :
$$
\pi^ {*} = \arg \max  _ {\pi} \mathbb {E} _ {\substack {O \sim p _ {O} (O), \tau_ {\mathbf {b}} \sim p _ {\tau_ {\mathbf {b}}} (\tau_ {\mathbf {b}} | O), \\ \mathbf {a} _ {t} \sim \pi (\cdot | \mathbf {J} _ {t}, \sigma_ {t}, H _ {t})}} [ \mathbb {1} (\text {success}) ] \tag{1}
$$
Eq. 1 poses a challenging objective since the policy should generalize not only to different objects  $O \sim p_{O}(O)$ , as required in conventional dexterous grasping, but also to different hand wrist trajectories  $\tau_{\mathbf{b}} \sim p_{\tau_{\mathbf{b}}}(\tau_{\mathbf{b}}|O)$ . In other words, the agent should be user-aware.
Overview: A user-aware policy needs to tackle the following two crucial questions: 1) How should the robot grasp the object considering the current relative pose between the user and the object? 2) When and at what speed should the robot execute the grasping action based on the user movement trajectory history? As illustrated in Figure 2, our key idea is to partition the task into two stages that address these questions individually: 1) Learning a primitive policy  $\pi_p^\theta (\mathbf{a}_t^p |\mathbf{J}_t,o_t,\mathbf{b}_t)$  that proposes a primitive action  $\mathbf{a}_t^p$  that can guide the fingers forming into a pre-grasp pose, from an success grasping pose example set. 2) Learning a residual policy  $\pi_r^\phi (\mathbf{a}_t^s,\mathbf{a}_t^r |\mathbf{J}_t,o_t,H_t)$  that outputs a scaling action  $\mathbf{a}_t$  to determine 'how fast' the joints should move with the primitive action and a residual action  $\mathbf{a}_t^r$  that further corrects the overall action, via RL. The combined policy  $\pi^{\theta ,\phi}(\mathbf{a}_t|\mathbf{J}_t,o_t,H_t)$  is as follows:
$$
\mathbf {a} _ {t} ^ {p} \sim \pi_ {p} ^ {\theta} \left(\mathbf {a} _ {t} ^ {p} \mid \mathbf {J} _ {t}, o _ {t}, \mathbf {b} _ {t}\right), \left(\mathbf {a} _ {t} ^ {s}, \mathbf {a} _ {t} ^ {r}\right) \sim \pi_ {r} ^ {\phi} \left(\mathbf {a} _ {t} ^ {s}, \mathbf {a} _ {t} ^ {r} \mid \mathbf {J} _ {t}, o _ {t}, H _ {t}\right) \tag {2}
$$
$$
\pi^ {\theta , \phi} (\cdot | \mathbf {J} _ {t}, o _ {t}, H _ {t}) = \mathbf {a} _ {t} ^ {p} \odot \mathbf {a} _ {t} ^ {s} + \mathbf {a} _ {t} ^ {r}
$$
Initially, we employ the score-matching to train the primitive policy  $\pi_p^\theta$  from a grasping poses dataset. Subsequently, the combined policy, which is constructed from the residual policy  $\pi_r^\phi$  is combined with the frozen  $\pi_p^\theta$ , and is trained under RL. In the following, we will introduce the motivations and the training procedures of the primitive policy (i.e., GraspGF) and the residual policy in Sec 3.1 and Sec 3.2, respectively. The implementation details of both policies are described in Appendix B.
Figure 2: We decompose the human-assisting dexterous grasping into learning a primitive policy  $\pi_p^\theta$  that learns to form a pre-grasp pose and a residual policy  $\pi_r^\phi$  that learns to adjust the proceeding of the primitive action. a) The primitive policy  $\pi_p^\theta$  is trained on success grasping examples via score-matching objective. b) The residual policy  $\pi_r^\phi$  is trained to adjust the primitive policy via RL.
3.1 Learning GraspGF from Synthetic Examples
To address the first question above, we aim to search for a primitive policy  $\pi_p^\theta$  that outputs action to maximize the likelihood of success, given a set of static conditions  $(\mathbf{J}_t, o_t, \mathbf{b}_t)$ . Inspired by [21], we can train such a policy by estimating the score function (i.e., gradient of the log-density) of a conditional distribution  $p_{\mathrm{success}}(\mathbf{J}|o_t, \mathbf{b}_t)$ :
$$
\mathbf {a} ^ {p} = \pi_ {p} ^ {\theta} (\cdot | \mathbf {J}, o, \mathbf {b}) = \nabla_ {\mathbf {J}} \log p _ {\text {s u c c e s s}} (\mathbf {J} | o, \mathbf {b}) \tag {3}
$$
the  $p_{\mathrm{success}}(\mathbf{J}|o,\mathbf{b})$  denotes a fingers-joints' distribution that can successfully grasp the objects, given the current observation  $o$  and the hand wrist  $\mathbf{b}$ . By definition, the score function of the distribution  $\nabla_{\mathbf{J}}\log p_{\mathrm{success}}(\mathbf{J}|o,\mathbf{b})$  indicates the fastest direction to increase the likelihood of the  $p_{\mathrm{success}}(\mathbf{J}|o,\mathbf{b})$ . Intuitively, if the fingers move in the direction of  $\nabla_{\mathbf{J}}\log p_{\mathrm{success}}(\mathbf{J}|o,\mathbf{b})$ , they will probably reach a feasible grasp-pose in the future with the success-likelihood  $p_{\mathrm{success}}(\mathbf{J}|o,\mathbf{b})$  increases. Hence, we formulate learning the primitive policy  $\pi_p^\theta$  as estimating the gradient field of the log-success-likelihood  $\nabla_{\mathbf{J}}\log p_{\mathrm{success}}(\mathbf{J}|o,\mathbf{b})$ , namely Grasping Gradient Field (GraspGF).
Thanks to the Denoising Score Matching (DSM) [22], we can obtain a guaranteed estimation of the GraspGF  $\nabla_{\mathbf{J}}\log p_{\mathrm{success}}(\mathbf{J}|o,\mathbf{b})$  from a set of success examples  $\mathcal{D}_{\mathrm{success}} = \{\left(\mathbf{J}_i^*,o_i^*,\mathbf{b}_i^*\right)\}_{i = 1}^{N}$  where the  $\mathbf{J}_i^*$  is the feasible finger joints for grasping conditioned on  $(o_i^*,\mathbf{b}_i^*)$ . In the following, we first revisit the preliminaries of DSM and then introduce how to employ DSM to estimate the GraspGF.
Denoising Score-Matching Given a set of data-points  $\{\mathbf{x}_i\sim p(\mathbf{x})\}_{i = 1}^N$  from an unknown data distribution  $p(\mathbf{x})$ , the score-based generative model aims at estimating the score function of a data distribution  $\nabla_{\mathbf{x}}\log p(\mathbf{x})$  via a score network  $\mathbf{s}_{\omega}(\mathbf{x}): \mathbb{R}^{|\mathcal{X}|} \to \mathbb{R}^{|\mathcal{X}|}$ . During inference, a new sample is generated by the Langevin Dynamics, which is out of our interest.
To estimate  $\nabla_{\mathbf{x}}\log p(\mathbf{x})$ , the Denoising Score-Matching (DSM) [22] proposes a tractable objective by pre-specifying a noise distribution  $q_{\sigma}(\widetilde{\mathbf{x}} |\mathbf{x})$ , e.g.,  $\mathcal{N}(0,\sigma^2 I)$ , and train the score network to denoise the perturbed data samples:
$$
\mathcal {L} (\omega) = \mathbb {E} _ {\widetilde {\mathbf {x}} \sim q _ {\sigma}, \mathbf {x} \sim p (\mathbf {x})} \left[ | | \mathbf {s} _ {\omega} (\widetilde {\mathbf {x}}) - \nabla_ {\widetilde {\mathbf {x}}} \log q _ {\sigma} (\widetilde {\mathbf {x}} | \mathbf {x}) | | _ {2} ^ {2} \right] \tag {4}
$$
where  $\nabla_{\widetilde{\mathbf{x}}}\log q_{\sigma}(\widetilde{\mathbf{x}} |\mathbf{x}) = \frac{1}{\sigma^2} (\mathbf{x} - \widetilde{\mathbf{x}})$  are tractable for the Gaussian kernel. DSM guarantees that the optimal score network holds  $\mathbf{s}_{\omega}^{*}(\mathbf{x}) = \nabla_{\mathbf{x}}\log p(\mathbf{x})$  for almost all  $\mathbf{x}$ .
Employing DSM to Estimate GraspGF To estimate the GraspGF  $\nabla_{\mathbf{J}}\log p_{\mathrm{success}}(\mathbf{J}|o,\mathbf{b})$  , we employ the DSM in Eq 4, the training objective of the primitive policy  $\pi_p^\theta$  is derived as follows:
$$
\mathcal {L} (\theta) = \mathbb {E} _ {\substack {\tilde {\mathbf {J}} \sim q _ {\sigma} (\tilde {\mathbf {J}} | \mathbf {J} ^ {*}), \\ \left(\mathbf {J} ^ {*}, o ^ {*}, \mathbf {b} ^ {*}\right) \sim \mathcal {D} _ {\text {success}}}} \left[ \left\| \pi_ {p} ^ {\theta} \left(\cdot \mid \widetilde {\mathbf {J}}, o ^ {*}, \mathbf {b} ^ {*}\right) - \frac {\mathbf {J} ^ {*} - \widetilde {\mathbf {J}}}{\sigma^ {2}} \right\| _ {2} ^ {2} \right] \tag{5}
$$
The Eq 5 is the L2 distance between the output of the primitive policy and the denoising direction  $\frac{\mathbf{J}^{*} - \widetilde{\mathbf{J}}}{\sigma^{2}}$ , i.e., a direction pointing from the perturbed joints  $\widetilde{\mathbf{J}}$  to the original joints  $\mathbf{J}^{*}$ . Intuitively, this objective is forcing the primitive policy to denoise the current joints to regions where the fingers are more likely to grasp the object.
3.2 Training Residual Policy via Reinforcement Learning
The human-assisting dexterous grasping cannot be effectively addressed solely through the primitive policy  $\pi_p^\theta$ . Although  $\pi_p^\theta$  predicts the fastest direction to form a pre-grasp pose, it fails to determine the appropriate 'velocities' at which the joints should move in that direction. If the fingers close too quickly or too slowly, the agent may struggle to grasp the object. Additionally, due to offline training, the primitive policy lacks awareness of the dynamics of the environment, leading to potential violations of physical constraints.
To overcome these limitations, we propose the training of a residual policy  $\pi_r^\phi$ , which complements the primitive policy. The role of  $\pi_r^\phi$  is twofold: firstly, it outputs a scaling action  $\mathbf{a}^s$  that adjusts the speed of the primitive action, and secondly, it produces a residual action  $\mathbf{a}^r$  that corrects the final output action. Eq 2 demonstrates that  $\mathbf{a}^s$  can be interpreted as 18-D 'pseudo-velocities' imposed on the finger joints. With outputs less than 1 for  $\mathbf{a}^s$ , the residual policy can decelerate the primitive action, whereas values greater than 1 accelerate it.
To effectively control the proceeding of the primitive action, the residual policy  $\pi_r^\phi (\mathbf{a}_t^r |\mathbf{J}_t,o_t,H_t)$  takes into account the history of the wrist  $H_{t}$  and the object's point cloud  $o$  as inputs. This enables the policy to infer the agent's speed of approach towards the object. Furthermore, the policy network incorporates the current joint state  $\mathbf{J}_t$  to infer how to correct the primitive action  $\mathbf{a}^p$ .
We employ Proximal Policy Optimization (PPO) [23] to search for a final policy  $\pi^{\theta, \phi}$  that maximises the following objective, where the primitive policy's parameters  $\theta$  are frozen during training:
$$
J (\phi) = \mathbb {E} _ {\substack {O \sim p _ {O} (O), \tau_ {\mathbf {b}} \sim p _ {\tau_ {\mathbf {b}}} (\tau_ {\mathbf {b}} | O), \\ \mathbf {a} _ {t} \sim \pi^ {\theta}, \phi (\cdot | \mathbf {J} _ {t}, o _ {t}, H _ {t})}} \left[ \sum_ {t = 0} ^ {T} \gamma^ {t} r _ {t} \right] \tag{6}
$$
where  $\gamma > 0$  denotes a discounted factor and  $r_t$  is the reward at time-step  $t$ . To encourage the policy to successfully grasp the object while leveraging the primitive action  $\mathbf{a}_t^p$  for efficient exploration, we assign the following simple reward function for training:
$$
r _ {t} = \left(1 - d _ {t}\right) \cdot \left(r _ {\text {s i m}} + r _ {\mathrm {h}}\right) + d _ {t} \cdot \lambda_ {s}, d _ {t} = \mathbb {1} (\text {s u c c e s s}, t = T)
$$
$$
r _ {\text {s i m}} = \lambda_ {a} \cdot \left\langle \frac {\mathbf {a} ^ {p}}{\| \mathbf {a} ^ {p} \| _ {2}}, \mathbf {J} _ {t} - \mathbf {J} _ {t - 1} \right\rangle , r _ {\mathrm {h}} = \lambda_ {h} \cdot \Delta h \tag {7}
$$
where  $\lambda_{a},\lambda_{h},\lambda_{s} > 0$  are hyperparameters and  $\Delta h$  represents the change in the height of the object's centre of gravity after lifting. The term  $d_{t}$  rewards the agent if the final grasp pose can successfully lift the target object. The term  $r_{\mathrm{sim}}$  is an intrinsic reward that encourages the agent when the joints-change follows the direction of the primitive action  $\mathbf{a}_t^p$ . We defer the PPO's hyperparameters to Appendix B.
Primitive Policy Network This module takes the hand joints  $\mathbf{J} \in \mathbb{R}^{18}$ , object point cloud  $o \in \mathbb{R}^{3 \times 1024}$  and the hand wrist  $\mathbf{b} \in \mathrm{SE}(3)$  as input, we first project the point cloud into the hand wrist's coordinate  $o_{\mathrm{b}}$ . We encode  $o_{\mathrm{b}}$  into a 512-D global feature by PointNet++ [24]. The hand joints  $\mathbf{J}$  and the noise-condition  $t \in [0,1]$  are further encoded into 1024-D and 512-D features respectively. Concatenating the features together, we feed the concatenated feature into MLPs to obtain the 18-D output.
Residual Policy Network Taking the hand joints  $\mathbf{J}_t\in \mathbb{R}^{18}$ , under-actuated joints  $\mathbf{J}_t^u\in \mathbb{R}^4$ , visual observation  $o_t\in \mathbb{R}^{3\times 1024}$  and the hand wrists' trajectory  $H_{t} = [\mathbf{b}_{t},\dots,\mathbf{b}_{t - 4}]$ ,  $\mathbf{b}_i\in \mathrm{SE}(3)$  as input, this module uses the PointNet [25] to encoder the point cloud. Similarly, we first obtain the global feature  $f_{o_{\mathrm{b}}}\in \mathbb{R}^{512}$  of the projected point cloud. The primitive action  $\mathbf{a}_t^p = \pi_p^\theta (\mathbf{J}_t,o_t,\mathbf{b}_t)$ , hand joints  $\mathbf{J}$  and the history  $H$  are concatenated and further encoded into 512-D features. Concatenating the above features together, we feed the concatenated feature into MLPs to obtain the 36-D output.
4 Experiment Setups
4.1 Task Simulation
Environment setup: We created a simulation environment based on the ShadowHand environment in Isaac Gym [26], using the ShadowHand model from IBS [27]. The simulation environment enables parallel training on hundreds of environments. Each environment consists of an object placed on the ground and a hand that follows a pre-generated human wrist trajectory. The agent can only control the joints of the hand. The episode horizon is 50 steps, and each episode only terminates at the final step. Following a similar setting to IBS [27], when the episode terminates, part of the joints of each finger will automatically close by 0.1 radians, and then the wrist of the hand will be lifted by  $1\mathrm{m}$ .