SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Human Wrist Trajectory Pattern Dataset: HandoverSim![28] provides handover trajectories collected from real humans, which can be utilized to generate human-like wrist trajectories during grasping in our task. However, these trajectories in HandoverSim![28] are only available for a limited set of objects and cannot be directly applied to all objects in our simulation environment. Therefore, to generate human-like grasping wrist trajectories in our simulation environment, we first collect wrist movement trajectories from the initial pose to the pose where the object is successfully grasped. To capture the movement patterns, we normalize the trajectories based on the max and min values of each axis  $(\mathrm{x},\mathrm{y},\mathrm{z})$ . Additionally, for the rotation component, we collect the distribution of wrist rotation change from the initial orientation to the final orientation of each axis (roll, pitch, yaw). We generate human trajectories according to these patterns.
A.3 Evaluations
For baseline comparison and ablation study, we conducted evaluations using 5 random seeds. For each seed, the evaluation of training instances was performed five times. The grasp pose for each object is different in each iteration. Throughout the evaluations, we utilized the human wrist trajectory patterns randomly sampled from their respective sets without introducing any additional noise. The human trajectory remained consistent across all comparisons for the same random seed.
B.1 Primitive Policy
Training Objective: In practice, we adopt an extension of DSM [19] that estimates a time-dependent score network  $\mathbf{s}_{\omega}(\mathbf{J}|o^{*},\mathbf{b}^{*},t)$  to denoise the perturbed data from different noise levels simultaneously:
$$
\mathcal {L} (\omega) = \mathbb {E} _ {t \sim \mathcal {U} (\epsilon , T)} \left\{\mathbb {E} _ {\substack {\widetilde {\mathbf {J}} \sim q _ {\sigma (t)} (\widetilde {\mathbf {J}} | \mathbf {J}), \\ \left(\mathbf {J} ^ {*}, o ^ {*}, \mathbf {b} ^ {*}\right) \sim \mathcal {D} _ {\text {success}}}} \lambda (t) \left[ \left\| \mathbf {s} _ {\omega} \left(\widetilde {\mathbf {J}}, \left| o ^ {*}, \mathbf {b} ^ {*}, t\right) - \frac {1}{\sigma^ {2} (t)} \left(\mathbf {J} - \widetilde {\mathbf {J}}\right) \right\| _ {2} ^ {2} \right] \right\} \right. \tag{8}
$$
where  $T = 1$ ,  $\epsilon = 10^{-5}$ ,  $\lambda(t) = \sigma^2(t)$ ,  $\sigma(t) = 1 - e^{-\frac{1}{2}t^2(\beta_{max} - \beta_{min}) - t\beta_{min}}$  and  $\beta_{max} = 10$ ,  $\beta_{min} = 0.1$  are hyper-parameters. The optimal time-dependent score network holds  $\mathbf{s}_{\omega}^{*}(\mathbf{J}|o^{*},\mathbf{b}^{*},t) = \nabla_{\mathbf{x}}\log q_{\sigma(t)}(\mathbf{J}|o^{*},\mathbf{b}^{*})$  where  $q_{\sigma(t)}(\mathbf{J}|o^{*},\mathbf{b}^{*})$  is the perturbed data distribution:
$$
q _ {\sigma (t)} \left(\mathbf {J} \mid o ^ {*}, \mathbf {b} ^ {*}\right) = \int q _ {\sigma (t)} \left(\widetilde {\mathbf {J}} \mid \mathbf {J}\right) p _ {\text {s u c c e s s}} \left(\mathbf {J} \mid o ^ {*}, \mathbf {b} ^ {*}\right) d \mathbf {J} \tag {9}
$$
With the trained score network  $\mathbf{s}_{\omega}$ , we parameterize the primitive policy as:
$$
\pi_ {p} ^ {\theta} \left(\cdot | \mathbf {J}, o ^ {*}, \mathbf {b} ^ {*}\right) = \mathbf {s} _ {\omega} \left(\mathbf {J} \mid o ^ {*}, \mathbf {b} ^ {*}, 0. 0 0 5\right) \tag {10}
$$
Network Architecture:
We utilize the Gradient Field backbone introduced in [21] along with the PointNet++ backbone from [47] to build our chosen backbone for this study.
For training the primitive policy, we initially extracted a maximum of five distinct poses for each object within the training instances to balance the training data, resulting in a total of 15,387 grasping examples. It is important to note that we exclusively employ grasp examples for the training of the primitive policy, without incorporating any trajectories. We use the Adam optimizer with a learning rate of 2e-4 for training. The batch size is set to the total number of grasping examples divided by 5. It takes 60 hours to train on a single A100 for primitive policy to converge.
B.2 Residual Policy
To incorporate object geometry information into the residual policy, we employ PointNet[25]. We begin by pretraining PointNet[25] using the same procedure as the primitive policy training with PointNet++[24]. Subsequently, during training for the residual policy, we continue fine-tuning PointNet[25].
**Reward Function:** Based on empirical observations, we found that the following intrinsic reward function (Eq. 11) which encourages the agent to simply follow the direction of primitive policy will lead to faster convergence.
$$
r _ {\mathrm {s i m}} = \lambda_ {a} \cdot \left\langle \frac {\mathbf {a} ^ {p}}{\left| \mathbf {a} ^ {p} \right|}, \mathbf {J} _ {t} - \mathbf {J} _ {t - 1} \right\rangle \tag {11}
$$
Note that since the intrinsic reward  $r_{\mathrm{sim}}$  is the only dense reward in our method, we set its frequency to 5 to avoid overfitting to the primitive policy. Empirically, we set  $\lambda_s = 1.0$ ,  $\lambda_a = 0.09$ , and  $\lambda_h = 0.5$  for our approach.
RL Backbone: We implemented the Proximal Policy Optimization (PPO) algorithm [23] using the PyTorch implementation provided in [2]. In our implementation, we incorporated the PointNet backbone from [17] into the PPO backbone.
Hyper-parameters: We mainly kept the hyper-parameters of PPO the same as those in [2], with the following exceptions: we set the number of update intervals (nsteps) to 50, the number of optimization epochs (noptepochs) to 2, the mini-batch size mini_batch_size(mini_batch_size) to 64, and the discount factor (gamma) to 0.99.
We trained the residual policy for a total of 10 million agent steps, which took approximately 15 hours using a single A100 GPU.
C Baselines Implementations
IBS: In the IBS baseline [27], two buffers are maintained: a demonstration buffer for warm-up RL training and an experiment buffer for storing the transition tuples. We set the size of both buffers to 15,000, which is three times larger than in our method. To generate the demonstration buffer, we follow a similar procedure in IBS [27]. Specifically, we randomly sample grasp poses from the same data used to train the primitive policy. We then use linear interpolation to compute joint actions based on the initial and target joint states. By applying these actions with the generated human trajectories, we store the transition tuples in the demonstration buffer until it reaches the maximum size.
PPO: Since the PPO baseline does not have a primitive policy, we substitute the intrinsic reward with the fingertip distance reward. It is important to note that we keep the  $r_{\mathrm{h}}$  and success reward the same as in our approach. We use PointNet++ [24] for this baseline, as the PointNet may lead to training collapse. All other training parameters remain the same as in our approach.
PPO(Goal): Similar to the PPO baseline, we use PointNet++ [24] and keep all training parameters the same as in our approach. We introduce a goal pose matching reward that computes the distance between the current joint position and the goal joint position, in addition to the rewards used in PPO.
ILAD: In the ILAD baseline [17], additional trajectories are required to train RL. We use the same data used to train the primitive policy to generate these trajectories following the same process as described in ILAD [17]. In total, we generate 125k transition tuples. We use PointNet [25] for this baseline and keep all training parameters the same as in our approach. The reward remains the same as PPO.
D.1 Additional Ablations
The necessity of a  $s$  module:
As shown in Figure 10, we visualise the grasp pose of the final state using different action modules  $\mathbf{a}^p$  and  $\mathbf{a}^p\odot \mathbf{a}^s$ , the results are generated without simulating the collision between hand and object in the simulation environment. We can observe that  $\mathbf{a}^p$  will usually cause a collision between the hand
and the object. When using the combination of  $\mathbf{a}^s$ , the finger still moves towards the similar grasp pose as with  $\mathbf{a}^p$ , while the  $\mathbf{a}^s$  module will also try to avoid the collision with the object by slowing down the movement of the  $\mathbf{a}^p$ .
Figure 10: Qualitative result of final grasp pose using  $\mathbf{a}^p$  and  $\mathbf{a}^p\odot \mathbf{a}^s$  action module
The effectiveness of intrinsic reward  $r_{\mathrm{sim}}$ :
As the primitive policy provides good guidance on "how to grasp", we introduce the  $r_{sim}$  reward to encourage the final output action  $a^p \odot a^s + a^r$  to explore in direction of the primitive policy's action  $a^p$ , this reward is the inner product of  $\frac{a^p}{\|a^p\|^2}$  and  $J_t - J_{t-1}$ , where  $J$  signifies the state of finger joints. A higher value of  $r_{sim}$  indicates a smaller angle between the gradient and  $J_t - J_{t-1}$ , implying a greater similarity in the movement of finger joints to the gradient.
As shown in Figure 11, when the intrinsic reward  $r_{\mathrm{sim}}$  is not included, the success rate of our method drops, and the training efficiency is significantly lower compared to when  $r_{\mathrm{sim}}$  is included. This indicates that  $r_{\mathrm{sim}}$  plays a crucial role in guiding the agent to initially follow the primitive policy and helps accelerate the learning of the residual policy. Furthermore, the in
Figure 11: Ablation Study on intrinsic reward  $r_{\mathrm{sim}}$
clusion of  $r_{\mathrm{sim}}$  encourages the residual policy to follow the primitive policy throughout the entire training process, leading to better utilisation of the generalisation capabilities of the primitive policy.
Action of each policy during the grasping process:
The primitive policy's actions solely depend on object-wrist relative orientation. Thus when the hand is far away, the primitive policy will still close the fingers, as shown in Figure 12 (a) Stage1. As the hand's posture progressively approaches the target grasp pose, the mean value of the primitive policy's actions decreases, as shown in Figure 12 (a) Stage2.
To further understand the action of residual policy, we utilise a measure called  $r_{sim}$ . A higher value suggests the final action more closely follows the primitive policy. As shown in Figure 12 (b) Stage1, when the hand is far from the object, the residual policy will restrict the finger's early closure to prevent the collision, as the hand approaches the object, residual policy starts to follow primitive policy, as shown in Figure 12 (b) Stage2. However, as shown in Figure 12 (b) Stage3, as the hand is about to grasp the object, the  $r_{sim}$  starts to decrease. At the last few steps, the residual policy will further refine the pose to hold the object firmly, leading to the negative  $r_{sim}$  value.
Figure 12: (a) Variation of the mean value of the primitive policy's action over time steps. (b) Variation of the  $r_{sim}$  reward over time steps.
D.2 Real World Evaluation
During the real-world grasping procedure, once the human wrist was detected to lift, the hand would automatically lift up, move towards a target box, and open the fingers to release the object, as illustrated in Figure 13. A grasp was considered successful if the object was lifted and moved to the box without falling before the fingers opened.
Figure 13: Real-World evaluation process for assessing grasp pose success: a) Detection of human wrist lift. b) Hand automatically lift up. c) Hand automatically move to the target box. d) Hand open to release the object.
D.3 Inference Speed for Each Module
We evaluate the inference speed on the GTX 1650, which is also used in our real-world experiment. We set the batch size equal to 1 and ran the policy 50 times to obtain a reliable average time for the inference speed of each module. As shown in Table 5, both modules take less than 0.004 seconds for each inference. This indicates that our integrated system is capable of seamless human interaction.
Device	Inference time for primitive policy	Inference time for residual policy
GTX1650	0.002766s	0.003646s
Table 5: Results of inference speed of primitive policy and residual policy on GTX1650.
D.4 Robustness to Perception Noise
To demonstrate the robustness of our method, we inject two levels of noise to the wrist pose observation following [48]. As indicated in Table 6, our approach yields comparable outcomes
with a 2-degree/2-cm estimation error while exhibiting approximately  $10\%$  reduction in performance under a 5-degree/5-cm error threshold, which indicates that our method can handle estimation error to some degree.
standard deviation of noise	0°, 0cm	2°, 2cm	5°, 5cm
success rate	56.69%	55.24%	44.35%
Table 6: Results of success rate under different levels of observation noise on unseen category instances using seed 0. The Gaussian noise is determined by the standard deviation of the specified threshold and added to observations of wrist position and orientation separately, any noise exceeding these bounds will be clipped. The pose estimation method can achieve  $80.99\%$  accuracy under  $2^{\circ}$ ,  $2\mathrm{cm}$ , and achieve  $95.80\%$  accuracy under  $5^{\circ}$ ,  $5\mathrm{cm}$ .