Learning Score-based Grasping Primitive for Human-assisting Dexterous Grasping
Authors: Tianhao Wu
Mingdong Wu
Jiyao Zhang
Yunchong Gan
Hao Dong
Venue: NeurIPS 2023 poster (2023)
Keywords: Human-asissting Dexterous Grasping
Score-matching
Reinforcement Learning
TL;DR: We introduce a novel human-asissting dexterous grasping task and propose a RL framework incorporated with score-matching.

Abstract: The use of anthropomorphic robotic hands for assisting individuals in situations where human hands may be unavailable or unsuitable has gained significant importance. In this paper, we propose a novel task called human-assisting dexterous grasping that aims to train a policy for controlling a robotic hand's fingers to assist users in grasping objects. Unlike conventional dexterous grasping, this task presents a more complex challenge as the policy needs to adapt to diverse user intentions, in addition to the object's geometry.  We address this challenge by proposing an approach consisting of two sub-modules: a hand-object-conditional grasping primitive called Grasping Gradient Field (GraspGF), and a history-conditional residual policy.  GraspGF learns 'how' to grasp by estimating the gradient of a synthesised success grasping example set, while the residual policy determines 'when' and at what speed the grasping action should be executed based on the trajectory history. Experimental results demonstrate the superiority of our proposed method compared to baselines, highlighting the user-awareness and practicality in real-world applications. The codes and demonstrations can be viewed at https://sites.google.com/view/graspgf.

GraspGF: Learning Score-based Grasping Primitive for Human-assisting Dexterous Grasping
Tianhao Wu 1,2.3" Mingdong Wu 1.3" Jiyao Zhangl22 Yunchong Gan', Hao Dong!
Center on Frontiers of Computing Studies, School of Computer Science, Peking University Beijing Academy of Artificial Intelligence
National Key Laboratory for Multimedia Information Processing,
School of Computer Science, Peking University
jiyao...yaazhanggJaa pku. edu. en, yunchong, hao. haw.dong! dong)@pku edu. en
Abstract
The use of anthropomorphic robotic hands for assisting individuals in situations where human hands may be unavailable or unsuitable has gained significant im- portance. In this paper, we propose novel task called human-assistie grasping that aims train policy for controlling a robotic hand's fingers assist users in grasping objects. Unlike conventional dexterous grasping, this task presents more complex challenge as the policy needs to adapt to diverse user intentions, in addition to the object's geometry. We address this challenge by proposing an approach consisting of two sub- sub-modules: hannd-bjject-conddtinall grasping primitive called Grasping Gradient Field (GraspGF), and a history- conditional residual policy. GraspGF learns 'how' to grasp by estimating the gradient from success grasping example set, while the residual policy deter- mines 'when' and at what speed the grasping action should be executed based the trajectory history. Experimental results demonstrate the superiority of our proposed method compared to baselines, highlighting user awareness and practi- cality in real-world applications. The codes and demonstrations can be viewed at attps://sites google..o.....oooooooooooo.o.... watggle..o.vviiiee com/view/ graspgf.
dexterous
Introduction
The significance of human hands in everyday life cannot be overstated. However, there are sit- uations where they may not always be available, especially in scenarios where an individual may have upper limb loss or need to interact with hazardous objects. In such instances, utilising an anthropomorphic dexterous robotic hand for assistance can be viable option [1]. Such dex- terous hand possesses a high degree of freedom, allowing it to handle diverse daily tasks [2, 3, 4], given that many everyday objects are designed match the structure of the human hand. This inspired us to propose novel task called human- assisting dexterous grasping, in which a policy is trained to assist users with upper limb loss in grasping objects by controlling the robotic hand's fingers, as illustrated in Figure la.
step
step
to grasp?
group?
Objects
trajectory
trajectory
gose
pose
Figure 1: a) Demonstration of human-assisting dexter- ous grasping. b) Challenges of our setting.
Traditional teleoperation methods [5, 6) are unsuitable for assisting upper limb amputees in grasping because no information about the human fingers can be accessed. Compared to conventional dexterous
equal contribution, f: corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

grasping, human- human-assisting dexterous grasping poses more complex challenge, as the policy must adapt to an exponentially growing number of pre-conditions. As shown in Figure Ib, human users may grasp an object with various intentions, such as grasping different parts for different purposes or moving the hand and wrist at different speeds due to the complexity and diversity of human behaviour. Consequently, the conditional policy must be tailored not only to the object's 8 geometry, as required by conventional dexterous grasping, but also to the user's intentions, requiring the policy to be user-aware. In this context, open-looped methods such as grasp pose generation [7, 8, 9, 10] and classification-basee methods [11, 12, 13] may fall short, as they do not factor in the user's intention. Reinforcement learning (RL) [2. 14, 15] presents natural solution by enabling the training of closed-loop human-objeet object-conditionee policy. However, in auman-assisting dexterous grasping, RL may encounter more severe generalisation issues, due to the need to generalise to diverse grasping pre-conditions. Prior RI.-based approaches [16, 17] have explored leveraging human-collected and ongineering-heay demonstrations to address this issue. However, collecting large volume of diverse demonstrations that encompass different objects, grasping timings, and locations may not be feasible.
To address the challenges associated with achieving dexterous grasping for assisting humans, an effective policy needs to tackle the following two crucial questions:
1) How should the robot grasp the object considering the current relative pose between the user and the object? 2) When and at what speed should the robot execute the grasping action based on the user movement trajectory history?
In this paper, we present a novel approach that consists of two sub-modules designed address the aforementioned questions individually: 1) a hand-ohject-condiioond grasping primitive, and 2) distory-conditionn Field (GraspGF), is trained to learn 'how to grasp' by estimating the score function, i.e., the gradient of the log-density, of success grasping example set. The GraspGF outputs gradient that indicates the fastest direction to increase the 'grasping likelihood' conditioned on an object and the user's wrist. The gradient can be translated into primitive controls on each finger joint, enabling the fingers to reach appropriate grasp pose iteratively. However, GraspGF is not capable of determining how fast the fingers should move along the gradient as is history-agnostic. To determine 'when to grasp', we train a residual policy that outputs a 'scaling action' that determines how fast the finger joints should move along with the primitive action, based on the history of the wrist's trajectory. Besides, as the primitive policy is not aware of the environment dynamics due to the offline training, we further require the residual policy to output a 'residual action' to correct the primitive action.
residual policy. The grasping primitive, which we call Grasping Gradient
Our proposed approach offers several conceptual advantages. GraspGF leverages the strong condi- tional generative modelling of score-based methods, as demonstrated in prior works [18, 19, 20], enabling it to output promising primitive actions conditioned on novel user's intentions. Additionally, the lesidual-learing design of GraspGF facilitates cold start exploration and enhances the efficiency of residual policy training. Compared to demonstration based methods, our approach only requires synthesised grasping example set and does not rely on exhaustive human labelling or extensive engineering effort, making it more practical for implementation in real-world applications.
In our experiments, we evaluate several methods a dexterous grasping environment that assists humans in grasping over 4900+ on-table objects with up to 200 realistic human wrist movement patterns. Our comparative results demonstrate that our proposed method significantly outperforms the baselines across various metrics. Ablation studies further confirm the effectiveness of our proposed grasping gradient field and residual-learning design. Our analysis reveals that our method's superiority lies in its user- awarenes,,,,,, i.e., our trained policy is more tailored to the user's 's intentions. Additionally. we conduct real-world experiments to validate the practicality of our approach. Our results indicate that our trained model can generalise to the real world to some degree without fine-tuning.
Our contributions are summarized follows:
We introduce a novel and challenging human- aasaaniisisiingggg dexterous grasping task, which may potentially help social welfare.
We propose novel two-stage framework that decomposes the task into learning a primitive policy via score-matee matching and training a residual policy to complement the primitive policy via RL.
We conduct experiments to demonstrate our method significantly outperforms the baselines and the effectiveness of our method deployed in the real world.
2

---

2 Human-assisting Dexterous Grasping
We study the human- mannaansssising dexterous grasping, in which a policy is trained to assist users in grasping objects by controlling the robotic hand's fingers. We formulate the problem as follows:
State and Action Spaces: In this task, we consider a human-assisting grasping scenario involving a 28-DoF 5-fingered robotic hand. The 18-DoF joints of the fingers are denoted as J E IR the 4-DoF under-actuated joints of the fingers are denoted as J IR and the 6-DoF pose of the wrist is represented by b = [bp, byl, where by E R denotes the 3-D position and by E R represents the 4-D quaternion. The action space A C IR corresponds to the 18-D relative changes applied to the hand joints. Unlike traditional dexterous grasping tasks, the action space does not include the 6-1) relative changes for the wrist, since the wrist pose is controlled by a human user.
Task Simulation:
To simulate the movement of the human user's wrist, sample wrist trajectory at the start of each episode Tb = (b,, by... by} (T' denotes the horizon). At each time step t, the wrist's pose is set to b,. Specifically, we initially sample target object O (O(O) from an object prior distribution. Then, the wrist trajectory is sampled Tb ~ Pr., (Tb O) conditioned on the target object O. The terminal wrist state, by, is designed to be 'graspable' In other words, there exists feasible hand joint J such that the grasp pose J by] can successfully grasp the object O.
This task requires the agent to adapt to both the wrist trajectories Th and different objects O. Consequently, the policy π ([ll-) should be conditioned on the finger joints J, visual observations and the history of hand wrist poses H, = [b,-k: b.- b.). where 1: is a hyper- parameter. In this work, we use the full point cloud of the target object as the visual observation o and
Observations:
consider the last five wrist states as the history Hi [b,-4, .... byl. Note that the RL-based policies used in all the experiments also take under-actuated joints J E I2d as input, for simplicity, we omit this input in the following notations.
Objective:
The objective is to find policy T(a o, H) that maximizes the expected grasping success rate over the initial distributions, i.e. O ~ (O) and Th ~ Pr, (T]/O):
π arg max E owpo by (o (1)))) (success)
(I)
Eq. poses a challenging objective since the policy should generalize not only to different objects 0 (O), required in conventional dexterous grasping, but also to different hand wrist trajectories Th ~ Pr. Pr. (T.(O). In other words, the agent should be user-aware.
3 Method
Overview: A user-aware policy needs to tackle the following two crucial questions: 1) How should the robot grasp the object considering the current relative pose between the user and the object? 2) When and at what speed should the robot execute the grasping action based on the user movement trajectory history? As illustrated in Figure 2, our key idea is to partition the task into two stages that address these questions individually: 1) Learning a primitive policy T (af J Ot, by) that proposes primitive action ad that can guide the fingers forming into a pre-grasp pose, from an success grasping pose example set. 2) Learning residual policy T (af, a. 0t, H2) that outputs scaling action a, to determine 'how fast' the joints should move with the primitive action and a residual action a that further corrects the overall action, via RL. The combined policy 0.0 (a Jr, o,, He) is as follows:
af ~ (a Jr, or, by), af) ~ mH (af J[ or, H.)
t (-J or, H.) = af O of +
Initially, we employ the score-matee moteenmt to train the primitive policy T.. from grasping poses dataset. Subsequently, the combined policy, which is constructed from the residual policy T. is combined with the frozen n.f. and is trained under RL. In the following, we will introduce the motivations and the training procedures of the primitive policy (i.e., GraspGF) and the residual policy See 3.1 and See 3.2, respectively. The implementation details of both policies are described in Appendix B.
3

---

0 n
x
b
R
Score
πt
O of
+
Matching
Perturb
Tsm
Og
di
2
πT 0
(b)
Figure decompose human human-assisting assisting dexterous grasping into learning primitive policy ny that learns form pre grasp pose and residual policy нद that learns adjust the proceeding of the primitive action. a) The primitive policy иp trained success grasping examples score score- mattcmig objective. b) The residual policy x trained to adjust the primitive policy via RL.
3.1 Learning GraspGF from Synthetic Examples
To address the first question above, we aim to search for primitive policy T that outputs action to maximize the likelihood of success, given set of static conditions (J,, or, b,). Inspired by [21], we can train such a policy by estimating the score function (i.e., gradient of the log-density) of conditional distribution Psnccess (Joor, b:):
(- = VJ log Paaccess (Joo, b)
the Psuccess (J b) denotes a fingers-joints' distribution that can successfully grasp the objects, given the current observation o and the hand wrist b. By definition, the score funetion of the distribution
VJ log (J b) indicates the fastest direction to increase the likelihood the Paucess (Joo, b). Intuitively, if the fingers move in the direction of V log Psaccess (J b). they will probably reach feasible grasp-pose in the future with the success-likelihood Psaccess (Jlo, o b) increases. Hence, we formulate learning the primitive policy TH as estimating the gradient field the log-success: -ikeeess--ikelhoood
JJ log Psaccess (J 0, b), namely Grasping Gradient Field (GraspGF).
Thanks to the Denoising Score Matching (DSM) [22], we can obtain guaranteed estimation of the GraspGF VJ log Pauccess (J lo, b) from of success examples D suscess [J; of. b where the J; is the feasible finger joints for grasping conditioned on (o b In the following, we first revisit the preliminaries of DSM and then introduce how to employ DSM to estimate the GraspGF.
Denoising Score-Matching Given a set of data-points {x; ~ ) from an unknown data distribution p(x), the score-based generative model aims estimating the score function of data distribution Vx log log)(x) via score network S., (x) IR [X) [ During inference, new sample is generated by the Langevin Dynamics, which is out of our interest.
To estimate Vx log p(x), the Denoising Score- Matte-ng (DSM) [22] proposes tractable objective by and train the score network to denoise
pre-specifying a noise distribution do (x(x), e.g., e.g...N AO, o²I). the perturbed data samples:
[(w) = E xxaa x-x(x)
[||s (x)
Vx go [x]
where Vx
(x are tractable for the Gaussian kernel. DSM guarantees that the optimal score network holds s (x) Vx log p(x) for almost all
go lo
Employing DSM to Estimate GraspGF To estimate the GraspGF VJ log Penccess (Jo,, b), employ the DSM in Eq 4, the training objective of the primitive policy TT.. is derived as follows:
t T- (b'',b) J J--
(()) = E
(d (3",0" Disseo
o2
(5)

---

The Eq is the L2 distance between the output of the primitive policy and the denoising direction J' J' ie., direction pointing from the perturbed joints J to the original joints J Intuitively, this objective is forcing the primitive policy to denoise the current joints to regions where the fingers are more likely to grasp the object.
3.2 Training Residual Policy via Reinforcement Learning
The human-assisting dexterous grasping cannot be effectively addressed solely through the primitive policy n.. Although = predicts the fastest direction to form pre-grasp pose, it fails determine the appropriate 'velocities' at which the joints should move in that direction. If the fingers close too quickly or too slowly, the agent may struggle grasp the object. Additionally, due offline training. the primitive policy lacks awareness of the dynamics of the environment, leading to potential violations of physical constraints.
To overcome these limitations, we propose the training of a residual policy xf. which complements the primitive policy. The role of n is twofold: firstly, it outputs a scaling action a" that adjusts the speed of the primitive action, and secondly, it produces a residual action a" that corrects the final output action. Eq 2 demonstrates that a" can be interpreted as 18-D "pseudo-velocitie" imposed on the finger joints. With outputs less than I for a", the residual policy can decelerate the primitive action, whereas values greater than accelerate it.
To effectively control the proceeding of the primitive action, the residual policy (a]]]J or, H₂) takes into account the history of the wrist H, and the object's point cloud o as inputs. This enables the policy to infer the agent's 's speed of approach towards the object. Furthermore, the policy network incorporates the current joint state J, to infer how to correct the primitive action aP.
We employ Proximal Policy Optimization (PPO) [23] to search for a final policy 8.6 that maximises the following objective, where the primitive policy's 's parameters 0 are frozen during training:
T
J(Ø) Eo-so mppp (n61)). (e)
Σ
(6)
Lt=
where 7 > 0 denotes a discounted factor and rr is the reward at time-step t. To encourage the policy to successfully grasp the object while leveraging the primitive action a for efficient exploration, we assign the following simple reward function for training:
= (1 - de) (rrsim + rn) + de As de = (ssuceess, t = T)
a²
ssm Aa-
Jr - JI-1) ,n A Ah
||a|| 2
where Aa An. A O are hyperparameters and Ah represents the change in the height of the object's centre of gravity after lifting. The term d, rewards the agent if the final grasp pose can successfully lift the target object. The term r'm is intrinsic reward that encourages the agent when the joints-change follows the direction of the primitive action aç. We defer the PPO's hyperparameters to Appendix B.
3.3 Implementation Details
Primitive Policy Network This module takes the hand joints J 1 object point cloud
3x310024 and the hand wrist SE(3) as input, we first project the point cloud into the hand wrist's s coordinate ob- We encode ob into a 512-D global feature by PointNet++ [24]. The hand joints J and the noise-condition E [0, 1] are further encoded into 1024-D and 512-D features respectively. Concatenating the features together, feed the concatenated feature into MLPs to obtain the 18-D
output.
Residual Policy Network Taking the hand joints J, E R188 under-actuated joints J E R4, visual observation 0 E R3x1021 and the hand wrists" trajectory H, = [b. ..., by-4], b, E SE(3) as input, this module use the PointNet [25]to encoder the point cloud. Similarly, we first obtain the global feature for g212 of the projected point cloud. The primitive action af = (J,, or, by), hand joints J and the history Hl are concatenated and further encoded into 512-D features. Concatenating the above features together, we feed the concatenated feature into MLPs to obtain the 36-D output.
5

---

Experiment Setups
4.1 Task Simulation
Environment setup: We created simulation environment based on the ShadowHand environment in Isaac Gym [26]. using the ShadowHand model from IBS [27]. The simulation environment enables parallel training on hundreds of environments, Each environment consists of object placed on the ground and a hand that follows a pre-generated human wrist trajectory. The agent can only control the joints of the hand. The episode horizon is 50 steps, and each episode only terminates at the final step. Following a similar setting to IBS [27], when the episode terminates, part of the joints of each finger will automatically close by 0.1 radians, and then the wrist of the hand will be lifted by Im.
Grasping pose generation: We created our success grasping pose based on the UniDexGrasp dataset [10]. We filtered the data to match our Shadow model's degrees of freedom. The dataset was split into three sets: training instances (3127 objects, 363,479 grasps), seen category unseen instances (519 objects, 2595 grasps), and unseen category instances (1298 objects, 6490 grasps)
Human grasping wrist patterns: To mimic real human grasping patterns, we resampled 200 real human grasping wrist trajectories from HandoverSim! [28], from which we extracted 200 wrist movement patterns. These patterns split into 150 training patterns and 50 testing patterns.
The details are deferred to Appendix A.
4.2
Metrics
Following the VIP [29], we report the following three metrics: 1) Success: The object can be lifted more than 0. off the table, while the change distance between the object and hand position after lifting is smaller than 0.05m. 2) Posture: The distance between the target human hand pose and the agent hand pose. It tells us how human-like the learned grasps are. Stability: The translation and rotation changes of the object during the agent's s grasping process. Translation is measured by the Euclidean distance between the final object position and the initial object position. Rotation is measured by the cosine distance between the x-axis of the initial object and the final object.
4.3 Baselines
We compare our method with the following RL-based methods. Note that all baselines are re- ertirined by taking the latest 5-frame wrist states as input. 1) IBS [27]: IBS explicitly compute rich contact information according to hand mesh and object mesh, which has good generalisation to objects and different contact situation. We modify the baseline only output joint actions, and keep the reward the same as previous. 2) PPO [23]: We adopt PPO as ours pure RL baseline. For pure RL, we use the reward of fingertip distance (distance of fingertip the closest point of object point cloud), and success reward. 3) PPO (Goal): We randomly sample the goal which corresponds the current object additional input the agent. We add another goal pose matching reward compared to PPO baseline. 4) ILAD [30]: We choose ILAD our RL+Imitation learning baseline, which will first generate sub-optimal grasping trajectories according to example grasp pose, then use these demonstrations for imitation learning and RL. We regenerate the trajectories for ILAD training based on our grasp data. We use the same reward as the PPO baseline. The details are deferred to Appendix C.
Initial
State
Final
State
GT
PPO
PPO
ILAD
IBS
Ours
GT
Traj1
Traj2
Traj3
Traj4
TrajS
(Goal)
Figure 3: Qualitative results of comparison with baselines and different trajectories. a): final grasp poses of different methods. b): final grasp poses of method under different human trajectories

---

50
62-
60
Success (%6
30
32-
30-
Ours
105
PPQ
PPO(Goa)
2.0
Success (%) f Posture (%)
1 Success (%6 Posture (%)
Samples (x10")
Seen category unseen instances
Unseen category instances
Figure 4: Quantitative comparative results. Left: training curve of different methods. Note that IBS takes 144 hours on V100 reach 3.5 million agent steps, while ours only takes 15 hours reach 10 million agent steps. Middle: Success and Posture of different methods on seen category unseen instances. Right: Success and Posture of different methods on unseen category instances.
5
Experimental Results
5.1 Comparative Results
As shown in Figure 4, Ours achieves comparable training efficiency to ILAD, even without the use of extra trajectory demonstrations, Ours can converge higher Success. Ours also surpasses other baselines of Success for both unseen and unseen category instances. This demonstrates the robust generalisation capabilities of our method. Ours also has the highest performance of Posture, which indicates that our method is better aware of the user's intentions. The qualitative results Figure b) demonstrate the successful grasping of objects across different user's intentions. As indicated in Table 1, Ours also causes the least disturbance to the object.
ILAD and IBS are considered as the strongest baseline. ILAD utilises additional trajectory demon- strations for RL training. However, Ours still surpasses this baseline because ILAD still relies on RL to generalise across diverse object categories. For IBS, we observed that IBS tends to fail when the human wrist trajectory has the potential to collide with the ground. This could be attributed to the reward design of IBS, which needs to balance between colliding with the scene and grasping the object. Additionally, computing the IBS representation is ten times more computationally expensive.
PPO primarily fails because it at- tempts learn general policy that can grasp most objects, as shown in Figure a), which is not suitable for diverse objects and grasp poses. PPO (Goal) faces challenges due to its inabil- ity to adjust the goal during the approach phase. Since the wrist continuously moving during the process, the goal set at the initial state may not be suitable.
Seen Category
Unseen Category
Tran(em) 4
Rot (rad) 4
Tran(em) 4
Rot (rad)
PPO(Gool)
2.621 10.415
0.589
2.537 10.296
0.543
PPO
2.745 168
0.594+0.046
2.771
0.563
IBS ILAD
2.653
0.57210.002
2.596
0.520
2 2.443+0.002
0.548+0.027
2.534+0.101
0.515
Ours
2.131 20.138
0.449
2.127
0.428
Table 1: Results of Stability on seen category unseen instances and category instances. Tran: translation of objects from the initial position to the final position. Rot: rotation of objects from initial orienta- tion to final orientation.
5.2 Ablation Studies and Analysis
We conduct the ablation studies to investigate: 1) The effectiveness of decomposing the policy into primitive policy and residual policy. 2) The necessity of different action modules. 3) The impact of different action modules on final action.
The ablation results depicted in Figure demonstrate the significance our proposed approaches. Ours w/o GF experiences significant performance decline, indicating the significance of combining RL policy with primitive action. Similarly, Ours w/o RL. shows a substantial drop in performance by focusing solely the 'how' of grasping, which results in collisions during the grasping process.
7

---

Although Ours w/o a" and Ours w/o a" exhibit higher training efficiency, both methods eventually experience training collapse. During this collapse, we observe that the agent tends either follow the primitive action more closely or deviate from it to greater extent. This suggests that the combination of both action modules allows for a better policy learning process that effectively utilises the primitive action.
The results shown in Table 2 indicate that the primitive action, which does not consider col- lisions due physical constraints the grasp- ing procedure, can achieve similar performance Ours. This suggests that the primitive ac- tion module has good understanding of how to grasp, but it moves quickly towards the tar- without considering potential collisions, as shown Figure 6. The inclusion the mod- ule slows down the progress of a" to avoid colli- sions, as illustrated in Appendix D.I. However, this conservative approach leads to drop in suc- cess. Nevertheless, after further addition a", the performance increases 56.50% which is comparable to a" w/o coll. By further combin- ing the a" module, the final action is corrected to achieve the 'how to grasp' knowledge learned by a" while also incorporating correct 'when to grasp' knowledge.
50
Success (%)
20-
Ours wio RL
Ours weo
Ours weo
Ours weo a²
Samples (x10')
Figure 5: Ablation Study decomposing policy and different action modules.
Action Type
Success
aP w/o coll a" の a'
19.49 %
55.60 %
5.08 %
aP a² + a"
56.50 %
Table Success rate of differ- ent combinations of action mod- ules.
Figure 6: Qualitative results of grasping procedure governed by the primitive policy. yellow circle highlights the collision between the finger the object caused by premature closure.
5.3 Adaptability Results
To further demonstrate the adaptability of our method, conduct both quantitative and qualitative experiments. Quantitatively, as shown Table 3, we calculate the success rate for each object across five different grasp poses. For the objects that have been successfully grasped, Ours excels grasping various parts of the objects. Qualitatively, Figure 7 demonstrates that the grasp pose generated by Ours closely resembles human's intended grasp pose. Furthermore, also show that GraspGF can adapt to wrist rotation; videos can be viewed at htttps://siteee http://ssiess..gglleeeeeee
Success Rate
1/5
3/5
4/5
GT
Ours
13.67%
27.70%
31.41 %
20.82 %
6,40 %
ILAD
25.44%
32.53%
26.61% %
12.53%
2,89%
IBS
27.70%
34.73%
25.67%
10.29%
1.61%
Qurs
PPO
48.20%
35 .9%%
13.10%
3.01%
0.20%
PPO (Goal)
77.04%
18.96%
3.62%
0.30%
0.08%
Pose
Table 3: Percentages ohjects with varying success rates grasp- ing under different methods. The results are averaged over five different random seeds.
Figure Qualitative results of grasp- ing diffeernnt parts shoes with the same shape but different scales.
5.4 Real-world Results
In this section, we construct real-world system to validate our method. As shown in Figure 8, the system consisted of calibrated multi-view RGB-D cameras (four Intel RealSense D415 sensors),

---

UR10e robot arm equipped with Shadow Hand as the end-effector, and an optical hand tracking sensor (Leap Motion Controller). To obtain the visual observation of the object, we reconstruct the scene point clouds from the multi-
view RGB-D cameras. We only get
the point cloud at the beginning of each grasping, assuming the point cloud observation does not change during the grasping. Then, we de- velop a teleoperation system inspired by the design of DexPilot [5] to en- able the robot arm to track the trajec- tory of a human wrist, but with the dif- ference of using a Leap Motion Con- troller as the hand tracking device to obtain human wrist and hand pose. It should be noted that we remove the estimated hand pose emulate the human-aae aasaannssiisinng setting where only the wrist can be controlled by humans.
100cm
wrist pose
pain
URI0e
wrist pose
Intel RealSense
D415
Shadow Hand
wrist pose!
wrist pose
Leap Motion
Controller
Figure 8: Real-world Experiment. Left: the real-world setup for real human-assisting dexterous grasping. Right: the qualitative result of real human-assisting dexterous grasping.
Following a similar setting in [16], evaluated our policy in the real world using ten objects from the YCB dataset [31]. For each object, we selected two grasp poses that were intended to be achieved by a human. Each pose was tested five times, with human randomly randomised initial object pose and wrist movement trajectories. The details of evaluation in the real world are deferred to D.2. As shown in Table 4, our policy achieved an average success rate of 66% in real-world human-assisted grasping, demonstrating its generalisation ability in real-world setting. The grasping policy tended to fail for certain large objects that were difficult for the dexterous hand to hold firmly, as well as for thin objects that were prone to colliding with the table. One of the reasons our policy still achieved a certain success rate in the real world is that when humans control the wrist of their hand, human will also try adjust according to the grasping pose of the joint, which may help to grasp the object. Our real-world demonstrations can be viewed at https: //tt///stttsseeeeeeee gowgle...oo...ooooooooo com/ com/view/ gr graspgf.
CHEEZ
Average Success
Object
Rate
Pose Pose 2
5/5
5/5
5/5
1/5
1/5
5/5
4/5
5/5
1/5
1/5
66 %
4/5
3/5
3/5
415
2/5
1/5
4/5
Table 4: Results of success rate on different YCB objects with two different poses the real world.
Related Works
6.1 Dexterous Hand Grasping
Existing studies on dexterous hand robotics mainly focus on dexterous grasping, assuming full control over the hand, including both the hand base and joint movements by the agent [27, 32, 14, 29, 30].
However, Reinforcement Learning (RL) based methods struggle with generalisation to different objects due object diversity, nigh-dimensional state and action spaces, and sparse rewards [33]. To address this issue, most existing approaches rely on human-collecter demonstrations [17, 16]. which require engineering heavy retargeting. The most relevant work, IBS [27], combines RL with a numan--esigged representation that explicitly incorporates contact information between the hand and objects to aid grasping. However, this computationally costly representation limits the practicality. Furthermore, in our proposed approach, we address different setting where the movement of the hand base determined by the user, requiring the agent be 'user-centric" rather than 'object-centric' Additionally, our approach only requires set of grasp examples and is free of human design.

---

In the medical AI field, there are studies with similar setting to ours, known as prosthetic dexterous grasping [11, 12, 13]. These studies aim control prosthetic hand to assist users in grasping and explore open-looped approaches that initially predict the grasp type and then execute predefined primitives for control. contrast, we explore a closed-loop control policy that adaptively adjusts the grasp pose based on the current user-object relationship and the user's movement trajectory history.
6.2 Score-based Generative Models
In the pursuit of estimating the gradient of the log- lilkiiikeoood associated with given data distribution, the score-based generative model, originally introduced by [34], has garnered substantial attention in the research community [22, 34, 35, 36, 37, 19, 38]. The denoising score-matching (DSM), as proposed by [22], further introduces a tractable surrogate objective for score-matching. To enhance the scalability of score-based generative models, [35] introduces a sliced score-matching objective that projects the scores onto random vectors before comparing them. Song et al. also introduce annealed training for denoising score matching [36], along with corresponding improved training techniques [37]. They also extend the discrete levels of annealed score matching to a continuous diffusion process and demonstrate promising results in image generation [19]. Recent works further explore the design choices of the diffusion process [39], maximum likelihood training [38], and deployment on the Riemann manifold [40]. These recent advances show promising results when applying score-based generative models in high-dimensional domains and promote wide applications in various fields, such as object rearrangement [21], medical imaging [,1], point cloud generation [42], scene graph generation [43], point cloud denoising [44], depth completion [45], and human pose estimation [18]. These works formulate perception problems into conditional generative modelling or inpainting, allowing the utilisation of score-based generative models to address these tasks.
In contrast, our focus lies in the application of score-based generative models for training low-level control policies. In this domain, [46] and [20] have proposed learning diffusion models from offline trajectories and leveraging these models for model predictive control. However, these approaches suffer from the drawback of requiring a substantial amount of offline data and inefficiency during test-time sampling. To the best of our knowledge, our method represents the first exploration of score-based generative models for learning closed-loop dexterous grasping policies.
Conclusion
In this work, we introduce human-assisting dexterous grasping, wherein a policy is trained to assist users in grasping objects by controlling the robotic hand's fingers. To search for user-aware policy, we propose a novel two-stage framework that decomposes the task into learning a primitive policy via ecore-matching and training residual policy to complement the primitive policy via RL. In experiments, we introduce a numan-assistinn dexterous grasping environment that consists of 4900+ on-table objects with up to 200 realistic human wrist movement patterns. Results demonstrate that our proposed method significantly outperforms the baselines across various metrics. Our analysis reveals that our trained policy is more tailored to the user's intentions. Our real-world experiments indicate that our learned policy can generalise to the real world to some degree without fine-tuning.
Limitations and Future works. Our method takes the full point cloud as the visual observation, which is not accessible in the wild. In the future, we may leverage teacher-student learning to generalise our method to a partial observation setting.
Ethics Statement. Our method has the potential develop home-assistant robots and assist individu- als with hand disabilities, thus contributing to social welfare. We evaluate our method in simulated environments, which may introduce data bias. However, similar studies also have such general concerns. We do not see any possible major harm in our study.
Acknowledgments and Disclosure of Funding
We would like to thank Qianxu Wang and Haoran Lu for supporting the real-world experiments. This work is supported by the National Natural Science Foundation of China - General Program (Project ID: 62376006), the National Youth Talent Support Program (Project ID: 8200800081) and the Beijing Municipal Science & Technology Commission (Project ID: Z221 222112000000 10000342204)
10

---

References
[1]
Francesca Cordella, Anna Lisa Ciancio, Rinaldo Sacchetti, Angelo Davalli, Andrea Giovanni Cutti, Eugenio Guglielmelli, and Loredana Zollo. Literature review on needs of upper limb prosthesis users. Frontiers in neuroscience, 10:209, 2016.
[2] Yuanpei Chen, Tianhao Wu, Shengjie Wang. Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAleer, Hao Dong, Song-Chun Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Systems, 35:5150-5115 2022.
[3] Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang. Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In Com- puter Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Purt XXXIX, pages 570-587. Springer, 2022.
[4] Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMLR, 2020.
[5] Ankur Handa, Karl Van Wyk, Wei Yang. Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, and Dieter Fox. Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164-9170. IEEE, 2020.
[6] Dandan Zhang, Weiyong Si, Wen Fan, Yuan Guan, and Chenguang Yang. From teleoperation to autonomous robot-assisted microsurgery: A survey. Machine Intelligence Research, 19(4):288 306, 2022.
[7] Min Liu, Zherong Pan, Kai Xu, Kanishka Ganguly, and Dinesh Manocha. Generating grasp poses for a high-dof gripper using neural networks. In 2019 IEEE/RSJ International Conference
on Intelligent Robots and Systems (IROS), pages 1518-1525. IEEE, 2019.
[8] Min Liu, Zherong Pan, Kai Xu, Kanishka Ganguly, and Dinesh Manocha. Deep differentiable grasp planner for high-dof grippers. arXiv preprint arXiv:2002.015500 2020.
[9] Puhao Li, Tengyu Liu, Yuyang Li, Yiran Geng, Yixin Zhu, Yaodong Yang. and Siyuan Huang. Gendexgrasp: Generalizable dexterous grasping. arXiv preprint arXiv:2210.00722) 2022.
[10] Yinzhen Xu, Weikang Wan, Jialiang Zhang. Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. arXiv preprint arXiv:2303.00m 00938, 2023.
[11] Ghazal Ghazaei, Ali Alameer, Patrick Degenaar, Graham Morgan, and Kianoush Nazarpour. Deep learning- artificial vision for grasp classification in myoelectric hands. Journal of neural engineering, 14(3):036025, 2017.
[12] Ghazal Ghazaei, Federico Tombari, Nassir Navab, and Kianoush Nazarpour. Grasp type estimation for myoelectric prostheses using point cloud feature learning. arXiv preprint 1908.0/900.225500 2019.
[13] Chunyuan Shi, Dapeng Yang, Jingdong Zhao, and Hong Liu. Computer vision-based grasp pattern recognition with application to myoelectric control of dexterous hand prosthesis. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2899))2000-2099, 2020.
[14] Yuzhe Qin, Binghao Huang, Zhao- Heng Yin, Hao Su, and Xiaolong Wang. Dexpoint: General- izable point cloud reinforcement learning for sim-to-real dexterous manipulation. In Conference on Robot Learning, pages 595-605. PMLR, 2023.
[15) Tianhao Wu, Fangwei Zhong, Yiran Geng, Hongchen Wang, Yongjian Zhu, Yizhou Wang, and Hao Dong. Grasparl: Dynamic grasping via adversarial reinforcement learning. arXiv preprint arXiv:2203.02119 2022.
[16] Zoey Qiuyu Chen, Karl Van Wyk, Yu-Wei Chao, Wei Yang, Arsalan Mousavian, Abhishek Gupta, and Dieter Fox. Dextransfer: Real world multi- fingered dexterous grasping with minimal human demonstrations. arXiv preprint arXiv:2209. 002.099422000 /4284, 2022.
[17] Yueh-Hua Wu, Jiashun Wang, and Xiaolong Wang. Learning generalizable dexterous manipula- tion from human grasp affordance. Conference on Robot Learning, pages 618-629. PMLR, 2023.
11

---

[18] Hai Ci, Mingdong Wu, Wentao Zhu. Xiaoxuan Ma, Hao Dong, Fangwei Zhong, and Yizhou Wang. Gfpose: Learning 3d human pose prior with gradient fields. arXiv preprint arXiv:2212.08641 2022.
[19] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1. 13456, 2020.
[20] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for makii?? arXiv preprint DxXiv:2211.156533 2022.
[21] Mingdong Wu, Fangwei Zhong, Yulong Xia, and Hao Dong. TarGF: Learning target gradient 2022.
field for object rearrangement. arXiv preprint
[22] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661 2011.
[23] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vwXivv:707.06344 2017.
[24] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[25] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[26] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv: 10470, 2021.
[27] Qijin She, Ruizhen Hu, Juzhan Xu, Min Liu, Kai Xu, and Hui Huang. Learning high-dof reaching-and- -nrggsppngg via dynamic representation of gripper-object interaction. ACM Transac- tions on Graphics (SIGGRAPH 2022), 41(4), 2022.
[28] Yu-Wei Chao, Chris Paxton, Yu Xiang, Wei Yang, Balakumar Sundaralingam, Tao Chen, Adithyavairavan Murali, Maya Cakmak, and Dieter Fox. Handoversim: A simulation framework and benchmark for human-to-robot object handovers. In 2022 International Conference on Robotics and Automation (ICRA), pages 6941-6947. IEEE, 2022.
[29] Priyanka Mandikal and Kristen Grauman. Dexvip:
Learning dexterous grasping with human hand pose priors from video. In Conference on Robot Learning, pages 651-661. PMLR, 2022.
[30] Yuch-Hua Wu, Jiashun Wang, and Xiaolong Wang. Learning generalizable dexterous manipula- tion from human grasp affordance. arXiv preprint arXiv:2204.023: 023200 2022.
[31] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The yeb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), pages 510-517. IEEE, 2015.
[32] Qiuyu Chen, Karl Van Wyk, Yu-Wei Chao, Wei Yang, Arsalan Mousavian, Abhishek Gupta, and Dieter Fox. Learning robust real-world dexterous grasping policies via implicit shape augmentation. In 6th Annual Conference on Robot Learning.
[33] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprini arXiv: 1709. 10087, 2017.
[34] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4). 2005.
[35] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pages 574-584. PMLR, 2020.
[36] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
12

---

[37] Yang Song and Stefano Ermon. Improved techniques for training score- based generative models. Advances neural information processing systems, 1112238--12444, 2020.
[38] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415-
1428, 2021.
[39] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364, 2022.
[40] Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet. Riemannian score-based generative modeling. arXiv preprint arXiv:220..02763. 2022.
[41] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models, arXiv preprint arXiv:2111.0000 2021.
[42] Ruojin Cai, Guandao Yang, Hadar Averbuch-Elon Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In European Conference on Computer Vision, pages 364-381. Springer, 2020.
[43] Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Eledath, Gerard Medioni, and Leonid Sigal. Energy-based learning for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13936-13945, 2021.
[44] Shitong Luo and Wei Hu. Score-based point cloud denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4592, 2021.
[45] Ruizhi Shao, Zerong Zheng. Hongwen Zhang, Jingxiang Sun, and Yebin Liu. Diffustereo: High quality human reconstruction via diffusion- stereo using sparse cameras. arXiv preprint arXiv:2207.08000, 2022.
[46] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205. 09991,, 2022.
[47] Kai Chen and Qi Dou. Sgpa: Structure gruuctre-guiied prior adaptation for category-level 6d object pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1773-2782. 2021.
[48] Hansheng Chen, Pichao Wang, Fan Wang, Wei Tian, Lu Xiong, and Hao Li. Epro-pnp: Generalized end-to-end probabilistic perspective-n-poins for monocular object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2781-2790, 2022.
13

---

A
Task Details
A.1 Simulation
Human Wrist Trajectory Generation: We generate the human wrist trajectories using the following steps based the human trajectory pattern dataset in A.2:
Randomly select one object sample for each environment and add random noise to the x and position change of the sampled object pose within the range of (-0.5, 0.5) meters.
Sample grasp pose that is different from the last sampled grasp pose for this object. We measure the distance between each grasp example of the object and the last sampled grasp example. If the sampled grasp example is already in the sample history, we randomly choose new grasp example and clear the history.
Based the sampled grasp pose, generate the initial wrist pose. First, compute the centre of the fingertip pose and the wrist pose to obtain a vector pointing from the fingertip centre to the wrist. Then, we randomly sample the initial wrist position within length range of 0.15 to 0.2 meters and randomly sample a deviation angle within the range of to 20 degrees according to the vector.
For the initial orientation, we randomly sample the delta angle change, denoted 6a. from distribution based on human rotation data. We set the initial angle as Atarget (1 5, 2**)) where Atarget represents the target angle.
After obtaining the initial and final poses of the wrist, we aim to add noise to the trajectories during training. However, simply adding random noise would result in shaking wrist trajectory that does not resemble human behaviour. To address this, we randomly sample two normalized trajectory patterns pl and p2 during training. We then combine these two patterns using random coefficient e sample from (0, to generate a new pattern: c p1 + (1 -)) p2. This generated pattern is then used to define the trajectory of the wrist, which serves as the human wrist trajectory, as shown in Figure 9.
By following the above generation process, we can obtain human trajectories with diverse human- like grasping patterns, diverse velocities, and diverse initial hand poses for diverse objects, while guaranteeing that the agent can grasp the object the final wrist pose.
0
030
0120
511
320
0311
0.3
018
626
of of os accdiig aosaoopoo 20 co6008C30
30 28 26 28
Z06
0.00
358
18.0
Figure 9: Visualizations of generated human trajectories.
State Space: For all joints J and under-actuated joints Ju we normalize the joint states the range of (-1, 1) based on their respective joint limits. As for the wrist pose b, we directly utilize the absolute pose relative the world frame. Regarding the object point cloud o, we sample 1024 points from the object mesh
Action Space: The action corresponds the relative change applied to the 18 joints J. Each action is clipped to the range of (-1, 1) and then scaled by factor of 0.05.
A.2 Datasets
The ShadowHand model used in IBS is equipped with 18 degrees of freedom, including four under-actuated joints that are not directly controlled by the agent. However, in the table-top dataset presented in [10], is assumed that the ShadowHand model has direct control over these four under-actuated joints. Consequently, the grasp examples from the dataset may not be directly applicable to ShadowHand model. To address this, incorporate the grasp poses from
Grasp Example Dataset:
14

---

the dataset into our simulation environment and reevaluate the grasp examples. We assess each grasp example based on the successful lifting of the object, with each grasp pose being tested twice for reliable evaluation.
To ensure fair comparison, we select five of the most diverse grasp poses for each object from the re-filtered dataset for both seen category unseen instances and unseen category instances.
Human Wrist Trajectory Pattern Dataset: HandoverSim!![8) provides handover trajectories collected from real humans, which can be utilized to generate human-like wrist trajectories during grasping in task. However, these trajectories in HandoverSim![22] set of objects and cannot directly applied to all objects in our simulation environment. Therefore, to generate human-like grasping wrist trajectories in our simulation environment, wrist movement trajectories from the initial pose to the pose where the object is successfully grasped. To capture the movement patterns, normalize the trajectories based on the max and min values of each axis (x.y.z). Additionally, for the rotation component, we collect the distribution of wrist rotation change from the initial orientation to the final orientation of each axis (roll, pitch, yaw). We generate human trajectories according to these patterns.
only available for limited
first collect
A.3 Evaluations
For baseline comparison and ablation study, we conducted evaluations using random seeds. For each seed, the evaluation of training instances was performed five times. The grasp pose for each object is different in each iteration. Throughout the evaluations, we utilized the human wrist trajectory patterns randomly sampled from their respective sets without introducing any additional noise. The human trajectory remained consistent across all comparisons for the same random seed.
B Implementation Details
B.1
Primitive Policy
Training Objective: In practice, adopt extension of DSM [19] that estimates time-dependent score network sw (J]o", b²,t) t) to denoise the perturbed data from different noise levels simultancously:
sw (J. 00 b).t) 0²(t) (J j)
((w) ELa
3mp (3* ott bb.b Donoess
A(t)
(8)
2
where T = L, = 10-5, s(t) = (t),,, (t) = - 2 (8mor -mmn)) and 3maz
10, 9min 0.1 are hyper-parameters. The optimal time-dependent score network holds
(J b", f) = Vx log (o b²) where (J b*) ) is the perturbed data distribution:
(J( o², b') = / (a(t) (T]J (Joo", b* b*)dJ
(9)
With the trained score network Sw, we parameterize the primitive policy as:
,"" = (J "0''"""",000000
(10)
Network Architecture:
We utilize the Gradient Field backbone introduced in [21] along with the PoinetNet++ backbone from [47] to build our chosen backbone for this study.
For training the primitive policy, we initially extracted a maximum of five distinct poses for each object within the training instances to balance the training data, resulting in a total of 15,387 grasping examples. is important to note that we exclusively employ grasp examples for the training of the primitive policy, without incorporating any trajectories. We use the Adam optimizer with learning rate of 2e-4 for training, The batch size set to the total number of grasping examples divided by It takes 60 hours to train on single A100 for primitive policy to converge.
15

---

to B.2 Residual Policy
To incorporate object geometry information into the residual policy, we employ PointNet[25]. We begin by pretraining PointNet[25] using the same procedure as the primitive policy training with Poin Subsequently, during training for the residual policy, we continue fine-tuning Poin
Reward Function: Based on empirical observations, we found that the following intrinsic reward function (Eq. 11) which encourages the agent to simply follow the direction of primitive policy will lead to faster convergence.
a² AP
Tsim
Aa
(11)
Note that since the intrinsic reward Fsim the only dense reward our method, we set its frequency to 5 to avoid overfitting to the primitive policy. Empirically, we set X 1.0, Xa- 0.09, and λy. 0.5 for our approach.
RL Backbone: We implemented the Proximal Policy Optimization (PPO) algorithm [23] using the PyTorch implementation provided in [2]. In our implementation, we incorporated the PointNet backbone from [17] into the PPO backbone.
Hyper-parameters: We mainly kept the hyper-parametern of PPO the same those in [2], with the following exceptions: we set the number of update intervals (nsteps) to 50, the number of optimization epochs (noplepochs) 2, the mini-batch size 64, and the discount factor (gammna) to 0.99.
We trained the residual policy for total of 10 million agent steps, which took approximately 15 hours using& a single A100 GPU.
C Baselines Implementations
IBS: In the IBS baseline [27], two buffers are maintained: demonstration buffer for warm-up RL training and an experiment buffer for storing the transition tuples. We set the size of both buffers to 15,000, which is three times larger than in our method. To generate the demonstration buffer, follow a similar procedure in IBS [27]. Specifically, we randomly sample grasp poses from the same data used to train the primitive policy. We then use linear interpolation compute joint actions based on the initial and target joint states. By applying these actions with the generated human trajectories, store the transition tuples in the demonstration buffer until it reaches the maximum size.
Since the PPO baseline does not have a primitive policy, we substitute the intrinsic reward with the fingertip distance reward. It is important to note that we keep the Th and success reward the same as in our approach. collapse. All other training parameters remain the same as in our approach.
PPO:
use PointNet++ [24] for this baseline, as the PointNet may lead to training
PPo(G:))): Similar to the PPO baseline, we use PointNet++ [24] and keep all training parameters the same as in our approach. We introduce a goal pose matching reward that computes the distance between the current joint position and the goal joint position, in addition to the rewards used in PPO.
In the ILAD baseline [17], additional trajectories are required to train RL. We use the same data used to train the primitive policy to generate these trajectories following the same process as described in ILAD [17]. In total, we generate 125k transition tuples. We use PointNet [25] for this baseline and keep all training parameters the as in approach. The reward remains the same as PPO.
ILAD:
D Additional Results
D.1 Additional Ablations
The necessity of a module:
As shown in Figure 10, we visualise the grasp pose of the final state using different action modules aP and a² @ a", the results are generated without simulating the collision between hand and object in the simulation environment. We can observe that a² will usually cause collision between the hand
16

---

and the object. When using the combination of a", the finger still moves towards the similar grasp pose with a", while the module will also try to avoid the collision with the object by slowing down the movement of the a².
a
a S
Final
Final state2
Final
Final state4
Final state5
Final state6
statel
state3
Figure 10: Qualitative result of final grasp pose using a" and a" @ @ action module
The effectiveness of intrinsic reward rsim
As the primitive policy provides good guidance on "how to grasp", we introduce the raim reward to encourage the final output action a² @ + a" explore in direction of the primitive policy's action a² this reward the inner product of [aai|2 and Jj Jt-1, where J signifies the state of finger joints. A higher value of Tsim indi- cates smaller angle between the gradient and Jj J,-1, implying greater similarity in the movement of finger joints to the gradient.
59
a²
Success
30
As shown in Figure 11, when the intrinsic re- ward rsim is included, the success rate of our method drops, and the training efficiency significantly lower compared when Tsim is included. This indicates that Tsim plays a crucial role guiding the agent initially follow the primitive policy and helps accelerate the learn- ing of the residual policy. Furthermore, the in- clusion of rsm encourages the residual policy to follow the primitive policy throughout the entire training process, leading to better utilisation of the generalisation capabilities of the primitive policy.
Ours weo Fam
Samples (x10')
Figure 11: Ablation Study intrinsic reward
Action of each policy during the grasping process:
The primitive policy's 's actions solely depend on object-wrist relative orientation. Thus when the hand far away, the primitive policy will still close the fingers, as shown in Figure 12 (a) Stagel. As the hand's s posture progressively approaches the target grasp pose, the mean value of the primitive policy's s actions decreases, as shown in Figure 12 (a) Stage2.
To further understand the action of residual policy, we utilise a measure called Taim A higher value suggests the final action more closely follows the primitive policy. As shown in Figure 12 (b) Stagel, when the hand is far from the object, the residual policy will restrict the finger's 's early closure to prevent the collision, as the hand approaches the object, residual policy start to follow primitive policy, as shown in Figure 12 (b) Stage2. However, as shown in Figure 12 (b) Stage3, as the hand is about to grasp the object, the rsim starts to decrease. At the last few steps, the residual policy will further refine the pose to hold the object firmly, leading to the negative raim value.
17

---

0.30
0.010
2.28
0.000
0.24
mean of
rsim
3.22
0.002
2.28
6.6
Stage1
Stage2
-0.002
Stagel
Stage2
Stage3
10
(a) Time step
(b) Time step
Figure 12: (a) Variation the mean value of the primitive policy's 's action over time steps. (b) Variation of the Taim reward over time steps.
D.2 Real World Evaluation
During the real-world grasping procedure, once the human wrist was detected to lift, the hand would automatically lift up, move towards a target box, and open the fingers to release the object, as illustrated in Figure 13. A grasp was considered successful if the object lifted and moved to the box without falling before the fingers opened.
b)
Figure 13: Real-We World evaluation process for assessing grasp success: Detection of human wrist lift. b) Hand automatically lift up. c) Hand automatically move the target box. d) Hand open release the object.
D.3 Inference Speed for Each Module
We evaluate the inference speed on the GTX 1650, which also used in our real-world experiment. We set the batch size equal and ran the policy 50 times obtain reliable average time for the inference speed of each module. As shown in Table 5, both modules take less than 0.004 seconds for each inference. This indicates that integrated system is capable of seamless human interaction.
Device
Inference time for primitive policy Inference time for residual policy 0.0036466
GTX1650
0.0027666
Table Results of inference speed of primitive policy and residual policy on GTX1650.
D.4 Robustness to Perception Noise
To demonstrate the robustness of our method, we inject two levels noise to the wrist pose observation following [48]. As indicated in Table 6, approach yields comparable outcomes
18

---

with 2-degree/2-cn em estimation error while exhibiting approximately 10% reduction in performance under 5-degree/5-cm error threshold, which indicates that our method can handle estimation error to some degree.
standard deviation of noise 0%, Ocm
2", 2cm 5" 5.5mm
success rate
56.69%
55.24%
44.35%
Table 6: Results of success rate under different levels of observation noise
category instances using specified threshold and added to
seed The Gaussian noise is determined by the standard deviation observations of wrist position and orientation separately, any noise exceeding these bounds will be clipped. The pose estimation method can achieve 80.99% accuracy under 2", 2cm, and achieve 95.80% accuracy under 5% 5em.
19


Reviews:
[{"review_id": "pL6g62AZHs", "reviewer": "Reviewer_ju1F", "summary": "This paper proposes a new task called assisting grasping. The main difference between this task and classical dexterous grasping is the wrist movement is controlled by a human instead of by the grasping algorithm. The authors propose a two stage method to solve this problem. First, they learn the grasping skill using a successful grasping dataset via score-matching loss. Then, they fine-tune this policy using RL in simulation. They show the proposed method is better than pure RL and score-matching is useful compared to imitation learning algorithms.", "questions": "As shown in Table 2, stage 1 results are already good if there is no collision. I’m curious if it’s possible to add a collision penalty loss in stage 1 training (similar to the \\delta h in RL training) to improve the stage 1 policy?\n\nIs there a more elaborated arguments or evidences that why the proposed task is harder than grasping? On the website, the simulation results look like a classical grasping algorithm while the hand poses are almost the same for all real-world results.", "limitations": "This paper is unlikely to have potential negative societal impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(+) The authors formulate the learning from a set of successful grasps as a denoising problem, which is quite interesting and novel. I think this is an effective design choice.\n\n(+) This paper proposes to separate finger target position and finger movement velocities as two stage problem. This design makes learning more efficient.\n\n(+) The experiments are comprehensive. The authors shows how each of the component affects the final performance of the policy.\n\n(+) It also demonstrates the method in the real-world. ", "weaknesses": "(-) My major concern of this paper is whether the proposed task is more challenging than classical grasping, as claimed by the authors. From my perspective, the grasping process can be roughly divided to 1) hand approaches the object and 2) finger closes. The proposed task use human teleoperation / predefined wrist trajectory for the approaching phase and only learn how / when the finger should grasp the object. In this sense, in terms of task difficulty, what is the difference from firstly moving the wrist to a close-enough position, and then grasp the object under a stationary wrist? Intuitively, I think this is an easier task. \n\n(-) As motivated by my previous argument, there should be more logical arguments on the task difficulty if the author want to emphasize this task “presents a more complex challenge”. \n\n(-) There are formatting issues in particular Table 2.\n\n(-) It relies on perfect point-cloud model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: My major concern of this paper is whether the proposed task is more challenging than classical grasping...; As motivated by my previous argument, there should be more logical arguments on the task difficulty...; Is there a more elaborated arguments or evidences that why the proposed task is harder than grasping?**: \n\nThank you  for bring this up. Due to the page limit, please refer to [Q1 of Common Response](https://openreview.net/forum?id=fwvfxDbUFw&noteId=DtPbwqFxR8).\n\n> **Q2: There are formatting issues in particular Table 2**:\n\nThank you for your suggestion. We will reexamine the formatting of our paper in the future revision.\n\n> **Q3: It relies on perfect point-cloud model**:\n\nThank you  for bring this up. Due to the page limit, please refer to [Q4 of Common Response](https://openreview.net/forum?id=fwvfxDbUFw&noteId=DtPbwqFxR8).\n\n> **Q4:  As shown in Table 2, stage 1 results are already good if there is no collision. I’m curious if it’s possible to add a collision penalty loss in stage 1 training (similar to the \\delta h in RL training) to improve the stage 1 policy?**:\n\nThank you for bring this up. We would like to clarify that incorporating a collision penalty loss in stage 1 does have the potential to enhance collision avoidance in grasp pose generation, but it does not specifically address collision avoidance during the grasping procedure. Due to the page limit, please refer to [Q3 of Common Response](https://openreview.net/forum?id=fwvfxDbUFw&noteId=DtPbwqFxR8).\n\n> **Q5: Website simulation results look like a classical grasping algorithm?**:\n\nThank you for pointing this out. The wrist movement in the website simulation demonstration actually follows a pre-generated human-like trajectory, while the algorithm only controls the finger movement. However, in the classical grasping setting, the algorithm also governs the wrist movement. In order to more effectively illustrate the differentiation between classical grasping and human-assisting grasping, we will render wrist movement trajectories in the revised version of the videos.\n\n> **Q6: While the hand poses are almost the same for all real-world results**: \n\nThank you for raising this point. We have also observed similar resemblances in grasp poses during our real-world experiments. We hypothesize that this phenomenon could be attributed to the sim2real gap, especially in the context of dynamic aspects. For example, we have discovered that the real hand fails to achieve precise poses when the applied action values fall below a certain threshold, resulting in more similar poses. \n\nHowever, our policy still demonstrates the capability to achieve successful grasps across various human movement trajectories, indicating the potential of our approach for real-world applications. To effectively showcase the adaptability of our method, we have undertaken other simulation-based experiments, as detailed in [Q2 of Common Response](https://openreview.net/forum?id=fwvfxDbUFw&noteId=DtPbwqFxR8)."}, {"review_id": "fsHWoPBY4C", "reviewer": "Reviewer_KjWd", "summary": "This paper focuses on addressing a task called human-assisting dexterous grasping. The aim is to create a finger controller to grasp objects with the robot's wrist conditioned on a human user's wrist. The authors propose 1) a Grasping Gradient Field (GraspGF) which estimates the gradient of a synthetic grasping example, and 2) a residual policy achieved through reinforcement learning. Experimental results demonstrate the superiority of the proposed method over previous ones.\n", "questions": "What are the outputs of GraspGF when the hand is at different stages, such as when the hand is far from the object at t_0 and close to the object at t_n? What actions are produced in these instances?\n\nGiven the same initial state and wrist trajectory, can we achieve diversified results?\n\nObserving Figure 4 and Table 1, there is not much difference between the seen and unseen conditions. Could the authors attempt to analyze this?\n\nOther datasets, such as DexYCB, could provide human wrist trajectory and thus increase the current 200 trajectories.", "limitations": "What is the tolerance for errors in wrist estimation? Often, people cannot carefully move their wrists or do not have a precise estimation tool like Leap Motion. \nFurther, Leap Motion has a requirement for a complete hand without occlusion, which make me doubt about the algorithm's ability to help people with hand disabilities. It might be beneficial to consider alternative wearable sensors for wrist SE3 estimation or additional vision algorithm for wrist pose estimation based on RGB input.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors are tackling an interesting problem - guiding a robot hand to follow human wrist trajectories and utilizing a learnt finger controller to manipulate objects. This bears resemblance to teleoperation but only provides wrist information. I would appreciate further discussion on this aspect.\n\nThe authors introduce a score-matching-based method for learning a primitive policy and a residual policy to aid the primitive policy. This combines synthetic data with reinforcement learning to accelerate training and achieve better performance.\n\nAuthors have conducted a large number of real-world robotic experiments, showcasing the practical applicability of the proposed method.", "weaknesses": "While I agree that human-assisting dexterous manipulation holds potential, it is concerning if this work only addresses the grasping task without considering other dexterous manipulation problems. What are the specific differences and motivations between an automatic dexterous grasping method and user-provided wrist? What is the practical application? If, as the authors suggest, grasping different parts meets varying needs, could the authors conduct experiments to demonstrate this? Or, stepping back, could the proposed method grasp the part that the user intends to grasp? Would it be possible to conduct experiments on it?\n\nIn Table 2, 'ap w/o coll' seems to achieve similar performance, and considering the increment from 55.6% to 56.5%, the residual policy seems not necessary.\n\nThe authors should continue to polish the paper. For instance, the subscript 't' in 'a' on lines 161, 163, and 165 lacks consistency. The formatting of Table 2 could also be improved.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: This bears resemblance to teleoperation ...? ..differences and motivations between an automatic dexterous grasping method...? What is the practical application?**:\n\nThank you  for bring this up. Due to the page limit, please refer to [Q1 of Common Response](https://openreview.net/forum?id=fwvfxDbUFw&noteId=DtPbwqFxR8).\n\n> **Q2: ... it is concerning if this work only addresses the grasping task without considering other dexterous manipulation problems**:\n\nThank you for raising this concern. Grasping serves as the foundational skill in manipulation tasks[1] ,such as picking up the hammer is the first step in nailing. Developing a generalized grasping algorithm forms the basis for more intricate manipulation tasks.\n\nIn fact, our current framework can also be adapted for manipulation. For example, we can collect expert demonstrations for manipulation tasks and employ diffuser[2] for learning primitive policies. Then combines with the residual policy for further refinement during the manipulation process.\n\n> **Q3: If, as the authors suggest, grasping different parts meets varying needs...could the proposed method grasp the part that the user intends to grasp? ...**:\n\nThank you  for bring this up. We have conduct additional experiments to demonstrate the adaptability of our method. Due to the page limit, please refer to [Q2 of Common Response](https://openreview.net/forum?id=fwvfxDbUFw&noteId=DtPbwqFxR8).\n\n> **Q4: In Table 2, 'ap w/o coll' seems to achieve similar performance...**:\n\nThank you  for bring this up. We would like to clarify that residual policy is necessary. Due to the page limit, please refer to [Q3 of Common Response](https://openreview.net/forum?id=fwvfxDbUFw&noteId=DtPbwqFxR8).\n\n> **Q5: The authors should continue to polish the paper...**:\n\nThank you for your suggestion.  We will review the formatting of the main pdf and ensure the consistency of the notation in the future revision.\n\n> **Q6:  What are the outputs of GraspGF when the hand is at different stages...**:\n\nThe primitive policy's actions are solely depend on object-wrist relative orientation. Thus when the hand is far away, the primitive policy will still close the fingers, as shown in **Figure 3 (a)  Stage1 of the rebuttal pdf**. \nAs the hand's posture progressively approaches the target grasp pose, the mean value of the primitive policy's actions decrease, as shown in **Figure 3 (a)  Stage2 of the rebuttal pdf**.\n\nTo further understand action of residual policy, we utilize a measure called \"$r_{sim}$\".  Higher $r_{sim}$ value suggests the final action more closely follows the primitive policy. As shown in **Figure 3 (b) Stage1 of the rebuttal pdf**, when the hand is far from the object, the residual policy will restrict finger's easrly closure to prevent collision, as the hand approach the object, residual policy start to follow primitive policy, as shown in **Figure 3 (b) Stage2 of the rebuttal pdf**. However, as shown in **Figure 3 (b) Stage3 of the rebuttal pdf**, as the hand is about to grasp the object, the \"$r_{sim}$\" start to decrease. At the last few steps, reisudal policy will further refine the pose to hold the object firmly, leading to the negative \"$r_{sim}$\" value.\n\n> **Q7: Given the same initial state and wrist trajectory, can we achieve diversified results?**:\n\nWe would like to clarify that our module is deterministic, adhering to the standard implementation of PPO.\n\n> **Q8: Observing Figure 4 and Table 1, there is not much difference between the seen and unseen conditions...**:\n\nFiltered from unidexgrasp[3], the current grasp dataset involves 3000+ training objects, totaling 0.36 million grasp poses. We hypothesize that this comprehensive dataset sufficiently captures diverse data distributions. Consequently, the distribution of seen and unseen datasets might be i.i.d with the training dataset, results in low difference. This observation aligns with our findings from unidexgrasp.\n\n> **Q9: Other datasets, such as DexYCB, could provide human wrist trajectory...**:\n\nThanks for your valuable suggestion. We actually use DexYCB as human trajectory dataset and extract trajectories from HandOverSim by default configurations. To increase the diversity, we have augmented the dataset by fusing pairs of trajectories  (See **Section 1.1 of Supplementary**). Also, we could extract more human wrist trajectoies by changing the configurations of HandOverSim, as suggested.\n\n> **Q10: What is the tolerance for errors in wrist estimation? ...**: \n\nThank you for bringing up this concern. To demonstrate the robustness of our method, we inject two levels of noise to the wrist pose observation following [4].\nAs indicated in **Table 3 in rebuttal pdf**, our approach yields comparable outcomes with a 2-degree/2-cm estimation error, while exhibiting approximately 10% reduction in performance under a 5-degree/5-cm error threshold, which indicate that our method can handle estimation error to some degree.\n\n> **Q11:  Further, Leap Motion has a requirement for a complete hand without occlusion...**: \n\nThanks for your valuable suggestion! We agree that occlusion is a major issue when deploying our system in the real world and it would be a great future direction. Actually, we are developing a system involves mounting a camera on the user's head and estimating the wrist pose using an instance-level pose estimation method.\n\n**Reference**:\n\n[1] Newbury, Rhys, et al. \"Deep learning approaches to grasp synthesis: A review.\"\n\n[2] Janner, Michael, et al. \"Planning with Diffusion for Flexible Behavior Synthesis.\"\n\n[3] Xu, Yinzhen, et al. \"Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy.\"\n\n[4]Chen, Hansheng, et al. \"Epro-pnp: Generalized end-to-end probabilistic perspective-n-points for monocular object pose estimation.\" "}, {"review_id": "QgfadCWzvr", "reviewer": "Reviewer_PG5Z", "summary": "This paper introduces a novel task called human-assisting dexterous grasping, which aims to train a policy for controlling a robotic hand's fingers to assist users in grasping objects. Unlike conventional dexterous grasping, this task is more complex as the policy must adapt to diverse user intentions and the object's geometry. The proposed approach consists of two sub-modules: Grasping Gradient Field (GraspGF) and a history-conditional residual policy. GraspGF learns 'how' to grasp by estimating the gradient of a synthesized success grasping example set, while the residual policy determines 'when' and at what speed the grasping action should be executed based on the trajectory history. Experimental results show that the proposed method outperforms baselines in terms of user-awareness and practicality in real-world applications.", "questions": "Regarding $r_{sim}$ in Equation 7, more clarification on its functionality would be helpful. Additionally, it would be beneficial to know the inference speed for each module. For human-assistance, quick response times are crucial for seamless human interaction. Since visual modules are utilized, a profiling analysis may be necessary. I am happy to raise the score if the concerned are addressed adequately.\n\n## After Rebuttal\n------------------------\nThe author response looks great to me. Some of the presentation issues are also addressed during the rebuttal phase. I agree with the author that this is more focus on assisting upper limb amputees with prosthetic hands instead of assisting normal persons. I would like to raise the score and glad to see it is accepted.", "limitations": "To address the weaknesses, the paper's authors could improve the presentation by creating more self-contained figures. Furthermore, the hand/object in several images, such as Figure 3, is too small to clearly discern the interaction patterns.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper's strengths can be outlined as follows:\n\n1. Introduction of a unique dexterous grasp task involving shared autonomy between humans and robots, a topic not extensively explored in prior research.\n2. Application of the Denoising Score Matching method to the grasping task.\n3. Explicit representation of robot finger velocity.\n5. A thorough acknowledgment of the system's limitations, including the requirement for a complete point cloud.\n", "weaknesses": "However, the paper also has some drawbacks:\n\n1. The proposed method is better suited for teleoperation settings compared to the reinforcement learning (RL) baselines used in the experiments. It is essential to include comparisons to teleoperation methods without assisted grasping, both qualitatively and quantitatively.\n2. The paper's presentation could be enhanced. For instance, the individual images in Figure 2 could be better explained, as it is currently difficult to comprehend and not highly informative.\n3. The residual policy, which corrects the primitive policy's action, does not consider the primitive policy action as input. This seems illogical for predicting velocity and bias terms without knowing the direction.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: The proposed method is better suited for teleoperation settings compared to the reinforcement learning (RL) baselines used in the experiments. It is essential to include comparisons to teleoperation methods without assisted grasping, both qualitatively and quantitatively.**:\n\nApologies for the confusion. To clarify, the main motivation and practical application of assisting grasping is assisting upper limb amputees with prosthetic hands instead of assisting normal persons, as show in **Figure 1 of main pdf**. Traditional teleoperation method[1] is unsuitable for assisting upper limb amputees to grasp, because we can not get information of human finger. In real-world experiments, we use teleoperation solely to get human wrist poses for mimicking grasping by upper limb amputees. We will elaborate the difference of teleoperation and human-assisting grasping to the introduction section of main pdf in the future revision.\n\n> **Q2:  The paper's presentation could be enhanced. For instance, the individual images in Figure 2 could be better explained, as it is currently difficult to comprehend and not highly informative.**:\n\nApologies for the confusion. We will add additional explanations and details to the pipeline shown in Figure 2 of the main PDF. This will include more detailed annotations that explicitly indicate how the primitive policy's actions are utilized as input for the residual policy. We will also provide further illustration of the functionality of the two components, as well as more compact and clear element in each image. Thanks again for your valuable suggestion!\n\n> **Q3: The residual policy, which corrects the primitive policy's action, does not consider the primitive policy action as input. This seems illogical for predicting velocity and bias terms without knowing the direction.**:\n\nThanks for reminding this. We would like to clarify that the residual policy does take the primitive policy's action as an input. The process involves the primitive policy initially taking the joint state $J_t$, object point cloud $o_t$, and wrist pose $b_t$ as inputs to generate the primitive action $a^p$. Subsequently, the residual policy takes both the primitive action $a^p$, joint state $J_t$, object point cloud $o_t$, and wrist trajectories $H_t$ as inputs to produce the residual actions $a^s$ and $a^r$.\nWe are sorry for the confusion by simplification of notation in the initial presentation. In the revised version of our paper, we will provide a more detailed and accurate description of the relationship between the primitive policy's action and the residual policy. Additionally, in the revised version of Figure 2, we will include detailed annotations that explicitly indicate how the primitive policy's action is utilized as input for the residual policy.\n\n> **Q4: Regarding $r_{sim}$ in Equation 7, more clarification on its functionality would be helpful**: \n\nThank you for pointing this out. As the primitive policy provides good guidance on \"how to grasp,\" we introduce the $r_{sim}$ reward to encourage the final output action ${a^p} \\odot {a^s} + {a^r}$ to explore in direction of the primitive policy's action $a^p$ , this reward is the inner product of $\\frac {a^p}{{\\parallel {a^p} \\parallel}2}$ and $J_t$ - $J_{t-1}$, where $J$ signifies the state of finger joints. A higher value of $r_{sim}$ indicates a smaller angle between the gradient and $J_t$ - $J_{t-1}$, implying a greater similarity in the movement of finger joints to the gradient. Actually, we have already conducted an ablation study of $r_{sim}$ in **Section 4.1 of the Supplementary** to demonstrate how $r_{sim}$ facilitates the acceleration of the residual policy's learning process, and leads to a more effective utilization of the generalization capabilities of the primitive policy.\n\n> **Q5: Additionally, it would be beneficial to know the inference speed for each module. For human-assistance, quick response times are crucial for seamless human interaction. Since visual modules are utilized, a profiling analysis may be necessary.**: \n\nThanks for bring up this concern. We evaluate the inference speed on the GTX 1650, which is also used in our real world experiment.\nWe set the batch_size equal to 1 and ran the policy 50 times to obtain a reliable average time for the inference speed of each module. \nAs shown in **Table 2 of the rebuttal pdf**, both modules take less than 0.004 seconds for each inference.\nThis indicates that our integrated system is capable of seamless human interaction.\n\n**Reference**:\n\n[1] Handa, Ankur, et al. \"Dexpilot: Vision-based teleoperation of a dexterous robotic hand-arm system.\"\n"}, {"review_id": "VBwv3OyJX1", "reviewer": "Reviewer_GtH3", "summary": "This paper introduced a novel and challenging task that performs dexterous grasping according to human wrist movements. This task is potentially useful for applications with prosthetic hands. \n\nThe paper further proposed a novel two-stage framework that solves the two challenging aspects of the proposed task and demonstrated strong performance in both simulated and real-world experiments.", "questions": "1. For the baseline methods, are all baselines re-trained to take the wrist pose as a condition? I didn’t find this piece of information in the paper. \n2. In figure 5, why do w/o $a^r$ and w/o $a^s$ success rates drop as more samples are seen after 5e6 samples?", "limitations": "1. The method assumes full point cloud observation which may limit its application in the real world. \n2. The qualitative results did not show how the proposed method adapts to different wrist poses relative to the object.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The proposed task is novel and potentially helpful to social welfare. The proposed framework is intuitive and is properly designed for the challenges of its task. The authors have also conducted extensive experiments to show the capacities of the proposed method. \n\nThe paper is well-structured and written.", "weaknesses": "From the qualitative results in the supplementary video, I noticed that for most objects, the graspings are from the same angle relative to the object. For example, with the chips can, all demonstrated graspings are from the side of the cylinder regardless of how the can is placed. This makes me wonder if the proposed method can truly adapt to different approach angles and different *user intentions*. Please correct me if I missed anything from the videos. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: From the qualitative results in the supplementary video, I noticed that for most objects, the graspings are from the same angle relative to the object. For example, with the chips can, all demonstrated graspings are from the side of the cylinder regardless of how the can is placed. This makes me wonder if the proposed method can truly adapt to different approach angles and different user intentions. Please correct me if I missed anything from the videos.**：\n\nThank you for bringing up this point. While the approach angles appear similar in the videos, it's important to note that the object's relative pose to the wrist is actually varied due to changing object poses. Principally, we have a world coordinate, and a wrist coordinate. The graspings looks like same angle relative to the object is because the movement of hand is similar in world coordinate. However, since the object pose in world coordinate is also changing (for instance, the Bleach Cleanser being positioned both vertically and horizontally), the object's pose in the wrist coordinate is consequently changing, resulting in different relative poses. Regarding the Chips Can, despite all demonstrations involving side grasps, it's essential to recognize that the object's relative pose to the wrist is in fact distinct.\n\nTo further illustrate the adaptability of our method to various approach angles and user intentions, we have conducted two additional experiments. As detalied in **Section 1 of the anonymous project page**, participants first move the hand to either the middle or the right side of the object, then start to approach. When dealing with chips, participants attempt to grasp the top of the chip, while for the mug, participants attempt to grasp the mug's edge. Alongside these real-world experiments, we have also conducted more comprehensive simulation experiments, as detailed in the [Q2 of Common Response](https://openreview.net/forum?id=fwvfxDbUFw&noteId=DtPbwqFxR8), to effectively demonstrate the adaptability of our method.\n\n> **Q2: For the baseline methods, are all baselines re-trained to take the wrist pose as a condition? I didn’t find this piece of information in the paper.**: \n\nThank you for pointing this out. To clarify, all baselines are re-trained by taking latest 5 frame wrist poses as input under human-assisting setting (e.g., the agent can not move the wrist), which aligns with \"Ours\" experiment setup. We will add additional detail of  this in the future revision.\n\n> **Q3: In figure 5, why w/o $a^r$ and w/o $a^s$ success rates drop as more samples are seen after 5e6 samples?**\n\nThank you for raising this concern. We hypothesis that this issue is tied to the fundamental exploration and exploitation challenge in classical reinforcement learning. As the primitive policy provides a strong guidance for the reinforcement learning algorithm, the agent might tend to focus solely on exploitation based on the primitive policy. This behavior can lead to getting stuck in local minima and ultimately result in a decrease in the success rate. Nevertheless, with the incorporation of both a_r and a_s, our policy is better equipped to strike a balance between exploration and exploitation. This balanced approach allows the agent to navigate both the exploration of novel strategies and the exploitation of existing knowledge more effectively, leading to improved overall performance. \n> **Q4: The method assumes full point cloud observation which may limit its application in the real world.**:\n\nThank you  for bring this up. Due to the page limit, please refer to [Q4 of Common Response](https://openreview.net/forum?id=fwvfxDbUFw&noteId=DtPbwqFxR8).\n\n> **Q5: The qualitative results did not show how the proposed method adapts to different wrist poses relative to the object.**:\n\nThank you  for bring this up. We have conducted additional experiments to demonstrate the adaptability of our method. Due to the page limit, please refer to [Q2 of Common Response](https://openreview.net/forum?id=fwvfxDbUFw&noteId=DtPbwqFxR8).\n"}]

Author rebuttal:
## **Common Response**:
We thank all reviewers for appreciating our ideas and experiments. “A unique dexterous grasp task **(PG5Z)**". "The proposed framework is intuitive and is properly designed for the challenges of its task **(GtH3)**". "Formulate the learning from a set of successful grasps as a denoising problem, which is quite interesting and novel **(ju1F)**".  "Demonstrated strong performance in both simulated and real-world experiments **(GtH3)**". "Showcasing the practical applicability of the proposed method **(KjWd)**".

However, we notice that some reviewers **(PG5Z, KjWd, ju1F)** might confused about the **(Q1) Distinction between Teleoperation, Automatic Dexterous Grasping (Classical Grasping),  and Human-Assisting Grasping**. 

The main motivation and practical application **(PG5Z, KjWd, ju1F)** of assisting grasping is assisting upper limb amputees with prosthetic hands, as show in **Figure 1 of main pdf**. 

 - **Assisting Grasping v.s. Teleoperation.** 
Traditional teleoperation method[1] **(PG5Z, KjWd)** is unsuitable for assisting grasping, because we can not get information of human finger. In real-world experiments, we use teleoperation solely to get human wrist poses for mimicking grasping by upper limb amputees. 

 - **Assisting Grasping v.s. Automatic Dexterous Grasping .** Compared to automatic dexterous grasping **(KjWd, ju1F)**, agent can not control the wrist of the prosthetic hand in assisting grasping, which poses challenge for user-aware grasping. Due to the complex and diverse behaviours when human controls the wrist, deciding how and when to grasp is challenging for agent without considering the human movement. For instance, if the human first moves the wrist close to the object and then grasp, the agent still must factor in the object-wrist relationship for how to grasp. On the other side, grasping with a fixed wrist pose instead of close the finger in advance can result in table obstructs finger closure, requiring humans to adjust their wrist. This affects grasp fluency and introduces burden, a crucial consideration for humans. Moreover, humans may face dynamic objects, as shown in **Section 1 of the anonymous project page**. Ignoring human movement could result in failed grasps.

We also noticed that some reviewers concerned about **Q2: The Necessity of Adaptability, and the Adaptability of Our Proposed Method (KjWd, GtH3, ju1F)**.

 - **Necessity of Adaptability** It's crucial to grasp different parts **(KjWd)** of objects in daily life, as many objects serve multiple purposes. For example, a hammer can be used for nailing and sweeping. Adaptability also requires for same purpose. When cleaning shoes, human need to hold different parts of shoes. Moreover, objects may be put in different places by various poses, which makes grasping a specific part challenging.

 - **Adaptability of Our Proposed Method** **(KjWd, GtH3, ju1F)** Quantitatively, we've evaluated "posture" measuring alignment between ours and intended grasp poses, as shown in **Figure 4 of the main pdf**. on the other side, as  shown in **Table 1 of the rebuttal pdf**, our method excels at grasping various parts of objects. Qualitatively, **Figure 1 of the rebuttal pdf**  shows human's intended grasp and our method's close-to-ground-truth pose. In another experiment, as shown in **Section 1 of the anonymous project page**, GraspGF produces diverse grasp poses as the wrist moves.

We futher noticed that some reviewers **(KjWd, ju1F)** might confused about **Q3: Distinction Between Collision in Grasp Pose Generation and Collision in Grasping Procedure**.

There are  two collision types in our current problem.

 - **Collision in Grasp Pose Generation**  The first arises during grasp pose generation, e.g., "how to grasp". Generated grasp poses can sometimes result in object collisions. To address this, we can heed the reviewer's advice **(ju1F)** to include a collision penalty, potentially improve the primitive policy's performance.

 - **Collision in Grasping Procedure** This collision relates to "when to grasp" regardless of direct object collision in the generated pose. Even if a pose has no collision with objects, collisions can still occur when the agent control fingers to reach the desired pose—colliding with the table or object itself.

 - **Necessity of Residual Policy** To clarify, results of "$a^p$ w/o coll" **(KjWd, ju1F)** are obtained by disabling collisions in grasping procedure. **Figure 2 of the rebuttal pdf** highlights that while the final grasp pose can grasp the object, poses are achieved before the agent intends to grasp. This shows a residual policy **(KjWd)**  is crucial to decide the ideal grasp timing.

Some reviewers **(ju1F, GtH3)** also highlights **Q4: Limitation of Perfect and Full Point Cloud**. 

We have discussed in **Section 7 of the main pdf**. Addressing this concern can involve using teacher-student learning[2] or adapting our pipeline to be trained with partial point cloud input **(ju1F, GtH3)**. For handling the gap between real point cloud and sim point cloud, we may add noises[3] to point cloud observations for training **(ju1F)**.

We sincerely hope our work contributes to the Machine Learning + Robotics research community and eventually improve the social welfare. Below we reply to reviewers’ questions point-by-point. Thanks again for your valuable comments and suggestions!

**Reference**:

[1] Handa, Ankur, et al. "Dexpilot: Vision-based teleoperation of a dexterous robotic hand-arm system." 

[2] Chen, Tao, Jie Xu, and Pulkit Agrawal. "A system for general in-hand object re-orientation."

[3] Dai, Qiyu, et al. "Domain randomization-enhanced depth simulation and restoration for perceiving and grasping specular and transparent objects."


Meta-review: The paper proposes a novel approach to fuse machine learning of motor skills with human input through tele-operation.

The reviewers found the method sound and novel, and the experimental results very convincing. They initially had questions about some details in the setting, the method, and assumptions/limitations. Some shortcomings in the presentation were pointed out.

After the rebuttal the main concerns of the reviewers have been successfully addressed and they are all vote in favor of accepting the paper.
Decision: Accept (poster)