ResPilot: Teleoperated Finger Gaiting via Gaussian Process Residual Learning
Authors: Patrick Naughton
Jinda Cui
Karankumar Patel
Soshi Iba
Venue: CoRL 2024 (2024)
Keywords: Teleoperation
Dexterous Manipulation
Gaussian Process
TL;DR: We achieve teleoperated finger gaiting using Gaussian Process residual learning to calibrate a human hand to robot hand motion retargeter.

Abstract: Dexterous robot hand teleoperation allows for long-range transfer of human manipulation expertise, and could simultaneously provide a way for humans to teach these skills to robots. However, current methods struggle to reproduce the functional workspace of the human hand, often limiting them to simple grasping tasks. We present a novel method for finger-gaited manipulation with multi-fingered robot hands. Our method provides the operator enhanced flexibility in making contacts by expanding the reachable workspace of the robot hand through residual Gaussian Process learning. We also assist the operator in maintaining stable contacts with the object by allowing them to constrain fingertips of the hand to move in concert. Extensive quantitative evaluations show that our method significantly increases the reachable workspace of the robot hand and enables the completion of novel dexterous finger gaiting tasks.

ResPilot: Teleoperated Finger Gaiting via Gaussian Process Residual Learning
Patrick Naughton 1.2* Jinda Cui², Karankumar Patel",Soshi Iha²
UIUC, Honda Research Institute USA
pm10@illinois. edu, {jinda_cul karankumar .aaael, siba @honda-ri "This work was completed during an internship at Honda Research Institute USA
Cube Rot
Cube Rot V
Palm to Quadpod
Card Pickup
Wrench Pickup
Controlled Pivot
Retarget
Figure 1: Our method retargets human hand configurations robot hand, enabling finger-gaited in-hand manipulation. We evaluate our method on six highly dexterous tasks with the palm facing upward and downward.
Abstract: Dexterous robot hand teleoperation allows for long-range transfer of human manipulation expertise, and could simultancously provide way for hu- mans to teach these skills to robots. However, current methods struggle to repro- duce the functional workspace of the human hand, often limiting them to simple grasping tasks. We present a novel method for finger-gaited manipulation with multi-fingered robot hands. Our method provides the operator enhanced flex- ibility in making contacts by expanding the reachable workspace of the robot hand through residual Gaussian Process learning. We also assist the operator in maintaining stable contacts with the object by allowing them to constrain fin- gertips of the hand to move in concert. Extensive quantitative evaluations show that our method significantly increases the reachable workspace of the robot hand and enables the completion of novel dexterous finger gaiting tasks. Project web- site: aapilot-hri.github..
Keywords: Teleoperation, Dexterous Manipulation, Gaussian Process
Introduction
Teleoperated dexterous manipulation has the potential enable the long-range transfer of human manipulation skills to remote environments and could simultancously serve mass data collection mechanism to enable fully autonomous manipulation. Fluent manipulation using multi-fingere robot hand is difficult even with human operator in the loop because it requires precise control of the many degrees of freedom (DoFs) of the hand simultaneously to coordinate finger motion. In particular, finger-gaited manipulation, where a subset of fingers are used to maintain a contact state while other fingers move change their contact state with the object and move it, has proven challenging because the motion of the target object is sensitive the locations and modes of con-
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

tacts. Such challenges are exacerbated by limitations of current interfaces such as a lack of haptic feedback and kinematic mismatches between the human and robot.
Existing approaches for teleoperation of dexterous hands typically use retargeter to map the opera- tor's hand to the robot's desired configuration. Current literature often evaluates these retargeters by how well they allow the robot hand to match the operator's 's hand visually, achieve grasping and pushing tasks [1, 2, 3, 4]. A few works have attempted basic in-hand manipulation [5, 6] but tend to require specially designed task spaces and provide no details about their reliability or efficiency. To the author's s knowledge, Handa et al. [7] is the only existing method that has demonstrated tele- operated finger gaiting with a multi- fingeree hand, and even this method was only able to achieve a single finger gaiting task (in hand block rotation).
The main contribution of this work is novel retargeting method that enables previously unseen teleoperated finger gaiting. We address two core challenges: 1) making contacts using free fingers at diverse locations on the target object and 2) using these contacts to stably change the object's pose. Our method uses a small set of hand-labeled calibration poses to learn a residual Gaussian Process between an optimization- based retargeting method and the labeled robot configurations. We also allow the operator to constrain the distance between the robot's fingertips while still tracking their input motion to allow the robot maintain stable contact with grasped target object the fingers move. We show that this method is fast to calibrate, expands the reachable workspace of the robot, and ultimately enables previously unseen teleoperated finger gaiting.
2 Related Work
There are many existing approaches for retargeting person's hand motion to an anthropomorphic robot hand. Here, we review the ones most related to our method. For a more comprehensive treatment, we refer readers to [8].
Joint-space retargeters directly map each joint of the operator's 's' hand to a joint of the robot hand and command the robot joints to follow the operator's 's hand [9, 10, 11]. While this approach allows the robot's fingers to approximate the shapes of the operator's. ss tends to make precise fingertip control difficult due to kinematic differences between the robot and operator. Conversely, inverse kinematics (IK) retargeters [12, 13], which use IK to directly command each robot fingertip to match the pose of the operator's fingertips, enable precise fingertip grasping, but result in unintuitive finger shapes [14, 9]. Additionally, since robot hands are often larger than the operator's, this approach can prevent the operator from reaching much of the robot's workspace. In contrast, our method, by calibrating for important configurations at multiple points in the robot's workspace, enables our retargeter to enjoy the advantages of both methods.
Recently, several systems [7, 2, 4] have used hand keypoint-vector matching (HKVM) approach where corresponding keypoints are labeled on the operator and robot hands (for example, the finger- tips and/or palm). A set of pairs of keypoints is chosen to define vectors on the robot and operator hands and the desired robot configuration is computed as the one that minimizes the deviation be- tween these vectors [7]. While Handa et al. [7] demonstrated some basic finger gaiting using this method, their focus on fingertip grasping typically limits them to grasping tasks. In contrast, we show that our retargeter enables a suite of highly dexterous finger gaiting tasks.
Finally, few works have proposed "pure-learning" approaches to the retargeting problem where the function from operator hand configuration to robot hand configuration is directly learned from set of labeled examples [15, 1]. While these methods are capable of both power and precision grasping, to the author's s knowledge researchers have not used these methods for finger gaiting.
3 Method
The goal of retargeting method is to map a given operator configuration to commanded robot configuration. Our method uses small set of calibration poses to learn a residual between an
2

---

optimization- based retargeter and the labeled robot poses conditioned on the operator's s hand con- figuration. We assume access to measurements of the operator's fingertip poses and joint angles.
3.1 Optimization- Retargeting Methods
For a given configuration of the operator's 's hand, existing optimization bassraeddo--aedddddd methods produce q by minimizing the error between the H sensed keypoint vectors the operator's hand and the scaled vectors between the same keypoints on the robot hand [7. 4]:
( = arg min
Σ (r.(())) - -
(q:)))) + +
qu
where r-))) and h.(-) compute the ith keypoint vector for the robot and human hands, q. and denote the robot and human hand configurations, and is a scaling parameter to account for size differences between the hands. 7 is regularization parameter that biases the robot configuration to be close zero (an open hand). Following Handa al. [7]. we use the same 10 keypoint vectors shown in Figure and 3 1.6, 7 0.0025. We refer to this retargeter the "hand keypoint vector matching" (HKVM) retargeter since it attempts to match vectors between keypoints the robot's hand corresponding vectors the operator's hand.
3.2 Residual Gaussian Process
While the HKVM retargeter alone can accomplish some tasks, struggles to reach parts of the fingers' workspace that are vi- tal for finger gaiting, such near-palm grasps. To expand the reachable workspace,
"base retar- geter," and collect small number C of paired human hand and robot hand configurations, D = f(gi.. 9r. 4r.) (C) where [C] denotes the sequence of integers (1....,C) C). Let F de- note the shared set of fingers between the human and robot hands (in the case of a four-fingered robot hand used here, F (thumb, index, middle, ring)) and denote by qif) the subset of hand joints associated with finger multi-output Gaussian Process (GP) to regress the residual between q (qr.) and qr; from qı, We denote this full residual with {(q)). ). In practice, we assume that the desired residuals of each finger are independent of each other and that they are only functions of the configuration of the cor- responding human finger. Thus, we learn a separate residual GP for each robot finger which takes as input only the joint angles of the corresponding human finger, g (qx. These assumptions make control of each finger more independent and thus easier for the operator reason about, and reduce the dimensionality of the learning problem. They also enable partial labelling of pairs of hand configurations, as done by Correia Marques et al. [15]: for example, in fingertip pinching configurations, only the fingers involved in the pinch can be reliably labeled with corresponding robot configuration. By only using these configurations to train the associated finger models, we keep the data for each finger model less noisy (for given hand configuration, we refer to the fingers being labeled as "active fingers").
use HKVM
Figure 2: Hand keypoint vectors.
F. Using D, learn the hyperparameters of
A GP represents a function, such as g as indexed set of random variables with the property that any finite subset has Gaussian distribution [16]. In this case, the index set is [x[f]. The GP is completely specified by its mean and covariance functions [f)) and ((qulf), If(f)):
é {(gn()) GP(n(qn [/)). [()x[)) ((D)
In our case, we assume that ju is uniformly zero (in expectation, we exactly follow the base retar- geter). We assume a parametric form for k:
M arccos (V(q, (/.))
((. = exp
(3)
26
3

---

where, for a vector of m angles q, V(q) denotes the matrix
(4)
sin(A)
V(g)= =
cos(gm) 4m)
|/] denotes the number of joints on finger f. and arccos is applied elementwise. We also use to compute the residual e = V()[//) - (q)]()))( to avoid the discontinuities encountered by learning directly in the angle space [17]. Thus, for each finger, we learn 2 -dimensoooaa vector- valued function. Rather than learning each component independently,
learn task- cavariovee matrix K,, where the element at row i column j is the covariance between components and of the output [18]. Assuming the calibration datapoints were labeled with Gaussian noise having variance o², we compute the input covariance matrix as K,, = K + o²1, where K is computed by evaluating k at every pair of collected human joint angles. The full covariance matrix is then K = K, & K,, where & is the Kronecker product.
We use GPyTorch [19, 20] to implement the GPs. We collect set of calibration configurations and compute the HKVM target robot configuration for each, saving an ordered pair (5:[/]. qb, [/]) for each active finger. We then fit a separate GP to each finger's 's dataset by optimizing the entries of K,, (&s) (E))) and a (collected into parameter 0) to maximize the marginal log likelihood of the data:
(og(p(Q,)D.0)) = - -,,, Q, log Kg
log 2n.
(4)
2
Qr the stacked vector of all residual targets for a finger. For each calibration point, we vectorize the ||] x 2 residual matrix and concatenate all of these vectors form Q We optimize this loss with respect to 0 with the Adam [21] optimizer, using learning rate of 0.01 and running for E = 3000 epochs, which we found to be sufficient to achieve convergence of the loss.
When using the retargeter, we first compute q using Equation 1. Then, for each finger, we find the posterior mean g [16] of the residual distribution the current finger joint configuration condi- tioned on the collected calibration poses for the current user. For each finger, we compute desired joint angles by converting the If 2 matrix, V(g + + into dilmeiinnso vector, finding the angle each row makes with the z-axis. We then collect each of these finger joint targets into qd, the full desired configuration of the robot hand.
3.3 Finger Constraints
Using the Residual GP retargeter, the operator can now move the robot fingers a large workspace. However, stably maintaining contact while moving the object can still be difficult because the oper- ator may not be able visualize the robot's target configuration, reason about the forces this will apply. To alleviate this problem, allow the operator to apply constraints on the motions of fingers to couple them together. Similar to Handa al. [7], our system allows the operator to constrain any of the robot's 's fingertips to stay a hand- specifiee distance d = 1 cm away from the robot's 's thumb tip. We compute the final constrained robot configuration by solving
qe = arg min l|q - qull3
q s.t. [r((g) = d Vie R.
(5)
where R. denotes the set of constrained fingertip vectors. In our implementation, Rs only ever contains vectors between one of the robot's fingertips and its thumb. The operator can toggle each of the vectors' inclusion in R. by tapping corresponding foot pedal.
4 Implementation
To instantiate our method, we designed set of calibration poses and tested the effectiveness of the retargeter when controlling a physical Allegro robot hand.

---

Figure 3: The full set of 24 calibration poses used to learn the residual GP. Active fingers for each calibration configuration are colored green.
Calibration. The goal of calibration is to expand the reachable workspace of the robot hand while maintaining an intuitive correspondence between the operator and robot hand shapes. We therefore selected calibration configurations at the boundary of the robot hand's workspace and labeled them using human hand configuration at a corresponding boundary of its workspace. This heuristic can be broadly applied other anthropomorphic hands as well. Figure 3 shows the full of 24 calibration configurations, where the active fingers of each colored green. We found that some pairs of configuration naturally defined finger trajectory (e.g., moving from pinch near the palm to far from the palm), and additionally recorded points halfway between these two extremes to constrain how the retargeter interpolated between them. After choosing a set robot and human hand pairs, to calibrate the retargeter for a new operator, the operator simply matches their hand shape to each of the 24 configurations while their joint angles at each point are recorded. GP for each finger then trained to calibrate the retargeter.
Inference. Algorithm 1 shows how the final desired joint configuration is computed. To solve the mathematical programs on Lines 3 and 8 we use the NLopt library [22] with the SLSQP algo- rithm [23]. We initialize the solver at its previous output (or the zero configuration at initialization). We use the PyToorc Kinematics library to compute and differentiate through the forward kinematics function of the robot hand [24]. The retargeter runs approximately 8 Hz when unconstrained and 4.5 Hz with constraint activated be suitably reactive for teleoperated control. The computation time dominated by the optimization routines, while the GP computation takes only 9 ms average.
AMD Ryzen 3955WX 2.2 GHz CPU, which we found to
Algorithm Full computation of the desired joint angles
1: function COMPUTEDESIREDD D, 0. B, y, Ro)
2: Update R. from user input (pedal presses)
3: Compute ("hi 3, 7) from Equation
4: Allocate qa
5: for f E do
6:
Compute rd (qi [f]: D,0) 7: quif) a(55" + V(gā
a(-) is the inverse of V(--)
8: Compute qe from Equation
return qe 9:
Hardware and Control. After the desired joint configuration computed, is sent to a lower level (gravity compensated) PD torque controller that tracks this set point at 300 Hz (smoothed by an exponential filter). This controller allows the operator control the force each finger exerts on grasped object by moving the set point for that finger into out of the object. We Allegro right hand outfit with Xela sensors as robot hand and use Manus Quantum Metaglove to track

---

the operator's right hand's fingertip poses and joint angles. The Xela sensors are not used in our teleoperation interface, but do alter the coefficients of friction between the hand and target objects. We use an IKKEGOL USB triple foot pedal to allow the operator to toggle constraints on each finger.
5 Experimental Results
To evaluate our retargeter, we calibrated it to two different operators and measured their ability to complete 6 tasks requiring substantial dexterity from the robot's fingers, including several finger gaiting tasks. Calibration takes on average 4.5 minutes (including training the GP), making it easy to calibrate the retargeter to each operator individually.
5.1 End-to-End Testing
We evaluate our system on the 6 challenging tasks shown in Figure 1:
1. Horizontal Cube Rotation (Rot H): The operator must lift the cube off of the robot's palm and rotate 180 about the horizontal axis before placing it back on the palm.
2. Vertical Cube Rotation (Rot V): The operator rotates the cube 90 about the vertical axis.
Screwdriver Palm to Quadpod (P-to-Q): The operator starts with a screwdriver on the robot's palm and transitions it to a "quadpod" grasp (suitable for turning the screwdriver).
Wrench Pickup (Wrench): The operator picks up a wrench off table by first pinching it, lifting it up, then transitioning to power grasping it.
5. Card Pickup (Card): The operator picks up card off of table by sliding it over the edge and pinching between two fingers.
6. Screwdriver Controlled Pivot (Pivot): The operator grasps screwdriver with all four fingers then loosens their grip until the screwdriver rotates to point downwards.
Compared to previous dexterous manipulation systems [7, 2, 4J, we demonstrate our retargeter with- out any arm motion so that all dexterity must come from finger-level manipulation. The retargeter kept the same across all tasks and attempts: no hand-tuning performed for individual tasks. Tasks 1-4 require finger gaiting, while the grasps required by tasks 4-6 demonstrate our retargeter retains basic grasping functionality and precise control. Previous retargeters we have tried [7, 6] were unable to complete the finger gaiting tasks.
We had two operators (paper authors) attempt to complete all 6 tasks times in row, both with and without the use of the finger constraints (subsection 3.3). Operators were allowed to attempt the tasks several times before starting their scored runs to get accustomed to the retargeter. runs with the finger constraints, operators were required use them least once. For each attempt, the operator earned 1 point for successful completion of the task, 0.5 points for reaching the desired final state but violating task constraint (c.g., introducing rotation about unwanted axes during Rot H or Rot V, failing to control the screwdriver's pivoting in Pivot), and 0 points otherwise. We recorded each operator's completion times for attempts where they scored any points. Figure shows the average scores achieved by each operator under each condition.
Both operators were able complete several tasks with relatively high reliability given their diffi- culty, average succeeding in 80% of trials (computed based the best of the two conditions in practice, the use of constraints is optional). Operators demonstrated advanced finger-gaited manipu- lation as well as common grasps (power and pinch grasps). Operators completed the tasks relatively quickly, taking on average only 43 and at most 139 s, indicating that the overall interface is fluent and usable. This is also fast enough for operators to collect a few hundred demonstrations in a few hours. Interestingly, the benefit of using finger constraints varies between tasks. Constraints are most effective in the Rot H. P-to-Q, Wrench, and Card tasks, where the operator must hold the tar- get object steady they use other fingers adjust its pose, precise finger control to grasp very thin object. However, in the Rot V and Pivot Tasks, the use of constraints sometimes reduces

---

1.0
Operetor 0 w/
Operator
0.0
Operater w/
Pivol
Figure 4: Task performance for two pilots on the 6 tasks with and without using the fingertip con- straints. The plot shows the average scores; error bars show the relative variability (0.2o).
Table Reachable workspaces of each of the retargeters.
Method
Joint Workspace (rad")
Fingertip Workspace (cm²)
Joint
0.1196
912.275
IK
0.1198
462.625
HKVM
0,2028
1090.750
DexPilot [7]
0.1577
908.950
NN
0,0627
500.325
GP
0.0718
706. 600
Res-NN
0.1749
1170.450
Res-GP (Ours)
0.2399
1511.050
the operator's performance. In the case of the Pivot task, constraints simply serve as a distraction, since once activated, the operator has no control over the force being applied between the finger and thumb. In the Rot task, once the constraint was activated, it difficult command the fingers in proper arcing motion achieve the desired rotation. This constraint is better suited to maintaining existing grasp of an object and translating it.
5.2 Comparison With Previous Methods
We hypothesize that this retargeter's ability to complete these difficult tasks stems from its combina- tion of an enlarged reachable workspace and preservation of precise operation regions that require it (for example, when pinching the fingers). To test this hypothesis, moving through all regions of its workspace and collected a long trajectory of N human hand joint angles by having an operator attempt mimic the hand in this video several times in row. We then used several baseline retargeters to compute a set of robot joint trajectories (q/, )E(N] where j denotes which retargeter was used. To approximate the robot hand's reachable workspace under retargeter, we considered each robot finger individually and voxelized its joint space resolution 51 0.05 rad. We report the volume of the voxels traversed by the retargeted robot trajectory, summed across all fingers. To compute the fingertip workspace, we performed the same procedure with each fingertip's 's trajectory, using spatial resolution of 82 5 mm, tracking only the fingertip's position (disregarding orientation). We performed this procedure with 5 operators (calibrating the relevant retargeters to each operator) and report the average results.
recorded video of hand
Table lists the retargeters we tested and their joint and fingertip workspaces. The "Joint" retargeter linearly rescales each of the 16 tracked human finger joint positions match the range of each robot joint. The inverse kinematies ("IK") retargeter uses IK solve for a joint configuration of each robot finger that places the fingertip at the same position as the corresponding human finger relative to the robot palm. HKVM directly uses 4 as the desired joint target, and DexPilot adds finger-snapping and collision avoidance heuristic to HKVM [7]. The neural network ("NN") and Gaussian Process ("GP") retargeters learn to output a desired joint configuration from a human finger configuration directly from the calibration data. Finally, the residual neural network ("Res-NN") uses neural network, rather than a GP, to learn a residual on top of HKVM.

---

Res-GP achieves by far the highest joint and fingertip workspaces of any retargeter. Even though the Joint retargeter can reach the full range of each robot joint independently, its workspace suffers from failing to consider correlated movements of human fingers. The fingertip retargeting methods (IK, HKVM, and DexPilot) are mostly concerned with configurations the vicinity fingertip grasps, and tend to severely limit the fingers' and thumb's mobility near the base of the palm. The other learningbbased methods (GP, NN, and Res-NN) tend to overfit the calibration data, "snapping" quickly between calibration poses rather than smoothly interpolating between them. Res-GP in contrast, uses strong prior of how to interpolate between fingertip configurations (encoded by HKVM as a base retargeter and our choice of kernel parameterization), allowing it to both reach the far extents of the workspace (specified by the calibration poses) while allowing for smooth control in its interior. Additionally, while drastically expanding the robot's workspace, Res-GP still allows the operator to precisely control the fingers in critical regions like fingertip pinches. Figure 5 shows how the Joint retargeter does poor job of approximating this pinch configuration, and while the HKVM retargeter
approximate the pinch, the location of the pinch remains nearly static relative to the palm. In contrast, Res-GP is able to pinch both near and far from the palm. This shows how Res-GP effectively captures where it can trade off' sensitivity for reachability.
6 Limitations
Our work has several limitations. First, rely on calibration poses tailored to specific robot hand. This assumes that operators will have time to collect calibration data before beginning operation. While empirically this can be performed quickly and only needs to be done once for each operator, introduces an additional step compared to calibration- retar- geters [7, 2, 4]. Additionally,
Joint
HKVM
Res-GP
rely on intuition of the required robot hand workspace to determine the set of calibration poses. We provide a heuristic for choosing these poses that could be applied to other hands, but a systematic approach to identifying them would make this method more automated.
Figure 5: Qualitative comparisons between retargeters. Res-GP reproduces the opera- tor's pinch many distances from the palm.
Second, while our finger constraints enable the operator to maintain stable contact with grasped object while translating it, they don't allow for intuitive adjustment of the applied force, and rotating the constrained fingers about center point proved difficult. Additionally. the operator is currently required to use a separate interface (foot pedals) to toggle the constraints, rather than only using their hand to control the robot. In the future, we'd like to address this by modifying how constraints are applied to allow the operator to more intuitively move objects in their grasp, using an approach similar to that presented in [25]. We'd also like investigate how the operator's intent to use constraint can be inferred directly from their hand tracking data.
Finally, as our method relies on optimization routines, the retargeter can get trapped in local minima and stop tracking the operator. We rarely encounter this problem in our experiments and typically the operator is able to recover from it, but more robust solution needed for real deployments.
Conclusion
We present a retargeter for finger-gaited dexterous manipulation with multi-fingered robot hands. Our method learns a residual GP on a small set of calibration poses to enhance the robot's reachable workspace, and allows the operator to constrain fingertip motions to maintain stable contacts. Based on real-world experiments, our method fast to calibrate and use, and enables previously unseen levels of teleoperated dexterity. In future work, we'd d like to investigate how calibration poses can be chosen in more systematic way, and how the raw retargeting and constraint satisfaction steps can be consolidated while maintaining the same level of dexterity.
8

---

Acknowledgments
We'd like thank Benjamin Evans for thoughtful comments on the paper and naming the method.
References
[1] E. Chong, L. Zhang, and V. J. Santos. A learning-based harmonic mapping: Framework, assessment, and case study of human-to-robot hand pose mapping. The International Journal of Robotics Research, 40(2-3):534-557, Feb. 2021. 060:1001177000866449966666
[2] A. Sivakumar, K. Shaw, and D. Pathak. Robotic Telekinesis: Learning Robotic Hand Imitator by Watching Humans on Youtube. New York City, NY. USA, July 2022. Robotics Science and Systems. URL https:/ http://wwwwoooooccprooeeeeeeeede / pdf.
org/rss18/p023
[3] Y. Qin, H. Su, and X. Wang. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera telcoperation. IEEE Robotics and Automation Letters, 7(4): 10873-10881, 2022.
[4] Y. Qin, W. Yang, B. Huang, K. Van Wyk, H. Su, X. Wang, Y.W. Chao, and D. Fox. AnyTelcop General Vision- Based Dexterous Robot Arm-Hand Teleoperation System, Aug. 2023. URL
http:://astii http://aaaiiv../abbb/////s/ wtw.ooog///bbaaaeeooeee 044777777777....
[5] S. P. Arunachalam, S. Silwal, B. Evans, and L. Pinto. Dexterous Imitation Made Easy: A Learning- Baase Framework for Efficient Dexterous Manipulation. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 5954-5961, London, United Kingdom, May 2023. IEEE ISBN 9798350323658. doi: 10. 109/11/////99991111 2023. URL ieee. org/ doocument/101000000
[6] S. P. Arunachalam, I. Güzey, S. Chintala, and L. Pinto. Holo-Dex: Teaching Dexterity with Immersive Mixed Reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5962-5969, London, United Kingdom, May 2023. IEEE. ISBN 9798350323658. 100:00.110//CCAAA4A1 2224889912222111101444444 URL https:///eeexplore. iee.. wttg///bccmeent11166557777
[7] A. Handa, K. Van Wyk, W. Yang, Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. Ratiff, and D. Fox. DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System, Oct. 2019. URL http: //tp:v///aiiv///// org/ abaaaaas//19 03135. 3.
[8] R. Li, H. Wang, and Z. Liu. Survey mapping human hand motion robotic hands for teleoperation. IEEE Transactions on Circuits and Systems for Video Technology. 32(5):2647-
2665, 2021.
[9] R. Meattini, R. Suárez, G. Palli, and C. Melchiorri. Human to robot hand motion mapping methods: Review and classification. IEEE Transactions on Robotics, 39(2):842-861, 2022.
[10] T. Mouri and H. Kawasaki. A novel anthropomorphii robot hand and its master slave system. Humanoid Robots, Human-like Machines, pages 29-42, 2007.
[11] H. Liu, X. Xie, M. Millar, M. Edmonds, F. Gao, Y. Zhu, V. J. Santos, B. Rothrock, and S.-C. Zhu. glove-based system for studying hand-object manipulation via joint pose and force sensing. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems IEEE, 2017.
(IROS), pages
[12] S. Li, X. Ma, H. Liang M. Görner, Ruppel, B. Fang. F. Sun, and Zhang. Vision-based Teleoperation of Shadow Dexterous Hand using End-to-Ent Deep Neural Network, Feb. 2019. URL http: /ttp/v/vvvvvv/////////ee///aaava wwwpxvvvvvr//... abs/ 1809. 06268.
[13] C. Wang, H. Shi, W. Wang, R. Zhang, L. Fei-Fei, and K. Liu. DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation, Mar. 2024. URL http: // //arxiv.. org/ abs wwwp///3.... 07788.

---

[14] Y. SAITOU, F. KOBAYASHL F. KOJIMA, H. NAKAMOTO, N. IMAMURA, K. SASABE, and H. SHIRASAWA. Haptie feedback in universal robot hand tele-operation. In SCIS & ISIS SCIS & ISIS 2008, pages 1123-1128, Japan Society for Fuzzy Theory and Intelligent Informatics, 2008.
[15] J. M. Correia Marques, P. Naughton, J.-C. Peng, Y. Zhu, J. S. Nam, Q. Kong, X. Zhang, A. Penmetcha, R. Ji, N. Fu, V. Ravibaskar, R. Yan, N. Malhotra, and K. Hauser. Immersive Commodity Telepresence with the AVATRINA Robot Avatar. International Journal of Social Robotics, Jan. 2024. ISSN 1875-4791, 1875-4805, 101:10.1100 123500007022 URL Sttps://link.sprinngg
www/co.mo/////cccc 1000.1100///111 101.000//12699-0220 01090-11
[16] C. E. Rasmussen and C. K. I. Williams. Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press, Cambridge, Mass, 2006. ISBN 978-0-262-
18253-9. OCLC: ocm6 188712.
[17] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5745-5753, 2019.
[18] E. V. Bonilla, K. Chai, and C. Williams. Multi-task Gaussian Process Predic- tion. In Advances in Nearal Information Processing Systems, volume 20. Curran Asso- ciates, Inc., 2007. URL https: /paperss//ppppeeeeeeeeeeee nips
/hash//
Abstracts..taaeee html.
[19] J. R. Gardner, G. Pleiss, D. Bindel, K. Q. Weinberger, and A. G. Wilson. Gpytorch: Black- box matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018.
[20] Ansel, E. Yang. H. He, N. Gimelshein, A. Jain, M. Voznesensky, B. Bao, P. Bell, D. Be- rard, E. Burovski, G. Chauhan, A. Chourdia, W. Constable, A. Desmaison, Z. Detooo, E. El- lison, W. Feng, J. Gong, M. Gschwind, B. Hirsh, S. Huang, K. Kalambarkar, L. Kirsch, M. Lazos, M. Lezcano, Y. Liang, J. Liang, Y. Lu, C. Luk, B. Maher, Y. Pan, C. Puhrsch, M. Reso, M. Saroufim, M. Y. Siraichi, H. Suk, M. Suo, P. Tillet, E. Wang, X. Wang, W. Wen, S. Zhang, X. Zhao, K. Zhou, R. Zou, A. Mathews, G. Chanan, Wu, and S. Chintala. Py- Torch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In 29th ACM International Conference on Architectural Support for Pro- gramming Languages and Operating Systems, Volume 2 (ASPLOS '24). ACM, Apr. 2024. URL
Noi:10011457622265536666333
pdf.
[21] D. P. Kingma and Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.6900 [cs], Jan. 2017. URL http: http////xvvvv/////////e///eaaevl wtt.aaawww....s.../rrrraaeeaaeee argg/abs/4112.995
[22] S. G. Johnson. The NLopt nonlinear-optimization package. stevengj/nlopt, 2007.
https://github.com
[23] D. Kraft. Algorithm 733: fOMP-fortran modules for optimal control caleulations. ACM Transactions Mathematical Software, 20:262-281, 1994. doi: 10::10.11455 1551114//9111..991111
[24] Zhong, T. Power, Gupta, and P. Mitrano. PyTorch Kinematics, Feb. 2024.
[25] B. Sundaralingam and T. Hermans. Relaxed-rigidity constraints: kinematic trajectory opti-
mization and collision avoidance for in-grasp manipulation. Autonomous Robots, 43:469-488
2019.
10

---

A Training and Implementation Details
Algorithm describes how the GP for each finger is trained.
Algorithm 2 Training the GP
1: function TRAINGP(D, a, E, f)
2: Initialize f randomly
3: Vectorize the sequence of 6:[f] in D into Q
4: for E [E] do
5:
Compute K for all pairs of (qr, qb, [f]) E D using Equation 3
6:
K, t- K + r²
7:
Kg f- K2 @ K, using the current value of K;
8:
Compute loss L from Equation 4
9:
Compute V and update 0 using Adam [21] with learning rate a
10:
return 0
In our implementation, we set the number epochs E = 3000 and the learning rate a = 0.01, which we found experimentally to sufficient to reach convergence. The full calibration procedure takes on average (across tested operators) 4.5 minutes. Data processing and parameter fitting for the GP takes total of 65 seconds, while explaining the calibration procedure and collecting the calibration data takes minutes and 20 seconds.
During inference, we need to compute the mean E of the posterior distribution of the residual conditioned on the observed calibration poses. If C' datapoints ((qr, (f)), )x) have already been observed for finger ʃ with additive independent and identically distributed Gaussian noise with variance o², the conditional distribution of & [f), at new input qž, [f] can be computed as
(<til/))~~N ~(KTK, "g, k.gkl/ (Xl) k k)
where
k(qn, (l.
k(qn, [f, ano, [f] : k(ano, [f] [f]. ahc, (n)) kaana, ((f)))
K,
: k(ano, (w,l1)
+o
k = [kqg,, (fl.gffff)
qa sl $c,,'
We use this expression compute the posterior mean when using the retargeter in Algorithm 1.
MEWW constraine
0.10
0.26
Figure 6: Time to run Res-GP for trajectory where R2 empty (Unconstrained) and one where contains the index-to-thie tooex--o-theee to-thumb vector (Constrained). The GP incurs little overhead.

---

When applying the constraint specified by Equation 5, we found that hand-tuning d to em applied sufficient force to our target objects to lift and manipulate them. As shown in Figure 2, the keypoints of each finger are located inside the fingertip geometry, so that constraining these points to be 1 cm away from each other causes the fingertips to intersect one another. This allows the hand to stably grasp very thin objects as well. In our implementation, we restricted Re to only contain vectors from fingertips to the thumb tip so that we could easily control membership in R. with foot pedals. With more extensive interaction interface, Rg could conceivably contain other keypoint vectors, for example, between the index and middle fingers, to grasp objects in different ways.
Figure 6 shows a breakdown of the time taken by different computations during inference. With no constraints activated the retargeter runs at approximately Hz; with the index-to- to-tummb vector constrained, the retargeter runs approximately 4.5 Hz. The optimization for the HKVM base re- targeter dominates the computation time, while the GP inference is quite fast, taking about 9 ms on average (performing inference for all four fingers takes a total of 9 ms). This shows that our residual GP can be added onto many different base retargeters with little additional overhead. We use NLopt to limit the optimization time in HKVM to 100 ms, and it uses up this entire time budget. When no constraints are activated, the constraint satisfaction problem has an overhead of approximately ms, but when a constraint is activated, it takes on the order of 100 ms to solve, As shown by our experiments, we found that this high-level control rate sufficed for quasistatic and quasidynamic ma- nipulation tasks. To handle highly dynamic tasks, higher control rate and more advanced feedback are likely required.
We use an Allegro right hand outfit with Xela sensors as our robot hand and use a Manus Quantum Metaglove to track the operator's s right hand's 's fingertip poses and joint angles. We use an exponential filter to smooth the values of qe commanded by the operator before using a PD torque controller to track this smoothed target the robot hand at 300 Hz. The smoothed target joint configuration targot q. + (1 -a) 4t-1, where 9o is initialized to O. We hand-
time t is computed q, jarget tuned 0.01 to achieve smooth tracking. When tracking the operator's hand, we remove Manus' "pinch correction," since we found that it introduced tracking artifacts, detecting that the fingers and thumb were performing a pinch even when they were quite far from one another. We prioritized stability and consistency of the hand tracking over absolute accuracy. We additionally opted not to use Manus' built-in glove calibration since we found that it visibly decreased the tracking accuracy of the glove.
B Baselines and Evaluation Metrics
Here, we provide more complete descriptions of the baseline retargeters tested:
Joint Space (Joint): We remap each human joint to the corresponding robot joint (ordered by proximity to the palm). For each human joint, we find its limits (from the recorded workspace) and linearly rescale its range to match the robot's joint range. To run the retar- geter, we apply these scalings to each of the human's 's current joint values to find the desired robot joint configuration.
We track each of the human's fingertips in their palm frame and solve an IK problem for each of the robot's fingers to place the corresponding fingertip at that position (ignoring orientation). The palm of the robot is placed as shown in Figure 2.
Inverse Kinematics (IK):
Hand Keypoint Vector Matching (HKVM): We directly use q (from Equation 1) as the desired joint configuration.
DexPilot [7] (DexPilot): Essentially the same method as HKVM but using the heuristics presented in Handa et al. [7] for precise fingertip pinching and collision avoidance.
Neural Network (NN): We train a neural network for each finger on the calibration data to directly output the finger's target joint configuration based on the operator's finger con- figuration. We use the same rotation representation for the NN retargeter that we used for the GP. The network is fully-connected multi-layer perceptron with an input size of 8,
12

---

4 hidden layers of 256 neurons each, and an output layer of size 2|f which we interpret as a lsl x 2 matrix, each row specifying joint angle as a vector. We use rectified- non-linearities after each of the hidden layers. The network trained for 5000 epochs using the AdamW optimizer with learning rate 0.001 (stepped down to 0.0001 after 2000 epochs), which we found to yield perfect loss on the training set. We use the negative mean cosine similarity loss between each of the rows of the network output and the V matrix representing the ground-truth Allegro hand joint position at each calibration pose.
Gaussian Process (GP): We directly train a GP for each finger on the calibration data to output the finger's target joint configuration. We use a constant mean function and the same kernel and rotation representation presented here.
Residual Neural Network (Res-NN): Use a neural network to learn a residual on top of HKVM. We learn a separate network for each Allegro hand finger. Each network has the same architecture and is trained in the same way as the NN retargeter, except its output is summed with the HKVM output (for given finger) to produce its prediction for each human hand configuration.
Residual Gaussian Process (Res-GP): The method presented here.
We found that the two operators whose results are presented in the paper were not able to complete any of the finger gaiting (Rot H. Rot V, P-to-Q, and Wrench) tasks with the baseline retargeters.
Workspace Analysis. To compute the reachable workspace of each of the retargeters, we collected long trajectory of N human hand joint angles by having operator attempt to mimic video of a hand moving through its workspace several times in a row. We use this approach to approximate the operator's hand workspace, rather than sampling on kinematic model of the operator's ' hand, to capture subtleties such as iniiguratiiondeeeeddee joint limits of human fingers. We then pro- cessed this trajectory using each retargeter to generate trajectories of robot joint angles and used Algorithm to approximate the reachable workspace of the retargeter, considering each finger inde- pendently and discretizing the robot's joint space into hypercubes of size 6. |s| denotes the number of joints associated with the robot finger (in the case of the Allegro hand, always equal 4). We then sum the workspace of each finger to arrive at the retargeter's total reachable workspace. To compute the workspace of the robot's 's fingertips under each retargeter, we similarly run Algorithm but replace Line 6 with X +- X U [p(a', [/])/ō] and Line 7 with w + w + |6 Here, p(-) gets the position of the relevant robot fingertip in its palm frame.
Algorithm 3 Compute joint workspace
1: function COMPUTTEOONYTWOORSPPPCCCC//
5)
2: W +-00 0
3: for F do
4:
X t 9
5: for i [N] do
6:
X - X U La [/]/8]
L.] and division applied elementwise
7: w - w + X 5
8: return
We performed this procedure for five different operators with hand widths (thumb tip pinky tip) ranging from 18.0 to 22.5 em and heights (base of palm to tip of middle finger) ranging from 16.5 to 18.5 cm. Table 2 shows the joint (J) and fingertip (F) workspaces for each of the subjects in rad and cm3 respectively. We see that not only is the average (across operators) workspace of Res-GP superior to the baseline retargeters, it is higher for every individual operator as well.
Qualitative Comparisons. Figure shows the robot configurations produced by all of the tested retargeters for the operator hand configurations shown in Figure 5, well as for extreme positions of the thumb. The IK retargeter tends to keep the robot's 's fingers quite close to the palm since no scaling
13

---

Table Reachable workspaces of each of the baseline retargeters for all tested operators. Joint (J)
and fingertip (F) workspaces for each of the subjects are reported in rad* and cm3 respectively.
Subject
Subject
Subject
Subject
Subject 4
Method
J
F
F
F
J
F
F
Joint
0.124
930
0.122
959
0.123
865
0.107
857
0.123
951
IK
0.120
422
0.123
446
0.113
395
0.108
526
0.135
524
HKVM
0.189
1000
0.200
998
0.197
1009
0.185
1140
0.244
1307
DexPilot [7]
0.145
842
0.150
811
0.150
824
0.151
966
0.192
1101
NN
0.080
632
0.072
582
0.064
506
0.034
310
0,064
472
GP
0.074
715
0.077
756
0.062
566
0.066
682
0.080
813
Res-NN
0.159
1106
0.173
1230
0.153
976
0.166
1174
0.224
1367
Res-GP (Ours)
0.241
1645
0.231
1407
0.236
1355
0.209
1457
0.282
1691
Joint
IK
HKVM
Dexpilot
NN
GP
Res-NN
Res-GP
W
W SAN
SMW
(b)
Figure 7: Qualitative comparison between retargeting methods for series of human hand configu- rations.
is applied to the operator's hand, and often gets trapped in local minima. DexPilot behaves quite similarly to HKVM in these regions. The learning based methods all produce similar results because the demonstrated human hand configurations are very similar to configurations in the calibration set (Figure 3). However, the NN, GP, and Res-NN retargeters tend to overfit to the calibration data, resulting in much lower reachable workspaces.
The higher workspace of Res-GP allows it to reach many parts of target objects, giving the operator many choices of contact location, and ultimately enabling advanced finger gaiting. In particular, it allows the operator to place the robot's thumb in many locations, as shown in Figure 7. The oper- ator able to move the thumb all the way to the base of the palm, which none of the uncalibrated retargeters can achieve. Additionally, using Res-GP, the operator able to keep the thumb outside the space above the palm where an object might sit, whereas other retargeters, such as Joint and HKVM, substantially intrude into this region during the demonstrated trajectory. When manipulat- ing object, this is likely to disturb it and introduce undesired motion. Res-GP gives the operator more freedom to reposition the thumb without touching the target object until they wish to.
14

---

Operator 0 wio
wi
Openetor
70
Operatoo wT
Time
40
Figure 8: Times taken by each operator under each condition to perform the tasks. The plot shows the average scores; error bars show the relative variability (0.2o). Operators take an average of about 43 seconds to collect each successful demonstration.
Timing Results. Figure 8 shows the mean and relative variation of the time taken for each operator complete each task across their attempts both with and without the constraints. Times for all tasks reasonable, typically less than minute, opening the potential for fast collection of many demonstrations for each task. The operators respond quite differently to the constraints. Operator appears to be less comfortable with the constraints, taking longer to complete most tasks when using them. Operator 1, contrast, fluidly integrates them into their operation, likely allowing them to more confidently complete each task and thereby reach the desired final state more quickly.
15


Reviews:
[{"review_id": "HrTKYB8J6d", "reviewer": "Reviewer_MZs1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a optimization-based method for teleoperation of dexterous hands to enable finger-gaiting and other complex dexterous hand motions with multi-fingered robot hands.", "summary_of_recommendation": "I think this paper yields an interesting and novel approach to improve pose retargeting for dexterous teleop, although some of the motivating tasks and examples do not fully convey this, nor are there comparisons to prior methods to strongly justify the additional complexity. However the results shown are fairly clearly, and would upgrade to a strong accept if these points are clarified/addressed.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Overall the paper was very well written, and clearly presented. Some remaining questions after reading through it and going through the supplemental materials include:\n\n- Does including this additional foot-pedal controller make the comparisons to the other methods fair when the workspace can be obviously expanded with additional degrees of freedom/controls for the operator? \n- Some of the tasks do not seem to clearly show the emphasized finger-gaiting, such as Pivot, which is more of a slip task. Were tasks such as rotating the screw driver once grasped possible for the operators (Task 3)?\n- There is no direct comparison between methods DexPilot and ResPilot on any of these tasks. Were these newly selected tasks simply not possible without residual GPs? If so, a note of this would be useful to further motivate.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nXwwmDsCiL", "reviewer": "Reviewer_zYVG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a novel teleoperation method for dexterous robot hands, enabling complex finger-gaited manipulation. Key innovations include: 1. A retargeting approach combining optimization-based hand keypoint matching with learned residuals via Gaussian Processes, expanding the robot's workspace. 2. Fingertip distance constraints to maintain stable object contacts.", "summary_of_recommendation": "The problem investigated by this paper is important, but the performance of the system, as well as the limitation in experiments make this paper not ready to be accepted.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "To better present this work, it is better to provide the following during rebuttal:\n1. A more comprehensive analysis of the system latency, and ways to improve the retargeting speed to a reasonable value.\n2. More experiments to compare with existing work, e.g. [7] and other calibration-free retargeters.\n3. Visualization and analysis of the calibrated robot pose, and the corresponding robot pose generated by other calibration-free retargeters using the same human pose.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gRG0MGe5W0", "reviewer": "Reviewer_5i54", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 5, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper focuses on teleoperation for dexterous hand manipulation. The authors introduce a novel retargeting method for mapping the human hand to the robot hand's configuration using Gaussian process. By evaluating the teleopration system through six challenging tasks, it succeeds in an average of about 80% of the time. There are also quantitative evaluations showing their method's superiority against baselines from the litrature.", "summary_of_recommendation": "The reviewer believes the proposed method has made significance progress compared against the state of the art, and potentially opens up new research opportunities for dexterous hand manipulation.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Can the authors conduct an additional set of experiments with another operator?\n- The reviewer noticed failure cases in the video, but is still curious about what's the main reason behind those failures. Can the authors elaborate on this?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper's strengths include:
The paper introduces a novel method for the teleoperation of dexterous hands by combining optimization-based keypoint matching with learned residuals via Gaussian Processes, improving complex finger-gaiting and manipulations.
The method shows promising results in both qualitative and quantitative evaluations, demonstrating improvements over baseline methods and effective performance in challenging tasks.
The paper is well-written and presented, with sufficient technical details for reproducibility and a thorough explanation of the proposed method.
The system is tested on real hardware with a detailed video showing experiments, supporting the practical applicability of the approach.

The paper's weaknesses include:
There is insufficient comparison with existing methods, especially calibration-free retargeting methods. The paper should provide more analysis of how the proposed method stands relative to others.
The system’s retargeting speed (4.5 Hz) is questioned as being potentially too slow for effective teleoperation, particularly for in-hand manipulation tasks.
The paper lacks experiments with multiple operators and a detailed analysis of failure cases. Additionally, certain experimental details, such as controller speed and constraints, need to be moved to the main body for clarity.
Some tasks in the evaluation do not clearly demonstrate the emphasized finger-gaiting, and there is no direct comparison of the performance of the new method with the baseline DexPilot on all tasks.

Post-rebuttal meta-review:
The paper addresses the challenge of teleoperating dexterous robot hands for complex manipulation tasks. Current methods struggle to replicate the full functional workspace of the human hand, particularly for intricate tasks like finger-gaiting.

The authors propose ResPilot, a novel method that enhances teleoperated dexterous manipulation by expanding the robot hand’s reachable workspace. This is achieved through Gaussian Process (GP) residual learning, which calibrates a human hand to a robot hand motion retargeter. The system introduces fingertip distance constraints to maintain stable object contacts, allowing for more precise and flexible manipulation.

The method is innovative, combining optimization-based keypoint matching with learned residuals via Gaussian Processes to enable complex finger-gaiting tasks. The approach is well-positioned within the existing literature, and the paper is well-written, with sufficient technical detail to ensure reproducibility. The method demonstrates significant improvements over baseline retargeting approaches and is validated through extensive quantitative evaluations on challenging tasks.

The paper lacks a comprehensive comparison with other existing methods, particularly calibration-free retargeting methods. The retargeting speed (4.5 Hz) may be too slow for effective real-time teleoperation, particularly for in-hand manipulation tasks, raising concerns about its practicality. The paper would benefit from additional experiments involving multiple operators to assess generalizability, and a more detailed analysis of failure cases is needed to clarify system limitations.

Final Recommendation: Strong Accept --The paper presents a promising approach to teleoperated dexterous manipulation with clear improvements over existing methods. However, the concerns about retargeting speed, the need for broader comparisons, and further operator testing must be addressed to strengthen the paper’s contributions.
Decision: Accept