DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human References
Authors: Xueyi Liu
Jianibieke Adalibieke
Qianwei Han
Yuzhe Qin
Li Yi
Venue: ICLR 2025 Poster (2025)
Keywords: Dexterous Manipulation
Neural Tracking Control
Homotopy Optimization
Primary area: applications to robotics, autonomy, planning
TL;DR: DexTrack presents a neural tracking controller for dexterous robot hand manipulation, with high adaptability, generalization, and robustness.

Abstract: We address the challenge of developing a generalizable neural tracking controller for dexterous manipulation from human references. This controller aims to manage a dexterous robot hand to manipulate diverse objects for various purposes defined by kinematic human-object interactions. Developing such a controller is complicated by the intricate contact dynamics of dexterous manipulation and the need for adaptivity, generalizability, and robustness. Current reinforcement learning and trajectory optimization methods often fall short due to their dependence on task-specific rewards or precise system models. We introduce an approach that curates large-scale successful robot tracking demonstrations, comprising pairs of human references and robot actions, to train a neural controller. Utilizing a data flywheel, we iteratively enhance the controller's performance, as well as the number and quality of successful tracking demonstrations. We exploit available tracking demonstrations and carefully integrate reinforcement learning and imitation learning to boost the controller's performance in dynamic environments. At the same time, to obtain high-quality tracking demonstrations, we individually optimize per-trajectory tracking by leveraging the learned tracking controller in a homotopy optimization method. The homotopy optimization, mimicking chain-of-thought, aids in solving challenging trajectory tracking problems to increase demonstration diversity. We showcase our success by training a generalizable neural controller and evaluating it in both simulation and real world. Our method achieves over a 10% improvement in success rates compared to leading baselines. The project website with animated results is available at  [DexTrack](https://meowuu7.github.io/DexTrack/).

Published as a conference paper at ICLR 2025
DEXTRACK: TOWARDS GENERALIZABLE NEURAL TRACKING CONTROL FOR DEXTEROUS MANIPULA- TION FROM HUMAN REFERENCES
Xueyi Liu Jianibieke Adalibicke", Qianwei Han', Yuzhe Qin', Li Yi
TTinnguua University *Shanghai Qi Zhi Institute Shanghai AI Laboratory 'UC San Diego
Project website: meowuu7. github. io/DexTrack
Oale
Geal
Noural Tracking
Objict
Stae
Goa.orl
Controler
Adtin
Vnvrrnn
Rouul (Stne e-1)
(1) Mirricingg inmotte nampulniior with thin objens and imiiinng in-lend memiionttion
(c) Tracking nekyy intractions with mntrachable goal strts and nol--aurt evaltation
(r) Infermoo fho of noural Stacting controllze
Figure 1: DexTrack learns a generalizable neural tracking controller for dexterous manipulation from human references. It generates hand action commands from kinematic references, ensuring close tracking of input trajectories (Fig. (af), generalizes to novel and challenging tasks involving thin objects, complex movements and intricate in-hand manipulations (Fig. (b)), and demonstrates robustness large kinematics noise and utility real-world scenarios (Fig. (c)). Kinematic references are illustrated in orange rectangles and background.
ABSTRACT
We address the challenge of developing a generalizable neural tracking controller for dexterous manipulation from human references. This controller aims to age dexterous robot hand to manipulate diverse objects for various purposes defined by kinematic human-object interactions. Developing such controller is complicated by the intricate contact dynamics of dexterous manipulation and the need for adaptivity, generalizability. and robustness. Current reinforcement learn- ing and trajectory optimization methods often fall short due to their dependence on task-specific rewards or precise system models. We introduce an approach that curates large-scale successful robot tracking demonstrations, comprising pairs of human references and robot actions, to train a neural controller. Utilizing a data flywheel, iteratively enhance the controller's performance, well as the num- ber and quality of successful tracking demonstrations. We exploit available track- ing demonstrations and carefully integrate reinforcement learning and imitation learning to boost the controller's performance in dynamic environments. At the time, obtain high-quality tracking demonstrations, we individually op- timize per-trajectory tracking by leveraging the learned tracking controller in a homotopy optimization method. The homotopy optimization, mimicking chain- of ohothhught, aids in solving challenging trajectory tracking problems to increase demonstration diversity. We showcase our success by training generalizable neural controller and evaluating it in both simulation and real world. Our method achieves over a 10% improvement in success rates compared to leading baselines. The project website with animated results is available at Dex Track.
INTRODUCTION
Robotic dexterous manipulation refers to the ability of a robot hand skillfully handling and manipu- lating objects for various target states with precision and adaptability. This capability has attracted

---

Published as a conference paper at ICLR 2025
significant attention because adept object manipulation for goals such as tool use is vital for robots to interact with the world. Many efforts have been devoted previously to push the ability of dexterous hand toward human-level dexterity and versatility (Rajeswaran et al., 2017; Chen et al., 2023; 2021; Akkaya et al., 2019; Christen et al., 2022: Zhang et al., 2023; Qin et al., 2022; Liu et al., 2022; Wu et al., 2023; Gupta al., 2016; Wang et al., 2023; Mordatch et al., 2012; Liu et al., 2024a; Li al., 2024). This also aligns with our vision.
Achieving human-level robotic dexterous manipulation is challenging due to two main difficul- ties: the intricate dynamics of contact-rich manipulation, which complicates optimization (Pang & Tedrake, 2021; Pang et al., 2023; Liu et al., 2024a; Jin, 2024), and the need for robots to master a wide range of versatile skills beyond specific tasks. Previous approaches mainly resort to model- free reinforcement learning (RL) (Chen et al., 2023; 2021; Akkaya et al., 2019; Christen et al., 2022: Zhang et al., 2023; Qin et al., 2022; Liu et al., 2022: Wu et al., 2023; Gupta et al., 2016; Wang et al., 2023) or model-based trajectory optimization (TO) (Pang & Tedrake, 2021; Pang et al., 2023; Jin, 2024; Hwangbo et al., 2018). While RL requires task-specific reward designs, limiting its generalization, TO depends on accurate dynamies models with known contact states, restricting adaptability to new objects and skills. A promising alternative is to leverage human hand-object ma- nipulation references, widely available through videos or motion synthesis, and focus on controlling dexterous hand to track these references. This approach separates high-level task planning from low-level control, framing diverse skill acquisition as the development of a universal tracking con- troller. However, challenges remain due to noisy kinematic references, differences in morphology between human and robotic hands, complex dynamics with rich contacts, and diverse object geome- try and skills. Existing methods struggle with these issues, often limiting themselves simple tasks without in hnd manipulation (Christen et al., 2022; Zhang et al., 2023; Wu et al., 2023; Xu et al., 2023: Luo et al., 2024; Singh et al., 2024; Chen et al., 2024) or certain specific skills (Qin et al., 2022; Liu et al., 2024a; Rajeswaran et al., 2017).
In this work, we aim to develop a general-purpose tracking controller that can follow hand-object manipulation references across various skills and diverse objects. In particular, given a collection of kinematics-only human hand-object manipulation trajectories, the controller is optimized to drive a robotic dexterous hand to manipulate the object that the resulting hand and object trajectories can closely mimic their corresponding kinematic sequences. We expect the tracking controller to exhibit strong versatility, generalize well to precisely track novel manipulations, and have strong robustness towards large kinematics noises and unexpected reference states.
To achieve the challenging goal above, draw three key observations: 1) learning is crucial for handling heterogeneous reference motion noises and transferring data prior to new scenarios, sup- porting robust and generalizable tracking control; 2) leveraging large-scale, high-quality robot track- ing demonstrations that pair kinematic references with tracking action sequences can supervise and significantly empower neural controllers, as demonstrated by data-scaling laws in computer vision and NLP (OpenAI, 2023; Brown et al., 2020); 3) acquiring large and high-quality tracking demon- strations is challenging but we could utilize the data flywheel (Chiang et al., 2024; Bai et al., 2023) to iteratively improve the tracking controller and expand the demonstrations in a bootstrapping manner.
Based upon the previous observations, we propose DexTrack, a novel neural tracking controller for dexterous manipulation, guided by human references. Specifically, given a collection of human hand-object manipulation trajectories, we first retarget the collection to kinematic robotic dexterous hand sequences to form a set of reference motions as data preparation. Our method then alternates between mining successful robot tracking demonstrations and training the controller with the mined demonstrations. To make sure the data flywheel functions effectively, we introduce two key designs. First, we carefully integrate reinforcement and imitation learning techniques to train a neural con- troller, ensuring its performance improves with more demonstrations while maintaining robustness against unexpected states and noise. Second, we develop per-trajectory tracking scheme that uses the trained controller mine diverse and high-quality tracking demonstrations through a homotopy optimization method. The scheme transfers the tracking prior from the controller to individual tra- jectories ease per-trajectory tracking for better demonstration quality. Moreover, the scheme will convert a tracking reference into a series of gradually simplified reference motions so that track- ing these references from simple to complex could help better track the original reference motion. This is akin to chain-of-thought and is very suitable for tracking complex reference motions to in- crease the demonstration diversity. The two designs above together with the iterative training enable Dexcrack to successfully track novel and challenging human references.
2

---

Published as a conference paper at ICLR 2025
We demonstrate the superiority of our method and compare it with previous methods on challenging manipulation tracking tasks in two datasets, describing expressive hand-object interactions in daily and functional tool-using scenarios, involving complex object movements, difficult and subtle in- hand re-orientations, interactions with thin objects, and frequent hand-object rich contact variations. We conduct both extensive experiments in the simulator, i.e., Isaac Gym (Makoviychuk al., 2021), and evaluations in the real world, to demonstrate the efficacy, generalization ability, and robustness of our tracker to accomplish wide range of manipulation tracking tasks and even excellently track novel manipulation trajectories (Figure 1). Our approach successfully surpasses the previous meth- ods both quantitatively and qualitatively, achieving more than 10% success rate than the previous best-performed method. Besides, we conduct further analysis and demonstrate the various recov- ery behaviors of our controller, demonstrating its robustness to unexpected situations. Thorough ablations are conducted to validate the efficacy of our designs.
Our contributions are threefold:
We present generalizable neural tracking controller that progressively improves its per- formance through iterative mining and incorporating high-quality tracking demonstrations.
We introduce a training method that synergistically combines reinforcement learning and imitation learning. This approach leverages abundant bigh-quality robot tracking demon- strations to produce a controller that is generalizable, versatile, and robust.
We develop per-trajectory optimization scheme that employs our tracking controller within a homotopy optimization framework. We propose a data-driven way to generate homotopy paths, enabling solving challenging tracking problems.
RELATED WORK
Equipping robots with human-level dexterous manipulation skills is crucial for future advancements. Previous approaches either rely on model-based trajectory optimization or model-free reinforce- ment learning (RL). Model-based methods face challenges due to the complexity of dynamics, often requiring approximations (Pang et al., 2023; Jin, 2024; Pang & Tedrake, 2021). Model-free ap- proaches, using RL (Rajeswaran et al., 2017; Chen et al., 2023; 2021; Christen et al., 2022; Zhang et al., 2023; Qin et al., 2022; Liu et al., 2022; Wu et al., 2023; Gupta et al., 2016; Wang et al., 2023; Mordatch et al., 2012), focus on goal-driven tasks with task-specific rewards, limiting their gener- alization across diverse tasks. Our work explores a general controller for dexterous manipulations. Besides, learning via mimicking kinematic trajectories has recently become a popular train to equip the agent with various expressive skills (Jenelten et al., 2023; Luo et al., 2023b;a). DTC (Jenelten al., 2023) proposes strategy that can combine the power of model-based motion planning and RL to overcome the sample inefficiency of RL. In the humanoid motion tracking space, PHC (Luo et al., 2023b) proposes an effective RL-based training strategy to develop general humanoid mo- tion tracker. Recently, OmniGrasp (Luo et al., 2024) proposes to train a universal grasping and trajectory following policy. The policy can generalize to unseen objects as well as track novel mo- tions. However, their considered motions are still restricted in grasping and trajectory following, leaving the problem of tracking more interesting and difficult trajectories such as those with subtle in-hand manipulations largely not explored. In this paper, we focus on these difficult and challeng- ing manipulations. Moreover, we are also related to recent trials on combining RL with imitation learning. To overcome the sample inefficiency problem of RL and facilitate the convergence, various approaches have been developed aiming to augment RL training with demonstrations (Sun al., 2018: Hester et al., 2017: Booher et al., 2024; Liu al.. 2023). In our work, we wish to leverage sish-uualtyy demonstrations to guide the agent's 's explorations. Unlike previous work where demonstrations are readily available, acquiring a sufficient amount of high-quality robot tracking demonstrations remains significant challenge in our task.
METHOD
Terminologies and notations. Dexterous manipulation "tracking" involves controlling robotic hand to mimic a kinematic hand-object state sequence, the goal trajectory, denoted as ($,, IN These "kinematic references" resenting the robot hand state and object pose at timestep n. A "tracking demonstration" pairs a kinematic reference { with expert action sequence { guiding the robot from So So to
retargeted from human manipulation trajectories, with 6, rep-
3

---

Published as a conference paper at ICLR 2025
Mining High-Quality Rohot Tracking Demonstrations
Learning a Neural Tracking Controller from Demonstrations
Homolopy
Tracking Demosstratoos
Pass
Stare &L
Optimization
Expet Adies
Neural Tracking
Action de
knvinomeess
Kinematic Reterences
Single Trajectury
Tracking
Relecngce
Kismrath
Demo (Eagen) Action di.
Controller
Rewantly Tp
cocovalle
Emsisation
Figure 2: DexTrack learns generalizable neural tracking controller for dexterous manipulation from human references. It alternates between training the tracking controller using abundant and high-quality robot tracking demonstrations and improving the data via the tracking controller through homotopy optimization scheme.
achieve a state sequence (s n aligned with (8,, - A "robust" controller can tolerate distur- bances like kinematics noise and unreachable states, It demonstrates high "generalization ability" if it performs well on unseen scenarios such as new objects and novel motions, and has "adaptivity" it maintains effectiveness despite shifting contexts, such changing contacts and dynamics.
Given a collection of kinematic human-object manipulation trajectories, we wish to learn gener- alizable neural tracking controller for dexterous robotic hand. The problem is challenging due to the difficulty in precise dexterous manipulation, which is challenged by the underlying complex dynamics, and the high demands for the controller's generalizability and robustness.
We address these challenges by combining reinforcement learning (RL.) and imitation learning (IL) to train generalizable tracking controller, jointly easing the difficulty in solving the complex prob- lem via supervision from high-quality and diverse robot tracking demonstrations and improving the policy's 's robustness via RL explorations. We introduce a single trajectory tracking scheme to mine tracking demonstrations, composed of paired kinematic references and action sequences. For each kinematic reference, we use RL train a trajectory-specifi policy that generates actions track the reference. To overcome RL's limitations. we propose to leverage the tracking controller through a homotopy optimization scheme to enhance the demonstration quality and diversity. By iteratively mining better demonstrations and refining the controller in a bootstrapping manner, we develop an effective, generalizable tracking controller. We will explain how we learn the neural tracking con- troller from demonstrations in Section 3. how we mine high-quality demonstrations in Section 3.2, and how we iterate between learning controller and mining demonstrations in Section 3.3.
3.1 LEARNING A NEURAL TRACKING CONTROLLER FROM DEMONSTRATIONS
Given a collection of human-object manipulation trajectories and a set of high-qualin robot track- ing demonstrations, our goal is to learn an effective and generalizable neural tracking controller. In the beginning, we retarget human hand-object manipulation to a robotic hand as a data preprocess- ing step. We combine RI. and II. to develop generalizable and robust neural tracking controller. Taking advantage of imitating diverse and high-quality robot tracking demonstrations, we can ef- fectively let the tracking controller master diverse manipulation skills and also equip it with high generalization ability. Jointly leveraging the power of RL, the controller avoids overfitting to suc- cessful tracking results limited to narrow distribution, thereby maintaining robust performance in the face of dynamic state disturbances. Specifically, we design an RL-based learning scheme for training the tracking controller, including the carefully-designed action space, observations, and the reward tailored for the manipulation tracking task. We also introduce an IL-based strategy that lets the tracking controller benefit from imitating high-quality demonstration data. By integrating these two approaches, we effectively address the complex problem of generalizable tracking control.
Neural tracking controller. In our formulation, the neural tracking controller acts an agent that interacts with the environment according to a tracking policy π. At each timestep n, the policy observes the observation on and the next goal š,+1 (designated as the target state for the robotic hand and the object). The policy then computes the distribution of the action. Then the agent sample an action a, from the policy, i.e., a, ~ T( oo,, $n+1). The observation o, primarily contains the current state S, and the object geometry. Upon executing the action with the robotic dexterous hand, the hand physically interacts with the object, leading both the hand and the object to transition into the next state according the environment dynamics, represented 8,+1 ~ p(-|s,,, a,). An effective tracking controller should ensure that the resulting hand and object states closely align with their respective next goal states.
Reinforcement learning, In the RL-based training scheme, the agent receives a reward rx r(S,, an, &n +1, 8,+1) after each transition. The training objective is to maximize the discounted
4

---

Published as a conference paper at ICLR 2025
cumulative reward:
W-1
J E (((ss)
Σ rT
where p(T[T) = p(so) II p(sn+1 Oz an) jectory of the agent determines the effective horizon length of the policy. In the tracking control problem, the next goal state S typically comprises the subsequent hand state and the object state from the kinematic reference sequence. We control the robotic hand using a proportional derivative (PD) controller, following previous literature (Luo et al., 2024; 2023b; Christen et al., 2022; Zhang et al., 2023). The action a, contains the target position commands for all hand joints. To enhance the sample efficiency of RL, rather than allowing the tracking policy to learn the absolute positional targets, we introduce residual action space. Specifically, we introduce baseline hand trajectory and train the policy to learn the residual relative target Aa, at each timestep. The baseline trajectory is consis- tently available for the tracking problem and can be trivially set to the kinematic reference trajectory. In each timestep n, we compute the position target via a, = 8% + >> Aak. where s is the n-th step hand state in the baseline trajectory. The observation at each timestep n encodes the current hand and object state, the baseline trajectory, actions, velocities, and the object geometry:
S is the likelihood of a transition tra- (So, ag, ro. .......SN--..... 1N-1, SN). The discount factor y E [0,
(2) where feat is the object feature generated by a pre-trained object point cloud encoder. We also introduce an auxiliary feature aux,, computed based on available states, to provide the agent with more informative context. Further details will be explained in Appendix A. Our reward for ma- nipulation tracking encourages the transited hand state and the object state to closely match their respective reference states, as well as promoting hand-object affinity:
On = {S Sns s a,, featebj: aux,),
D.pp + Wo.g"o. + Wwiist wrist + Winnerr + WallimtyTalliiitt
(3)
where To.p. go wristt Finger. Faftiiity represent rewards for object position, object orientation, hand wrist, hand fingers, and hand-object affinity, respectively, while Wo,ps Wo.q. Wwrist; Wfingers Waffinity are their corresponding weights. Details regarding the reward computation are deferred to Appendix A.
Imitation learning. The RL-based learning scheme, hindered by sample inefficiency and its in- ability to handle multiple tracking problems, struggles to solve the generalizable tracking control problem, as demonstrated by our early experiments. Therefore, we propose an IL-based strat- egy to distill successful, abundant, and diverse "tracking knowledge" into the tracking controller. Specifically, train the tracking agent to imitate large number of high-quality robot tracking demonstrations. This approach effectively guides the agent to produce "expert actions" that can successfully track the reference state. Additionally, by imitating diverse tracking demonstrations, the agent can avoid repeatedly encountering low rewards in challenging tracking scenarios, while also preventing over-exploitation of easier tasks. Formally, robot tracking demonstration of length N consists of a kinematic reference sequence (So...,, S..,,,)) and an expert's 's state-action trajectory (sf a ..., sty 1: ali- sk). In addition to the actor loss, we incorporate an action supervision loss to bias the policy's 's predictions at each timestep towards the corresponding expert action in the demonstration:
C. = Ea,, ron(- an |nn /
(4)
This guidance allows the policy's s exploration be informed by these demonstrations, ultimately speeding up convergence and enhancing performance in complex problems. From the perspective of IL, the RL exploration introduces noise into the states, making the imitation more robust, in a similar flavor to DART (Laskey et al., 2017). Since the agent should not and will not explore states too far from the reference states in the tracking control task, it is feasible to optimize the policy using both the imitation loss and RL reward simultaneously.
3.2
MINING QUALITY ROBOT TRACKING DEMONSTRATIONS USING NEURAL CONTROLLER THROUGH HOMOTOPY OPTIMIZATION SCHEME
To prepare demonstration from kinematic reference trajectory (So. ..., Sx) for training the track- ing controller, we need to infer the action sequence (a/ a that successfully tracks the ref- erence sequence. A straightforward approach is leverage RI. train single-trajectory tracking
5

---

Published as a conference paper at ICLR 2025
policy and use its resulting action sequence directly. However, relying solely this strategy of- ten fails to provide a diverse and high-quality dataset of tracking demonstrations, as RL struggles with the inherent challenges of dexterous manipulation. To enhance both the diversity and quality of robot tracking demonstrations, we propose utilizing the tracking controller in conjunction with a homotopy optimization scheme to improve the per-trajectory tracking results.
RL-based single trajectory tracking. A basic approach to acquiring demonstrations is to leverage RL address the per-trajectory tracking problem, using the resulting action sequence as the demon- stration. Given a kinematic reference trajectory, our goal is to optimize a tracking policy π that can accurately track this reference trajectory. At each timestep, т observes the current state Sn and the next goal state Sx+1, and it predicts the distribution of the current action an. The policy æ is opti- mized to minimize the difference between the transited state S p(-Is,, an) at each timestep and the goal state +.. Similar to our design for the RL--based learning scheme for the tracking controller, we adopt a residual action space. For each tracking task, we introduce a baseline trajec- tory (s). .e,,....., sk.) and only learn a residual policy using RL. The observations and rewards follow the same design as for the tracking controller (see Eq. 8 and Eq. 10). Once π has been optimized, we can sample the action a., from и at each timestep. By iteratively transitioning to the next state using the predicted action and querying the policy π to generate a new action, we can obtain the expert action sequence (at ..... - for the input kinematic reference trajectory.
Transferring "tracking prior". To improve the demonstrations, we devise a strategy that takes advantage of the tracking controller, which has already encoded "knowledge" that can track lots of trajectories (named as "tracking prior"), to improve trajectory-specific tracking policies. Specifi- cally, track a reference trajectory (So...,, we first utilize the tracking controller to track it, with the baseline trajectory set to thew reference sequence. We then adjust the baseline trajectory to the resulting action sequence and re-optimize the residual policy using the RL-based single tra- jectory tracking method. This approach can help us find a better baseline trajectory, facilitating single-trajectory tracking policy learning and improving the per-trajectory tracking results.
A homotopy optimization scheme. Training the controller with data mined from itself can intro- duce biases and reduce diversity, hindering its generalization capabilities. To address this issue, we propose homotopy optimization scheme to improve the per-trajectory tracking performance and to tackle previously unsolvable single-trajectory tracking problems. For the tracking problem To. instead of solving directly, the homotopy optimization iteratively solves each tracking task in an optimization path, e.g., (TK. T.k.......... To), and finally solves To. in a similar flavor to "chain-or thought" (Wei et al., 2022). In the beginning, we leverage the RL-bbased tracking method solve Tx. After that, we solve each task Tm via the RI.-based tracking method with the baseline trajectory set to the tracking result of m+1- This transfer of tracking results from other tasks helps us establish better bascline trajectories, ultimately yielding higher-quality tracking outcomes.
Finding effective homotopy optimization paths. While leveraging the homotopy optimization scheme to solve previously unsolvable problems has proven effective for many tasks, it depends on identifying effective optimization paths. straightforward approach is brute-force searching. Specifically, given a set of kinematic references. we first optimize their per-trajectory tracking out- comes. We then identify neighbors for each task based on the similarity between pairs of kinematic reference trajectories. Next, we iteratively transfer optimization results from neighboring tasks and re-optimize the residual policy for each tracking task. For a specific task, we consider a neighbor that provides a better baseline trajectory than its kinematic trajectory an effective "parent task". After reaching the maximum number of iterations, K, we can find effective homotopy paths for a task, i.e., To. by starting from To and backtracing effective "parent tasks". We define (Tk.... .... To) as an effective homotopy path for Tp if, for each 0 S m S K, Tm+1 is an effective parent task for Tm-
Learning: homotopy generator for efficiint homotopy path planning. Finding effective homo- topy optimization paths for tracking each trajectory is computationally expensive and impractical during inference. To address this, we propose learning homotopy path generator M from a small dataset, enabling efficient generation of effective homotopy paths for other tracking tasks. The key problem in identifying homotopy paths lies in finding effectivve "parent tasks". We reformulate this problem as tracking task transformation problem, aiming for a generator M that provides a dis- tribution of effective "parent tasks" for each tracking task To: M(-)To), considering the fact that a single tracking task may have multiple effective "parent tasks". Once M is trained, we can find a homotopy path by iteratively finding parent tasks. We propose training a conditional diffusion model

---

Published as a conference paper at ICLR 2025
as the tracking task transformer, leveraging its strong distribution modeling capability. Given set of tracking tasks, characterized by the kinematic reference trajectories of the hand and object, as well as object geometry, we first train a diffusion model to capture the distribution of tracking tasks. To fine-tune this diffusion model into a conditional model, we first search for effective homotopy paths within the tracking task dataset. This yields set of paired data of the tracking task T. and their corresponding effectíve "parent task" Tp- We then use this data to tune the diffusion model into conditional diffusion model, such that Tp (M)2))). A well-trained M can efficiently propose an effective homotopy path for a tracking task, i.e., To. by starting from To and recursively finding the parent task, resulting in (TK, ..., To), where Tm+1 ~ (M for all I m I K
3.3 IMPROVING THE TRACKING CONTROLLER VIA ITERATIVE OPTIMIZATION
We adopt an iterative approach that alternates between training the tracking controller with abun- dant robot tracking demonstrations and curating more diverse, higher-quality demonstrations using the controller. Our method is divided into three stages. In the first stage, we sample small set of tracking tasks and generate an initial demonstration set by applying RL to obtain tngje-trajyctory tracking results for each task. Using these demonstrations, we train the tracking controller with both RI. and II., At this stage, we do not train the homotopy path generator, as the model's generalization ability would be limited by the small number of effective homotopy paths available for training. In the second stage, we sample a dataset of trajectories from the remaining tasks, weighted according to the controller's tracking error. We then use RL, incorporating tracking priors, to optimize per- trajectory tracking, search for homotopy paths, and train a homotopy path generator based on the resulting data. The best tracking results from all successfully tracked trajectories are curated into a new demonstration set, which is used to re-train the tracking controller. In the third stage, we resample tracking tasks from the remaining set and leverage RL, the tracking controller, and the ho- motopy generator to curate another set of tracking demonstrations. This final set is used to optimize the tracking controller, resulting in our final model.
4 EXPERIMENTS
We conduct extensive experiments to evaluate the effectivenesss, generalizability, and robustness of our tracking controller. Tested on two HOI datasets featuring complex daily manipulation tasks, our method is assessed through both simulation and real-world evaluations (see Sec. 4.1). We compare our approach to strong baselines, showing its superiority. Our controller successfully handles novel manipulations, including intricate movements, thin objects, and dynamic contacts (see Sec. 4.2), while previous methods fail to generalize well. On average, our method improves the tracking suc- cess rate by over 10% compared to the best prior methods. Additionally, analyze its robustness to significant kinematic noise, such as unrealistic states and large penetrations (see Sec. 4.3).
4.1 EXPERIMENTAL SETTINGS
Datasets. We test our method on two public human-object interaction datasets: GRAB (Taheri et al., 2020). featuring daily interactions, and TACO (Liu et al., 2024b), containing functional tool-use interactions. In simulation, we use the Allegro hand, with URDF adapted from Isaac- GymEnvs (Makoviychuk et al., 2021), and in real-world experiments, the LEAP hand (Shaw et al., 2023), due to hardware constraints. Human-object interaction trajectories are retargeted to create robot hand-object sequences using TyToochKKinematoe the GRAB and TACO datasets, producing 1,269 and 2,316 robot hand manipulation sequences, respectively. The evaluation GRAB focus on testing the model's generalization action sequences. Specifically, we use sequences from subject s (197 sequences) as the test data while the remaining trajectories as the training set. For the TACO dataset, we follow the gener- alization evaluating setting suggested by the authors (Liu et al., 2024h) and split the dataset into a training set with 1,565 trajectories and four distinct test sets different difficulty levels. The primary quantitative results are reported on the first-level set. More details are provided in Appendix C. Metrics. We introduce five metrics to evaluate the tracking accuracy and task success: 1) Per-frame average object rotation error: Rer = 1 Ew Diff Angle (qn, qn), where ån. and qn are refer- ence and tracked orientation. 2) Per-frame average object translation error: Tex = N+I Σ ||t., E. where t, and t, are the tracked and reference translations. 3) Per-frame average wrist position and rotation error: Ewiss प Σ (0.5 Diff Angle(q,,, wrist ( + 0.5||t wrist - i []), where q wrist and qurist wrist orientations, and t.yrist and f.wrist are translations. 4) Per-frame per-joint
(Zhong et al., 2024). We fully retargeted
unseen inter-
N+1
7

---

Published as a conference paper at ICLR 2025
Quantitative evaluations. Bold red and italic blue values for hest and the second best-performed ones respectively. "Ours (w/)) data" and "Ours (w/a data, w/o homotopy)" are two ablated versions w.r.t.
Table 1:
quality of rohot tracking demonstrations used in imitation learning (see Section 5 for details).
Dataset
Method
Rai (rad, ()
Tar (cm, 4)
Ewiu (4)
Einga (rad, ()
Success Rate (%, 1)
DGrasp
0.4493
6.75
0.1372
0.6039
34.52/52.77
PPO (OmniGraasp rew.)
0.4404
6.69
0.1722
0.6418
35.53/54,8
PPO (wfo sup,, tracking rew.)
0.3945
6.11
0.1076
0.5899
38.58/54.88
GRAB
Ours (w/o data, homotopy)
0.3443
7.81
0.1225
0.5218
39.59/57.88 8.87
Ours (w/o data)
0.3415
4.97
0.1483
0.5264
43.15562.44 44
Ours
0.3303
4.53
0.1118
0,5048
46.70/65.48
DGrrsp
0,5021
5.04
0.1129
0.4737
38.42/47.78
PPO (OmmiGGrssp rew.)
0,5174
5.43
0.1279
0.4945
33,5/46.31
PPO (wfo sup.. tracking rew.)
0.4815
4.82
0.1195
0.4682
34.98/57.0
TACO
Ours (w/o data, w/o homotopy)
0.4444
2.33
0.1782
0,5438
44.83/67.08 .0
Ours (wlo data) Ours
0.4854
2.21
0.1698
0.4772
47.78172.41
0.4953
2.10
0.1510
0.4661
48.77/74.38
average position error: Eninger N+1 Σ
- (11) where 0 denotes finger joint
positions, is the degrees of freedom. 5) Success rate: A tracking attempt successful if Ter. Ram. and 0.5 + 0.5 0.5ger are all below the thresholds. Success is calculated with two thresholds: 10cm-209-0.8 and 10cm-409-1.2.
Baselines. To our knowledge, no prior model-based methods have directly tackled tracking control for dexterous manipulation. Most existing approaches focus on single goal-driven trajectory opti- mization with simplified dynamics models (Jin. 2024; Pang et al., 2023; Pang Tedrake, 2021), limiting their adaptability for our framework's s generalizable tracking controller. Thus, we primarily compare our method with model-free approaches: 1) DGrasp (Christen et al., 2022): Adapted to track by dividing sequences into subsequences of 10 frames, with each subsequence solved incre- mentally. 2) PPO (OmniGrasp rew.): We re-implemented OmniGrasp's reward (Luo et al., 2024) to train a policy for tracking object trajectories. 3) PPO (w/o sup.. tracking rew.): We trained policy using PPO with our proposed tracking reward and observation design.
Training and evaluation settings. We use PPO (Schulman al., 2017), implemented in rl-games (Makoviichuk & Makoviychuk, 2021), with Isaac Gym (Makoviychuk et al., 2021) for simulation. Training is performed with 8192 parallel environments for both the per-trajectory tracker and the tracking controller. For the dexterous hand, the position gain and damping coefficient set to 20 and 1 per finger joint. Evaluation results are averaged across 1000 parallel environments, and for real-world evaluations, use LEAP (Shaw et al., 2023) with Franka arm and Foundation- Pose (Wen et al., 2023) for object state estimation. Additional details are provided in Appendix C.
4.2
GENERALIZABLE TRACKING CONTROL FOR DEXTEROOUS MANIPULATION
Kisznzti Reforece
Figure 3: Robustness w.rt. unreasonable states. Please check our website and video animated results.
We demonstrate the generalization ability and robustness of our tracking controller on unseen trajec- tories involving challenging manipulations and novel, thin objects. Our controller has no difficulty in handling intricate motions, subtle in-hand re-orientations, and expressive functional manipulations, even when dealing with thin objects. As shown in Table 1, we achieve significantly higher success rates, calculated under two different thresholds, compared both datasets. Figure 4 provides qualitative examples and comparisons. We show the real-world effectiveness of our method and the superiority over best- best-performing baselines (Figure 4,Table 2). For animated results, please visit project website and the accompanying video.
the best-performing baseline across
Intriguing in-hand manipulations. Our method effectively generalizes to novel, complex, and challenging functional manipulations, featured by subtle in-hand re-orientations, essential for pre- cise tool-use tasks. For instance, in Figure 4a, the shovel is lifted, tilted, and reoriented with intricate finger movements to complete stirring motion. Similarly, in Figure 4c, the small shovel is reori-
8

---

Published as a conference paper at ICLR 2025
Table 2:
Real-world quantitative comparisons. Bold red numbers for best values.
apple
banana
dlick
elegbant
Eight:
plone
redo tooking men
10:220 5188255//33
0001 252/23
25,0O0
66 TEN TO MANNNLOK
21
51 33 SURRLAAL
25,000
ented using minimal wrist adjustments. These results demonstrate the robustness and generalization of our controller, outperforming the PPO baseline, which struggles with basic lifting.
Intricate manipulations with thin objects. Our method also generalizes well to challenging manipulations involving thin objects. In Figure 4b, despite the thin shovel's complexity and missing CAD model parts, our method successfully lifts and controls it using a firm grasp with the second and third fingers. Similarly, in Figure 4e, our controller adeptly lifts and manipulates a thin flute, while the best- best-performing baseline struggles with the initial grasp. These results highlight the advantages of our approach handling complex and delicate manipulations.
Real-world evaluations and comparisons. We directly transfer tracking results to the real world to assess tracking quality and evaluate the robustness of the state-based controller against noise in the state estimator. Success rates are measured under three thresholds and compared with the best baseline. Table 2 summarizes the per-object success rates averaged over their manipulation trajectories in the transferred controller setting. As demonstrated in Figures 4f and 4g, enable the robot to track complex object movements and successfully lift a hard-to mond-to-gret to-grasp round apple in real-world scenarios. While the baseline fails. Further details can be found in Appendix B.2.
4.3 FURTHER ANALYSIS AND DISCUSSIONS
Robustness to noise in the kinematic reference motions. Despite severe hand-object penetrations in Figure 4c (frames 2 and 3) and Figure 4a (frame 2), the hand still interacts effectively with the object, highlighting the resilience of our tracking controller in challenging scenarios.
Robustness to unreasonable references. As shown in Figure 3, our method remains unaffected by significant noise in the kinematic references with unreasonable states. We effectively track the entire motion trajectory, demonstrating the controller's robustness in handling unexpected noise.
ABLATION STUDIES
Diversity and quality of robot tracking demonstrations. We propose enhancing the diversity and quality of tracking demonstrations using the tracking controller and homotopy generator. We ablate these strategies by creating two variants: "Ours (w/o data, w/o homotopy)". where the dataset is built by optimizing each trajectory without prior knowledge, and "Ours (w/o data)", which uses only the homotopy optimization scheme to improve demonstrations. Despite using the same number of demonstrations, both variants produce lower-quality data. As shown in Table 1, they underperform compared to our full method, underscoring the importance of data quality in training the controller. Scaling the number of demonstrations. To in- vestigate the relationship between the tracking con- troller's performance and the number of demonstra- tions, we vary the size of the demonstration dataset during training and tested performance the TACO dataset. Specifically, down-sampled the dataset to 0.1, 0.3, 0.5, and 0.9 of its original size and re-trained the model. As shown in Figure there is a clear correlation be- tween the amount of demonstrations and model per- formance. Since the curve has not plateaued, we hy- pothesize that increasing the amount of high-quality data could further improve performance.
Scaling the number of demonstrution 24.38
72.91
Sucues Rate (59)
70
the final training iteration,
62.
0..07
39.61
0.0
0.2
0.4
0.66
0.8
1.0
Proportion of Total Demonstrations Used
Figure 5: Scaling the amount of demonstrations.
CONCLUSIONS AND LIMITATIONS
We propose DexTrack to develop a generalizable tracking controller for dexterous manipulation. Leveraging high-quality tracking demonstrations and a pre-trajectory tracking scheme, we refine the controller through bootstrapping. Extensive experiments confirm its effectiveness, establishing strong foundation for future advancements. Limitations. A key limitation is the process of acquiring high-quality demonstrations. Future work could explore faster, approximate methods for homotopy optimization to speed up training.

---

Published as a conference paper at ICLR 2025
Brorgeed
Trieetey
Oar
t6 Minaiiiskiiigg slovd (Chillagee difaak in haaa oriennttzess using Tegers,oonglonn dijett (IOVERSOT)
Kinenvik
Tnjesuey
(b) Manipslaing in extirely Gia stowd, (Chullnges difiiah hald Eun the esd, infodd re-otanaaioos using fagers, comples dbject marenans))
Retrreted
Kinetatie
Ingestory
Retareta
Kinsmatio
Trapctory
PPO
tracking
rew.
Our
Uiig chamns, (Challonrs difiiall oumele doed morenari
Reaggee Kinenate Ingerary
PPO
Oap
(6) Playlng (Chilleggr grap, usigg firgon)
Aetageted
Kinemati
Tryectry
r90
tracking
rew)
Onn
i Cosplic divect sesoneso will is Good sutot rewnistaiioss
(e) Liliig upole taa is linel to smare and
Figure 4: Qualitative comparisons. Please check website and the accompanying video for animated results.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Igge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXiv: 1910.07 13, 2019. 2
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenhang Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, K. Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang. Jian Yang, Jian Yang, Shusheng Yang. Yang Yao, Bowen Yu, Yu Bowen, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xing Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen techni- cal report. ArXiv, abs/2309. 16609, 2023. URL https: htapps:///ppp/ /api semanti icchoola ar aatoooarrorr/// .rrg/ Corpus TD 263134555
Jonathan Booher, Khashayar Rohanimanesh, Junhong Xu, Vladislav Isenbaev, Ashwin Balakrishna, Ishan Gupta, Wei Liu, and Aleksandr Petiushko. Cimrl: Combining imitation and reinforcement learning for safe autonomous driving. ArXiv, abs//2006.08878, 2024. URL https::///// //api.
semanticscholes ort/Corpo/ :2704444133
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Verbert--oss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc- Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020. URL https: /ttp///ppp..////// semant iommanicscholar. org/ Co
2
Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. Confer on Robot Learning, 2021. 2,3
Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes. Science Robotics, 8(84): eadc9244, 2023. doi: 10. 1126/scioootticssacc2444 URL https:// https:://wwww.e http///wwwiiinccc.cecc
doi/abs/ 10. 1tw.1126//00 http://crroboiic..coooooeeocecce adc 2,
Zerui Chen, Shizhe Chen, Cordelia Schmid, and Ivan Laptev. Vividex: Learning vision-based dexterous manipulation from human videos. ArXiv, 15s22404.15099, 2024. URL https:
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng. Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Sto- ica. Chatbot arena: An open platform for evaluating Ilms by human preference. ArXiv, 0bs//403..0413.. 2024. 268264163.2
wesccholar..og//oopuuuu URL https://apt semant wwscchcar......... wttp://gg/////eeeeeeee
Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
20577-20586, 2022. 2, 5,
Abhishek Gupta, Clemens Eppner, Sergey Levine, and Pieter Abbeel. Learning dexterous manip- ulation for soft robotic hand from human demonstrations. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3786-3793, IEEE, 2016. 2,
Todd Hester, Matej Vecerík, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Amold, John P. Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations. In AAAI Conference Artificial Intelligence, 2017. URL https: : tttp:/p.....iiiiiioo....am....as semanticscholars .rg.aarrrrrgg wwt. htwp://cccooaaaoooooooorroaaaoaa 1200847.
11

---

Published as a conference paper at ICLR 2025
Jemin Hwangbo, Joonho Lee, and Marco Hutter. Per-contact iteration method for solving contact dynamics. IEEE Robotics and Automation Letters, 3(2):895 9025, 2018. 2
Fabian Jenelten, Junzhe He, Farbod Farshidian, and Marco Hutter. Dte: Deep tracking control. Science Robotics, 9, 2023. URL atttps://api. http://pp.ssmaancccsscccccooco/a
org/CorpusID:
263152143.
Wanxin Jin. Complementarity-fiee multi-contact modeling and optimization for dexterous manip- ulation. 2024. https://apls semanticscholar. 2,8
Michael Laskey, Jonathan Lec, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on robot learning, pp. 143-156. PMLR, 2017.
Zhongyu Li, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, and Koushil Sreenath. Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control. ArXiv, abs/2401. 16889, 2024. URL https:/tp//....... /api semant www.aattscccocccrooooooo..
CoIpusID:267320454.
Xingyu Liu, Deepak Pathak, and Kris M Kitani. Herd: Continuous human-to-robot evolution for learning from human demonstration. arXiv preprint arXiv:2212.04355 2022. 2,
Xuefeng Liu, Takuma Yoneda, Rick L. Stevens, Matthew R. Walter, and Yuxin Chen. Blending imitation and reinforcement learning for robust policy improvement. ArXiv, abs/2310.01737, 2023. URL https: http:///appp.aaaaiiieeeeeeemeeee www.ntccscoooeeeeeeooo.o.cccccce
Xueyi Liu, Kangbo Lyu, Jieqiong Zhang, Tao Du, and Li Yi. Quasisim: Parameterized quasi- physical simulators for dexterous manipulations transfer. arXiv preprint arXiv:2404.07988 2024a. 2
Yun Liu, Haolin Yang, Xu Si. Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, and Li Yi. Taco: Benchmarking generalizable bimanual tool-action-object understanding. arXiv preprint arXiv:2401.08399 2024b. 7,21
Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, and Weipeng Xu. Universal humanoid motion representations for physics-based control. ArXiv, abs/2310.04582, 2023a. URL https:/ //api. semant commantcscchoarr org/ Corppss 263829555.
Zhengyi Luo, Jinkun Cao, Alexander W. Winkler, Kris Kitani, and Weipeng Xu. Perpetual humanoid control for real-time simulated avatars. In International Conference Computer Vision (ICCV), 2023b. 5
Zhengyi Luo, Jinkun Cao, Sammy Joe Christen, Alexander Winkler, Kris Kitani, and Weipeng Xu. Grasping diverse objects with simulated humanoids. ArXiv, 1 2024. URL. https: //ap: semant cascho wogp/ooprssoooooooooooooopo TD: 2, 3,
Denys Makoviichuk and Viktor Makoviychuk. rl-games: A high-performance framework for rein- forcement learning. https://gitt wwwpwwwww........eeeoaaeeeeee Denys r1 aarl_lggaaa__ May 2021. 8
com
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance basebased physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021. 3, 7,
Igor Mordatch, Zoran Popović, and Emanuel Todorov. Contact-invariant optimization for hand manipulation. In Proceedings of the ACM SIGGRAPPI/Eurogaahiis symposium computer animation, pp. 137-144, 2012. 2.3
OpenAl. Gpt-4 technical report. 2023. URL
CorpusID :ee.III:::77211111
Tao Pang and Russ Tedrake. A convex quasistatic time-stepping scheme for rigid multibody systems with contact and friction. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6614-6620. IEEE, 2021. 2,
12

---

Published as a conference paper at ICLR 2025
Tao Pang, HJ Terry Suh, Lujie Yang, and Russ Tedrake. Global planning for contact-rich manipula- tion via local smoothing of quasi- dyssmii contact models. IEEE Transactions on Robotics, 2023. 2, 3.8
Yuzhe Qin. Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, pp. 570-587. Springer, 2022. 2.
Wang. Dexmv:
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 1709, 10087, 2017. 2,3
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proxi-
mal policy optimization algorithms. ArXiv, abs/1707.06347 2017. URL https://api.
semant mommantisccolass wrt. :I: 28895055. 8
Kenneth Shaw, Ananye Agarwal, and Deepak Pathak. Leap hand: Low-cost, efficient, and an- 064440, 2023. URL https://apt
thropomorphic hand for robot learning. ArXiv, abs/2309 10442309.00440,, semanticschen org Corpus :::::/ooopuuus:::20220 259327055.. 7,8
Himanshu Gaurav Singh, Antonio Loquercio, Carmelo Sferrazza, Jane Wu, Haozhi Qi, Pieter Abbeel, and Jitendra Malik. Hand-object interaction pretraining from videos. 2024. URL
https: / //tpp/apppp.pppe/ wwwant www.anntcscchlarr. wttg/oorpuueeeoooooo ID:
Wen Sun, J. Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining 2018. URL https://
reinforcement learning & imitation learning. ArXiv, api www.ssseaannccccccceooemmmo
hegg/CorpusiD. 3333p3333 3533333.
Omid Taheri, Nima Ghorbani, Michael Black, and Dimitrios Tzionas. Grab: A dataset of whole- body human grasping of objects. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part IV 16, pp. 581-600. Springer, 2020. 7, 21
Yinhuai Wang, Jing Lin, Ailing Zeng. Zhengyi Luo, Jian Zhang, and Lei Zhang. Physhoi: Physics- based imitation of dynamic human-object interaction. arXiv preprint arXiv:2312.04393, 2023. 2,
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi. F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language mod- els. ArXiv, abs/2201.11903. 2022. URL https:/ //apii semant
CorpusID: 246 621.
Bowen Wen, Wei Yang. Jan Kautz, and Stanley T. Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. 2024 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 17868-17879, 2023. URL. https://api.
Cemanticschol. org/ www.aaaurrrooraaaaaaaaaaaa 2661 91 252. 8,22 22
Yueh-Hua Wu, Jiashun Wang, and Xiaolong Wang. Learning generalizable dexterous manipulation from human grasp affordance. In Conference on Robot Learning, pp. 1188-29, PMLR, 2023. 2,
Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. arXiv preprint arXiv:2303.00938, 2023. 2
Hui Zhang. Sammy Christen, Zicong Fan, Luocheng Zheng, Jemin Hwangbo, Jie Song, and Otmar Hilliges. Artigrasp: Physically plausible synthesis of bi-manual dexterous grasping and articula- tion. arXiv preprint arXiv:2309.33891, 2023. 5,
Sheng Zhong, Thomas Power, Ashwin Gupta, and Peter Mitrano. PyTorch Kinematics, February
2024. 7, 15
13

---

Published as a conference paper at ICLR 2025
Overview. The Appendix provides list of materials to support the main paper.
Additional Technical Explanations (Sec. A). We give additional explanations to comple- ment the main paper.
Detailed Method Overview Figure. We include an overview figure for the method (Figure 6) that illustrates the method more detailed than the figure in the method sec- tion.
Data Preprocessing (Sec. A.I). We present details of the kinematics retargeting strat- egy we leverage to create a dexterous kinematic robot hand manipulation dataset from human references.
Tracking Controller Training (Sec. A.2). We explain additional details the RL- based training scheme design, including the observation space and the reward. We also explain the control strategy for floating base dexterous hand.
Homotopy Generator Learning (Sec. A.3). We explain details the homotopy gener- ator learning-
Additional Details (Sec. A.4). We present additional details w.r.t. the techniques. Additional Experimental Results (Sec. B). We include more experimental results in this section to support the effectiveness of the method, including
Dexterous Manipulation Tracking Control (Sec. B. We present additional experi- ments of our methods well as additional comparisons, including results achieved by using different training settings and additional qualitative results. We will also discuss more generalization ability evaluation experiments.
Real-World Evaluations (Sec. B.2). We include more results of the real-world evalua- tions. We will discuss failure cases the real-world evaluation as well.
Analysis on the Homotopy Optimization Scheme (Sec. B.3). We present qualitative results achieved by the proposed homotopy optimization method and comparisons to demonstrate the capability of the homotopy optimization as well as the effectiveness of the homotopy optimization path generator. Additionally, we discuss the generalization ability of the homotopy path generator.
Failure Cases (Sec. B.4). We discuss the failure cases for comprehensive evaluation and understanding w.r.t. the ability of our method.
Experimental Details (Sec. C). We illustrate details of datasets, models, training and eval- uation settings, simulation settings, real-world evaluation settings, and the running time as well as the complexity analysis. Additionally, we attempt to ground some key properties of the tracking controller computationally and present quantified evaluations regarding these properties.
We include video and website introduce our work. The website and the video contain animated results. We highly recommend exploring these resources for an intuitive understanding of the challenges, the effectiveness of our model, and its superiority over prior approaches.
A ADDITIONAL TECHNICAL EXPLANATIONS
Overview figure of the method with details. In figure 6, to present comprehensive overview of our method, we draw a detailed overview figure that includes more important details than the figure of the original method in Section 3.
A.I DATA PREPROCESSING
Kinematic retargeting. We curate kinematic robot-object interaction data from human references by retargeting robot hand manipulation sequences from human hand trajectories. For instance, given human hand-object interaction trajectory describing the human hand pose sequences represented in MANO and the object pose sequences (Huhmmn)) ), O as well as the description of the articulated robot hand, retarget Hhmman to acquire the robot hand trajectory H. We manually define corre- spondences between the robot hand mesh and the MNAO hand mesh. After that, the sequence the robot hand DoF positions is optimized so that the resulting robot hand mesh sequence is close the
14

---

Published as a conference paper at ICLR 2025
Learning Neural Tracking Controller from Demonstrations
Mining Hligh-Qulltn Robot Tracking Demonstrations
Tracing Contrellerns
Actin
Envirosment
Expan Acdioa
Trajetity
Object Saaiiaa
Cial At
Thicking
Kinsratk Ackoowa
Expat Tiacki
Imitaton Late
Reiafoceemen Leasing Objectives
Tracking Controller
Teajockory Tracking Policy
Haap-tthhe ebl
Chic
Seead
Tracking Carnolle
Lose
Ruba Tracking Demonerationa
Homotopy Oytımization
Puth Generatir
Kinteiii Reienzoo
Zxpot Trudkitd Rorda
Rowi
Rona
Bazatogy Optimiationn
Figure 6: DexTrack leams generalizable neural tracking controller for dexterous manipulation from human references. It alternates between training the tracking controller using abundant and high-quality robot tracking demonstrations, improving the data via the tracking controller through homotopy optimization scheme.
Table 3:
Weights of different reward components.
Ma.p
Mlog
Mwiisr
Mhrans
Warist Wontt
Mhnger
Weight 1.0
0.33
0.3
0.05
0.05
human hand sequence. Formally, let Khhuman and denote the human hand keypoint sequence and the robot hand keypoint sequence respectively, the optimization objective is:
minimize| K
K
Kchuman ||.
(5)
We use P Kinematics (Zhong et al., 2024) to calculate the forward kinematics. Specifically, given the robot hand per-joint DoF position 6, at the timestep n. we calculate h,, and k,, as follows:
h Forward (innamttiiin
kp KeyPoints(Forrard_KK Kinematics(O,.),
where Forward Kinematics(-) computes the forward kinematics using the function provided in Py- Jorch_Kinematice (eyPoints()) reads out keypoints from the converted articulated mesh. We use a second-order optimizer, ie.. L--BGG implemented in PyTorch, solve the optimization problem 5.
A.2 TRACKING CONTROLLER TRAINING
Controlling floating-base articulated hand. The articulated hand is represented in the reduced coordinate ffnnger, We additionally add three translation joints and three revolute joints to control the global position and orientation of the hand, resulting in 0 (0)s grot (rot). For the Allegro hand and the LEAP hand that we use in our experiments, (linger is a 16-dimensional vector. Therefore, a 22-dimensional vector.
Observations. The observation at each timestep n encodes the current hand and object state, the next goal state, baseline trajectory, actions, and the object geometry:
= {sn, Sn, $.+1. Sh an featabj, aux.).
(8)
where aux, is the auxiliary features, computed as follows:
aux,, (S fn. S e Sn,
where $n+1 O s, calculates the difference between two states, including the hand state difference and the object state difference, f, indicates the hand finger positions in the world space.
Reward. Our reward for manipulating tracking encourages the transited hand state and the object state to be close to their corresponding reference states and the hand-object affinity:
Top Wo.gTog + Wwrist wrist
Wingerrlingg + Waliinayaalliiity allinity
(10)
15

---

Published as a conference paper at ICLR 2025
Table Quantitative evaluations and comparisons. Bold red numbers for best values. Models are trained on training tracking tasks from both the GRAB and the TACO datasets.
Dataset
Method
Raw (rad, !)
Tar (cm, J)
Eviu (L)
Einger (rad, J)
Success Rate (%, 1)
PPO (w/o sup., tracking rew.)
0.5813
6.03
0.1730
0.5439
36.04/55.84
GRAB
Ours
0.4515
4.82
0.14574
0.4574
42.64/61.42
PPO (wlo sup., tracking rew.)
0.6751
6.37
0.1264
0.5443
21.67/50.25
TACO
Ours
0.4782
3.94
0.1329
0.4228
32.02/62.07
where ro.p: To.g. l'wiis, I llinger are rewards for tracking object position, object orientation, hand wrist,
hand fingers, Wo,p; Wo.g; Wwrist: Wiineer; Waltinity are their weights. Topps To.gs Twrists Fiigger are computed as follows:
Top 0.9 | pnll2,
(11)
To.q np.pi Diff Diif_Anglee(ff. - -g
(12)
Twria
- (Wiann | 3] :: 3]]1 + Wore || [ :6]
- [3 6|1l
(13)
T'inger Winger
6
6 |6 =]I|
(14)
where Pg and q. denote the position and the orientation, represented in quaternion, of the current object, s, denotes the current hand state. In addition to these rewards, we would add an additional bonus reward 1 if the object is accurately tracked, i.e., with the rotation enror kept in 5-degree and the translation error kept 5-cm. Table summarizes weights different reward components we in our experiments.
Pre-processing object features. We train PonintNet-based auto-encoder on all objects from the two datasets considered, namely GRAB and TACO. After that, use the latent embedding of each object as its latent feature feeding into the observation the tracking controller. The object feature dimension is 256 in experiments.
A.3 HOMOTOPY GENERATOR LEARNING
Mining effective homotopy optimization paths. The maximum number of iterations K is set to 3 our method balance between the time cost and the effectiveness.
We need to identify neighbors for each tracking task so that we can avoid iterating over all tasks and reduce the time cost. We use the -oosmmiinemtiic trajectory similarity to filter neighboring tasks. We pre-select Kaii 10 neighbouring tasks for each tracking task.
A.4 ADDITIONAL EXPLANATIONS
In the reward design, we do not include the velocity-related terms since it is impossible for us to get accurate velocities from kinematic references. One can imagine calculating the finite differences between adjacent two frames as the velocities. However, it may not be accurate. Therefore, we do use them to avoid unnecessary noise.
B ADDITIONAL EXPERIMENTS
B.1 DEXTEROUS MANIPULATION TRACKING CONTROL.
Training the tracking controller on two datasets. In the main experiments, the training data and the test data come from the same dataset. We adopt such a setting considering the large cross- dataset trajectory differences. Specifically, GRAB mainly contains manipulation trajectories with daily objects, while TACO mainly covers functional tool-using trajectories. However, jointly us- ing the trajectories from such two datasets to train the model can potentially offer a stronger controller considering the increased labeled data coverage. Therefore, we additionally conduct this experiment where train single model using trajectories provided by the two datasets and test the performance on their test sets respectively. Results are summarized Table 4, which can still demonstrate the effectiveness of our approach.
16

---

Published as a conference paper at ICLR 2025
Table 5:
Generalizability evaluations on the TACO dataset.
Test set
Rex (rad. ))
Tan (en,
Euna (L)
Ennger (rud, L)
Success (%,,
$1
0.5787
2.43
0.1481
0.4703
35.97/67.66 .63
S2
0.6026
2.46
0.1455
0.4709
30.8366.00 00
S3
0.6508
8.06
0.1513
0.4683
10.18/46.32
Table 6:
Performance comparisons across models trained with different amount of demonstration
data.
Proportion
Re: (rad, ()
Tex (cm,
Eune (1)
Eingr (rud. L)
Success Rate (%, 1)
0.0
0,4985
4.42
0.1435
0.4767
31.03/57.6
0.4730
3. 3.86
0.1502
0.4921
36.45/59,6 61
0.3
0.49003
2.94
0.1256
0.4804
40.89/62.00
0.4682
41.38/67
0.9
0.4776
2.29
0.1437
0.4839
44.83/72.91
1.0
0.4953
2.10
0.1510
0.4661
48.77/74.38
Comparisons with labeling the whole dataset for training the controller. In our method we only try to label a small fraction of data in progressive way leveraging the power of the tracking prior provided by the tracking controller and the homotopy paths proposed by the homotopy generator so that we can get high-quality demonstrations within an affordable time budget. One may wonder whether it is possible to label all the training dataset trajectories and train the tracking controller using those demonstrations. And if possible, what's 's the performance of the model trained via this approach? We therefore manage to label each trajectory in the training dataset of the GRAB dataset by running the per-trajectory tracking experiments in parallel in 16 GPUs using two machines. We complete the optimization within one week. After that, we use the resulting demonstrations to train the trajectory controller. The final model trained in this way achieves 42%13% and 60.41% % success rates under two thresholds. The performance can still not reach our original method where we only try to optimize high-quality demonstrations for part of the data (see Table for details). We suppose it is the quality difference between the two labeled datasets that causes the discrepancy. This further validates our assumption that both the quantity and the quality of the labeled dataset matter train a good tracking controller.
Kinematz Reforesce
Oan
Figure 7: Robustness towards out-of-distribution objects and manipulations. Please refer to our website and the accompanying video for animated results.
Further generalization ability evaluations TACO dataset. We further evaluate the model's generalization ability across various test sets within the TACO dataset. As shown in Table 5, the controller performs well the category-level generalization setting (SI), where object categories are known but manipulation trajectories and object geometries are novel. Performance on S2, in- volving novel interaction triplets, is satisfactory, demonstrating the controller's capacity handle new manipulation sequences. However, results from S3 reveal challenges when dealing with new object categories and unseen interaction triplets. For instance, generalizing from interactions with shovels and spoons to using bowls for holding objects is particularly difficult. As shown in Figure 7, despite unfamiliar objects and interactions, we successfully lift the knife and mimic the motion, though execution is imperfect, highlighting areas for improvement and adaptability in challenging scenarios.
Additional results. We present additional qualitatively results in Figure further demonstrate the capability of our method.
Scaling the number of demonstration data. In Table 6, present the full evaluation results all five types metrics of each model traiend in the ablation study regarding the influence of the amount of demonstration data model's performance (see section for details).
17

---

Published as a conference paper at ICLR 2025
Teekkw
Resegesal Kinematee
Trjenoyy
PN) moding sow)
Ous
Waviaa cricstziion
Retrptted
Kinematic
Ina
P8O (tacking
Ous
ifting hand moed (Chollagge comple movemone))
Figure 8: Additional qualitative comparisons. Please refer to our website and the accompany- ing video for animated results.
Table 7:
Real- world quantitative comparisons (GRAB dataset). Bold red numbers for hest values.
Mushod
apple
Danana
dack
ateplant
Stanhtighs
Fluce
haraac
hacd
phace
Hacebbotile
PeO Inio buabing
GND
25
25.0000.0 SRASSL850
S0LO25A0 15
25 A5.0225
255.20 66.766.76667
35,300 58.51130
Ous
75 A50OOOS
B.2
REAL- WORLD EVALUATIONS
Tumkay
Figure 9: Additional real-world qualitative results. Please refer to our website and the accom- panying video for animated results.
Success thresholds. We define three levels of success rates. The first level of success is defined as reaching the object, finding a good grasp pose, and exhibiting the potential movements to lift the object up, i.e., one side of the object is successfully lifted up from the table. The second level of success is defined as finding a way to manage to lift the whole object up from the table. The third level of success is lifting the object up, followed by keeping tracking the object's trajectory for more than 100 timesteps.
More results. For direct tracking results transferring setting, we present the quantitative success rates evaluated
method and the best-performed baseline in Table 7 (for the dataset GRAB) and Table 8 (for the dataset TACO). As observed in the table, the tracking results achieved by our method can be well transferred to the real- world robot, helping us achieve obviously better results than the baseline methods. validates the seal-world applicability of our tracking results.
Please refer to the main text (Sec. for the quantitative comparisons between the transferred trollers. We include more qualitative results in Figure 9 to demonstrate the real-world application value of our method.
18

---

Published as a conference paper at ICLR 2025
Table 8: Real-world quantitative comparisons (TACO dataset). Bold red numbers for best values.
Methed
soap
shovel
bzush
so_les
kniča
spcor:
PRO two sup., traking cow)
33,3AD0
25 DADA
25.0/W0 20
BADA 25
Ours
TORAN66.7/6A.
25.0025.000.0
50A/2500/55.0
30.050.0/250
Figure 10: Failure cases in real-world experiments. Please refer to our website for animated results.
Failure cases. A typical fail mode is that the object tends to drop from the hand as contact varies when attempting the in-hand manipulations, as shown in Figure 10.
B.3 ANALYSIS ON THE HOMOTOPY OPTIMIZATION SCHEME
Resageeed
w! Tooking
Canicalan
Lifiin en fur.
Retageted
Kannatic
Irppstory
Itadkou Cunicolio
Trackiu
(bi samal
Rstageted
Tnjattey
w/l Trickleg
Trecking
Dariation
Guapiis
Figure 11: Effectiveness of the homotopy optimization scheme. Please refer to our website and the accompanying video for animated results.
We conduct further analysis of the proposed homotopy optimization scheme and the homotopy path generator to demonstrate their effectiveness. As shown in Figure 11, by optimizing through the homotopy optimization path,
get better results in per-trajectory tracking.
Lifting thin objects. As demonstrated in Figure 1a, for the originally unsolvable tracking problem where we should manage lift very thin flute up from the table, finally the tracking
19

---

Published as a conference paper at ICLR 2025
Table 9: Generalization experiments the homotopy path generator.
Homotopy test (aj
Homotopy test (b)
Hoenotopy test (c)
Homotopy test (d)
Effectivenees Ratio
64.0
56.0
28.0
52.0
difficulty by gradually solving each tracking problem in the homotopy optimization path proposed by the generator.
Grasping small objects. As shown in Figure 11b, the original per-trajectory tracker fails to find a proper way to grasp the small sphere and lift it up from the table. However, empowered by the homotopy optimization, we can finally find way to lift it up from the table.
Lifting a round apple. Figure 1lc demonstrates an effective homotopy optimization path that can let us lift apple up from the table which would previously challenge the policy due to the round surface.
Generalization experiments the homotopy path generator: To further understand the general- ization ability of the homotopy generator, we conduct the following tests:
(a) Train the path generator via homotopy paths mined from GRAB's training set and eval- uate it on the first test set that contains 50 tracking tasks uniformly randomly selected from remaining tracking tasks in GRAB's training set that are not observed by the homotopy generator.
(b) Evaluate the path generator trained in (a) on the second test set that contains 50 tracking tasks uniformly randomly selected from the test tracking tasks of GRAB's 's test set.
(c) Evaluate the path generator trained in (a) on 50 tracking tasks uniformly randomly selected from the test tracking tasks of TACO's first-level test set.
(d) Train the path generator via homotopy paths mined from both GRAB's and TACO's training set and evaluate it on the test set used in (c).
For each tracking task, if the tracking results obtained by optimizing through the optimization path are better than the original tracking results produced by RL-based per-trajectory tracking, we regard the generated homotopy optimization path as an effective one. Otherwise, we regard it as ineffective. We summarize the ratio of the effective homotopy optimization paths in Table 9.
In summary:
(a) the homotopy path generator can perform relatively well in the in-distribution test set- ting;
(b) the performance would decrease slightly as the manipulation patterns shift bit (please refer to section 4.1 for the difference between GRAB's 's training split and the test split);
(c) the path generator would struggle to generalize to relatively out-of-distribution tracking tasks involving brand new objects with quite novel manipulation patterns;
(d) Increasing the training data coverage for the homotopy path generator would let it get obviously better.
Teekeep
Trenliny
Figure 12: results.
Failure Cases.
Please refer to our website and the accompanying video for animated
B.4
FAILURE CASES
Our method may fail to perform well in some cases where the object from a brand new category with challenging thin geometry, demonstrated in Figure 12.
20

---

Published as a conference paper at ICLR 2025
C
ADDITIONAL EXPERIMENTAL DETAILS
Figure 13: Examples of novel objects from the seen object category (TACO).
Figure 14: Examples of objects from new object categories (TACO).
Datasets. Our dexterous robot hand-object manipulation dataset is created by retargeting two public human-object datasets, namely GRAB Taheri et al. (2020), containing single-hand interactions with daily objects, and TACO Liu et al. (2024b), featured by functional tool using interactions. We retarget the full GRAB dataset and the fully released TACO dataset, obtaining 1269 and 2316 robot hand manipulation sequences respectively. For GRAB, we use sequences of the sub ject s1, with 197 sequences in total, as the test dataset. The training dataset is constructed by remaining sequences from other subjects. For the TACO dataset, we create one training set with four distinct test sets with different generalization levels for detailed evaluation of the model's generalization performance. Specifically, the whole dataset is split into 1) a training dataset, containing 1565 trajectories, 2) test set SO where both the tool object geometries and the interaction triplcts are seen during training containing 207 trajectories in total, 3) test set S1 where the tool geometry is novel but the interaction triplets are seen during training with 139 trajectories, 3) test set S2 with novel interaction triplets but seen object categories and geometries, containing 120 trajectories in total, and 4) test set S3 with 285 trajectories where both the object category and interaction triplets are new the training dataset. Figure 13 and Figure 14 draw the examples of unseen objects from seen categories and the objects from new categories respectively. The original data presented in TACO often contains noisy initial frames where the hand penetrates through the table or the object. Such noise, though seems subtle, would affect the initial dynamics, however. For instance, if the hand initially penetrates through the table, a large force would be applied to the hand at the beginning, which would severely affect the simulation in subsequent steps. Moreover, if the hand initially penetrates through the object, the object would be bounced away at the start of the simulation. To get rid of such phenomena, we make small modifications to the original sequences. Specifically, we interpolate the phone pass sequence of the subject s2 from the GRAB dataset with such TACO sequences the final modified sequence. Specifically, we take hand poses from the initial 60 frames of the GRAB sequence. We then linearly interpolate the hand pose in the 60th frame of the GRAB sequence with the hand pose in the 60th frame of the TACO sequence. For details, please refer to code in the supplementary material (refer "README.md" for instructions).
Table 10: (TACO dataset).
Total training time consumption
Training and evaluation settings. For both GRAB and TACO, in the first stage, we first sample 100 trajectories from the training dataset. We train their per-trajectory trackers to obtain their action- data to construct our first version of the labeled dataset. After that, the first tracking controller is trained and evaluated on all trajectories. We then additionally sam- ple 100 trajectories from the remaining trajectories using the weights positively proportional to the tracking object position error. These sampled trajectories and those sampled in the first stage then form our second version of the dataset to label. We leverage both per- pejetrajettory tracker optimization and the tracking prior from the first version of the trained tracking controller to label the data aim- ing to get high- qualiiyality labeled trajectories. After that, we search tracking curricula from such 200 trajectories and train tracking curriculum scheduler using the mined curricula. We then construct
Ours
PPO (w/osup))
Ours
(wfo prior. w/o curri.)
Ours
(wlo prior)
Time
~1 day
22 days
4 days
4 days
21

---

Published as a conference paper at ICLR 2025
the second version of the action-labele, dataset using the final best-optimized trajectories. We then re-train the tracking controller and evaluate its performance on each trajectory. In the third stage, sample additionally 200 trajectories from the remaining not selected trajectories. We then label them using the joint power of per-trajectory tracking optimization, tracking prior from the tracking controller, and the curriculum scheduled by the curriculum scheduler. After that, third version is constructed using best-optimized labeled trajectories. We then re-train the tracking controller using this version of the labeled dataset. When training the tracking controller, we set threshold in the reward, i.e., 50. Only trajectories with reward above the threshold is used to provide supervision. Both the simulation and the policy run at 60Hz. The hand's gravity is ignored in the simulation. Please refer to the code in the supplementary materials for detailed settings. We train all models in a Ubuntu 20.04.6 LTS with eight NVIDIA A AO cards and CUDA version 12.5, All the models are trained in single card without multi-gpu parallelization.
Camara
Foom Rnd Forld per
Figure 15: Real-world experiment setup.
Real world experiment setup. We use the Franka and LEAP hand to conduct real-world evaluations (Figure 15). When transferring the state-based policy, we use FoundationPose Wen et al. (2023) to estimate object poses. We use finite difference estimate both the hand joint velocities as well as the object linear and angular velocities. Considering the gap between the control strategy we use in the simulator and the control of the Franka arm and the LEAP hand, we devise strategy to mitigate the discrepancy between these two control methods. Specifically, instead of directly applying the control signal to the LEAP hand and the Franka arm, we set up a simulator with physical and control-related parameters same as our simulation settings during training. Then, in each timestep. we first apply the control commands to the simulated LEAP hand. We then simulate the hand. After that, we read the simulated state out from the simulator. We then calculate the positional target signal that should be applied the real LEAP hand using the current obtained state from the simulator and the current state of the real LEAP hand. The calculated positional target signal is then directly fed to the real LEAP hand controller. In our observation, once the real hand has been commanded, it can almost reach exactly the same position in the command. Thereby, in practice, we directly use the state obtained from the simulator as the positional target command fed the real LEAP controller. Experiments demonstrate the effectiveness of this control strategy.
Time consumption and time complexity. Table 10 summarizes the total time consumption of different methods on the TACO dataset. Directly training PPO without any supervision is the most efficient approach while the performance lagged behind due to no proper guidance. Solving the per-trajectory tracking problem for providing high-quality data for training the general tracking controller would additionally increase the time consumption due to the requirement in optimizing per-trajectory trackers. Since we only select a subset from the whole training dataset, the time consumption is still affordable. Improving the per-trajectory trackers via mining tracing curricula would introduce additional time costs. Since the number of trajectories we consider for learning the curriculum scheduler is still controlled to relatively small value. The final time cost is still relatively affordable. Experiments are conducted on a Ubuntu 20.04 machine with eight A10 GPU cards. For per- trajtrajyytory tracker optimization, we train eight trackers in parallel at one time.
The overall time complexity of the training process is O + K Kuei SI). S denotes the training dataset.
"Robustness", "Generalization Ability", and "Adaptivity". We try to give computational defi- nitions of some crucial characteristies of the neural tracking controller. Please note that there is no standard formal computational definitions of such concepts, to our best knowledge. The definition and the quantification we present here is only from our perspective.
22

---

Published as a conference paper at ICLR 2025
Table 11:
Generlaization score (GRAB dataset), Bold red numbers for best values.
Method DGrasp
(OmniGrasp
PPO
(wlo sup. tracknn
PPO
(w/o data, w/o hometopy)
Ours
(wlo
Ours
Ours
2,44
2.389
2,688
2.725
3.050
3.251
Table 12: Robustness score (GRAB dataset). Bold red numbers for best values.
PPO
Method
(w/o sup., tracking rew.) Ours
8y
2.665
3.276
To quantify the "generalization ability", we need to first quantify 1) the distribution gap between two trajectory distributions, that can define the levels of generalization, 2) the generalization ability from the training distribution to the test distribution via the gap between the model performance:
Denote as the distribution of the tracking task of the training set, and D as the test distribution, we define their gap as follows:
d(D;E) =
min (Tracking Task Diff(TT, Tirain))
(15)
[
where Tracking. Task -iff(:-)) measures the differnece between two manipulation trajec- tory tracking problem. For a trajectory tracking task described by the kinematic hand state sequence (sh: [s n=0 the kinematic object pose sequence {pa q. JN Ja=o- and the ob- ject geometry PC, e.g,, represented the point cloud, we calculate the trajectory tracking task difference between two tracking tasks, i.e., T = ((sh.A), }, {p weighted sum of the hand trajectory difference,
A. 1...A. J. PCA) and
Tn ( (pa. ge.C), PC ), p object pose sequence difference and the object geometry difference:
N
Tracking Task Diff(T Tn)
(wi bifkhyllss s/
fls
+ w diff |pp.
N
m=0
(16)
W diff 1l
(17)
+
PC diff Chammer--isttaaae Distancce PC
(18)
y 1, w diff 0.3, w = 0.5. For tracking policy æ, define performance on tracking tasks from the distribution & via the expectation of tracking errors:
where w/ys = 0.1, w 2.
diff
Le(n) = E [Tracking Error, (T))
(19)
where Tracking Error, (.) evaluates the tracking error of the tracking policy л on trajec- tory tracking problem. It is weighted sum of the difference between the tracking result and the reference trajectory of the robot hand and that of the object:
Tracking-Erros (T)- ".P TerP 0.0 Rerr + werr
h.rrist Eurist + w finger Efingers (20)
wer
where h.wrist = 0.1, wh. finger 0.1, wo. 1.0, wo.9 0.3. err Therefore, the genralization ability of the policy x trained to solve the trajectories from trianing distribution E the distribution D measured as:
d(D;E)
(21)
min(Lo(T).c)
where is a small value avoid numerical issue. The score 8g increases as the train-test distribution gap increases or the tracking error on the test distribution decreases. Using the above quantification w.r.t. the tracking task distribution gap and the general- ization ability score, we summarize the generalization ability score of different kinds of models achieved on the GRAB dataset in Table 11.
23

---

Published as a conference paper at ICLR 2025
Table 13: Adaptativity score (GRAB dataset). Bold red numbers for best values.
PPO
Method
Ours
(w/o sup., tracking rew.)
8n
0.317/0.098/0.0
0.537/0.415/0.998
To quantify the "robustness", since is quite difficult to analyze the dynamic function with a neural controller, we measure the robustness by the performance discrepancy between the tracking policy's s performance on the tracking tasks with relatively high-quality kinematic trajectories to those with disturbed kinematic references.
To measure the "quality" of kinematic manipulation trajectories, introduce three trends quantities:
Smoothness: it calculates the difference between state finite differentiations:
N-1
t,
1
Σ
Ws
(8i+1 s;) W
(s:
8;
(22)
N
At
At
At
i=1
where w. is the weight vector of each state DoF, At is the time between two neigh- bouring frames.
Consistency: it calculates the hand object motion consisteney:
P
P
P
P
Le (T)
(23)
At
At
where P is the object position at frame i while p is the hand wrist position at the frame
Penetrations: it calculates the penetration between hand and object across frames: Depthhs,,
N
(T)
(24)
N
where po represents the object point cloud, Pene. calculates the maximum pen- etration depth between hand and the object.
"Quality" of kinematic manipulation references can be grounded using such three measure- ments. Combining the above quantities, define the overall "quality" of the test distribution L as:
(T) t. (T) + tp (T)
Squality (C) ET
(25)
The "robustness" can be measured by the performance gap of the model on tracking tasks with "high-quality" references and those with "perturbed" thus "low-quality" track- ing tasks. Denote the "high-qualtiy" tracking task distribution as H. while those with low quality as L. Thus we can quantify the "robustness" as:
Syuality (C)
sy (T)
(26)
min( Le(T),))
As the quality of the trajectory distribution gets worse and the tracking error decreases, the "robustness score" would increase.
To evaluate this, construct disturbed test set by adding random noise the hand trajectory and the object position trajectory to test the trajectories of the GRAB dataset. After that, test the performance of our method and the best-performed baseline, PPO (w/o sup., tracking rew.). We summarize the results in Table 12.
We mainly focus on the real-world adaptivity to evaluate the "adaptivity" the tracking controller. Since it is hard to quantitatively measure the discrepancy between the real-world dynamics and that in the simulator, we use the model's performance gap as the metric to directly evaluate the adaptivity. We use real-world per-trajectory average success rate to measure the "adaptivity". For our world test on trajectories from the GRAB dataset, summarize the result in Table 14.
24

---

Published as a conference paper at ICLR 2025
Table 14: Generlaization score (GRAB dataset), Bold red numbers for best values.
PPO
Method
Ours
(w/o sup., tracking rew.)
8a
0.317/0.098/0.0
0.537/0.415/0.293
Table 15: Trajectory difficulty statistics.
S Smoothh (m ==)
Vicntaat
Schape Schape !)
GRAB TACO
3,426 1.978
1.641
0.275
2.285
0.497
Computational interpretation of "hard-to-track". The most direct quantification is defining a score related to the tracking method's performance. For the tracking method M and the trajectory T, define π the best tracking policy that we can optimize for the trajectory T (i.e., ा = M(T).. Given the tracking error Tracking Error,(T) the "hard-to-track" score could be defined as: 8ht
Bin[Trccing.Eroor. (T).e) where is small value to avoid numerical issue.
Besides, use some statisties of the hand and object kinematic trajectories to quantify the "hard-to-track" characteristic. Here we introduce three types of statistics: 1) object movement smoohtness "mmooth" quantifies the motion smoothness by calculating the per-frame average ob- ject accelerations, i.e., 5% smooth N-T ΣmT ll | (P=
Di -i-Pi-1)]l )ll 2) hand-object con- tact shifting velocity Veontoct: it quantifies the per-frame velocity of the contact map change, i.e., Vcontact = +/4 Σ>> [-=---11] ||. where c, E {0, 1 is the binary contact map which encodes the contact flag between sampled points from the hand surface and the object, N, is the number of points sampled from the hand, 3) object shape score Sahope: it is the z-axis extent of the object's bounding box to quantify the shape of the object: Sahope min where extent E R is the extent of the bounding box of the object. We can jointly these three trends of scores quantify the "hard-to-track" characteristic. As s Somooth increases, Ucontact increases, and Sshope increases, the trajectory would get more "difficult", thus "hard" train policy track it. As shown in Ta- ble 15, the test trajectories in GRAB are more "difficult" than those in TACO w.r.t the trajectory smoothness, while test trajectories TACO are more difficult than those in GRAB regarding the contact velocity and the shape difficulty.
Additional details. For tracking error metrics, we report the medium value of per-trajectory result in the test set, under the consideration that the average value may be affected by outliers.
25


Reviews:
[{"review_id": "ChlfivvuAq", "reviewer": "Reviewer_4aCE", "summary": "This paper proposes an approach for training generalizable neural tracking controllers from human reference trajectories for dexterous manipulation tasks with robot hands. The approach consists of three steps: The first step involves using reinforcement learning to successfully track single, per-task trajectories for a subset of tasks. These policies are then distilled into a single policy using imitation learning. Next, a dataset of trajectories is sampled from the remaining tasks. The trained IL policy is now used to generate reference policies and residual RL is used to improve per-trajectory tracking. The best tracking results are used to re-train the IL policy. To enhance the diversity of trajectories tracked, a homotopy optimization scheme is introduced to facilitate “chain-of-thought” learning of the tracking controller over the set of related tasks. Once a sufficient set of high-quality tracking trajectories is generated, a conditional diffusion model is trained to serve as a homotopy generator. Finally, RL, the trained tracking and the trained homotopy generator are used to curate a final set of demonstrations and optimize the tracking controller.", "questions": "- How does a simple open loop baseline perform? I would be interested in the authors’ experience with sampling a reference trajectory from the dataset based on a simple heuristic to determine a nearest neighbor trajectory. \n- With human-object interaction, the trajectory of the object is generally the main variable of interest. Given the significant morphology gap between human hands and the robot hands used in this work, the error in position and orientation of the wrist as well as the robot hand fingertips can be a noisy signal resulting from artifacts in the trajectory retargeting scheme. I would like the authors to reflect on the relative weighting schemes attempted and any related conclusions that informed their design of the reward function as well as metrics.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Meticulous and concise description of the details of the proposed algorithm.\n- Tackles an important problem of learning a tracking controller for dexterous manipulation that is often overlooked, but could have significant impact as our data collection devices continue to improve with the rapid rise of AR/VR.\n- Thorough evaluations of the proposed method in simulation as well as real-world setup.\n- Paper provides useful insights into the importance of high-quality demonstration data for dexterous manipulation, and outlines a procedure for curating it from existing datasets.", "weaknesses": "- Approach relies on the presence of object states in the dataset which are difficult to obtain and are limited to very few datasets. It could be useful to investigate the effectiveness of this dataset without using object states, especially given the relatively small performance gap with the `(w/o data)` ablation. Based on the current results, it is unclear to me that the use of object state data, which is especially hard to come by in real-world datasets is significantly useful to the proposed approach.\n- In the same vein, it might be insightful to present results with the `Ours (w/o data)` ablation in the real-world experiments to better understand the effectiveness of data when crossing the significant domain gap between simulation and real-world.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K3R9kNlw3C", "reviewer": "Reviewer_CSzf", "summary": "The authors propose a neural IK controller for controlling dexterous robot hands. The results are validated in both simulated environments and the real world. The authors claim the proposed IK controller can achieve better tracking results than existing approaches.\n\n(12/11/2024 - The score was updated to a five after discussion with other reviewers)", "questions": "Please see my comments in the weakness section. Thanks.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- **Real-world experiments are valuable**. I appreciate the authors' efforts to conduct real-world validations of their approach.\n- **Baseline comparisons add value**. I appreciate the authors' efforts to compare against DGrasp and PPO baselines.", "weaknesses": "**A. Presentation**.\n\nThe paper's presentation is substantially below the ICLR standard in figure and writing. I had a very difficult time understanding the paper.\n\nThe teaser figure is confusing. First, it is too dense for the readers to quickly catch what the paper is doing. There are simply too many display items in the teaser figure, and the readers could be overwhelmed. More importantly, **I could not comprehend the input and output behavior of the work from the teaser figure**. I scrolled to the method section just to understand what the paper is tackling.\n\nNow, the second figure of the method is just as dense. I don't know what viewing order the authors want to present, but it left me confused about the most essential input and output behavior of the method.\n\nRegarding writing, the authors had trouble situating their contribution and the problem they are solving using the conventional language in robotics. **The authors could have communicated very clearly that they were building an inverse kinematics controller that takes Euclidean space coordinates as input and outputs joint space position-based commands, in just a few sentences in the abstract, introduction, and figures.**\n\nNext, there is a mixture of overuse and incorrect usage of terms from deep (reinforcement) learning, multi-body dynamics, control theory, and robotics in general. This leads me to question the credibility of this work. \n\nI already provided one example of the authors' clarity in communicating their input-output behavior.  Another example I will give is the use of \"robustness of tracking controller\" by the authors. It can be easily confused with robustness analyses from control theory in the context of trajectory tracking and IK, e.g., using toolsets from Lyapunov Analysis. This, in my view, irresponsible use of terms is also evidenced in line 176, when the authors casually throw out words like \"adaptability,\" \"generalizability,\" and \"robustness\" without grounding these properties in any computational ways, leaving the readers to interpret themselves. These terms should not be easily thrown into publications without computational interpretations. Another example - what are the \"hard-to-track\" trajectories mentioned in Figure 2 vs. the \"Parent\" trajectory? Can the authors computationally state what makes them hard to track?\n\nWhether the approach can handle second-order transients is also not communicated effectively. The proposed approach seems to eventually use a PID-like controller to command the included robotic systems. I could not assess this part due to a lack of descriptions. Can the system perform torque-level control, which is crucial for contact-rich manipulations?\n\n**B. Confusing related work section, Missing critical experiments**. \n\nI am currently unconvinced and cannot fully assess the performance of the work. The related work section mentions the sample inefficiency problem of RL. Yet, the evaluation protocol doesn't ablatively analyze how the proposed approach can generalize other than a very fuzzy success rate criterion in Figure 5. \n\nI am worried that this work will not inform the community why the proposed approach brings benefits. The proposed approach is a very complicated pipeline, piecing together existing toolsets. First, tracking is a highly continuous task and should not be solely evaluated in a single scalar of success rate when the authors want to show that their approach is more \"sample-efficient.\" Together with the presentation issues I have listed above, I could not understand the take-home message of the work to the robotics community. What limitations of existing controllers (e.g., OmniGrasp & DGrasp) are addressed by this pipeline?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9jyJgZ82OX", "reviewer": "Reviewer_gJWM", "summary": "This paper presents DexTrack, a generalizable neural tracking controller for dexterous manipulation, trained on large-scale robot tracking demonstrations based on human-object interactions. The method integrates reinforcement learning, imitation learning, and homotopy optimization to improve controller performance, diversity, and adaptivity in dynamic environments. DexTrack achieves over 10% higher success rates than leading baselines in both simulation and real-world tests, demonstrating robust manipulation capabilities across various objects and tasks.", "questions": "Are there any limitations to the proposed homotopy generator beyond its speed? While speed is certainly a factor, there may also be other limitations, such as the generator’s ability to generalize across a diverse range of objects and trajectories. It would be valuable to understand if the homotopy generator maintains high performance across these variations or if specific configurations present challenges.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality: The method is original, and combining RL and IL makes perfect sense for training a controller with high-quality demonstrations. I especially like the use of homotopy optimization as a path generator to enhance data quality.\n\nQuality: The experimental results are thorough and solid, both qualitatively and quantitatively. A substantial number of simulation experiments are conducted, and I appreciate the inclusion of real-world results as well.\n\nClarity: The paper is clear, well-written, and easy to understand.\n\nSignificance: This work represents a solid advancement toward solving the challenging task of dexterous hand manipulation, which is a significant objective in robotics.", "weaknesses": "The real-world robot results are not very impressive, as the grasp does not appear to be consistently robust, which may limit the method’s applicability in practical settings. It’s understandable, however, that real-world evaluations can be influenced by various factors, such as hardware limitations, sensor noise, and environmental variability, all of which can impact performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ooaFWJv19", "reviewer": "Reviewer_4aMi", "summary": "The authors proposed a neural tracking controller that integrates the reinforcement learning and imitation learning by iteratively improving the controller and generating demonstrations in simulation. Specifically, they proposed a per-trajectory tracking scheme to generate diverse and high-quality tracking demonstrations through a homotopy optimization framework. The authors further show qualitative and quantitive results to evaluate their proposed method.", "questions": "1) In section 3.1 Imitation Learning: Does expert state-action trajectory come from human demonstration? If so, how to address the kinematic and dynamic gaps here?\n\n2) In section 3.2 Finding effective homotopy optimization paths: Could the authors clarify “better baseline trajectory” in the sentence “For a specific task, we consider a neighbor that provides … parent task.\"?\n\n3) The real-world deployment seems impressive. I wonder beyond grasping, have the authors tried in-hand manipulation on real-world robot or were there any failure cases?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This work has solid contributions including novel approach and substantial experimental results. The work is well-motivated to present a generalizable tracking controller (policy learning) especially that can tackle dexterous manipulation. The framework the authors proposed was well-formulated. The experiments are thorough and results are well-presented and analyzed.", "weaknesses": "I found the method section was hard to follow and understand, and I would suggest to improve the writing.\n1) It would be great to clearly define each terminology at the beginning such as for kinematic reference trajectory, expert trajectory, baseline trajectory, kinematic reference sequence, tracking prior and etc and use them consistently through the paper.\n2) I found the pipeline in section 3.3 helpful for me to understand the whole framework, so I would suggest to move it to the beginning of the section 3. \n3) Figure 2 has a lot information, but it was hard for me to find a proper order to read it. Maybe it can be improved with a more clear flowchart. It would be great to use consistent icons (multiple different ones for object) or remove redundant icons (such as what is the magnifying glass icon?)\n\nTypo in section 4.1 Metric: racking -> tracking", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)