Cross-Embodiment Dexterous Grasping with Reinforcement Learning
Authors: Haoqi Yuan
Bohan Zhou
Yuhui Fu
Zongqing Lu
Venue: ICLR 2025 Poster (2025)
Keywords: dexterous grasping
cross-embodiment learning
reinforcement learning
Primary area: applications to robotics, autonomy, planning

Abstract: Dexterous hands exhibit significant potential for complex real-world grasping tasks. While recent studies have primarily focused on learning policies for specific robotic hands, the development of a universal policy that controls diverse dexterous hands remains largely unexplored.
In this work, we study the learning of cross-embodiment dexterous grasping policies using reinforcement learning (RL). Inspired by the capability of human hands to control various dexterous hands through teleoperation, we propose a universal action space based on the human hand's eigengrasps. The policy outputs eigengrasp actions that are then converted into specific joint actions for each robot hand through a retargeting mapping. We simplify the robot hand's proprioception to include only the positions of fingertips and the palm, offering a unified observation space across different robot hands. Our approach demonstrates an 80\% success rate in grasping objects from the YCB dataset across four distinct embodiments using a single vision-based policy. Additionally, our policy exhibits zero-shot generalization to two previously unseen embodiments and significant improvement in efficient finetuning. For further details and videos, visit our project page (https://sites.google.com/view/crossdex).

Published as a conference paper at ICLR 2025
CROSS-EMBODIMENT DEXTEROUS GRASPING WITH REINFORCEMENT LEARNING
Haoqi Yuan', Bohan Zhou', Yuhui Fu', Zongqing Lu
School of Computer Science, Peking University BBeijing Academy of Artificial Intelligence
ABSTRACT
Dexterous hands exhibit significant potential for complex real-world grasping tasks. While recent studies have primarily focused on learning policies for spe- cific robotic hands, the development of a universal policy that controls diverse dexterous hands remains largely unexplored. In this work, we study the learn- ing of tross-embodiment dexterous grasping policies using reinforcement learn- ing (RL). Inspired by the capability of human hands to control various dexterous hands through teleoperation, we propose universal action space based on the human hand's eigengrasps. The policy outputs eigengrasp actions that are then converted into specific joint actions for each robot hand through refargeting map- ping. We simplify the robot hand's 's proprioception to include only the positions of fingertips and the palm, offering a unified observation space across different robot hands. Our approach demonstrates an 80% success rate in grasping objects from the YCB dataset across four distinct embodiments using a single vision- based pol- icy. Additionally, our policy exhibits zero-shot generalization to two previously unseen embodiments and significant improvement in efficient finetuning. For fur- ther details and videos, visit our project page.
INTRODUCTION
Robotic dexterous grasping (Bicchi, 2000; Duan et al., 2021) has been studied for decades, estab- lishing a foundation for embodied agents to interact with the world through robotic hands. Recent advances have focused on grasping various objects using multi-fingered dexterous hands, employing data-driven methods (Qin et al., 2022b; Mandikal & Grauman, 2022; Liu et al., 2023) reinforce- ment learning (RI.) (Xu et al., 2023; Wan et al., 2023; Wu et al., 2024b), with real-world deployment of the learned policies (Qin et al., 2023a; Agarwal et al., 2023). However, existing approaches typi- cally learn policies tailored to specific dexterous hands, such as ShadowHand, Developing a policy for new embodiment often necessitates substantial data collection costly simulations and hyper- parameter tuning in RL.
In this paper, we aim to develop cross-embodiment dexterous grasping policy (CrossDex) that is applicable to various dexterous hands. Cross-embodiment learning leverages shared structural fea- tures among different robots acquire generalized skills, thus facilitating enhanced generalization to new robotic embodiments. Recent research has utilized these shared structures to learn cross- embodiment policies for different robot arms (Chen et al., 2024), robots with varied morphologies (Liu et al., 2022; Hejna III et al., 2021), and embodiments for navigation and manipulation (Yang et al., 2024; Doshi et al., 2024). However, applying cross-ents coos-emmooi learning to diverse dexterous hands remains largely unexplored and presents several challenges. Dexterous hands feature high gogrees-offfreedone (DoFs) and vary in the number of fingers and DoF's which complicates the unification of the action space for control across different robots. For instance, it is not straight- forward to map a control command for a 22-DoF. 5-fingered ShadowHand to a 16-DoF, 4-fingered LEAP Hand (Shaw et al., 2023). Moreover, even if actions could be aligned among different hands, their varied sizes and shapes make it challenging to implement a single policy in contact-rich grasp- ing tasks. For example, Patel & Song (2024) develops policy for LEAP Hands with varied link
Corresponneece Zongqing Lu <zongqing. lu pku.edu.cn

---

Published as a conference paper at ICLR 2025
ShadowHand
Allegro Hand
Schunk SVH Hand
Ability Hand
Leap Hand
Inspire Hand
Cup
Spotula
Power drill
Training hands
Unseen hands
We propose CrossDex, learning a cross-embodiment policy for dexterous grasping. The learned RL policy can grasp diverse objects with a variety of dexterous hands and transfer to hands not seen during training.
Figure 1:
morphology using graph neural networks (Scarselli et al., 2008), but this approach struggles to gen- eralize between L.APP Hand and Allegro Hand, which, despite their structural similarities, differ in the joint limits of DoFs and shapes of links.
We take inspiration from how humans teleoperate robot hands to tackle this challenge. Humans not only adept manipulating everyday objects with their own hands but also capable of transferring such manipulation skills to various robotic dexterous hands. In teleoperation systems (Li et al., 2019; Handa et al., 2020; Qin et al., 2023b), by observing the robot hand and simulta- neously adjusting their own hand poses, human operators can remotely control any robotic hand grasp objects without needing prior knowledge of the specific robot hand extensive practice. We adopt a human-like policy that can execute actions within the space of human hand poses to "teleoperate" different dexterous hands. In our approach, we use cigengrasps (Ciocarlie al., 2007) of the MANO (Romero et al., 2022) hand model as the unified action space, which efficiently com- presses 45 45-dimensional hand poses into oow-dimensional eigenvectors. These output hand poses are then converted into each dexterous hand's 's joint positions through retargeting algorithm (Handa al., 2020), providing commands for controller. We propose training neural networks to replace the optimization-basee hand's 's proprioception within the observation space, we eliminate the hand-specific joint positions and adopt the 3D positions of fingertips and palms. These positions crucial in grasping tasks (Handa et al., 2020) and are consistent across different embodiments.
retargeting process, significantly improving the training speed. To unify the
We train the policy to grasp objects from the YCB dataset (Calli et al., 2015) using reinforcement learning within the IsaacGym (Makoviychuk et al., 2021) simulation environment. Following the teacher-student framework (Jia et al., 2022; Wan et al., 2023), first train individual state-based policies for each object using PPO (Schulman et al., 2017), and then distill these into a vision-based policy using DAgger (Ross et al., 2011) grasp all objects. To train each policy, different dexterous hands in parallel environments, each attached to 6-DoF robot arm with its base fixed on table. Experimental results demonstrate that our CrossDex policy outperforms baseline methods on the four training hands and two hands not seen during training. CrossDex achieves high success rates in grasping YCB objects and exhibits zero-shot generalization to unseen embodiments with a single policy. Additionally, the policy shows significant increase in learning efficiency on unseen embodiments and objects via finetuning.
four
Our contributions are summarized as follows:
We propose cross embodiment training for robotic dexterous grasping, marking a step to- wards universal grasping policy.
2

---

Published as a conference paper at ICLR 2025
We adopt techniques from teleoperation to construct universal action space and observa- tion space for policy learning, enabling transfer across various dexterous hands.
Our experimental results demonstrate the superior training performance and generalization capabilities of CrossDex.
RELATED WORK
Dexterous grasping. Grasping is a fundamental skill that enables robotic manipulators to inter- act with the real-world environment. Multi- Multi-fingered dexterous hands (Pons et al., 1999; Kappassov al., 2015; Shaw et al., 2023), in contrast to traditional parallel grippers (Yi al., 2002; Hu al., 2019; Fang et al., 2020), offer advanced hardware capabilities. However, the complexity of using dexterous hands to grasp objects brings significant challenges. To address how to grasp, a substantial number of studies focus on generating grasping poses (Zhu et al., 2021; Shao et al., 2020; Li et al., 2023; Liu et al., 2023) and constructing grasping datasets (Wang et al., 2023; Chao et al., 2021). Real-world applications require not only static grasping poses but also closed-loop grasping policies that can execute complete trajectories. Recent studies utilize learning from demonstrations (Qin al., 2022b; Mandikal & Grauman, 2022; Qin et al., 2022a; Zhou al., 2024) and deep reinforce- ment learning (RL) (Xu et al., 2023; Wan et al., 2023; Wu et al.. 2024b; Qin et al., 2023a; Kannan al., 2023) develop dexterous grasping policies. RL offers distinct advantages due its inde- pendence from human supervision and its scalability to numerous objects (Xu et al., 2023; Huang al., 2024; Zhang al., 2025). In this paper, we study cross crosseembodiment dexterous grasping using RL. marking significant step towards a universal policy.
Foossmmmbodimnnt Learning lays the foundation for developing generalized agents by utilizing data across various embodiments. In the field of embodiment transfer (Niu et al., 2024), leveraging data from source embodiments to learn target embodiment policies facilitates data-efficient transfer learning. Techniques include modeling mappings between embodiments (Zhang al., 2021; Chen et al., 2024), learning invariant representations (Zakka et al., 2022; Salhotra et al., 2023; Wang et al., 2024), and employing hierarchical learning to share high-level policies (Hejna et al., 2020).
Another line of research focuses on multi-robot training to develop universal, generalizable policies. Studies by Hejna III et al. (2021); Liu al. (2022); Patel & Song (2024) explore generating varied morphologies and utilizing graph neural networks (Scarselli et al., 2008) to learn embodiment rep- resentations. Yu et al. (2023); Bousmalis al. (2023) use Transformers (Vaswani et al., 2017) to adapl policies to different embodiments in-context. Recent efforts (Doshi et al., 2024; Yang et al., 2024; Xu al., 2024) aim train a universal policy that accommodates embodiments with diverse functionalities, such as combining manipulation and navigation tasks.
However, existing research has not yet developed universal policy for various dexterous hands. GET-Zero (Patel & Song, 2024) learns a policy for in-hand reorientation tasks using different mor- phological variants of LEAP Hand (Shaw et al., 2023), but this method cannot generalize across different types of dexterous hands. Our research is the first to introduce universal grasping policy for diverse dexterous hands, achieving zero-shot generalization on hands not seen during training.
Hand Motion Retargeting is primarily studied within the context of robotic teleoperation (Antot- siou et al., 2018; Qin et al., 2023b; Fu et al., 2024), where detected human hand poses are trans- formed into target joint positions of robotic hands. Practical approaches include direct mapping (Fu et al., 2024), supervised learning (Li et al., 2019), and optimization for energy functions (Antotsiou et al., 2018; Handa et al., 2020; Sivakumar et al., 2022). DexPilot (Handa et al., 2020), notable optimization-bal based method for dexterous hand teleoperation, explicitly considers the relationships between fingertip positions. In research, integrate DexPilot with RL policy, enabling the control of various dexterous hands with a single policy.
3 PRELIMINARIES
3.1 REINFORCEMENT LEARNING FOR UNIVERSAL DEXTEROUS GRASPING
We consider a tabletop grasping scenario where a dexterous hand is mounted on a 6-DoF robot arm with its base fixed (see Figure 1). The goal is grasp and lift object initially placed on the table.
3

---

Published as a conference paper at ICLR 2025
We assume that tasks differ in the dexterous hand h E H and the object w E Ω2 where H. denotes the embodiment space and Q denotes the object set. Each task, characterized by a unique combination Partially Observable Markov Decision Process (POMDP) Mi, = (O, S, A. T. R, 4), which represents the observation space O, the state space S, the action space A, the transition dynamics ((8++1 |8,, at), the reward function R(s;, dt), and the observation emission function Ld ((() At each timestep f, the agent observes or E O and takes an action or A. then receives a reward r; R(s,, at). The environment then transitions to the next state S(+1 T(s++al (tt at). The objective of cross-embodiment dexterous grasping is to maximize the expected return across all tasks ΣKER,WED and y discount factor.
of embodiment and an object, is formulated
Σ 1= >'r)
T-1
where is the time limit
Existing literature (Qin et al., 2023a; Xu et al., 2023; Wu et al., 2024b) on dexterous grasping focuses on using& single dexterous hand (/H] 1) with m DoFs and n fingers to grasp any object. The observation E O consists of: (1) Robot proprioception, including the joint positions of the arm J E R the joint positions of the hand J E IR"", and the positions of the fingertips and palm ar (R,++1)x3; (2) Object perception, which includes the object pose b IR? consisting of the object position and quaternion in simulation and visual code v" representing the static shape of the object. In the real world, however, we access object information through visual observations. We choose to use the object point cloud E RNN3 which contains N points captured by cameras. The action E AA includes the target joint positions for the arm je and the target joint positions for the hand jhh which are passed to PD controller for joint torque control. The objective is to learn ? a vision-based policy T (arlJf, Jh at Pt. 0t-1) parameterized by á to maximize the expected return across all objects.
Training a vision- based policy for all objects directly using RL encounters optimization challenges inherent in multi-task RL (Yu et al., 2020), as well as the high- tote--mmmiiinalttty of point cloud obser- vations, which can slow down the learning process. Recent works (Jia et al., 2022; Xu et al., 2023; Wan et al., 2023) propose teacher-student framework with curriculum learning address these is- sues. We adopt a simplified approach from these studies. First, the object set is clustered into several groups, and state-based policy T.. (oelJ J/ ax be, vn ac-1) parameterized by 1/s trained using RL for each group i. Then, all state-based policies { } are distilled into a single vision- based policy n using DAgger (Ross et al., 2011), online imitation learning algorithm. In our research, we leverage this framework to address the challenge of learning a universal vision-based dexterous grasping policy for various embodiments and objects.
3.2 HAND POSE MODELING AND RETARGETING
The main challenge in coos-emmbodiment learning arises from the misalignment of hand joint posi- tions Jh that occurs in both observations and actions among different dexterous hands. This diver- gence results from differences in dimensions, joint limits, and the functionality of each joint across various embodiments. Then, can we find a shared representation of hand joint positions across these embodiments and establish mappings between them?
The literature on hand teleoperation (Handa et al., 2020; Qin et al., 2023b) suggests that human hand poses can effectively represent poses for dexterous hands. A human hand pose represented by 3). where 0 E 15 denotes the 16 axis angles of the hand joints and wrist, while 8 e R 10 denotes the hand shape. Using the MANO hand model (Romero al., 2022), we can derive the 3D positions of 21 keypoints on the hand from the hand pose: rM = MANO 8), x e 21 The human hand pose can then be mapped to the joint positions of any dexterous hand through retargeting process. We adopt optimization-base retargeting methods that optimize J.t at each timestep t:
min S (J))))), + || 12 J. Jinwer S J. Jh
(1)
s.t.
uppert
(2)
where f is the forward kinematics function of the dexterous hand, providing the positions of the fingertips and palm. J and Hh upper represent the joint limits of the dexterous hand, and S is a predefined function that measures the similarity between the robot hand pose and the human hand
4

---

Published as a conference paper at ICLR 2025
Arm joint positions
Retargeting Mapping
Fingertips palm positions
Eigengrasp
Target hand positions
Policy
Object pose
Lastaction
Target arm positions
Cross-embodimentimmuadde
Figure 2: CrossDex employs unified observation and action space facilitate the learning of universal policy across various dexterous hands. Rather than relying on joint angles specific to each hand, our policy utilizes the positions of the fingertips and palm discern the spatial relationship between the hand and the object. Actions are represented using eigengrasps from the MANO hand model, which are mapped process. This design, akin to teleoperation, enables consistent control across different dexterous hands. The policy is trained using reinforcement learning within a cross-embodiment simulation environment built on IsaacGym. To learn a vision-based policy, we substitute the object pose in this pipeline with the object's s point cloud.
position targets of each hand's PD controller through retargeting
pose. For example, DexPilot (Handa et al., 2020) evaluates the distances of the relative positions for each pair of fingertips of the robot hand and the human hand within S. This problem can be solved using quadratic programming algorithms. Thus, we can leverage the human hand pose (0, 3) as universal representation for dexterous hand poses.
METHOD
In this section, we present the proposed method, CrossDex, which utilizes shared representations of actions and observations to learn a cross-embodiment policy for dexterous grasping. Figure provides an overview of our approach.
4.1 EIGENGRASP ACTIONS
We propose using human hand poses as universal action interface for dexterous hands. The output human hand pose is converted target joint positions for each dexterous hand through retargeting. Since human hand pose can be translated into similar grasping poses across various dexterous hands, these actions have the potential to generalize across embodiments, sharing similar trajectories for grasping each object.
However, the dimensionality of human hand poses is significantly higher than the DoFs of most dexterous hands, which make RL less efficient. Additionally, because the joints of human hands do not move independently and the limits each joint are unknown, defining the policy's output space becomes challenging. We aim for the policy's output space be compact subspace of human hand poses that contains meaningful and natural poses. To achieve this, we adopt eigengrasps (Ciocarlie al., 2007) compress human hand poses.
Given dataset D = } capturing human hand poses in daily life, where each 0; represents the axis angles of hand pose (with 0 to disregard the variability shapes), we apply PCA to the dataset and retain the first-k eigenvectors, {e, je known as eigengrasps. Each eigengrasp represents distinctive hand pose, and we can generate diverse, novel hand poses by linearly combining these eigengrasps. In addition to the arm actions, our cross-embodiment policy outputs k-dimensional hand actions The hand pose 9 is then converted into keypoint positions using MANO and subsequently mapped to the target joint positions of each dexterous hand via DexPilot retargeting.
(W1, ............ wk). which represent weighted sum of eigengrasps: 0 == Σ w,e,-
5

---

Published as a conference paper at ICLR 2025
Agarwal al. (2023) also used eigengrasps to improve RL for dexterous manipulation. While their work applies eigengrasps in the LEAP Hand's action space, we leverage eigengrasps of human hand poses universal action interface, enabling cross-embodiment control through retargeting.
4.2 A UNIFIED OBSERVATION SPACE
The hand joint positions Jh within the observation space, which require embodiment-specee inter- pretation, are likely to hinder the policy when adapting to new embodiments with unknown propri- oception structures. Therefore, retaining only the fingertips and wrist positions x². These positions are also used in the retargeting process, shown in Equation providing shared hand state representations across different em- bodiments. Notably. fingertips and palm positions are crucial for grasping tasks, as they enable the agent effectively infer actions and value based on the relative positions of the object and the hand.
simplify the observation by discarding the hand joint positions,
In practice, we use the concatenated 3D positions of the palm and the first four fingers: thumb, index, middle, and ring. For five-fingered hands, we omit the little finger's position. This simplification is practical as the little finger is typically less critical in grasping tasks. For hands less than four fingers, the method can be generalized by setting the unused finger positions to zero in the observations. In principle, this approach extends to any hands with up to N fingers, maintaining compatibility by zero-padding unused positions.
At this point, we establish the structures for cross-embodiment policies. Our state-based policy takes the form T (Ji w. [f, xt, b,, wt, J..,,,,,..... w.-1). while the rision-based policy takes the form To (Jr, wi, Jf, æt pt, Ji-1, w)-1). where w, E 12 denotes the eigengrasp action.
4.3 MULTI-TAN LEARNING
Following the teacher-student framework, we divide the entire task space of embodiments and ob- jects, H S2, into groups. We learn state-based policies individually for each group. then distill these policies into vision-based policy. Our experimental observations, detailed in Section 5.2, sug- gest that our policy architecture benefits from co-training across embodiments, exhibiting slightly enhanced stability and comparable performance when trained simultaneously on all embodiments compared to individual training on each dexterous hand. Therefore, we choose divide only the object set for training the state-based policies.
We establish cross-embodiment simulation environments using IsaaeGym (Makoviychuk et al., 2021) to train policies. During the teacher learning phase, we train state-based policy T for each object w using PPO (Schulman et al., 2017) within parallel environments that iterate through all dexterous hands. In the distillation phase, set up parallel environments for every combination of objects and hands. Here, we iteratively use the vision-based policy To to collect trajectories.
Actions in these trajectories are then relabeled using the learned teacher policies (# Twen, and we optimize with behavior cloning.
While the parallel environments in IsaacGym run at frame rate of tens of thousands fps, the re- targeting process, which involves iterative optimization, runs at only 300 fps, significantly slowing down the training process. Fortunately, due the continuous nature of the retargeting mappings from the human hand to dexterous hands, we propose to use neural networks to accelerate the retar- geting process. For each dexterous hand h. we use MANO and the retargeting algorithm to label all samples 0, E DD from the dataset of human hand poses with the corresponding joint positions of the dexterous hand J Then, we train a neural network Ph parameterized by & to fit the mapping from 0, to J These trained retargeting networks {P jiSH enable batch computations within parallel environments, enhancing overall training efficiency.
Due to the limited number of dexterous hand embodiments available, we incorporate some ran- domization to enrich the embodiment space for training. aiming at enhancing the robustness and transferability of the policy. Specifically, add Gaussian noise to the origin position of the joint connecting the hand and the arm, which encourages the policy to adapt to variations in the hand attachment.

---

Published as a conference paper at ICLR 2025
Table Comparison of grasping success rates for YCB objects between CrossDex and baseline methods. The "Training hands' columns show average performance on the four hands provided during training, while the 'Unseen hands' columns show average zero-shot performance on two hands not seen during training. "State" denotes state-based RL policies, and 'Vision' denotes vision- based policies after multi-task distillation.
Training hands (State)
Training hands (Vision)
Unseen hands (State)
Unseen hands (Vision)
Method
MT--aw--AA
0.914
0.782
0.054
0.162
MT-Raw-A
0.823
0.728
0.272
0.210
MT-Raw-O
0.884
0.779
0.046
0.145
CrossDex
0.885
0.800
0.391
0.352
We summarize our training process in Appendix A. Further implementation details can be found in Appendix B.
EXPERIMENTS
5.1
TASKS SETUP
We evaluate CrossDex on 45 daily objects from the YCB dataset (Calli et al., 2015) and dexterous hands, with URDFs provided by Ding et al. (2024). We use four of these dexterous hands in training and the remaining two a 16-DoF, 4-fingered LEAP Hand and a 12-DoF, 5 5-fingered Inspire Hand are reserved for testing the model's generalization capabilities. In simulation, implement unified reward to train across all objects and dexterous hands:
y total = dis + p height + r + y success
where dis encourages to minimize the distance between the fingertips and the palm to the object, height encourages lift the object, yay discourages horizontal object displacement, and success indicates successful task completion. Success is defined as maintaining the object at specific height for 30 steps while keeping the hand close the object. The episode ends either when the task is completed or the object falls off the table. Detailed descriptions of the tasks and simulation settings are provided Appendix B.
To obtain eigengrasps and train the retargeting networks, we use the GRAB dataset (Taheri et al., 2020), which includes 1.6M frames depicting human hands interacting with various objects. For RL and DAgger, deploy 8192 parallel environments that encompass combinations of all the required objects and the four training hands. Additional training details can be found in Appendix B.
5.2 PROSS-EMBODIMENT LEARNING
To evaluate the performance of method, compare against multi-task RL methods for cross- embodiment learning. Highlighting our principal technical contribution of unifying observations and actions for dexterous hands, we introduce MT-Raw-OA. This baseline trains policies using the complete proprioception of the robot and the original hand actions of target joint positions. To align the dimensions of hand joint positions across different hands, we pad them with zeros. To diminish the ambiguity in such observations and actions, we include a one-hot label of the embodiment within observations. Similarly, MT-Raw-A uses original actions with our uni- fied observations and MT-Raw-O uses original observations but integrates our proposed eigengrasp actions. baselines are trained using the same teacher-student framework as our method.
fixed dimension
Table presents the success rates of these approaches on YCB objects. Our vision-based policy surpasses all baselines on both training hands and unseen hands, demonstrating its superior per- formance. According to Figure 5 in the Appendix, vision-based policy achieves success rates greater than 60% on 42 of the 45 objects across the four training hands. This performance highlights its capability to effectively control various embodiments with single policy. CrossDex particularly excels in zero-shot performance on new hands, showcasing unique transferability of the learned skills across different embodiments.

---

Published as a conference paper at ICLR 2025
Table 2: Finetuning performance of CrossDex compared baselines on the unseen LEAP Hand. We evaluate the finetuning of state-based policies and the multi-task iision-based policy for YCB objects. Additionally, we evaluate multi-task finetuning on unseen objects using 55 objects from the GRAB dataset. Training curves for these experiments can be found in Appendix C.2.
YCB
YCB
GRAB
Method
(5 objects, state)
(multi-task, vision)
(multi-task, vision)
No-Pretrain
0.758=+.122
0.436=10.159
0.313=0.373
MT-Raw-OA
0.701
0.417=000007
0.651
MT-Raw-A
0.798+00.002
0.390:10.005
0.655
MT-Raw-O
0.708 0.872=10.013
0.385=0.003
0.616=0.018
CrossDex
0.643=10.009
0.740=0.009
Table 3: Abaltion study examining the selec- tion of optimization objectives used in retarget- ing. "Position retargeting" focuses on optimiz- ing the absolute positions of the fingertips and palm. "Vector retargeting" focuses on optimiz- ing the relative positions from the palm each fingertip. "DexPilot" includes optimization of the relative positions between the fingertips.
Success Rate
0.6
0.5
0.4
0.3
0.2
CrossDex
Individual
2000
4000
6000
8000
10000
Retarget
Training hands
Unseen hands
Iteration
Figure Comparison RL training curves: eenedsmmbodiment learning average per- formance of individual training.
Position 0.884
0.500:10.037 0.435=1.007 0.482=#0.043
Vector 0.841=0.042
DexPilot
0.892=#0.005
MT-Raw-OL and MT-Raw-O demonstrate high performance in state- based training. However, their performance decreases a lot when distilled into vision-based policy. The primary reason that these approaches incorporate embodiment labels in observations during state-based training, which makes the policy to rely on this information. During distillation, to enhance generalization across embodiments, remove these robot labels in the lision-based policy, resulting in less informative observations. Consequently, on unseen hands, these two methods perform worse with state-based policies than with vision-based policies, the state-based policies utilize the specific unseen label of the new robot, whereas vision-based policies discard this information to enhance generalization.
We notice that even when using the action space IMT-Raw-OA and MT-Raw-A), the policy generalizes to unseen hands, albeit with a lower success rate. This generalization is facilitated by the careful alignment of the URDFs for all dexterous hands, ensuring that the arrangement of joint position elements (from the thumb to the little finger) and the corresponding movements (opening the hand with increasing values and closing with decreasing values) are consistent. This manual alignment underperforms our method that leverages human hand eigengrasps to effectively align embodiments.
All methods achieve success rates exceeding 80% during state-based training, demonstrating the effectiveness of jointly training various embodiments. Figure 3 illustrates the training curves of CrossDex alongside those from training each dexterous hand individually. We observe that cross- embodiment training slightly enhances training efficiency and stability compared to individual train- ing, while achieving comparable performance. This observation supports approach of co- training across embodiments rather than dividing the embodiment space for individual training.
5.3 EMBODIMENT TRANSFER
The zero-shot performance unseen hands, shown in Table 1, highlights the cross-em coos-emmbod tranferability of the learned policy. Such capabilities not only allow for the direct deployment of the policy new hands but also facilitate efficient finetuning by treating the policy pre-trained
8

---

Published as a conference paper at ICLR 2025
1.0
Success Rate
0.6
0.4
Training Hands
0.2
Unseen Hands
0.0
3-E
9-E
18-E
36-E
MANO
Figure 4: Abaltion study the eigengrasp action space. "MANO" refers using raw axis angles of the MANO hand model as actions. "k-E" refers to using the first-k cigengrasps as actions. Results show average success rates across five YCB objects.
Table 4: Ablation study on embodiment randomization and the inclusion of embodiment-speccfie observations. "Rand." refers to applying randomization to the hand mount position. "Obs." refers to including one-hot robot labels and the randomized positions in the observations.
Training hands (State)
Training hands (Vision)
Unseen hands (State)
Unseen hands (Vision)
Rand.
Obs.
0.834
0.738
0.044
0.146
X
0.836
0.735
0.259
0.209
V
/
0.840
0.726
0.040
0.187
/
X
0.885
0.800
0.391
0.352
model for universal grasping skills across all embodiments. We conduct RL finetuning experiments using a simple yet effective approach based on PPO. We initialize the finetuning policy æ with the pre-trained CrossDex policy no. and during each training iteration, update using weighted sum of the surrogate loss from PPO and the KL divergence to the pre-trained policy, DKL (- [o)), mitigate forgetting.
The results are presented in Table 2 and detailed training curves are provided in Appendix C.2. These results include finetuning state-based policies on the unseen hand, finetuning the multi-task vision-based policy on the unseen hand with all objects, and finetuning the vision-based policy to learn 55 unseen objects from the GRAB dataset (Taheri al., 2020). We observe that CrossDex offers a superior pre-trained policy compared to other methods, excelling in all finetuning tasks involving an unseen hand. Notably, while a policy learned from scratch (No-Pretrain) struggles in mulli-task learning, finetuning the CrossDex policy proves to be promising approach to learn tasks across all objects simultaneously. Furthermore, finetuning approaches demonstrate enhanced stability compared to learning from seratch, evidenced by the reduced variation and smoother training curves.
5.4 ABLATION STUDY
We conduct ablation studies to assess the impact of key components in CrossDex, particularly fo- cusing on the unified actions and observations.
Retargeting Methods: In our main results, we use DexPilot retargeting to map eigengrasp actions robot joint positions. To evaluate it, we compare DexPilot with two other optimization- retargeting methods by training state-based policies 5 randomly chosen YCB objects. As shown in Table 3, all retargeting approaches yield similar training and test performance, which underscores the robustness of CrossDex to the choice of retargeting algorithm.
We explore the importance using eigengrasps and the impact of varying the number of eigenvectors k. Results presented in Figure 4 indicate that modifying k from to 36 leads to consistent training and test performance, highlighting our method's ' insensitivity to changes k. However, CrossDex-1Aa -IE shows decreased training performance, likely due to the limited capacity of
Eigengrasp Actions:

---

Published as a conference paper at ICLR 2025
the action space. CrossDex-MANO, which uses the 45- 45-dimensional axis angles of the human hand as actions, performs worse on zero-shot adaptation than the configurations using varied numbers of eigengrasps.
Embodiment Randomization and Observations: Table 4 investigates whether applying embod- iment randomization adding embodiment-speccffe information observations affects perfor- mance. Incorporating randomization appears to improve performance across both training and test hands, suggesting greater generalizability of the policy. However, including embodiment specific observations increases the performance gap between state-based and vision-based policies on train- ing hands, as these observations benefit state-based policies but are unavailable for vision-based policies. Moreover, these observations lead to poorer generalization to unseen hands, supporting our argument for unifying observations for cross-embodiment learning.
5.5 REAL-WOOLLD EXPERIMENTS
We establish real-world setup using RealMan RM65 robot arm paired with a LEAP Hand. The object point cloud is captured using Intel RealSense D435i RGBD cameras. We provide video records of the real-world test on our project page.
6 CONCLUSION AND LIMITATIONS
In this paper, we propose CrossDex to address the challenges in cross-embodiment learning for dex- terous grasping. Our main technical contributions include unifying hand actions with human hand eigengrasps to control various dexterous hands, utilizing pre- pre-trained neural networks for efficient hand pose retargeting, and proposing a shared cmbodiment-unawaat observation space to enhance generalization. Our experimental results demonstrate that CrossDex facilitates robust training com- pared to individual embodiment training, successfully controls various hand embodiments with a single policy, and effectively transfers to unseen embodiments through zero-shot generalization and finetuning.
Our work has some limitations that can be addressed in future research. First, train policies on only four dexterous hands due to limited access to dexterous hand models. Future work should include a wider range of dexterous hand embodiments, which is likely to improve embodiment generalization. Additionally, while this work focuses on grasping tasks, there are many real-words tasks for robotic hands such as in-hand reorientation, dynamic handover, and functional grasping that remain unexplored in the literature of cross-embodiment learning. Future work could extend CrossDex to these tasks.
ACKNOWLEDGMENTS
This work was supported by NSFC under Grant 62450001 and 62476008. The authors would like thank the anonymous reviewers for their valuable comments and advice.
REFERENCES
Ananye Agarwal, Shagun Uppal, Kenneth Shaw, and Deepak Pathak. Dexterous functional grasping. In 7th Annual Conference Robot Learning, 2023.
Dafni Antotsiou, Guillermo
and Tae-Kyun Kim. Task-oriented hand motion re- targeting for dexterous manipulation imitation. In Proceedings of the European conference on computer vision (ECCV) workshops, 2018.
Antonio Bicchi. Hands for dexterous manipulation and robust grasping: A difficult road toward simplicity. IEEE Transactions on robotics and aulomation, 2000.
Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Manon Devin, Alex X Lee, Maria Bauza Villalonga, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, et al. Robocat: A self-im self-improving generalist agent for robotic manipulation. Transactions on Machine Learning Research, 2023.
10

---

Published as a conference paper at ICLR 2025
Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference advanced robotics (ICAR). 2015.
Yu-Wei Chao, Wei Yang. Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
Lawrence Yunliang Chen, Kush Hari, Karthik Dharmarajan, Chenfeng Xu, Quan Vuong, and Ken Goldberg, Mirage: Cross-embodiment zero-shot policy transfer with cross-painting. arXiv preprint arXiv:2402. 19249, 2024.
Matei Ciocarlie, Corey Goldfeder, and Peter Allen. Dexterous grasping via eigengrasps: A low- dimensional approach to a high-complexity problem. In Robotics: Science and systems manipu- lation workshop-senste and adapting to the real world, 2007.
Runyu Ding, Yuzhe Qin, Jiyue Zhu, Chengzhe Jia, Shiqi Yang, Ruihan Yang. Xiaojuan Qi, and Xiao- long Wang. Bunny-visionpro: Real-time bimanual dexterous teleoperation for imitation learning. arXiv preprint arXiv:2407.03162 2024.
Ria Doshi, Homer Rich Walke, Oier Mees, Sudeep Dasari, and Sergey Levine. Scaling cross- embodied learning: One policy for manipulation, navigation, locomotion and aviation. In 8th Annual Conference on Robot Leurning, 2024.
Haonan Duan, Peng Wang, Yayu Huang. Guangyun Xu, Wei Wei, and Xiaofei Shen. Roboties dex- terous grasping: The methods based on point cloud and deep learning. Frontiers in Neurorobotics, 2021.
Hao-Shu Fang. Chenxi Wang, Minghao Gou, and Cewu Lu. Graspnet- Ibillioo:::: A large-scale bench- mark for general object grasping. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, and Chelsea Finn. Humanplus: Humanoid shadowing and imitation from humans. arXiv preprint arXiv:2406.10454 2024.
Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, and Dieter Fox. Dexpilot: Vision-based teleoperation of dexterous robotic hand- arm system. In 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020.
Donald Hejna, Lerrel Pinto, and Pieter Abbeel. Hierarchically decoupled imitation for morphologi- cal transfer. In International Conference on Machine Learning, 2020.
Donald Joseph Hejna III, Pieter Abbeel, and Lerrel Pinto. Task-agnostic morphology evolution. In International Conference on Learning Representations, 2021.
Zhengtao Hu, Weiwei Wan, and Kensuke Harada. Designing mechanical tool for robots with two-finger parallel grippers. IEEE Robotics and Automation Letters, 2019.
Ziye Huang, Haoqi Yuan, Yuhui Fu, and Zongqing Lu. Efficient residual learning with mixture-of- experts for universal dexterous grasping. arXiv preprint 10.iv:24100024755 2024.
Zhiwei Jia, Xuanlin Li, Zhan Ling, Shuang Liu, Yiran Wu, and Hao Su. Improving policy opti- mization with generalist-speccillss learning. In International Conference on Machine Learning, 2022.
Aditya Kannan, Kenneth Shaw, Shikhar Bahl, Pragna Mannam, and Deepak Pathak. Deft: Dexterous fine-tuning for hand policies. In Conference on Robot Learning, 2023.
Zhanat Kappassov, luan-Antonia Corrales, and Véronique Perdereau. Tactile sensing in dexterous robot hands. Robotics and Autonomous Systems, 2015.
Puhao Li, Tengyu Liu, Yuyang Li, Yiran Geng, Yixin Zhu, Yaodong Yang, and Siyuan Huang. Gendexgrasp: Generalizable dexterous grasping. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023.
11

---

Published as a conference paper at ICLR 2025
Shuang Li, Xiaojian Ma, Hongzhuo Liang, Michael Gömer, Philipp Ruppel, Bin Fang, Fuchun Sun, and Jianwei Zhang. ussionbased telcoperation of shadow dexterous hand using end-to-end deep neural network. In 2019 International Conference Robotics and Automation (ICRA), 2019.
Shaowei Liu, Yang Zhou, Jimei Yang. Saurabh Gupta, and Shenlong Wang. Contactgen: Generative contact modeling for grasp generation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, 2023.
Xingyu Liu, Deepak Pathak, and Kris Kitani. Revolver: Continuous evolutionary models for robot- to-robot policy transfer. In International Conference on Machine Learning, 2022.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
Priyanka Mandikal and Kristen Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. In Conference on Robot Learning, 2022.
Haoyi Niu, Jianming Hu, Guyue Zhou, and Xianyuan Zhan. A comprehensive survey of cross- domain policy transfer for embodied agents. arXiv preprint arXiv:2402.04580 2024.
Austin Patel and Shuran Song, Get-zero: Graph embodiment transformer for zero-shot embodiment generalization. arXiv preprint arXiv:2407. /5002, 2024.
Jose I. Pons, R Ceres, and Friedrich Pfeiffer. Multifingered dextrous robotics hand design and control: review. Robotica, 1999.
Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Yuzhe Qin, Hao Su, and Xiaolong Wang. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation. IEEE Robotics and Automation Letters, 2022a.
Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, 2022b.
Yuzhe Qin, Binghao Huang, Zhao-Heng Yin, Hao Su, and Xiaolong Wang. Dexpoint: Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation. Conference on Robot Learning. 2023a.
Yuzhe Qin, Wei Yang, Binghao Huang. Karl Van Wyk, Hao Su, Xiaolong Wang, WeiWei Chao, and Dieter Fox. Anyteleop: A general vision-based dexterous robot arm- arm-hand teleoperation system. In Robotics: Science and Systems, 2023b.
Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201 022610, 2022.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statistics, 2011.
Gautam Salhotra, I-Chun Arthur Liu, and Gaurav S Sukhatme. Learning robot manipulation from cross-morphology demonstration. In 7th Annual Conference on Robot Learning. 2023.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.
The graph neural network model. IEFE transactions on neural networks, 2008.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
12

---

Published as a conference paper at ICLR 2025
Lin Shao, Fabio Ferreira, Mikael Jorda, Varun Nambiar, Jianlan Luo, Eugen Solowjow, Juan Apari- cio Ojea, Oussama Khatib, and Jeannette Bohg. Unigrasp: Learning a unified model to grasp with multifingered robotic hands. IEEE Robotics and Automation Letters, 2020.
Kenneth Shaw, Ananye Agarwal, and Deepak Pathak. Leap hand: Low-cost, efficient, and anthro- pomorphic hand for robot learning. arXiv preprint 0rXiv:2309.064440 2023.
Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak. Robotic telekinesis: Learning robotic hand imitator by watching humans on youtube. arXiv preprint arXiv:2202.10442 2022.
Omid Taheri, Nima Ghorbani, Michael Black, and Dimitrios Tzionas. Grab: A dataset of whole- body human grasping of objects. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems (NeurIPS), 2017.
Weikang Wan, Haoran Geng, Yun Liu, Zikang Shan, Yaodong Yang, Li Yi, and He Wang. Unidex- grasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iter- ative generalist--pecialin learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Ruicheng Wang, Jialiang Zhang, Jiayi Chen, Yinzhen Xu, Puhao Li, Tengyu Liu, and He Wang- Dexgraspnet: A large-scale robotic dexterous grasp dataset for general objects based on simula- tion. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023.
Tianyu Wang, Dwait Bhatt, Xiaolong Wang, and Nikolay Atanasov. Cross-embodiment robot ma- nipulation skill transfer using latent space alignment. arXiv preprint arXiv:2406.0/966 2024.
Tianhao Wu, Yunchong Gan, Mingdong Wu, Jingbo Cheng, Yaodong Yang, Yixin Zhu, and Hao Dong. Unidexfpm: Universal dexterous functional pre-grasp manipulation via diffusion policy.
arXiv preprint a4Xiv:2403. /242/, 2024a.
Tianhao Wu, Mingdong Wu, Jiyao Zhang, Yunchong Gan, and Hao Dong. Learning score-based grasping primitive for human-assisting dexterous grasping. Advances in Neural Information Pro- cessing Systems, 2024b.
Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasp- ing via learning diverse proposal generation and goal-conditioned policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Zhixuan Xu, Chongkai Gao, Zixuan Liu, Gang Yang, Chenrui Tie, Haozhuo Zheng, Haoyu Zhou, Weikun Peng, Debang Wang. Tianyi Chen, al. Manifoundation model for general-purpose robotic manipulation of contact synthesis with arbitrary objects and robots. arXiv preprint arXiv:2405.06664, 2024.
Jonathan Yang. Catherine Glossop, Arjun Bhorkar, Dhruv Shah, Quan Vuong, Chelsea Finn, Dorsa Sadigh, and Sergey Levine. Pushing the limits of cross- cross-embodiment learning for manipulation and navigation. arXiv preprint arXiv:2402. 19432, 2024.
Byung-Ju Yi, Heung Yeol Na. Jae Hoon Lee, Yeh-Sun Hong, Sang-Rok Oh, Il Hong Suh, and Whee Kuk Kim. Design of parallel-type gripper mechanism. The International Journal of Robotics Research, 2002.
Chen Yu, Weinan Zhang, Hang Lai, Zheng Tian, Laurent Kneip, and Jun Wang. Multi-emboodment legged robot control sequence modeling problem. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023.
Tianhe Yu. Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances Neural Information Processing Systems, 2020.
13

---

Published as a conference paper at ICLR 2025
Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. Conference on Robot Learning. 2022.
Hui Zhang, Sammy Christen, Zicong Fan, Otmar Hilliges, and Jie Song. Graspxl: Generating grasping motions for diverse objects at scale. In European Conference on Computer Vision, 2025.
Qiang Zhang, Tete Xiao, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Learning cross-domain correspondence for control with dynamics cycle-consistency. International Conference on Learning Representations, 2021.
Bohan Zhou, Haoqi Yuan, Yuhui Fu, and Zongqing Lu. Learning diverse bimanual dexterous ma- nipulation skills from human demonstrations. arXiv preprint arXiv:2410.02477, 2024.
Tianqiang Zhu, Rina Wu, Xiangbo Lin, and Yi Sun. Toward human-like grasp: Dexterous grasp- ing via semantic representation of object-hand. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
14

---

Published as a conference paper at ICLR 2025
A ALGORITHM
Algorithm 1: CrossDex training process.
Input: Dexterous hand models H4; Object set f2: Human hand pose dataset D; Untrained retargeting networks {P& hemH state-based policies { wen and the vision-based
policy T
Output: The learned dision-based policy x
Preprocessing:
{ei) + PCA(D); for / H do
Construct D' = {(O,, J. using MANO and DexPilot; Train P on D' to minimize MSE loss.
Teacher learning:
for w do
Create IsaacGym environments for object w and all hands; while not converge do
Get state- based observations of;
Sample actions Je wy T (o );
Compute target joint positions Jhh +- Ph (Ein ;i;;
Step the environments, get rewards rt;
Save (of" Ja, ww; ri) into the PPO buffer;
Update T using PPO.
Distillation:
Create IsaacGym environments for all objects and hands; while not converge
Get state-based observations of and vision- based observations oi:
Sample actions Jr. un ~ π V (o
Step the environments;
Get teacher actions â, + - (of)) for each object w;
Save (oy ( into the buffer;
Update T minimize MSE loss using the buffer.
B IMPLEMENTATION DETAILS
B. TASKS SETUP
For our experiments, we use objects from the YCB dataset (Calli al., 2015) evaluate CrxssDex. We exclude objects that cannot be modeled as rigid bodies, objects composed of more than 30 convex hulls after VHACD decomposition (to fit within the IsaacGym simulation), and objects that either too large too flat, necessitating functional grasping rewards (Wu et al., 2024a) outside the scope of our study. The resulting dataset comprises 45 objects, representing diverse array of everyday items with varied geometries, such cups, scissors, and boxes. The full object list is available in Figure 5. For testing the finetuning performance on unseen objects, we use 55 objects from the GRAB dataset (Taheri al., 2020). For state- based finetuning and certain ablation studies, we select five random objects from the YCB dataset, including mustard bottle, mug, spoon, softball, and cup_j.
Our reward function, adapted from UnidexGraspt** (Wan et al., 2023), is defined as follows:
total dis height Sy + success.
15

---

Published as a conference paper at ICLR 2025
Here, phs penalizes the weighted L2 distance between the object center and the fingertips and palm: dis 2 x average_ saarngefingertis_ohhgee Lineeriiss_ohheec_ditneeee distance aaal__ojjet_deee distance.
The phight term encourages lifting the object to specific height:
0, if average fingerrifnnnntsss_ pepeetippsobjectt.oo object distance 0.12 and palm_object distance 2 0.15
height
0.9 H 0.6] + (H- 0.6) | otherwise,
where H denotes the current height of the object, starting from an initial height of 0.3 meters. pay aims to minimize horizontal displacement of the object:
y 0.3 x ||objectxxy - object_initial_
Finally, y. awards 200 points upon task completion, defined as maintaining H - 0.6] S 0.05 and either average_ average_fingertips tongertppsobject.t.o object distance I 0.12 palm_ bimm__jeet__tt distance I 0.15 for steps. object_distance
n is set to 30 in test and we increase it to 60 in training to enhance policy robustness.
B.2 EMBODIMENTS
We use six dexterous hand models provided by Ding al. (2024), allocating four for training and two for test.
The training embodiments consist of:
ShadowHand: a 5-fingered hand with 22 DoFs.
Allegro Hand: a 4-fingered hand with 16 DoFs.
Schunk SVH Hand: 5-fingered hand with 20 DoFs.
Ability Hand: 5-finerred hand with 10 DoFs.
For testing, we use:
Leap Hand: 4-fingered hand with 16 DoFs.
Inspire Hand: 5-fingered hand with 12 DoFs.
Refer to Figure for a visualization
six dexterous hands.
Regarding the robot arm, employ the 6-DoF RealMan RM65 arm. The base of the arm is mounted on the side panel of the table, and the dexterous hand is fixed to the end-effector, which aligns our hardware configuration.
B.3 TRAINING RETARGETING NETWORKS
The retargeting networks are designed to map 45-dimensional axis angles of fingers to joint positions for each dexterous hand. For simplicity, the 3-dimensional wrist rotation and the shape parameters of the human hand are set to zero. Each retargeting network is four-layer MLP with hidden layers of 512 dimensions and ReLU activation functions. Each network trained for 200 epochs, minimizing the MSE loss. Figure 11 visualizes the performance of the learned retargeting networks.
B.4 TRAINING POLICIES
Our codebase for RI. and DAgger based on UniDexGrasp (Xu al., 2023). For each state- based policy, use five-layer MLPs for both the actor and the critic, featuring hidden layers with dimensions [1024, 1024, 512, 512] and using ELU activation functions. For the vision-based policy, simplified PointNet (Qi et al., 2017) backbone incorporating two 1D convolution layers and two MLP layers to process the object point cloud, which has dimensions of 512. Both the actor and the critic share the output of this backbone.
In state- based RL, we train the policies using PPO across 40,000 iterations within parallel simula- tions of 8, 192 environments. Hyperparameters provided in Table 5. For vision-based distillation,
16

---

Published as a conference paper at ICLR 2025
use DAgger for 20,000 iterations within parallel simulations of 16,384 environments. Hyperpa- rameters are provided Table 6. In our finetuning experiments, state-based finetuning is conducted for 10,000 iterations using 8,192 parallel environments, while vision-based multi-task finetuning is conducted for 20000 iterations using 16,384 parallel environments. Our experiments within en- vironments of 8,192 can be completed a single NVIDIA RTX 4090 GPU. For the larger scale experiments involving 16,384 environments and PointNet backbones, we use single NVIDIA A800 GPU.
To acquire object point clouds for vision-based policies, we follow the approach in prior work (Wu al., 2024b). simulation, object point clouds are constructed from the objects' mesh data. At each timestep, the point clouds are transformed based on the objects' poses. During training, we apply Farthest Point Sampling (FPS) to sample 512 points, which are then fed into PointNet to extract features. The PointNet is trained jointly with the policy during the distillation process. In the real world, to acquire similar object point clouds, we use RGB-D cameras and instance seg- mentation capture the object point cloud at the first timestep. For subsequent timesteps, we use instance tracking and pose estimation to transform the object point cloud, ensuring consistency of the shape across execution steps. Our approach of constructing object point clouds from mesh data in simulation avoids the need for simulated cameras, which would significantly limit the number of parallel environments. Using this approach, IsaacGym can support thousands of parallel environ- ments, allowing us to train vision-based policies efficiently with 16,384 environments on a single NVIDIA A800 GPU in approximately 10 hours. We have also validated that same performance can be achieved using 4,096 environments on an NVIDIA RTX 4090 GPU, which takes approximately 2 days.
Table 5: Hyperparameters of PPO.
Name
Symbol
Value
Parallel rollout steps per iteration
8
Training epochs per iteration
5
Minibatch size
16384
Optimizer
Adam
Learning rate
"
3e-4
Discount factor
7
0.96
GAE lambda
X
0.95
Clip range
0.2
Table 6: Hyperparameters of DAgger.
Name
Symbol
Value
Parallel rollout steps per iteration Training epochs per iteration
8
Minibatch size
4096
Optimizer
Adam
Learning rate
77
3e-4
C ADDITIONAL RESULTS
C. SUCCESS RATES PER OBJECT
Figure shows the success rates achieved by the learned vision-based policy for each YCB object.
C.2 TRAINING CURVES
Figure 6, 7, and illustrate the reward vs. training iteration curves for the finetuning experiments.
17

---

Published as a conference paper at ICLR 2025
Herr
Figure 5: Average success rates for each YCB object across various embodiments.
2.0
1.5
1.0
Reward
0.5
0.0
CrossDex
-0.5
MT-Raw-A
MT-Raw-O
-1.0
MT-Raw-OA
No-Pretrain
0
2000
4000
6000
8000
10000
Iteration
Figure 6: Training curves depicting state-based finetuning performance on five YCB objects using the unseen LEAP Hand.
Reward
005
CrassDex
MT-Raw-A
-1.0
MT-Raw-O
MT-Raw-OA
No-Pretrain
-1.5
2500
5000
7500
10000
12500
15000
17500
20000
Iteration
Figure 7: Training curves depicting multi-task vision-based finetuning performance on all YCB objects using the unseen LEAP Hand.
18

---

Published as a conference paper at ICLR 2025
1.0
Reward
-0.5
CrossDex
MT-Raw-A
-1.0
MT-Raw-O
MT-Raw-OA OA
-1.5
No-Pretrain
2500
5000
7500
10000
15000
17500
Iteration
Figure 8: Training curves depicting multi-task vision-based finetuning performance on the 55 un- seen GRAB objects using the unseen LEAP Hand.
C.3 TRAINING CURVES FOR DIFFERENT ACTION SPACE CONFIGURATIONS
Figure shows the PPO training curves CrossDex with different numbers of eigengrasp actions and CrossDex with the 45-dimensional MANO actions.
2.0
1.5
Reward
0.5
0.0
9-E
18-E
-1.0
E
MANO
-1.5
2000
4000
6000
8000
10000
Iteration
Figure 9; Training curves of CrossDex with different human hand action spaces.
19

---

Published as a conference paper at ICLR 2025
C.4 VISUALIZATION OF EIGENGRASP ACTIONS AND RETARGETING
Figure 11 displays samples from the eigengrasp action space and the corresponding joint positions of various dexterous hands predicted by our retargeting networks.
Figure 10: Visualization of hand poses for some eigengrasp actions and the corresponding poses of the six dexterous hands as predicted by our retargeting neural networks.
20

---

Published as a conference paper at ICLR 2025
C.5 THE IMPACT OF THE NUMBER OF TRAINING HANDS
We conduct experiments training CrossDex with different numbers of dexterous hands. The sults in Table demonstrate that increasing the number training hands consistently improves performance unseen hands, highlighting the potential for further improvement by including more dexterous hands in the training set.
Table 7: Performance on the two test hands given different numbers of training hands.
Number Training Hands
2
3
4
Success Rate on Unseen Hands 0.129
0.328
0.434
0.482
C.6
RESULTS FOR A FINGERFINGERED HAND
Figure 11: Comparison of the retargeted hand poses between the three-fingered D'Claw Gripper and other dexterous hands, with each row corresponding to a specific eigengrasp action.
It is essential to evaluate our method on 3-fingered hands, even though hand retargeting algorithms are less frequently applied to such hands. To explore this, we conduct an experiment using the three-fingered D' Claaw Gripper. The retargeting algorithm is adapted by ignoring the ring and small fingers the human hand and mapping the other three fingers to the robotic fingers. We also use the GRAB dataset to train the retargeting network. We train the CrossDex policy using the four previous training hands along with the D'Claw Gripper on the five randomly selected YCB objects. The results presented in Table 8.
The results indicate that our method is scalable to the three-fingeree C'alww Gripper, achieving high success rates.
21

---

Published as a conference paper at ICLR 2025
Table 8: Performance on the three-fingered D' Claw Gripper trained with the other four hands.
Allegro Hand
Ability Hand
Schunk SVH Hand
D'Claw Gripper
Hand
ShadowHand
Num.
4
5
5
3
Fingers
Success
0.891 + 0.009
0.892 + 0.008 0.753 + 0.009
0.911 + 0.009 0.853 + 0.033
Rate
In Figure 11, demonstrate that the retargeting network effectively enables the three-fingered hand to open and close, similar to other dexterous hands. However, we observe lower consistency in achieving specific hand poses compared anthropomorphii hands, due the significant differences in morphology. For instance, the D' O'Claw Gripper struggles with certain grasping poses requiring fingers curve inward. Future work could address these challenges by developing more tailored retargeting methods for hands with distinct morphologies.
22


Reviews:
[{"review_id": "oHRanXQAom", "reviewer": "Reviewer_hjEm", "summary": "This paper learns a unified vision-based reinforcement learning policy via teacher-student learning to control various dexterous hands for grasping different objects. The authors propose a unified observation space across various hands, and a universal action space inspired by human hand teleoperation. The learned policy can generalize to unseen dexterous hands in a zero-shot manner.", "questions": "- The authors used four dexterous hands for training and would like to include more different hands in the future. It would be interesting to investigate the impact of the number of hands used for training on the learning performance, e.g., generalizability. \n- The dexterous hands for training and testing consist of either 4-fingered or 5-fingered configurations with different DoFs. What would be the impact if for example one or more 3-fingered hand is also included for training?\n- The authors randomized the origin position of the joint connecting the hand and the arm to increase robustness and transferability of the policy. What would be the impact of extending this to randomize additional hand parameters?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The authors conducted a comprehensive benchmark and ablation study. \n- The presentation of the paper is clear and well-structured.", "weaknesses": "- Only LEAP Hand was tested in the real-world experiments.\n- Some failure cases should be included in the video.\n- Joint position of the arm is included for training policies, which might affect the generalizability to different arms.\n- More design choices should be analyzed regarding the impact on generalizability. See the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "syJbgU4nC9", "reviewer": "Reviewer_B2t5", "summary": "This paper presents a method for learning a unified policy for dexterous grasping that can transfer to unseen hand morphologies. It achieves this by first designing an observation space that contains only fingertip positions and palm pose, which can be shared between different hands. Then, the action space is based on the eigengrasp from the MANO hand model. By using the shared observation and action space, the authors can train a reinforcement learning policy on four different hands and transfer the learned policy to new hands.", "questions": "For policy observation, how should different numbers of fingers be handled? From Figure 2, it seems the authors simply consider four fingers of the hand and ignore the others. If the paper only considers anthropomorphic hands with more than four fingers, this should be clearly stated.\n\nInstead of predicting the eigengrasp and performing retargeting mapping, what is the performance of directly predicting fingertip positions?\n\nWhen evaluating on unseen hands, does the retargeting mapping network use new hand data for training? If this is the case, I believe there should also be a comparison with baselines using those data.\n\nCan you clarify how “MT-Raw-A” makes predictions on the unseen hands? From my understanding, it outputs joint positions for the unseen hands. But what if the unseen hand has a different number of joints compared to the training hands?\n\nMinor questions and suggestions:\n* The YCB dataset contains more than 70 objects. Why were only 45 YCB objects used? What was the criterion for selection?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper presents a clear story on training policies with multiple embodiments. The paper is well-written and illustrates the motivation from human teleoperation well.\n\nThe design of the action space (i.e., a linear combination of eigengrasp coefficients) is interesting.\n\nThe idea of using a neural network to approximate the retargeting is also interesting.\n\nThe authors also show that the policy can successfully transfer to the real world by training a student policy with vision as the input.", "weaknesses": "From my understanding, the observation space contains four fingertip positions. It is not discussed how to deal with three or five fingers, or if the method is specifically designed for anthropomorphic hands. If this is an intrinsic assumption or limitation, it should be discussed in the main paper.\n\nThe authors design several ablation experiments to show the importance of the proposed observation and action space. However, there is still one design missing: What if we train a policy that directly predicts the fingertip positions and palm positions while keeping the observation space the same as the proposed method? The fingertip positions could then be converted to joint positions (independently for each finger) using inverse kinematics. This can also generalize to new hands as long as we have the kinematics structure of the new hand.\n\nThe retargeting mapping network needs training for each morphology, which means that when evaluating a new, unseen hand morphology, it requires extra data from the target hand. I’m concerned that this is not a fair comparison to the baselines, which does not use these data.\n\nIt is also concerning that increasing the number of eigengrasp does not significantly improve performance (Figure 4). This might indicate that the learned grasping actions are quite similar and may be limited for future applications such as functional grasping.\n\nThe real experiments are also limited; only three successful trials are demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PGugFNzE15", "reviewer": "Reviewer_FoGX", "summary": "This paper introduces CrossDex, a method for learning cross-embodiment dexterous grasping by defining a unified, embodiment-independent observation and action space. The action space utilizes eigengrasps of the human hand based on the MANO hand model. To enable efficient training, pre-trained neural networks replace the optimization-based retargeting process from eigengrasps to the joint angles of various embodiments. The policies are trained on grasping tasks using objects from the YCB dataset across different robot hands and evaluated for transferability to unseen embodiments. Additionally, visual policies operating on object point clouds are distilled using DAgger.", "questions": "- What exactly are the observations used by the visual policies, and how are they transferred to the real robot system? In lines 809-810 of the appendix, you mention that the vision policies are trained using DAgger in 16,384 parallel environments, which is double the number of environments used for state-based PPO training. Generating visual observations for such a large number of environments seems particularly challenging, especially given that prior works using student-teacher learning to distill vision policies in Isaac Gym have used two orders of magnitude fewer environments (https://arxiv.org/abs/2211.11744, https://arxiv.org/abs/2111.03043). You also state that \"the object point cloud p ∈ R^(N×3), which contains N points captured by cameras.\" Is this captured by one camera per environment? If so, at what resolution, and how does the object point cloud differ from the point cloud obtained directly from the camera?\n- Has a quantitative evaluation been conducted on real-robot deployment?\n- In lines 346-347 it is stated that \"The episode ends either when the task is completed or the object falls off the table\". Are episodes also terminated after a certain number of time-steps?\n- Is the object shape or ID represented to the state-based policies? If not, is the grouping of objects intended to reduce variability between them enough to allow learning a single grasping policy? If so, how is this grouping performed, and what are the characteristics of the resulting groups? A clearer explanation of the grouping process and its effectiveness in handling object diversity would help clarify how the policy generalizes across different objects.\n- In Figure 3, the training performance of cross-embodiment learning is compared to individual training with the Shadow Hand. Is there a reason to specificallly use the Shadow Hand here, rather than the mean of the performance of all single-embodiment policies?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Relevance of the Problem**\n- The huge diversity of embodiments presents a significant challenge in embodied AI, making research on cross-embodiment learning highly relevant.\n\n**Method**\n- Leveraging pre-trained neural networks to approximate the optimization-based retargeting process across different embodiments is a intuitive approach to ensure efficient, parallelized training.\n\n**Presentation**\n- The writing is clear, and the logical flow of the paper effectively outlines studied problem and proposed solution, making it intuitive to follow.\n- The visual presentation in Figures 1 and 2 is well-executed, providing a clear overview of the simulation setup and the input-output mapping from observations through policy actions to target joint positions.", "weaknesses": "**Method and Evaluation**\n- The baselines used in the evaluations are ablations of the proposed method and do not appear to be particularly strong for this task.\n- Training on 4 embodiments and transferring to 2 novel ones is indeed quite limited, as acknowledged in the limitations section. However, attributing this limitation to 'restricted access to dexterous hand models' is not a compelling justification. Incorporating a simulation with a broader range of dexterous hand models and exploring how the performance of cross-embodiment policies scales with an increasing number of embodiments would add significantly value to the study.\n- Details on how the vision policies are trained including camera resolution and a definition of the object point cloud are missing.\n\n\n**Results and Claims**\n- In section 5.3 it is stated that the cross-embodiment transferabiliy of the learned policy allows for direct deployment of the policy to new hands. This statement is not supported by the performance drop of over 50% between seen and unseen embodiments oberved for CrossDex in Table 1.\n- In lines 254-255 you state that the \"the dimensionality of human hand poses is significantly higher than the DoFs of most\ndexterous hands, which can make RL less efficient.\" However, this high-dimensionality does not seem to be a challenge according to the results in Table 1, where policies using the joint-based action space outperform or perform simlarly to CrossDex in the PPO training stage. Further, related work has shown good performance with PPO in parallized simulators even with very high-dimensional action spaces (https://arxiv.org/abs/2305.12127).\n- The comparison to baseline methods does not convincingly support the claim that using eigengrasps is an effective approach to solving this problem. Baseline policies based on joint-level control are not expected to transfer well across embodiments with different and unknown joint limits, especially when there is no information about the function of each joint. Using a baseline that directly controls keypoints in the MANO hand model, rather than using eigengrasps, would be a more appropriate baseline to validate this claim. This approach would more naturally align with the task of positioning the fingers into stable grasping configurations on an object’s surface. Moreover, Figure 9 in the Appendix highlights substantial differences in the resulting joint configurations for the same eigengrasp across different embodiments. For instance, in the first row of the figure, the second hand shows three fingers touching at the tips, whereas in the fifth hand, there is a significant gap between the fingers. These very different grasp configurations raise questions about the reliability of grasp transfer under such variation.\n- In the ablation on 'Eigengrasp Actions,' Figure 4 examines the effect of varying the number of eigengrasps on performance it is inferred that the method is relatively insensitive to this change. However, the finding that comparable performance can be achieved with just a single eigengrasp raises important questions. This outcome may reflect a characteristic of the problem being studied, which might not require the nuanced control offered by a multi-fingered hand, rather than demonstrating a strength of the proposed method itself.\n\n\n**Minor Points on Writing**\n- In line 483, \"affect\" should be \"affects\".\n\n\n***\n\nOverall, the fact that baselines not specifically designed for cross-embodiment transfer still achieve non-negligible success raises doubts about the suitability of the proposed approach. Moreover, a success rate below 40% on unseen embodiments does not justify the claim of direct transferability. This calls into question the overall feasibility of the framing. Additionally, the observation that a single eigengrasp performs comparably well on the task suggests that the task itself may not be sufficiently challenging. More complex manipulation problems are needed to meaningfully assess the impact of the proposed action space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)