Efficient Residual Learning with Mixture-of-Experts for Universal Dexterous Grasping
Authors: Ziye Huang
Haoqi Yuan
Yuhui Fu
Zongqing Lu
Venue: ICLR 2025 Poster (2025)
Keywords: dexterous grasping
residual policy learning
reinforcement learning
Primary area: applications to robotics, autonomy, planning

Abstract: Universal dexterous grasping across diverse objects presents a fundamental yet formidable challenge in robot learning. Existing approaches using reinforcement learning (RL) to develop policies on extensive object datasets face critical limitations, including complex curriculum design for multi-task learning and limited generalization to unseen objects. 
To overcome these challenges, we introduce ResDex, a novel approach that integrates residual policy learning with a mixture-of-experts (MoE) framework. ResDex is distinguished by its use of geometry-agnostic base policies that are efficiently acquired on individual objects and capable of generalizing across a wide range of unseen objects. Our MoE framework incorporates several base policies to facilitate diverse grasping styles suitable for various objects. By learning residual actions alongside weights that combine these base policies, ResDex enables efficient multi-task RL for universal dexterous grasping.
ResDex achieves state-of-the-art performance on the DexGraspNet dataset comprising 3,200 objects with an 88.8% success rate. It exhibits no generalization gap with unseen objects and demonstrates superior training efficiency, mastering all tasks within only 12 hours on a single GPU. For further details and videos, visit our project page.

Published as a conference paper at ICLR 2025
EFFICIENT RESIDUAL LEARNING WITH MIXTURE-OF- EXPERTS FOR UNIVERSAL DEXTEROUS GRASPING
Ziye Huang", Haoqi Yuan', Yuhui Fu', Zongqing Lu
1 School of Computer Science, Peking University
Schooo of EECS, Peking University
3 Beijing Academy of Artificial Intelligence
ABSTRACT
Universal dexterous grasping across diverse objects presents a fundamental yet formidable challenge in robot learning. Existing approaches using reinforcement learning (RL) to develop policies on extensive object datasets face critical limi- tations, including complex curriculum design for multi-task learning and limited generalization to unseen objects. To overcome these challenges, we introduce ResDex, novel approach that integrates residual policy learning with a mixture- of-experts (MoE) framework. ResDex is distinguished by its use of geometry- agnostic base policies that are efficiently acquired on individual objects and ca- pable of generalizing across a wide range of unseen objects. Our MoE frame- work incorporates several base policies to facilitate diverse grasping styles suitable for various objects. By learning residual actions alongside weights that combine these base policies, ResDex enables efficient multi-task RL for universal dexter- ous grasping. ResDex achieves ofte--of-the-att performance on the DexGraspNet dataset comprising 3,200 objects with an 88.8% success rate. It exhibits no gen- eralization gap with unseen objects and demonstrates superior training efficiency, mastering all tasks within only 12 hours on a single GPU. For further details and videos, visit our project page.
INTRODUCTION
Dexterous robotic hands (Pons et al., 1999; Shaw et al., 2023) provide advanced capabilities for complex grasping tasks, similar to those performed by human hands. However, achieving universal dexterous grasping across a wide range of objects remains a significant challenge due to the high degrees of freedom (DoFs) for dexterous hands and the high variability in object geometry in the real world. Previous works (Qin et al., 2022a; Agarwal et al., 2023) develop dexterous grasping policies using reinforcement learning (RL), but these policies
limited small range of objects that similar to the training objects. To improve the scalability of universal dexterous grasping, recent studies (Chao et al., 2021; Wang al., 2023; Hang et al., 2024) introduce datasets that contain a wide variety of objects, each labeled with grasping poses. Xu et al. (2023); Wan et al. (2023); Wu et al. (2024a) leveraged these datasets learn universal grasping policies through a teacher- student framework, which addresses the challenges of multi-task optimization. They first train state- based policies using RL to master all objects within the dataset, and then distill these policies into a universal vision-basse based policy.
However, these approaches exhibit certain limitations. UniDexGrasp (Xu al., 2023) involves a complicated curriculum learning design, requiring iterative training across an expanding set of ob- jects, which significantly increases training time and necessitates careful design for the curriculum. Similarly. UniDex (Wan et al., 2023) requires training various state-based policies on large number of object clusters. This not only consumes substantial training time but may also lead to overfitting, training is conducted individually vestigate how to directly learn a multi-task dexterous grasping policy across thousands of objects, which enables both efficient learning and enhanced generalization.
separate object groups. In this study, we in-
Corresponndece Zongqing Lu <zongqing. lu @ pku.edu.cn2.

---

Published as a conference paper at ICLR 2025
Residual policy learning (Silver et al., 2018; Johannink et al., 2019) offers efficient approach to learning challenging tasks by training a policy to output residual actions using RL, where a subopti- mal base policy is provided. This approach has the potential to address the optimization challenges in multi-task RI (Wu et al., 2024b), particularly when the base policy can effectively explore all tasks. Motivated by this, propose ResDex train a residual multi-task policy for universal dexterous grasping. The key question then becomes, how to efficiently acquire a base policy that possesses some generalizability to grasp wide range of objects? Directly applying multi-task RL to all objects leads to worse results due to multi-task gradient interference (Yu et al., 2020) and re- quires extensive training time. Conversely, training a policy to grasp a specific object often results in poor generalization to unseen objects.
Recent work (Agarwal al., 2023) suggests that a blind grasping policy, relying solely on robot proprioception, can robustly grasp unseen objects placed close to the palm. This is because the pol- icy does not overfit to specific object information, leveraging feedback from the hand's joint angles and fingertip forces adapt to various object geometries inherently. Given this insight, propose training geometry-agnostic base policies that only observe proprioception and the 3D positions of objects to infer the object location. Experimental results demonstrate that our geometry-agnonee policy, even trained on a single object, generalizes better to a broad range of objects compared to policies with full object perception.
To enhance the diversity of grasping poses across various objects, we introduce a mixture-of-expprrs (MoE) approach that learns multiple base policies to represent different grasping styles. We use geometric clustering to categorize all objects and train a geometry ter's center. In our multi-task learning framework, we train residual policy that not only outputs residual actions but also assigns weight to each base policy. The final control action for the robot determined by a weighted sum of the base policies' actions and the residual action. This method effectively diversifies grasping poses by varying the weights for the base policies, thereby adapting to different object geometries.
policy for each clus-
ResDex achieves ofate-of-the-art training performance and generalization capabilities, successfully grasping 3,200 objects in DexGraspNet (Wang et al., 2023). It achieves a success rate of 88.89 18.8% across all training objects and exhibits no generalization gap when applied to unseen objects and categories. Additionally, ResDex demonstrates remarkable training efficiency, mastering such a single NVIDIA RTX 4090 GPU. In our ablation study,
wide range of tasks in only 12 hours highlight the critical roles of residual policy learning and geometry-agnoste experts in enhancing multi-task learning efficiency and generalization. We also demonstrate the importance of the MoE approach in achieving proper grasping poses.
Our main contributions be summarized as follows:
We introduce ResDex, a novel residual policy learning approach that significantly ad- dresses the problem of efficient multi-task learning and generalization for universal dex- terous grasping.
Our technical contributions lie in the novel combination of residual multi-task reinforce- ment learning, geonetry-agnostic base policies, and a mixture of experts framework, which together enable the development of more generalizable and effective grasping policy.
ResDex achieves state-of-the-art performance on the DexGraspNet dataset, demonstrat- ing its superior training performance and generalization capabilities compared to existing methods.
2 RELATED WORK
Dexterous Grasping (Pons et al., 1999; Kappassov et al., 2015) continues to be formidable chal- lenge, given the high degrees of freedom in multi-fingered robotic hands and the complex geometries and physical properties of real-world objects. A fundamental task dexterous grasping is to gen- erate grasping poses. Recent studies have employed various methods such contact points (Shao et al., 2020; Wu et al., 2022), affordance maps (Brahmbhatt et al., 2019; Jiang et al., 2021), natural hand annotations (Wei et al.. 2023; Hang et al., 2024), and grasping datasets (Chao et al., 2021; Wang et al., 2023) to train models for synthesizing hand grasping poses. While generating target grasping poses is crucial, successfully completing a grasp also requires close-loop policies that can manage
2

---

Published as a conference paper at ICLR 2025
the entire trajectory. In learning dexterous grasping policies, both imitation learning (Qin et al., 2022b; Mandikal & Grauman, 2022) and reinforcement learning (RL) (Rajeswaran et al., 2017; Wu et al., 2024b; Yuan et al., 2024; Zhou et al., 2024; Zhang et al., 2025) have shown promise. The latter offers scalable advantages across a variety of objects due to its independence from human data collection and the efficiency of simulation environments (Makoviychuk
al., 2021). Recent advancements in research explore universal dexterous grasping using RL for thousands of objects. UniDex (Xu et al., 2023) and Urapex (Wan et al., 2023) introduce curriculum learn- ing and a teacher-student framework to enable training on numerous objects. UniDexFPM (Wu et al., 2024a) extends these approaches to universal functional grasping tasks. In our study, we propose an improved RL method for universal dexterous grasping that is more efficient and demonstrates superior performance and generalizability.
Residual Policy Learning provides an effective approach to learn challenging RL tasks when base policy is available. In robotics, residual policy learning is extensively applied in both manipulation (Alakuijala et al., 2021; Davehev et al., 2022: Schoettler et al., 2020) and navigation tasks (Rana et al., 2020). Typically, the residual policy is constructed upon base policies that employ classical model-based control methods (Johannink et al., 2019; Silver et al., 2018). Farcia-Hernance et al. (2020) investigates residual policy learning based on human data. GraspGF (Wu et al., 2024b) explores residual policy learning on a pre-trained score-based generative model (Vincent, 2011). Zhang et al. (2023) and Jiang et al. (2024b) explore using residual policy learning to finetune RI. policies. Barekatain et al. (2019) extends residual policy learning to adaptively reweight multiple expert policies. In our work, we adopt residual policy learning to tackle the challenges in universal dexterous grasping. Our method, which integrates residual RL with a mixture of geometry-agnostic experts, significantly improves multi-task learning to grasp diverse objects.
Mixture-of-Expee Ef-prrss (MoE) is initially introduced by Jacobs et al. (1991); Jordan & Jacobs (1994) and typically comprises set of expert models alongside gating network (Shazeer et al., 2017; Fedus et al., 2022) that learns to weight the output of each expert. Recently, the MoF framework has gained substantial interest in fields such as natural language processing (Jiang et al., 2024a) and multi-modal learning (McKinzic et al., 2024). MoE has also been applied in RL policies (Doya et al., 2002; Peng et al., 2019), where each expert policy learns a distinct probability distribution that is subsequently integrated. Recent works (Cheng et al., 2023; Celik et al., 2024) use MoE to enhance multi- learning in robotics. In our research, we use the MoE framework to improve the diversity of grasping poses in the multi-task learning of dexterous grasping policies. Each expert within our framework is geometry-agnoste unique grasping style and achieve broad generalization across a variety of objects.
policy, trained on an individual object develop
3 PRELIMINARIES
3.1 PROBLEM FORMULATION
We consider tabletop grasping tasks using a 5-fingered ShadowHand to grasp and lift objects initially placed on a table. The hand features 18 DoFs that control total of 22 joints, including coupled joints. Our goal is to enable grasping any object within large object set, denoted as w E Q. For each object, the task formulated as a Partially Observable Markov Decision Process (POMDP) M = (0, S, A. T, R,, representing the observation space O, the state space S, the action space A. the transition dynamics T(s (t, at), the reward function R(st, az), and the observation emission function ((, (s), respectively. At each timestep t, the agent observes o, O and takes an action a, E A, then receives a reward r, = R(s,, az). The environment then transitions to the next state 81+1 ~ ((st+1 (8,, az). The agent's ss objective is maximize the expected return across all objects Σwen E. 0 yrr) where T is the time limit and y is the discount factor.
For task learning in simulation, the observation o E O includes: (1) Robot proprioception J E TR 123 including wrist position and orientation, joint angles (values of DoF positions) of the hand, fingertip states and forces fingertip sensors; (2) Object pose, including position 62 E I and quaternion ba E IR:; (3) An object code c" E 64 representing the object geometry via pre-trained PointNet (Qi et al., 2017). In real-world settings, while precise object pose is unavailable, we opt to the object point cloud p E IR N x3, which contains N points captured by cameras. The action a E A consists of target joint angles of the hand and the 6D force applied at the wrist. Our aim is to learn
3

---

Published as a conference paper at ICLR 2025
vision-based policy Th (aolJt: Pt, a-1),, parameterized by 8, to maximize the expected return across all objects.
DexGraspNet (Wang al., 2023) provides dataset that associates each object with grasping pro- posals. Each grasping proposal defined as a triplet g = (R, t, q). representing the wrist's relative rotation R (3) and position t E R$3 to the object and the hand's joint angles R22 for successful grasp. Following Xu et al. (2023), these data can be integrated into the reward function facilitate policy learning:
pask + ar, ar
proposal
f = = lgg gell.
proposal
(2)
where rass is predefined reward for the grasping tasks. The reward term it proposal penalizes the distance to the grasping proposal, where a a hyperparameter adjusting its weight and gr = (R2, te, q:) represents the current relative pose of the hand to the object. The full description of the reward functions used in our framework is provided in Appendix A.2.
3.2 THE TEACHER- -SSUDET FRAMEWORK FOR UNIVERSAL DEXTEROUS GRASPING
Directly optimizing the vision-based policy using RL faces challenges due gradient interference (Yu et al., 2020) in multi-task RL and the high dimensionality of point cloud observations. Recent works (Xu et al., 2023; Wan et al., 2023; Wu al., 2024a) have adopted a teacher-student framework in two stages to address these issues. First, a based policy Tे (at Jr, by b/. c" ax-1) is trained using privileged object information to master all tasks. Then, this policy distilled into a vision- based policy using DAgger (Ross et al., 201 I), an online imitation learning method.
To address the multi-task optimization challenge learning the state-based policy, UniDexGrasp (Xu et al., 2023) proposed curriculum learning approach. The RL training starts with a single object and, after a certain number of iterations, gradually includes more objects. This process con- tinues until all objects are included and the policy achieves high success rate. UniDexGrasp++ (Wan et al., 2023) introduced an improved method based on generalist-speciallss learning (Jia et al., 2022). The entire object set is divided into groups through geometry-awart clustering. Numerous specialist state-based policies are then trained and subsequently distilled into generalist state-based policy, with iterative training implemented through curriculum. These methods require meticulous curriculum design and are time-consuming, as various policies are trained across different sets of objects. Additionally, their learned rision-based policies exhibit a significant decrease of about 7% in success rates when tested on unseen objects, indicating limited generalization capabilities.
4 METHOD
We propose ResDex, a framework that leverages residual policy learning combined with a mixture of experts to provide an efficient approach for universal dexterous grasping, significantly enhancing generalization capabilities. Figure I illustrates an overview of our framework.
4.1 LEARNING GEOMETRY-AGNOSTE POLICIES
To enable efficient multi-task RL using residual policy learning, is essential to build base policy that can effectively explore all the involved tasks. Training base policy directly on single type of object often results in overfitting, which significantly decreases its generalizability to other objects. Conversely, training a policy on all objects using RL presents unique challenges, as different tasks can lead to gradient interference in the learning processes, making the training highly inefficient.
We propose to build base policies that can generalize effectively across broad range of objects. Each base policy is trained on a single object. Multiple base policies can then be combined as a mixture-of-experts (MoE) to facilitate efficient multi-task learning.
Empirical insights from Agarwal et al. (2023) suggest that a blind grasping policy, trained solely on robot proprioception without specific object information, can better generalize to unseen objects. We hypothesize that limiting observations helps the policy avoid overfitting to specific object features. When a policy does not have complete information about object poses and geometric features,
4

---

Published as a conference paper at ICLR 2025
Base Policy!
proprioception
2
6D object
Hyper- Policy
Base Policyy
Robot proprioception
Base Policy
Object visual
a
at
representation
3D object
positon
1. Learning Geometry-Agoostic Experts
Residual Multi-Task RL with MoE
Figure 1: We propose ResDex, an efficient learning framework for dexterous grasping across thou- sands of objects. The learning process consists of two stages: (1) For each representative object from the cluster center, we train geometry-agnont tion across a broad range of objects. (2) To develop a universal policy applicable to all objects, we use residual multi-task reinforcement learning (RL) to train a hyper-policy, incorporating the base policies within a mixture-of (MoE) framework. ResDex demonstrates efficient training and robust generalization to unseen objects.
base policy, which provides weak generaliza-
tends to learn more generalizable grasping strategies and rely on the proprioceptive feedback to adjust actions. Although we cannot use a fully blind policy in our setting as the agent must know the object's location to approach it integrate this insight by proposing base policy, x (at Jr, bj, a1-1), which uses only robot proprioception J and the 3D position of the object 6P
-agnosii
The grasping proposal reward propossil inherently leaks the object's geometric information, as the target relative wrist pose specifies "where to grasp on the object". To mitigate this unwanted infor- mation leakage and enhance generalization, replace this term with pose reward:
(3) where q. represents the current hand joint angles. This reward encourages the hand to reach the target joint angles, focusing on the hand pose rather than the specific region to grasp on the object.
pose -11 - qe
Experimental results (Section 5.3) show that our aeonetry- policy, trained on a single object, demonstrates remarkable generalizability to unseen objects and significantly outperforms policies that incorporate full observations or those trained using the full grasping proposal reward.
4.2 RESIDUAL MULTI-TASK REINFORCEMENT LEARNING
While the base policy trained on a single object offers some degree of generalizability across various objects, it typically achieves low overall success rate. To address this, we introduce residual policy learning to develop policy that masters all objects.
The state-based residual policy, denoted TH [ar]Jr, Dū, bi, -=,,a,,). is parameterized by Φ. utilizes all available state-based observations to better maximize performance in solving POMDPs.
Given the pre-trained base policy = at each timestep, the base policy uses the required observations from the complete observations to compute a base action a. arg max,, T (at Je. b. a--1)- f Simultancously, the residual policy samples residual action a." म (at Je, b, by, et a.,;)),
B
and these actions are combined element-wise form the final action a; a. a.
The generalizability of the base policy reduces the need for extensive exploration by the residual policy across diverse object geometries, making it practical to train under multi-task settings. For objects already successfully grasped by the base policy, the residual policy refines the grasping process, enhancing the success rate. For objects not successfully grasped by the base policy, the residual policy can efficiently explore in the residual action space, benefiting from the significant exploration bias provided by the base policy. We train this residual policy across the entire object set using RL, aiming maximize the average return across all objects. Our experiments demonstrate that a single base policy, when aided by the residual policy, can achieve high success rates across thousands of objects.
5

---

Published as a conference paper at ICLR 2025
4.3 INCORPORATING A MIXTURE OF EXPERTS
Utilizing diverse poses to grasp different objects is not only crucial feature for dexterous hands but also essential for post-grasping manipulations in real-world tasks. While residual policy learning based on single base policy can achieve commendable success rates, it often struggles to perform various grasping poses for different objects. This limitation arises because the base policy typically provides only a single grasping pose for its training object, thus posing significant challenge for the residual policy to explore diverse grasping poses for certain objects.
To enhance the diversity of grasping poses, we propose mixture-of-experts (MoE) approach. In this setup, several base policies are trained, each capable of executing distinct grasping styles, and their actions can be combined to generate variety of novel grasping poses. To acquire base policies that exhibit diverse behaviors and grasping poses, we use geometry-aware clustering (Wan et al., 2023) to divide the object set into clusters based on object shape representations. Objects at the cluster centers are used to train the base policies { k==1. leveraging their distinct and representative shapes to foster diversified grasping styles.
In multi-task learning, to integrate the base policies while learning residual actions, replace the residual policy with hyper-policy, denoted as क (af Ar|Jr, bb, by, et', a1-1). This hyper-policy predicts the residual action at E AA along with a weight X E R for the MoE. At each timestep, all base policies predict base actions {af, }== using partial observations, and the hyper-policy samples the weights and the residual action. The final action is computed as follows:
R + a2 at
B A
(4)
using a normalized weighted sum of base actions in addition to the residual action. This hyper- policy aims at efficiently learning diverse, natural grasping poses by adjusting the MoE weights and enhancing multi-task performance through residual learning.
4.4 METHOD SUMMARY
Here, we outline the complete pipeline for training ResDex, which consists of three phases. The pseudocode provided in Appendix A.I.
Training Base Policies: Using the entire training set of objects, we apply K-Means clustering (Lloyd, 1982) on the PointNet (Qi et al., 2017) features of objects to generate clusters. From each cluster, select the object closest to the center and train goometry-agnostic base policy for each object using RL, as detailed in Section 4.1.
Training the Hyper-Policy: We train the hyper-policy across parallel environments that span all objects in the training set, as described in Section 4.3. During the training process, the hyper- policy is continually updated while the base policies remain fixed. To cultivate diverse and effective grasping poses while maximizing success rates, we employ a two-stage reward function:
First stage: We use the reward function that includes the grasping proposal reward: r plask pproposal This reward function guides the policy to follow the reference grasping poses provided by the dataset, resulting in more natural and human-like grasps.
We remove the grasping proposal term in the reward function and elim- inate terms that encourage approaching the object within yosk focusing solely terms related to object lifting and task completion. This adjustment further enhances the policy's performance by removing constraints imposed by these reward terms. Further details on the reward functions are provided in Appendix A.2.
Second stage:
Vision-based Distillation: To learn vision-based policy T. that operates without privileged object information, we adopt the teacher-student framework. The state-based hyper-policy serves as the teacher, and the vision-based policy to learn acts as the student. We use DAgger (Ross et al.. 201 1) to train, which involves iteratively collecting trajectories with the student policy and supervising it using the teacher policy.

---

Published as a conference paper at ICLR 2025
Table 1: Success rates of state-based policies. We evaluate our method on three different random seeds. The hyper-policy is trained with four geometry-agnosti base policies. We present the success rates after each multi-task training stage.
Test( %)
Method
Train (
Uns. Obj. Seen Cat.
Uns. Cat.
UniDexGrasp
79.4
74.3
70.8
UniDexGrasp++
87.9
84.3
83.1
ResDex (stage-1) 1)
90.6+00.6
89.7+0.88
90.9=0.1
ResDex (stage-2)
94.6+1.66
94.4+1.7
95.4=1.0
5 EXPERIMENTS
EXPERIMENT SETTINGS
We evaluate the effectiveness of method DexGraspNet (Wang et al., 2023), large-scale robotic dexterous grasping dataset for thousands of everyday objects. The dataset is split into one training and two test sets, including one that contains unseen objects in the seen categories and the other that contains unseen objects in unseen categories. The training set includes 3,200 object instances, while the test sets contain total of 241 object instances.
To train RL policies, we set parallel simulation environments using IsaacGym (Makoviychuk al., 2021). For vision-based distillation, we sample 512 points on each object's mesh provide point cloud observations. We compare our ResDex with state-of of-the-art methods including UniDex- Grasp (Xu et al., 2023) and UniDexGrasp++ (Wan al., 2023).
5.2
MAIN RESULTS
We compare method with baseline methods using the hyper-policy trained with four geometry- agnostic base policies, which is our best-performing policy. The ablation study on the number of base policies used to train the hyper-policy presented in Section 5.3.
Table shows that our method outperforms UniDexGraspt+ by 6.7%, 10.1%, and 12.3% on the training and test respectively. Unlike previous methods, our approach shows no generalization gap, achieving consistent success rates on both the training and test sets. This consistency indicates that our method can provide grasping policy that is more robust and generalizable.
During distillation, we use the hyper-policy trained with four base policies as the teacher to learn rision-based policy. Performance of vision-based policies are presented in Table Our vision- based policy outperforms UniDexGrasp+* by 3.4%, 8.9% and 10.5% in success rates on the three object sets respectively, demonstrating strong generalization capabilities unseen objects.
Table 2: Success rates of vision-based policies.
Test( %)
Methods
Train %
Uns. Obj-
Uns. Cat.
Seen Cat.
Unapp
73.7
68.6
65.1
UniDex
85.4
79.6
76.7
ResDex
88.8
88.5
87.2

---

Published as a conference paper at ICLR 2025
73.1
Success Rate 180
20
Date 6
Rate 190
70
15
50
50
11.3
20
21.2
30
29.3
10
Ours
Full Obs
Full Pose
Ours
Full Cbs
Pose
Ours
Full Obs
Full Pose
Cell Phone
Toy Figure
Bottle
64.4
71.8
Success Rate (1S)
90
60
Dato
30
27.6
40
20
30
20
0.02
Video Game Console
Toilet Paper
Mug
Figure 2: Generalization performance to all objects using policies with different observations and reward, each trained on single object. Ours: Geometry-Agnose policy. Full Obs: Policy trained with the complete state- based observations. Full Pose: Policy trained using the reward function that includes the full grasping proposal reward.
5.3 ABLATION STUDY
Geometry-Agnoste Experts. We compare generalizability between geometry-agnostic policies and policies trained with full state-based observations. We train 3 types of policies on objects, including cell phone, toy figure, bottle, video game console, toilet paper and mug, and we eval- uate their performance on the training set, which comprises more than 3000 objects. The results are shown in Figure 2. Our geometry-agnostie policies achieve higher success rates compared to other policies, achieving over 70% success rates when trained on some objects, which demonstrates remarkable generalizability. Policies with the full observations the full grasping proposal reward demonstrate poor generalization when trained on some specific objects.
Table 3: Ablation study residual reinforcement learning. We assess success rates of policies on the training set. Method indicates the number of base policies used. MoE shows the results for hyper-policies without residual actions, while MoE+Res shows the results for policies that output both normalized weights for MoE and residual actions.
Method
k 2
k
4
k
6
MoE 61.4
71.1
79.4
80.3
72.1
81.6
MoE+Res 83,2
82.8
88.1
90.6
87.6
88.7
Residual Reinforcement Learning. To demonstrate the multi-task learning ability provided by residual reinforcement learning, we implement ablation method that combines base policies ing hyper- policy which only outputs the weights without residual actions. We evaluate the perfor- mance on the training set. The results, as shown in Table demonstrate that for different number of base policies, the method with residual learning can notably boost the performance.
Eixters-of-Experrs.. We further demonstrate that mixture base policies can generate better grasping poses. We assess the quality of the grasping poses executed by our policies by computing D -= Σ proposal The term r proposal is negative reward that punishes the difference between the current grasping pose and the grasping proposal (R, t, q). Therefore, the higher the value of D, the less natural the grasping poses executed by the policy. The results, shown in Table 4, reveal
8

---

Published as a conference paper at ICLR 2025
==
==
k=
Figure 3: Grasping poses achieved by hyper-policies trained with various numbers of base policies. Each row displays grasping poses for a kettle, tape measure, mug, and headphone, respec- tively. Columns show hyper-policies trained with 3, and base policies, arranged from left to right.
that for the hyper-policies trained over two stages, although their success rates are very close, those with more base policies generally display better grasping poses.
Table 4:
Quality of grasping poses achieved by different policies. We evaluate the D values of ResDex policies with various & on the test set of unseen objects in unseen categories. The lower D means the better grasping poses achieved.
Methods
k
2
4
==
=
D
223.6
174.5
194.3
176.3
204.6
176.1
Moreover, Figure 3 illustrates the various grasping poses executed by our policies with different numbers of base policies on randomly selected objects (kettle, tape measure, mug and headphones). We observe that ResDex trained with more base policies tends to learn grasping strategies that are more appropriate and natural.
The Number of Base Policies. We investigate how k. the number of base policies used to train the hyper-policy, affects the performance. Table 5 shows that ResDex with > 2 consistently outperforms all the baselines according to Table 1. Furthermore, we notice that there is gap in success rates between the configurations of base policies enables the hyper- policy to better align with the guidance provided by the grasping proposal reward. Table demonstrates that the second training stage significantly boosts the success rates of our policy regardless of the value of k. highlighting the stability of method.
< 2 and > This indicates that using a mixture

---

Published as a conference paper at ICLR 2025
Table Success rates of state-based policies after the first training stage. We evaluate our method on three different random seeds. k denotes the number of geometry-agnostic base policies used train the hyper-policy.
Test( %)
Method
Train %)
Uns. Obj-
Uns. Cat.
Seen Cat.
ResDex (k 1)
83.2 1.5
82.8+1.0
85.1=0.9
ResDex (k 2)
82.84 3.9
82.61:3.2
85.0.1..3
ResDex (k 3)
88.1# 1.2
88.2:10.4
89.3+11.0
ResDex (k 4)
90.6=0.6
89.7+00.8
90.911011
ResDex (k 5)
87.61
87.310.8
88.1+0.2
ResDex (k 6)
88.71 0.6
87.810.5
88.8+1.1
Table 6: Success rates of state-based policies after the second training stage. We evaluate our method on three different random seeds. k denotes the number of geometry-agnostic base policies used train the hyper-policy.
Test( %)
Method
Train %)
Uns. Obj-
Uns. Cat.
Seen Cat.
ResDex (A 1)
9..3#1.6
93.8+1..8
94.5+11.
ResDex (k 2)
94.5+0.9
94.3#1.11
95.2+1..
ResDex (A 3)
94.1#0.9
93.910.9
94.4=112
ResDex (k =
94.6#11.6
94.4+:1.7
95.4+110
ResDex (k 5)
94.2+0..5
93.7+00.9
94.2+#0.6
ResDex (k -))
93.9
93.611..6
94.5
CONCLUSION AND LIMITATIONS
We propose ResDex for universal dexterous grasping, framework that effectively addresses the challenges of training efficiency and generalization that are prevalent in existing methods. Our technical contributions include residual policy learning framework designed for efficient multi-task reinforcement learning in dexterous grasping, a method to train geometry-agnostic base policies that enhances generalization and facilitates exploration across multiple tasks, and MoE framework that enriches the diversity of the Iearned grasping poses. We demonstrate the superior performance of ResDex compared existing methods on the large-scale object dataset DexGraspNet, notably achieving a zero generalization gap to unseen objects. The framework also showcases promising simplicity and training efficiency, marking significant step towards scaling up dexterous learning.
The limitations of our work include: (1) Although we incorporate grasping proposal reward to refine grasping poses, we have not yet considered the task as functional grasping. Future work could extend our approach to functional grasping tasks to further enhance general robotic manipulation in real-world settings. (2) We have not deployed the vision-based policy on hardware. Future efforts should focus this aspect and overcome the sim-to-real gap.
The failure cases of our method arise with objects specific sizes and shapes. For example, some large objects may unexpectedly collide with the dexterous hand upon initialization in the simulator, leading to failures. Similarly, extremely small or thin objects, such as scissors and knives, pose chal- lenges under the tabletop grasping setting. Additionally, the policy sometimes generates unstable grasps that result in objects falling off the table before reaching the goal position. However, the policy's closed-loop nature allows itself to adapt to such cases by performing regrasping.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by NSFC under Grant 62450001 and 62476008. The authors would like to thank the anonymous reviewers for their valuable comments and advice.
REFERENCES
Ananye Agarwal, Shagun Uppal, Kenneth Shaw, and Deepak Pathak. Dexterous functional grasping. In 7th Anmual Conference on Robot Learning. 2023.
Minttu Alakuijala, Gabriel Dulac-Amold, Julien Mairal, Jean Ponce, and Cordelia Schmid. Residual reinforcement learning from demonstrations. arXiv preprint arXiv:2106.08055 2021.
Mohammadamin Barekatain, Ryo Yonetani, and Masashi Hamaya. Multipolar: Multi-source policy aggregation for transfer reinforcement learning between diverse environmental dynamies. arXiv preprint arXiv:1909. 131 11, 2019.
Samarth Brahmbhatt, Ankur Handa, James Hays, and Dieter Fox. Contactgrasp: Functional multi- finger grasp synthesis from contact. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2386-2393. IEEE, 2019.
Berk Calli, Arjun Singh, James Bruce, Aaron Walsman, Kurt Konolige, Siddhartha Srinivasa, Pieter Abbeel, and Aaron Dollar. Yale-cmu bele-eeu-berkeley dataset for robotic manipulation research. The International Journal of Robotics Research, 36(3):261-268, 2017.
Onur Celik, Aleksandar Taranovic, and Gerhard Neumann. Acquiring diverse skills using curricu- lum reinforcement learning with mixture of experts. arXiv preprint (41V::2200306666 2024.
Yu-We Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, 2021.
Guangran Cheng, Lu Dong, Wenzhe Cai, and Changyin Sun. Multi-task reinforcement learning with attention- based mixture of experts. IEEE Robotics and Automation Letters, 16):3812-3819, 2023.
Djork-Amé Clevert. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 2015.
Todor Davchev, Kevin Sebastian Luck, Michael Burke, Franziska Meier, Stefan Schaal, and Sub- ramanian Ramamoorthy. Residual learning from demonstration: Adapting dmps for contact-rich manipulation. IEEE Robotics and Automation Letters, 7(2):4888444) 2022.
Kenji Doya, Kazuyuki Samejima, Ken-ichi Katagiri, and Mitsuo Kawato. Multiple model-based reinforcement learning. Neural computation, 14(6):1347-136) 2002.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(10)1:1-39, 2022.
Guillermo
Edward Johns, and Tae-Kyun Kim. Physics-based dexterous manipu- lations with estimated hand poses and residual reinforcement learning. In 2020 JEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pp. 9561-9568. IEEE, 2020.
Jinglue Hang, Xiangbo Lin, Tianqiang Zhu, Xuanheng Li, Rina Wu, Xiaohong Ma, and Yi Sun. Dexfunegrasp: A robotic dexterous functional grasp dataset constructed from cost-effective real- simulation annotation system. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10306-10313) 2024.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.

---

Published as a conference paper at ICLR 2025
Zhiwei Jia, Xuanlin Li, Zhan Ling, Shuang Liu, Yiran Wu, and Hao Su. Improving policy opti- mization with generalist-specian speciallst learning. International Conference on Machine Learning, 2022.
Albert Q Jiang, Alexandre Sablayrolles. Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.00 104088, 2024a.
Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11107-11116, 2021.
Yunfan Jiang, Chen Wang, Ruohan Zhang, Jiajun Wu, and Li Fei-Fei. Transic: Sim-to-real policy transfer by learning from online correction. arXiv preprint 1rXxi:2405.10315, 2024b.
Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Residual reinforcement learning for robot control. In 2019 international conference on robotics and automation (ICRA), pp. 6023- 6029. IEEE, 2019.
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994.
Zhanat Kappassov, Juan-Antonic Corrales, and Véronique Perdereau. Tactile sensing in dexterous robot hands. Robotics and Autonomous Systems, 2015.
Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 1982.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint 200. 10470, 2021.
Priyanka Mandikal and Kristen Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. In Conference on Robot Learning, pp. 651-661. PMLR, 2022.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre-training, arXiv preprint arXiv:2403.09611, 2024.
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. Mcp: Learning composable hierarchical control with multiplicative compositional policies. Advances in neural information processing systems, 32, 2019.
Jose L Pons, R Ceres, and Friedrich Pfciffer. Multifingered dextrous robotics hand design and control: review. Robotica, 1999.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Yuzhe Qin, Binghao Huang, Zhao-Heng Yin, Hao Su, and Xiaolong Wang. Dexpoint: Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation. Conference Robot Learning (CoRL), 2022a.
Yuzhe Qin, Yueh- Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, pp. 570-587. Springer, 2022b.
Wang. Dexmv:
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: / 1709. 10087, 2017.
12

---

Published as a conference paper at ICLR 2025
Krishan Rana, Ben Talbot, Vibhavari Dasagi. Michael Milford, and Niko Sünderhauf. Residual reac- tive navigation: Combining classical and learned navigation strategies for deployment in unknown environments. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp- 11493-11499 IEEE, 2020.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statisties, 2011.
Gerrit Schoettler, Ashvin Nair, Jianlan Luo, Shikhar Bahl, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Deep reinforcement learning for industrial insertion tasks with visual inputs and natural rewards. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5548-5555. IEEE, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (XXiv=1777.06477 2017.
Lin Shao, Fabio Ferreira, Mikael Jorda, Varun Nambiar, Jianlan Luo, Eugen Solowjow, Juan Apari- cio Ojea, Oussama Khatib, and Jeannette Bohg. Unigrasp: Learning unified model to grasp with multifingered robotic hands. IEEE Robotics and Automation Letters, 5(2):2286- 2020.
Kenneth Shaw, Ananye Agarwal, and Deepak Pathak. Leap hand: Low-cost, efficient, and anthro- pomorphic hand for robot learning. arXiv preprint arXiv:2309.0644/ 2023.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gat mixture-of-experts layer. arXiv preprint arXiv: 011017011..5533 2017.
Tom Silver, Kelsey Allen, Josh Tenenbaum, and Leslie Kaelbling, Residual policy learning. arXiv preprint 18/2.0629// 2018
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- tation, 2011.
Weikang Wan, Haoran Geng, Yun Liu, Zikang Shan, Yaodong Yang, Li Yi, and He Wang. Unidex- grasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iter- ative seppraaisttspeccialist learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3891-3902, 2023.
Ruicheng Wang, Jialiang Zhang, Jiayi Chen, Yinzhen Xu, Puhao Li, Tengyu Liu, and He Wang. Dex- graspnet: A large-scale robotic dexterous grasp dataset for general objects based simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11359-11366. IEEE, 2023.
Wei Wei, Peng Wang, and Sizhe Wang. Generalized anthropomorphic functional grasping with minimal demonstrations. arXiv preprint arXiv:2303.178000 2023.
Albert Wu, Michelle Guo, and C Karen Liu. Learning diverse and physically feasible dexterous grasps with generative model and bilevel optimization. arXiv preprint 2xXiv:2207.00195 2022.
Tianhao Wu, Yunchong Gan, Mingdong Wu, Jingbo Cheng, Yaodong Yang, Yixin Zhu, and Hao
Dong. Unidexfpm: Universal dexterous functional pre-grasp manipulation via diffusion policy.
arXiv preprint 2rXiv:2403.124211 2024a.
Tianhao Wu, Mingdong Wu, Jiyao Zhang, Yunchong Gan, and Hao Dong. Learning score-based grasping primitive for buman-assisting dexterous grasping. Advances in Neural Information Pro- cessing Systems, 36, 2024b.
Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasp- ing via learning diverse proposal generation and goal-conditioned policy. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 4737-4746, 2023.
13

---

Published as a conference paper at ICLR 2025
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances Neural Information Processing Systems, 2020.
Haoqi Yuan, Bohan Zhou, Yuhui Fu, and Zongqing Lu. Cross-embodiment dexterous grasping with reinforcement learning. arXiv preprint arXiv:2410.022 02.49999 2024.
Hui Zhang, Sammy Christen, Zicong Fan, Otmar Hilliges, and Jie Song. Graspxl: Generating grasping motions for diverse objects at scale. In European Conference on Computer Vision, 2025.
Xiang Zhang, Changhao Wang, Lingfeng Sun, Zheng Wu, Xinghao Zhu, and Masayoshi Tomizuka. Efficient sim-to-real transfer of contact-rich manipulation skills with online admittance residual learning. Conference on Robot Learning, pp. 1621-1639. PMLR, 2023.
Bohan Zhou, Haoqi Yuan, Yuhui Fu, and Zongqing Lu. Learning diverse bimanual dexterous ma- nipulation skills from human demonstrations. arXiv preprint 2410:2240.00477, 2024.
14

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
A.1
ALGORITHM SUMMARY
Algorithm The Training Pipeline of ResDex
Input: Dataset D of objects and grasping proposals; Number of clusters k.
Preprocess:
Apply K-means to divide objects E D into k clusters C₁, Cz...., Cs based on PointNet fea- tures.
for each cluster C,, i 1, ... k do
Find the object w, that closest to the center of C₁. end for
Training Base Policies:
for i = -I.... A do
Train base policy ne on wi, maximizing reward p-oose plsk
end for
Multi-Task Residual RL:
Train hyper-policy x with frozen base policies {m
all objects.
Training stage I: maximize = pankk + prrppoaal
Training stage 2: maximize plnck reach
Vision-based Distillation:
Train the based policy n. on all objects using DAgger, with frozen n
A.2 SIMULATION SETUP
We conduct all our experiments in IsaacGym (Makoviychuk et al., 2021). a GPU-accelerated plat- form for physics simulation and reinforcement learning. Each environment features table that is 60 cm tall, with an object initialized 10 cm above the tabletop, which then falls onto The shadow hand is initialized 20 above the desktop. The task is to grasp the object and lift its center to 20 cm above the center of the tabletop.
The dataset is split into one training set and two test sets. The training contains 3,200 object instances. The test sets include 141 instances of unseen objects within seen categories from the training set and 100 instances unseen objects in unseen categories. For state-based policies, we use PPO (Schulman et al., 2017) for training. For vision-based policies, we distill the state-based expert policy into a vision-based policy using DAgger (Ross et al., 2011). Each geometry-agnoste policy is trained with 4,096 environments in parallel for 5,000 iterations. The hyper policy is trained with 11,000 environments in parallel for 20,000 iterations for every training stage. The vision-based policy is trained with 11,000 environments in parallel for 8000 iterations.
Reward Function for Base Policy We use a modified goal-conditioned reward function to train geometry-agnostic base policies. The reward function is defined as:
pose task
Xjoint denotes the joint angles. The pose defined as follows:
pose -0.0 ||g X Xjoinelli
p task is defined as follows:
task
reach + fifft + move
bonus
The reach encourages the hand to reach the object, as it penalizes the distance between the object and different parts of the hand. Here, Xobj and Xhand denote the position of the object and the hand, and X Xfinger denotes positions of all the fingers. The reach is defined follows:
reach -1.0 ||obj
Xhand 0.5
Σ ||
X finner |
15

---

Published as a conference paper at ICLR 2025
The plift encourages the hand to lift the object. It gives positive reward when this condition can be
satisfied: f1 1 (Σ X fingerllz I 0.6) + 1 oob Xhandllz S 0.12). as is the scaled force applied to the hand root along the z-axis. The plift is defined as follows:
0.1 0.1 a if f1 = otherwise
Lift
0
The mooe encourages the hand to move the object to the target position. Xtarget de- notes the target position. It gives positive reward when this condition is satisfied: f2 1 XXob - X finger ll2 I 0.6) + 1 Xory -handll2 S 0.12) + 1 (olq--jointlll S 6). The mooe is defined follows:
2al - Xtargetll2 00.9
if f2 =3
mone
00
otherwise
The ybouus gives an extra reward when the object close to the target position. We denote X Xtargetl22 dabj. The poonue is defined follows:
|Xobs
if dobb 0.05
bonus
T0edobs
otherwise
Reward Function for Hyper Policy At the first training stage for a hyper policy, we use the goal- conditioned reward function exactly the same the one proposed in UniDexGrasp(Xu al., 2023).
At the second training stage for hyper policy, we use a loosened reward function defined as follows:
r = liftt + mooe + bonus
The definitions of lift and bonus are the same as those mentioned above. The move has loosened its condition. It is defined as follows:
10.9 0
Xol Xtarget if fi =
move
otherwise
A.3 TRAINING DETAILS
Network Architecture We use a MLP architecture which consists of layers (1024, 1024, 512, 512) for base policies and the hyper policy, For the vision-based policy, we a simplified PointNet (Qi et al., 2017) encoder to represent the object point cloud and apply MLPs with the same hidden layer sizes for the actor and the critic. We use ELU (Clevert, 2015) the activation function.
Training Device and Training Time All the based policies are trained on on single NVIDIA RTX 4090 GPU. Training base policy takes about 20 minutes, while training a hyper- policy takes about 11 hours. For the vision-based policy, we train on single A800 GPU, taking about 16 hours.
Analysis of Training Efficiency To demonstrate the training efficiency of our method compared with UnipexGrasp and UniDexGrasp++, we provide a comparative analysis based on the number of training rounds, as detailed in their papers. UniDexGrasp implements progressive training strategy starting with single object, expanding to several objects within the same category, and finally covering the full training set - requiring three multi-lask training stages in practice. UniDexGraspt+ is more complex, involving the training of 20 multi-task policies along with several distillation stages. In contrast, our method only necessitates the training of a single multi-task policy in one trial, using between simpler but also efficient. As demonstrated in our experiments, our method achieves high success rates with just one base policy. Table compares the training efficiency of different methods in terms of the number of training rounds.
to six low-cost, single-tasi, task base policies. Our approach is not only
The hyperparameters of PPO and DAgger are described in Table 8 and Table 9.
16

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of the number of training rounds required by different methods.
Rounds for Single-Ohject
Rounds for Multi-Task
Method
Training (< 20 minutes)
Training (1 ~ 10 hours)
UniDexGrassp
0
2 3
UniDexGrasp*+
20
ResDex
1
Table 8: Hyperparameters of PPO.
Name
Symbol
Value
Episode length
200
Num. envs (base policy)
4096
Num. envs (hyper-policy)
11000
Parallel rollout steps per iteration
Training epochs per iteration
5
Num. minibatches per epoch
Optimizer
Adam
Clip gradient norm
1.0
Initial noise std.
0.8
Clip observations
5.0
Clip actions
1.0
Learning rate
3e-4
Discount factor
0.96
X
GAE lambda
0.95
Clip range
0.2
Table Hyperparameters of DAgger.
Name
Symbol
Value
Episode length
200
Num. envs
11000
Parallel rollout steps per iteration
Training epochs per iteration
5
Num. minibatches per epoch
Optimizer
Adam
Clip observations
5.0
Clip actions
1.0
Learning rate
3e-4
Clip range
0.2
17

---

Published as a conference paper at ICLR 2025
B ADDITIONAL RESULTS
B.1 RESULTS ON YCB DATASET
To further demonstrate the generalizability of our method, we test the learned policy on the YCB Dataset (Calli et al., 2017), which comproses 75 objects. We evaluate our vision-based policy, trained on DexGraspNet, in zero-shot manner. It achieves success rate of 65.55%, which under- scores the strong generalizability of our policy to unseen datasets. It is important to note that 30% of YCB objects very flat and thin, which significantly challenges tabletop grasping. Additionally, because the models of YCB objects are scanned from real-world objects, they often feature irregular, non-convex shapes. This leads to differences between visual observations and collision meshes in IsaacGym, increasing the difficulty for the grasping policy, which relies on visual point clouds but interacts with mismatched physical shapes.
B.2 RESULTS ON LEAP HAND
We additionally evaluate ResDex the low-cost LEAP Hand, which is more accessible in labora- tories. We implement a simulation setup for the LEAP Hand attached to a 6-DoF robot arm that is fixed table. The action space includes PD control targets for both the hand joints and the six arm joints. This setup enhances the practicability for sim-to-real deployment.
We train ResDex without modifying any hyperparameters and achieved an average success rate of 60.71 % on the 3.2K objects in DexGraspNet. Several factors affect the LEAP Hand's performance, which is lower than that of the ShadowHand: (1) LEAP Hand is significantly larger and has thicker fingertips, posing challenges for grasping small objects in DexGraspNet; (2) LEAP Hand policies are trained without the grasping proposal reward due to the absence of corresponding data; (3) LEAP Hand has fewer degrees of freedom compared to ShadowHand, which can limit its capabilities; (4) The attachment to a robot arm reduces the effective workspace and alters the mechanism for controlling wrist pose, potentially affecting training performance.
Figure visualizes the setup for the simulation and the learned grasping poses of the LEAP Hand.
Fooditem
Tank
Bottle
Marker
Figure 4: The learned grasping behaviors of the LEAP Hand for different objects. Using the low-cost LEAP Hand attached robot arm, this configuration offers greater accessibility for sim- to-real deployment.
B.3 DIVERSITY OF THE LEARNED MOE WEIGHTS
We demonstrate that the diversity of the learned correlates with the diversity of grasping styles, as evidenced by Table 4. To further investigate the diversity of the normalized weights À produced by the syper-policy, we calculate the sum of each dimension, which corresponds to each base policy, throughout the evaluation process across the entire object set. The results presented in Figure 5.
When &o = and A: = 6, the hyper-policy can utilize different base policies. In contrast, it almost exclusively selects one base policy when & 5. The result is consistent with the result in Table 4, which shows that the quality of grasping is relatively low for = 5 compared to and = 6. Since reward functions have no limit the diversity of X, there could be many reasons for these
18

---

Published as a conference paper at ICLR 2025
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.0
0.0
2
3
1
2
0
3
(a) l =
(b) l: 5
(c) k 6
Figure 5: Diversity of the learned X. k denotes the number of geometry-agnostie base policies used to train the hyper-policy, Each number on the x-axis corresponds specific base policy. The height of each bar indicates the sum of weights for the corresponding base policy, with the largest sum normalized to 1 and other bars adjusted proportionally.
results. We suggest that explicitly diversifying À during the training process might lead to polices that exhibit more natural grasping styles.
Furthermore, we demonstrate the variation of along the executed trajectories for different objects. As shown in Figure 6, varies at different timesteps and the variation of X follows distinct patterns for different objects, which indicates that the hyper-policy leverages actions from different base polices to grasp different objects. Instead of collapsing to one-hot vectors or constant vectors, exhibits diversity both across different objects and different timesteps within grasping trajectory.
5.3
30
(a) Drinking Utensil
(b) Mug
(c) Pistol
Figure 6: The learned À for different objects their executed trajectories. The figures show the variation of X through grasping trajectories for different objects (drinking utensil, mug, and pistol)
when k = 6. Weights for each base policy plotted using distinct color.
19


Reviews:
[{"review_id": "eD0fIfWGa1", "reviewer": "Reviewer_8Don", "summary": "This work combines concepts from residual policy learning, mixture of experts and student-teacher distillation to train generalizable grasping policies with a dexterous hand in simulation. The proposed method ResDex has multiple stages of reinforcement learning in simulation, including the training of proprioception only policies on different types of objects, training of a residual mixture of experts policy and training of policies with a curriculum of reward functions. The resulting policies are shown to achieve a high performance for grasping unseen object instances and categories.", "questions": "Is a sophisticated robot hand necessary to reach a high performance, or could similar performance be reached with simpler hands like the LEAP or Allegro?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The authors propose a residual mixture-of-experts policy for dexterous grasping, where the individual base policies are trained on different datasets. This is both a novel and a very interesting idea. In particular, the individual policies are trained on clusters of object geometries using only proprioceptive information, whereas the high level mixing policy is trained with state information.\n2. The work further includes a curriculum of two reward functions: the first reward function encourages similarity to demonstrated grasps, whereas the second reward only encourages grasping success. This is a good trade-off between encouraging natural and optimal grasps.\n3. The method is compared to prior work on the reinforcement learning of dexterous grasping and it is shown to achieve a higher zero-shot grasping success rate. Appropriate ablations for the various parts of the mixture policy are included.", "weaknesses": "1. The paper lacks any real-world experiments. Therefore, it is not clear if the specific design decisions made in this work, which increase the performance in the simulator, lead to a higher real-world grasping success rate. Further, real-world evaluation might be challenging because the Shadow Hand is very expensive. The work could be strengthened by also running experiments with the LEAP Hand, for example, which is more accessible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sCVGjlRvkw", "reviewer": "Reviewer_HhuW", "summary": "For universal dexterous grasp execution, this work introduces a residual RL policy based on a mixture of experts for the base geometry-unaware policies.  The major motivation is to address the training inefficiency and limited generalization issues in the previous work. Technically, the authors propose to combine residual RL and the mixture of experts to tackle the gradient interference issues in training multi-task RL and the limited diversity of using a single base policy. The simulated benchmarking results and ablation study demonstrate superior generalization performance against the baseline.", "questions": "- The actions from different base policies are summed together based on the predicted weights of the hyper policy. I am wondering about the rotation representation used in this summation as they lie in a different space than the Euclidean one. \n\n- Is it seemingly contradictory to first perform geometry-aware clustering and then learn a geometry-unware policy? In the end, the mixture of experts is geometry-aware. For the presentation part, it would be clearer to refine the texts for such differences. For the technical part, can they be merged into a single step in a more intelligent way?\n\n- How is the part of grasp synthesis done? \n\n- It seems that the number of experts doesn't represent the specific grasp styles as the model performs the best with only 4 experts, which is counter-intuitive for a dataset with more than 3k objects.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This work tackles an important problem in learning-based grasping, i.e., how to learn a performant policy for grasp execution;\n\n- The creative combination of existing ideas to address the problems in the previous work is well-motivated and sensible;\n\n- The idea of using geometry-unware training to enhance the generalization of the base policy is interesting and meaningful;\n\n- The presentation is easy to follow and possesses good readability; \n\n- Comprehensive comparison and ablation study in the experiments.", "weaknesses": "- The definitions of different reward functions are scattered across several sub-sections. It would be more clear for the reader if they could be grouped and discussed together.\n\n- It would be clearer to reframe the technical contributions in the draft so that the readers can grasp the key idea more conveniently. It's because the main technical contribution is to develop a novel combination of existing techniques and demonstrate its effectiveness in learning generalizable grasping policies. \n\n-  There is no specific comparison on this aspect. It would be nicer to also compare the training time with Unidexgrasp as the authors claim that the previous approach is inefficient, and this has been addressed by the proposed idea. \n\n- In the experiment part, for conciseness, the ablation of different numbers of experts in Tables 1 and 2 can be taken out and put into the ablation study subsection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zS5M2GfOU9", "reviewer": "Reviewer_3ip3", "summary": "This paper proposes residual learning with an MoE method for generalized grasping in simulation. The proposed method includes a set of k geometry-unaware base policies and a hyper policy that learns the weights of each base policy. It also includes a residual action based on the geometry and position of the target object and the robot's proprioception. \n\nThe proposed method avoids complex curriculum design and can be trained within 12 hours on a single 4090 GPU. Its performance peaks SOTA methods and shows no performance drop when generalized to unseen objects and categories. All claims are supported by solid experimental evidence from simulation.", "questions": "- In line 064, what do you mean by “base policies that only observe … 3D positions of objects to infer the object location”? What’s the difference between the 3D positions of objects and the object location?\n- When training the base policy, do you train it with randomized object positions? What about orientations?\n- Tables 1 and 2 suggest that the proposed method’s performance peaks with four base policies. Why is it not the case that more base policies always yield better performance?\n- What is the setup for the vision-based policy? How many cameras are used? How are the cameras placed? Are there any treatments for the observed point cloud before feeding it into the policy? Is the vision-based policy evaluated in simulation or on real robots?\n- Around line 421 “… and we evaluate their performance on the training set”, does the training set refer to the training set of the ablation study (i.e., the six objects), or the training set of DexGraspNet?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- This paper proposed a novel combination of residual learning and MoE for general dexterous grasping.\n- The proposed method significantly outperforms SoTA grasping methods on a large-scale simulation benchmark.\n- The proposed method avoids complex curriculum design and observes almost zero performance drop when generalizing to unseen objects and categories.\n- The proposed method can be trained within 12 hours on a single 4090 GPU.\n- Extensive experiments in simulation support the authors’ claims.\n- Overall, the paper is well-organized and written.", "weaknesses": "- The authors did not discuss the proposed method's limitations and failure cases. It will be interesting to see and discuss what cases still challenge the proposed method.\n- There is no real robot experiment to test if the learned policy adapts well to noises and challenges in the real world.\n- In line 353, the authors wrote, “Increasing k leads to a slight performance gain.” This is not true, as the proposed method performs best when k=4 and the performance drops with k larger than 4. It would be better to discuss why the model performs best when k=4.\n- When reading subsections 4.1 and 4.2, I am confused about whether the base policy is trained on a single object or multiple objects. The paper contains both descriptions. This confusion is quickly resolved when I discover MoE in subsection 4.3. I suggest specifying how the base policy is used early in subsection 4.1 to avoid this confusion in the future.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XJvPHKg4fn", "reviewer": "Reviewer_BKwA", "summary": "In this work, the authors introduce ResDex, which integrates residual policy learning with a Mixture-of-Experts (MoE) framework for learning universal dexterous grasping policies. The method addresses drawbacks in conventional methods such as UniDexGrasp and UniDexGrasp++, including limited generalization and complex multi-task curriculum design, by leveraging geometry-unaware base policies. ResDex achieves efficient training and superior generalization, performing state-of-the-art on the DexGraspNet dataset.", "questions": "1. How is $g$ in equation 2 sampled? Will randomly sampling $g$ cause gradient interference?\n2. In lines 160 and 237, are $q$ and $q_t$ hand joint positions or hand joint angle configurations?\n3. One reason for using MoE is that \"the base policy typically provides only a single grasping pose for its training object.\" Does this limitation arise due to the use of argmax for base actions (line 252)? Will other multimodal policy training methods also address this?\n4. Could the authors provide more insights into how combining residual policy learning with MoE improves learning? Given that residual RL typically combines known, stable controllers with RL, what role does the MoE play? How does $\\pi^H_{\\phi}$ learn to weight $a^B_{t,i}$ dynamically without having $a^B_{t,i}$ as input? Could $\\lambda_t$ collapse to a mean or one-hot value?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Thorough experiments: The experiments are comprehensive, including comparisons with baselines and ablation studies that validate the importance of the method's components.\n2. Performance: ResDex demonstrates state-of-the-art success rates on the DexGraspNet dataset, achieving 88.8% success in grasping unseen objects.\n3. Clarity: The method is well-explained, and the presentation is enhanced by figures and tables that clearly illustrate key components of the approach.", "weaknesses": "1. Complexity of Approach: While simpler than UniDexGrasp and UniDexGrasp++, the combination of multiple base policies and MoE adds complexity, which goes against the original spirit of residual RL to reduce exploration burden.\n2. Training Efficiency: The claim of training efficiency is not substantiated through controlled experiments. Although training times are given in the appendix, there is no comparison to baselines using comparable parameter counts and hardware.\n3. Generalizability: While generalization is a key claim, the evaluation is limited to simulation on DexGraspNet data. In contrast, both UniDexGrasp and UniDexGrasp++ evaluated generalizability in different experimental settings, providing stronger support for their claims.\n4. Minor Writing Issues: There are some citation issues (e.g., misuse of \\citep vs. \\citet in lines 101-102, line 296). Section 4.4 would benefit from a \\begin{algorithm}. Additionally, the term \"geometry-unaware\" could be more appropriately named \"geometry-agnostic.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)