Efficient Residual Learning with Mixture-of-Experts for Universal Dexterous Grasping

## Abstract

Abstract Universal dexterous grasping across diverse objects presents a fundamental yet formidable challenge in robot learning. Existing approaches using reinforcement learning (RL) to develop policies on extensive object datasets face critical limitations, including complex curriculum design for multi-task learning and limited generalization to unseen objects.
To overcome these challenges, we introduce ResDex, a novel approach that integrates residual policy learning with a mixture-of-experts (MoE) framework. ResDex is distinguished by its use of geometry-unaware base policies that are efficiently acquired on individual objects and capable of generalizing across a wide range of unseen objects. Our MoE framework incorporates several base policies to facilitate diverse grasping styles suitable for various objects. By learning residual actions alongside weights that combine these base policies, ResDex enables efficient multi-task RL for universal dexterous grasping.
ResDex achieves state-of-the-art performance on the DexGraspNet dataset comprising 3,200 objects with an 88.8% success rate. It exhibits no generalization gap with unseen objects and demonstrates superior training efficiency, mastering all tasks within only 12 hours on a single GPU.

## 1 Introduction

Dexterous robotic hands (Pons et al., ; Shaw et al., ) provide advanced capabilities for complex grasping tasks, similar to those performed by human hands. However, achieving universal dexterous grasping across a wide range of objects remains a significant challenge due to the high degrees of freedom (DoFs) for dexterous hands and the high variability in object geometry in the real world. Previous works (Qin et al., ; Agarwal et al., ) develop dexterous grasping policies using reinforcement learning (RL), but these policies are limited to a small range of objects that are similar to the training objects. To improve the scalability of universal dexterous grasping, recent studies (Chao et al., ; Wang et al., ; Hang et al., ) introduce datasets that contain a wide variety of objects, each labeled with grasping poses. Xu et al. (); Wan et al. (); Wu et al. () leveraged these datasets to learn universal grasping policies through a teacher-student framework, which addresses the challenges of multi-task optimization. They first train state-based policies using RL to master all objects within the dataset, and then distill these policies into a universal vision-based policy.

However, these approaches exhibit certain limitations. UniDexGrasp (Xu et al., ) involves a complicated curriculum learning design, requiring iterative training across an expanding set of objects, which significantly increases training time and necessitates careful design for the curriculum. Similarly, UniDexGrasp++ (Wan et al., ) requires training various state-based policies on a large number of object clusters. This not only consumes substantial training time but may also lead to overfitting, as training is conducted individually on separate object groups.
In this study, we investigate how to directly learn a multi-task dexterous grasping policy across thousands of objects, which enables both efficient learning and enhanced generalization.

Residual policy learning (Silver et al., ; Johannink et al., ) offers an efficient approach to learning challenging tasks by training a policy to output residual actions using RL, where a suboptimal base policy is provided. This approach has the potential to address the optimization challenges in multi-task RL (Wu et al., ), particularly when the base policy can effectively explore all tasks. Motivated by this, we propose ResDex to train a residual multi-task policy for universal dexterous grasping. The key question then becomes, how to efficiently acquire a base policy that possesses some generalizability to grasp a wide range of objects? Directly applying multi-task RL to all objects leads to worse results due to multi-task gradient interference (Yu et al., ) and requires extensive training time. Conversely, training a policy to grasp a specific object often results in poor generalization to unseen objects.

Recent work (Agarwal et al., ) suggests that a blind grasping policy, relying solely on robot proprioception, can robustly grasp unseen objects placed close to the palm. This is because the policy does not overfit to specific object information, leveraging feedback from joint positions and fingertip forces to adapt to various object geometries inherently. Given this insight, we propose training geometry-unaware base policies that only observe proprioception and the 3D positions of objects to infer the object location. Experimental results demonstrate that our geometry-unaware policy, even trained on a single object, generalizes better to a broad range of objects compared to policies with full object perception.

To enhance the diversity of grasping poses across various objects, we introduce a mixture-of-experts (MoE) approach that learns multiple base policies to represent different grasping styles. We use geometric clustering to categorize all objects and train a geometry-unaware policy for each cluster’s center. In our multi-task learning framework, we train a residual policy that not only outputs residual actions but also assigns weight to each base policy. The final control action for the robot is determined by a weighted sum of the base policies’ actions and the residual action. This method effectively diversifies grasping poses by varying the weights for the base policies, thereby adapting to different object geometries.

ResDex achieves state-of-the-art training performance and generalization capabilities, successfully grasping 3,200 objects in DexGraspNet (Wang et al., ). It achieves a success rate of 88.8% across all training objects and exhibits no generalization gap when applied to unseen objects and categories. Additionally, ResDex demonstrates remarkable training efficiency, mastering such a wide range of tasks in only 12 hours on a single NVIDIA RTX 4090 GPU. In our ablation study, we highlight the critical roles of residual policy learning and geometry-unaware experts in enhancing multi-task learning efficiency and generalization. We also demonstrate the importance of the MoE approach in achieving proper grasping poses.

Our main contributions can be summarized as follows:

- •
We introduce ResDex, a novel residual policy learning approach that significantly addresses the problem of efficient multi-task learning and generalization for universal dexterous grasping.
- •
Our technical contributions, including residual multi-task RL, geometry-unaware base policies, and a mixture of experts, demonstrate marked improvements in developing a universal grasping policy.
- •
ResDex achieves state-of-the-art performance on the DexGraspNet dataset, demonstrating its superior training performance and generalization capabilities compared to existing methods.

## 2 Related Work

Dexterous Grasping (Pons et al., ; Kappassov et al., ) continues to be a formidable challenge, given the high degrees of freedom in multi-fingered robotic hands and the complex geometries and physical properties of real-world objects.
A fundamental task in dexterous grasping is to generate grasping poses. Recent studies have employed various methods such as contact points (Shao et al., ; Wu et al., ), affordance maps (Brahmbhatt et al., ; Jiang et al., ), natural hand annotations Wei et al. (); Hang et al. (), and grasping datasets (Chao et al., ; Wang et al., ) to train models for synthesizing hand grasping poses.
While generating target grasping poses is crucial, successfully completing a grasp also requires close-loop policies that can manage the entire trajectory.
In learning dexterous grasping policies, both imitation learning (Qin et al., ; Mandikal & Grauman, ) and reinforcement learning (RL) (Rajeswaran et al., ; Wu et al., ) have shown promise. The latter offers scalable advantages across a variety of objects due to its independence from human data collection and the efficiency of simulation environments (Makoviychuk et al., ).
Recent advancements in research explore universal dexterous grasping using RL for thousands of objects. UniDexGrasp (Xu et al., ) and UniDexGrasp++ (Wan et al., ) introduce curriculum learning and a teacher-student framework to enable training on numerous objects. UniDexFPM (Wu et al., ) extends these approaches to universal functional grasping tasks.
In our study, we propose an improved RL method for universal dexterous grasping that is more efficient and demonstrates superior performance and generalizability.

Residual Policy Learning provides an effective approach to learn challenging RL tasks when a base policy is available. In robotics, residual policy learning is extensively applied in both manipulation (Alakuijala et al., ; Davchev et al., ; Schoettler et al., ) and navigation tasks (Rana et al., ).
Typically, the residual policy is constructed upon base policies that employ classical model-based control methods (Johannink et al., ; Silver et al., ).
Garcia-Hernando et al. () investigates residual policy learning based on human data. GraspGF (Wu et al., ) explores residual policy learning on a pre-trained score-based generative model (Vincent, ). Zhang et al. () and Jiang et al. () explore using residual policy learning to finetune RL policies. Barekatain et al. () extends residual policy learning to adaptively reweight multiple expert policies.
In our work, we adopt residual policy learning to tackle the challenges in universal dexterous grasping. Our method, which integrates residual RL with a mixture of geometry-unaware experts, significantly improves multi-task learning to grasp diverse objects.

Mixture-of-Experts (MoE) is initially introduced by Jacobs et al. (); Jordan & Jacobs () and typically comprises a set of expert models alongside a gating network (Shazeer et al., ; Fedus et al., ) that learns to weight the output of each expert. Recently, the MoE framework has gained substantial interest in fields such as natural language processing (Jiang et al., ) and multi-modal learning (McKinzie et al., ). MoE has also been applied in RL policies (Doya et al., ; Peng et al., ), where each expert policy learns a distinct probability distribution that is subsequently integrated. Recent works (Cheng et al., ; Celik et al., ) use MoE to enhance multi-task learning in robotics.
In our research, we use the MoE framework to improve the diversity of grasping poses in the multi-task learning of dexterous grasping policies. Each expert within our framework is a geometry-unaware policy, trained on an individual object to develop a unique grasping style and achieve broad generalization across a variety of objects.

## 3 Preliminaries

### 3.1 Problem Formulation

We consider tabletop grasping tasks using a 5-fingered ShadowHand to grasp and lift objects initially placed on a table. The hand features 18 DoFs that control a total of 22 joints, including 4 coupled joints. Our goal is to enable grasping any object within a large object set, denoted as $\omega\in\Omega$. For each object, the task is formulated as a Partially Observable Markov Decision Process (POMDP) $M^{\omega}=\langle\mathcal{O},\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},
\mathcal{U}\rangle$, representing the observation space $\mathcal{O}$, the state space $\mathcal{S}$, the action space $\mathcal{A}$, the transition dynamics $\mathcal{T}(s_{t+1}|s_{t},a_{t})$, the reward function $\mathcal{R}(s_{t},a_{t})$, and the observation emission function $\mathcal{U}(o_{t}|s_{t})$, respectively. At each timestep $t$, the agent observes $o_{t}\in\mathcal{O}$ and takes an action $a_{t}\in\mathcal{A}$, then receives a reward $r_{t}=\mathcal{R}(s_{t},a_{t})$. The environment then transitions to the next state $s_{t+1}\sim\mathcal{T}(s_{t+1}|s_{t},a_{t})$. The agent’s objective is to maximize the expected return across all objects $\sum_{\omega\in\Omega}\mathbb{E}\left[\sum_{t=0}^{T-1}\gamma^{t}r_{t}\right]$, where $T$ is the time limit and $\gamma$ is the discount factor.

For task learning in simulation, the observation $o\in\mathcal{O}$ includes: (1) Robot proprioception $\bm{J}\in\mathbb{R}^{123}$, including wrist position and orientation, joint positions of the hand, fingertip states and forces on fingertip sensors; (2) Object pose, including position $\bm{b}^{p}\in\mathbb{R}^{3}$ and quaternion $\bm{b}^{q}\in\mathbb{R}^{4}$; (3) An object code $\bm{c}^{\omega}\in\mathbb{R}^{64}$, representing the object geometry via a pre-trained PointNet (Qi et al., ). In real-world settings, while precise object pose is unavailable, we opt to use the object point cloud $\bm{p}\in\mathbb{R}^{N\times 3}$, which contains $N$ points captured by cameras.
The action $a\in\mathcal{A}$ consists of target joint positions of the hand and the 6D force applied at the wrist.
Our aim is to learn a vision-based policy $\pi_{\theta}^{V}\left(a_{t}|\bm{J}_{t},\bm{p}_{t},a_{t-1}\right)$, parameterized by $\theta$, to maximize the expected return across all objects.

DexGraspNet (Wang et al., ) provides a dataset that associates each object with grasping proposals. Each grasping proposal is defined as a triplet $\bm{g}=(R,\bm{t},\bm{q})$, representing the wrist’s relative rotation $R\in\mathbb{SO}(3)$ and position $\bm{t}\in\mathbb{R}^{3}$ to the object and the hand’s joint positions $\bm{q}\in\mathbb{R}^{22}$ for a successful grasp. Following Xu et al. (), these data can be integrated into the reward function to facilitate policy learning:

$$ $\displaystyle r_{t}=r_{t}^{task}+\alpha r_{t}^{proposal},$ (1) $\displaystyle r_{t}^{proposal}=-\|\bm{g}-\bm{g}_{t}\|,$ (2) $$

where $r_{t}^{task}$ is a predefined reward for the grasping tasks as detailed in Section A.1. The reward term $r_{t}^{proposal}$ penalizes the distance to the grasping proposal, where $\alpha$ is a hyperparameter adjusting its weight and $\bm{g}_{t}=(R_{t},\bm{t}_{t},\bm{q}_{t})$ represents the current relative pose of the hand to the object.

### 3.2 The Teacher-Student Framework for Universal Dexterous Grasping

Directly optimizing the vision-based policy using RL faces challenges due to gradient interference (Yu et al., ) in multi-task RL and the high dimensionality of point cloud observations. Recent works (Xu et al., ; Wan et al., ; Wu et al., ) have adopted a teacher-student framework in two stages to address these issues. First, a state-based policy $\pi^{S}_{\phi}\left(a_{t}|\bm{J}_{t},\bm{b}^{p}_{t},\bm{b}^{q}_{t},\bm{c}^{
\omega},a_{t-1}\right)$ is trained using privileged object information to master all tasks. Then, this policy is distilled into a vision-based policy using DAgger (Ross et al., ), an online imitation learning method.

To address the multi-task optimization challenge in learning the state-based policy, UniDexGrasp (Xu et al., ) proposed a curriculum learning approach. The RL training starts with a single object and, after a certain number of iterations, gradually includes more objects. This process continues until all objects are included and the policy achieves a high success rate.
UniDexGrasp++ (Wan et al., ) introduced an improved method based on generalist-specialist learning (Jia et al., ). The entire object set is divided into groups through geometry-aware clustering. Numerous specialist state-based policies are then trained and subsequently distilled into a generalist state-based policy, with iterative training implemented through a curriculum.
These methods require meticulous curriculum design and are time-consuming, as various policies are trained across different sets of objects. Additionally, their learned vision-based policies exhibit a significant decrease of about $7\%$ in success rates when tested on unseen objects, indicating limited generalization capabilities.

## 4 Method

We propose ResDex, a framework that leverages residual policy learning combined with a mixture of experts to provide an efficient approach for universal dexterous grasping, significantly enhancing generalization capabilities. Figure 1 illustrates an overview of our framework.

### 4.1 Learning Geometry-Unaware Policies

Figure: Figure 1: We propose ResDex, an efficient learning framework for dexterous grasping across thousands of objects. The learning process consists of two stages: (1) For each representative object from the cluster center, we train a geometry-unaware base policy, which provides weak generalization across a broad range of objects. (2) To develop a universal policy applicable to all objects, we use residual multi-task reinforcement learning (RL) to train a hyper-policy, incorporating the base policies within a mixture-of-experts (MoE) framework. ResDex demonstrates efficient training and robust generalization to unseen objects.
Refer to caption: x1.png

To enable efficient multi-task RL using residual policy learning, it is essential to build a base policy that can effectively explore all the involved tasks. Training a base policy directly on a single type of object often results in overfitting, which significantly decreases its generalizability to other objects.
Conversely, training a policy on all objects using RL presents unique challenges, as different tasks can lead to gradient interference in the learning processes, making the training highly inefficient.

We propose to build base policies that, while trained on a limited number of objects, can generalize effectively to a wider range of objects.
Empirical insights from Agarwal et al. () suggest that a blind grasping policy, trained solely on robot proprioception without specific object information, can better generalize to unseen objects.
We hypothesize that limiting observations helps the policy avoid overfitting to specific object features. When a policy does not have complete information about object poses and geometric features, it tends to learn more generalizable grasping strategies and rely on the proprioceptive feedback to adjust actions.
Although we cannot use a fully blind policy in our setting – as the agent must know the object’s location to approach it – we integrate this insight by proposing a geometry-unaware base policy, $\pi^{B}_{\psi}\left(a_{t}|\bm{J}_{t},\bm{b}^{p}_{t},a_{t-1}\right)$, which uses only robot proprioception $\bm{J}$ and the 3D position of the object $\bm{b}^{p}$.

The grasping proposal reward $r^{proposal}$ inherently leaks the object’s geometric information, as the target relative wrist pose specifies “where to grasp on the object”. To mitigate this unwanted information leakage and enhance generalization, we replace this term with a pose reward:

$$ $r^{pose}_{t}=-\|\bm{q}-\bm{q}_{t}\|,$ (3) $$

where $\bm{q}_{t}$ represents the current hand joint positions. This reward encourages the hand to reach the target joint positions, focusing on the hand pose rather than the specific region to grasp on the object.

Experimental results (Section 5.3) show that our geometry-unaware policy, trained on a single object, demonstrates remarkable generalizability to unseen objects and significantly outperforms policies that incorporate full observations or those trained using the full grasping proposal reward.

### 4.2 Residual Multi-Task Reinforcement Learning

While the base policy trained on a single object offers some degree of generalizability across various objects, it typically achieves a low overall success rate. To address this, we introduce residual policy learning to develop a policy that masters all objects.

The state-based residual policy, denoted as $\pi^{R}_{\phi}\left(a_{t}|\bm{J}_{t},\bm{b}^{p}_{t},\bm{b}^{q}_{t},\bm{c}^{
\omega},a_{t-1}\right)$, is parameterized by $\phi$. It utilizes all available state-based observations to better maximize performance in solving POMDPs.

Given the pre-trained base policy $\pi^{B}_{\psi}$, at each timestep, the base policy uses the required observations from the complete observations to compute a base action $a_{t}^{B}=\operatorname*{arg\,max}_{a_{t}}\pi^{B}_{\psi}\left(a_{t}|\bm{J}_{t}
,\bm{b}^{p}_{t},a_{t-1}\right)$. Simultaneously, the residual policy samples a residual action $a_{t}^{R}\sim\pi^{R}_{\phi}\left(a_{t}|\bm{J}_{t},\bm{b}^{p}_{t},\bm{b}^{q}_{t
},\bm{c}^{\omega},a_{t-1}\right)$, and these actions are combined element-wise to form the final action $a_{t}=a_{t}^{B}+a_{t}^{R}$.

The generalizability of the base policy reduces the need for extensive exploration by the residual policy across diverse object geometries, making it practical to train under multi-task settings. For objects already successfully grasped by the base policy, the residual policy refines the grasping process, enhancing the success rate. For objects not successfully grasped by the base policy, the residual policy can efficiently explore in the residual action space, benefiting from the significant exploration bias provided by the base policy.
We train this residual policy across the entire object set using RL, aiming to maximize the average return across all objects. Our experiments demonstrate that a single base policy, when aided by the residual policy, can achieve high success rates across thousands of objects.

### 4.3 Incorporating a Mixture of Experts

Utilizing diverse poses to grasp different objects is not only a crucial feature for dexterous hands but also essential for post-grasping manipulations in real-world tasks.
While residual policy learning based on a single base policy can achieve commendable success rates, it often struggles to perform various grasping poses for different objects. This limitation arises because the base policy typically provides only a single grasping pose for its training object, thus posing a significant challenge for the residual policy to explore diverse grasping poses for certain objects.

To enhance the diversity of grasping poses, we propose a mixture-of-experts (MoE) approach. In this setup, several base policies are trained, each capable of executing distinct grasping styles, and their actions can be combined to generate a variety of novel grasping poses.
To acquire base policies that exhibit diverse behaviors and grasping poses, we use geometry-aware clustering (Wan et al., ) to divide the object set into $k$ clusters based on object shape representations. Objects at the cluster centers are used to train the base policies $\{\pi^{B}_{\psi_{i}}\}_{i=1}^{k}$, leveraging their distinct and representative shapes to foster diversified grasping styles.

In multi-task learning, to integrate the base policies while learning residual actions, we replace the residual policy with a hyper-policy, denoted as $\pi^{H}_{\phi}\left(a^{R}_{t},\bm{\lambda}_{t}|\bm{J}_{t},\bm{b}^{p}_{t},\bm{b
}^{q}_{t},\bm{c}^{\omega},a_{t-1}\right)$. This hyper-policy predicts the residual action $a_{t}^{R}\in\mathcal{A}$ along with a weight $\bm{\lambda}_{t}\in\mathbb{R}^{k}$ for the MoE.
At each timestep, all base policies predict base actions $\{a^{B}_{t,i}\}_{i=1}^{k}$ using partial observations, and the hyper-policy samples the weights and the residual action. The final action is computed as follows:

$$ $a_{t}=a^{R}_{t}+\frac{1}{\|\bm{\lambda}_{t}\|}\sum_{i=1}^{k}{\lambda_{t,i}a^{B }_{t,i}},$ (4) $$

using a normalized weighted sum of base actions in addition to the residual action. This hyper-policy aims at efficiently learning diverse, natural grasping poses by adjusting the MoE weights and enhancing multi-task performance through residual learning.

### 4.4 Method Summary

Here, we outline the complete pipeline to train ResDex, which encompasses three training phases:

Training Base Policies: Using the entire training set of objects, we apply K-Means clustering (Lloyd, ) on the PointNet Qi et al. () features of objects to generate $k$ clusters. From each cluster, we select the object closest to the center and train a geometry-unaware base policy for each object using RL, as detailed in Section 4.1.

Training the Hyper-Policy: We train the hyper-policy across parallel environments that span all objects in the training set, as described in Section 4.3.
During the training process, the hyper-policy is continually updated while the base policies remain fixed.
To cultivate diverse and effective grasping poses while maximizing success rates, we employ a two-stage reward function:

- •
First stage: We use the reward function that includes the grasping proposal reward: $r=r^{task}+r^{proposal}$. This reward function guides the policy to follow the reference grasping poses provided by the dataset, resulting in more natural and human-like grasps.
- •
Second stage: We remove the grasping proposal term in the reward function and eliminate terms that encourage approaching the object within $r^{task}$, focusing solely on terms related to object lifting and task completion. This adjustment further enhances the policy’s performance by removing constraints imposed by these reward terms. Further details on the reward functions are provided in Appendix A.1.

Vision-based Distillation: To learn a vision-based policy $\pi_{\theta}^{V}$ that operates without privileged object information, we adopt the teacher-student framework. The state-based hyper-policy serves as the teacher, and the vision-based policy to learn acts as the student. We use DAgger (Ross et al., ) to train, which involves iteratively collecting trajectories with the student policy and supervising it using the teacher policy.

## 5 Experiments

### 5.1 Experiment Settings

We evaluate the effectiveness of our method on DexGraspNet (Wang et al., ), a large-scale robotic dexterous grasping dataset for thousands of everyday objects.
The dataset is split into one training set and two test sets, including one that contains unseen objects in the seen categories and the other that contains unseen objects in unseen categories.
The training set includes 3,200 object instances, while the test sets contain a total of 241 object instances.

To train RL policies, we set up parallel simulation environments using IsaacGym (Makoviychuk et al., ). For vision-based distillation, we sample 512 points on each object’s mesh to provide point cloud observations. We compare our ResDex with state-of-the-art methods including UniDexGrasp (Xu et al., ) and UniDexGrasp++ (Wan et al., ).

### 5.2 Main Results

Table 1 shows that our method outperforms UniDexGrasp++ by 2.7%, 5.4%, and 7.8% respectively on the training set and two test sets when $k=4$ after the first training stage. Our method consistently outperforms the baselines using $k$ larger than $2$.
Additionally, we notice that increasing $k$ leads to a slight performance gain. This indicates that using a mixture of base policies enables the hyper-policy to better align with the guidance provided by the grasping proposal reward.

**Table 1: Success rates of state-based policies after the first training stage. We evaluate our method on three different random seeds. $k$ denotes the number of geometry-unaware base policies used to train the hyper-policy.**
| Method | Train(%) | Test(%) |  |
| --- | --- | --- | --- |
| Uns. Obj. | Uns. Cat. |  |  |
| Seen Cat. |  |  |  |
| UniDexGrasp | 79.4 | 74.3 | 70.8 |
| UniDexGrasp++ | 87.9 | 84.3 | 83.1 |
| ResDex ($k=1$) | 83.2$\pm$ 1.5 | 82.8$\pm$1.0 | 85.1$\pm$0.9 |
| ResDex ($k=2$) | 82.8$\pm$ 3.9 | 82.6$\pm$3.2 | 85.0$\pm$3.3 |
| ResDex ($k=3$) | 88.1$\pm$ 1.2 | 88.2$\pm$0.4 | 89.3$\pm$1.0 |
| ResDex ($k=4$) | 90.6$\pm$0.6 | 89.7$\pm$0.8 | 90.9$\pm$0.1 |
| ResDex ($k=5$) | 87.6$\pm$ 0.5 | 87.3$\pm$0.8 | 88.1$\pm$0.2 |
| ResDex ($k=6$) | 88.7$\pm$ 0.6 | 87.8$\pm$0.5 | 88.8$\pm$1.1 |

**Table 2: Success rates of state-based policies after the second training stage. We evaluate our method on three different random seeds. $k$ denotes the number of geometry-unaware base policies used to train the hyper-policy.**
| Method | Train(%) | Test(%) |  |
| --- | --- | --- | --- |
| Uns. Obj. | Uns. Cat. |  |  |
| Seen Cat. |  |  |  |
| UniDexGrasp | 79.4 | 74.3 | 70.8 |
| UniDexGrasp++ | 87.9 | 84.3 | 83.1 |
| ResDex ($k=1$) | 94.3$\pm$1.6 | 93.8$\pm$1.8 | 94.5$\pm$1.3 |
| ResDex ($k=2$) | 94.5$\pm$0.9 | 94.3$\pm$1.1 | 95.2$\pm$1.0 |
| ResDex ($k=3$) | 94.1$\pm$0.9 | 93.9$\pm$0.9 | 94.4$\pm$1.2 |
| ResDex ($k=4$) | 94.6$\pm$1.6 | 94.4$\pm$1.7 | 95.4$\pm$1.0 |
| ResDex ($k=5$) | 94.2$\pm$0.5 | 93.7$\pm$0.9 | 94.2$\pm$0.6 |
| ResDex ($k=6$) | 93.9$\pm$1.3 | 93.6$\pm$1.6 | 94.5$\pm$1.1 |

Table 2 demonstrates that the second training stage significantly boosts the success rates of all policies, regardless of the value of $k$.
Specifically, our best-performing policy ($k=4$) outperforms UniDexGrasp++ by 6.7%, 10.1%, and 12.3% on the training and test sets respectively.
Unlike previous methods, our approach shows no generalization gap, achieving consistent success rates on both the training and test sets.
This consistency indicates that our method can provide a grasping policy that is more robust and generalizable.

During distillation, we use the hyper-policy trained with four base policies as the teacher to learn a vision-based policy.
Performance of vision-based policies are presented in Table 3. Our vision-based policy outperforms UniDexGrasp++ by 3.4%, 8.9% and 10.5% in success rates on the three object sets respectively, demonstrating strong generalization capabilities to unseen objects.

**Table 3: Success rates of vision-based policies.**
| Methods | Train(%) | Test(%) |  |
| --- | --- | --- | --- |
| Uns. Obj. | Uns. Cat. |  |  |
| Seen Cat. |  |  |  |
| UniDexGrasp | 73.7 | 68.6 | 65.1 |
| UniDexGrasp++ | 85.4 | 79.6 | 76.7 |
| ResDex ($k=4$) | 88.8 | 88.5 | 87.2 |

### 5.3 Ablation study

Geometry-Unaware Experts.  We compare generalizability between geometry-unaware policies and policies trained with full state-based observations. We train 3 types of policies on 6 objects, including cell phone, toy figure, bottle, video game console, toilet paper and mug, and we evaluate their performance on the training set. The results are shown in Figure 2. Our geometry-unaware policies achieve higher success rates compared to other policies, achieving over 70% success rates when trained on some objects, which demonstrates remarkable generalizability.
Policies with the full observations or the full grasping proposal reward demonstrate poor generalization when trained on some specific objects.

Figure: Cell Phone
Refer to caption: extracted/5898775/pictures/Cell_Phone.png

Figure: $k=1$
Refer to caption: extracted/5898775/pictures/objects/kettle.png

**Table 4: Ablation study on residual reinforcement learning. We assess success rates of policies on the training set. Method indicates the number of base policies used. MoE shows the results for hyper-policies without residual actions, while MoE+Res shows the results for policies that output both normalized weights for MoE and residual actions.**
| Method | $k=1$ | $k=2$ | $k=3$ | $k=4$ | $k=5$ | $k=6$ |
| --- | --- | --- | --- | --- | --- | --- |
| MoE | 61.4 | 71.1 | 79.4 | 80.3 | 72.1 | 81.6 |
| MoE+Res | 83.2 | 82.8 | 88.1 | 90.6 | 87.6 | 88.7 |

Residual Reinforcement Learning.  To demonstrate the multi-task learning ability provided by residual reinforcement learning, we implement an ablation method that combines base policies using a hyper-policy which only outputs the weights without residual actions. We evaluate the performance on the training set. The results, as shown in Table 4, demonstrate that for different number of base policies, the method with residual learning can notably boost the performance.

Mixture-of-Experts.  We further demonstrate that a mixture of base policies can generate better grasping poses.
We assess the quality of the grasping poses executed by our policies by computing $D=-\sum_{t=1}^{T}r_{t}^{proposal}$. The term $r_{t}^{proposal}$ is a negative reward that punishes the difference between the current grasping pose and the grasping proposal $(R,\bm{t},\bm{q})$. Therefore, the higher the value of $D$, the less natural the grasping poses executed by the policy.
The results, as shown in Table 5, reveal that for the hyper-policies trained over two stages, although their success rates are very close, those with more base policies generally display better grasping poses.

**Table 5: Quality of grasping poses achieved by different policies. We evaluate the $D$ values of ResDex policies with various $k$ on the test set of unseen objects in unseen categories. The lower $D$ means the better grasping poses achieved.**
| Methods | $k=1$ | $k=2$ | $k=3$ | $k=4$ | $k=5$ | $k=6$ |
| --- | --- | --- | --- | --- | --- | --- |
| $\bm{D}\downarrow$ | 223.6 | 174.5 | 194.3 | 176.3 | 204.6 | 176.1 |

Moreover, Figure 3 illustrates the various grasping poses executed by our policies with different numbers of base policies on randomly selected objects (kettle, tape measure, mug, and headphones).
We observe that ResDex trained with more base policies tend to learn grasping strategies that are more appropriate and natural.

## 6 Conclusion and Limitations

We propose ResDex for universal dexterous grasping, a framework that effectively addresses the challenges of training efficiency and generalization that are prevalent in existing methods. Our technical contributions include a residual policy learning framework designed for efficient multi-task reinforcement learning in dexterous grasping, a method to train geometry-unaware base policies that enhances generalization and facilitates exploration across multiple tasks, and an MoE framework that enriches the diversity of the learned grasping poses. We demonstrate the superior performance of ResDex compared to existing methods on the large-scale object dataset DexGraspNet, notably achieving a zero generalization gap to unseen objects. The framework also showcases promising simplicity and training efficiency, marking a significant step towards scaling up dexterous learning.

The limitations of our work include: (1) Although we incorporate a grasping proposal reward to refine grasping poses, we have not yet considered the task as functional grasping. Future work could extend our approach to functional grasping tasks to further enhance general robotic manipulation in real-world settings. (2) We have not deployed the vision-based policy on hardware. Future efforts should focus on this aspect and overcome the sim-to-real gap.

## Appendix A Appendix

### A.1 Simulation Setup

We conduct all our experiments in IsaacGym (Makoviychuk et al., ), a GPU-accelerated platform for physics simulation and reinforcement learning.
Each environment features a table that is 60 cm tall, with an object initialized 10 cm above the tabletop, which then falls onto it.
The shadow hand is initialized 20 cm above the desktop.
The task is to grasp the object and lift its center to 20 cm above the center of the tabletop.

The dataset is split into one training set and two test sets.
The training set contains 3,200 object instances. The test sets include 141 instances of unseen objects within seen categories from the training set and 100 instances of unseen objects in unseen categories.
For state-based policies, we use PPO (Schulman et al., ) for training.
For vision-based policies, we distill the state-based expert policy into a vision-based policy using DAgger (Ross et al., ).
Each geometry-unaware policy is trained with 4,096 environments in parallel for 5,000 iterations.
The hyper policy is trained with 11,000 environments in parallel for 20,000 iterations for every training stage.
The vision-based policy is trained with 11,000 environments in parallel for 8000 iterations.

Reward Function for Base Policy  We use a modified goal-conditioned reward function to train geometry-unaware base policies. The reward function is defined as:

$$ $r=r^{pose}+r^{task}$ $$

$X_{joint}$ denotes the joint positions. The $r^{pose}$ is defined as follows:

$$ $r^{pose}=-0.05*\|\bm{q}-X_{joint}\|_{1}$ $$

$r^{task}$ is defined as follows:

$$ $r^{task}=r^{reach}+r^{lift}+r^{move}+r^{bonus}$ $$

The $r^{reach}$ encourages the hand to reach the object, as it penalizes the distance between the object and different parts of the hand. Here, $X_{obj}$ and $X_{hand}$ denote the position of the object and the hand, and $X_{finger}$ denotes positions of all the fingers. The $r^{reach}$ is defined as follows:

$$ $r^{reach}=-1.0*\|X_{obj}-X_{hand}\|_{2}-0.5*\sum\|X_{obj}-X_{finger}\|_{2}$ $$

The $r^{lift}$ encourages the hand to lift the object. It gives a positive reward when this condition can be satisfied: $f_{1}=\bm{1}\left(\sum\|X_{obj}-X_{finger}\|_{2}\leq 0.6\right)+\bm{1}\left(\|
X_{obj}-X_{hand}\|_{2}\leq 0.12\right)$. $a_{z}$ is the scaled force applied to the hand root along the z-axis. The $r^{lift}$ is defined as follows:

$$ $r^{lift}=\begin{cases}0.1+0.1*a_{z}&\text{if }f_{1}=2\\ 0&\text{otherwise}\end{cases}$ $$

The $r^{move}$ encourages the hand to move the object to the target position. $X_{target}$ denotes the target position. It gives a positive reward when this condition is satisfied: $f_{2}=\bm{1}\left(\sum\|X_{obj}-X_{finger}\|_{2}\leq 0.6\right)+\bm{1}\left(\|
X_{obj}-X_{hand}\|_{2}\leq 0.12\right)+\bm{1}\left(\|\bm{q}-X_{joint}\|_{1}
\leq 6\right)$. The $r^{move}$ is defined as follows:

$$ $r^{move}=\begin{cases}0.9-2\|X_{obj}-X_{target}\|_{2}&\text{if }f_{2}=3\\ 0&\text{otherwise}\end{cases}$ $$

The $r^{bonus}$ gives an extra reward when the object is close to the target position. We denote $\|X_{obj}-X_{target}\|_{2}$ as $d_{obj}$. The $r^{bonus}$ is defined as follows:

$$ $r^{bonus}=\begin{cases}\frac{1}{1+10*d_{obj}}&\text{if }d_{obj}\leq 0.05\\ 0&\text{otherwise}\end{cases}$ $$

Reward Function for Hyper Policy  At the first training stage for a hyper policy, we use the goal-conditioned reward function exactly the same as the one proposed in UniDexGrasp(Xu et al., ).

At the second training stage for a hyper policy, we use a loosened reward function defined as follows:

$$ $r=r^{lift}+r^{move}+r^{bonus}$ $$

The definitions of $r^{lift}$ and $r^{bonus}$ are the same as those mentioned above. The $r^{move}$ has loosened its condition. It is defined as follows:

$$ $r^{move}=\begin{cases}0.9-2\|X_{obj}-X_{target}\|_{2}&\text{if }f_{1}=2\\ 0&\text{otherwise}\end{cases}$ $$

### A.2 Training Details

Network Architecture  We use a MLP architecture which consists of 4 layers (1024, 1024, 512, 512) for base policies and the hyper policy. For the vision-based policy, we use a simplified PointNet (Qi et al., ) encoder to represent the object point cloud and apply MLPs with the same hidden layer sizes for the actor and the critic. We use ELU (Clevert, ) as the activation function.

Training Device and Training Time  All the state-based policies are trained on on a single NVIDIA RTX 4090 GPU. Training a base policy takes about 20 minutes, while training a hyper-policy takes about 11 hours.
For the vision-based policy, we train on a single A800 GPU, taking about 16 hours.

The hyperparameters of PPO and DAgger are described in Table 6 and Table 7.

**Table 6: Hyperparameters of PPO.**
| Name | Symbol | Value |
| --- | --- | --- |
| Episode length | – | 200 |
| Num. envs (base policy) | – | 4096 |
| Num. envs (hyper-policy) | – | 11000 |
| Parallel rollout steps per iteration | – | 8 |
| Training epochs per iteration | – | 5 |
| Num. minibatches per epoch | – | 4 |
| Optimizer | – | Adam |
| Clip gradient norm | – | 1.0 |
| Initial noise std. | – | 0.8 |
| Clip observations | – | 5.0 |
| Clip actions | – | 1.0 |
| Learning rate | $\eta$ | 3e-4 |
| Discount factor | $\gamma$ | 0.96 |
| GAE lambda | $\lambda$ | 0.95 |
| Clip range | $\epsilon$ | 0.2 |

**Table 7: Hyperparameters of DAgger.**
| Name | Symbol | Value |
| --- | --- | --- |
| Episode length | – | 200 |
| Num. envs | – | 11000 |
| Parallel rollout steps per iteration | – | 1 |
| Training epochs per iteration | – | 5 |
| Num. minibatches per epoch | – | 4 |
| Optimizer | – | Adam |
| Clip observations | – | 5.0 |
| Clip actions | – | 1.0 |
| Learning rate | $\eta$ | 3e-4 |
| Clip range | $\epsilon$ | 0.2 |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: Is a sophisticated robot hand necessary to reach a high performance, or could similar performance be reached with simpler hands like the LEAP or Allegro?
soundness: 4
strengths: 1. The authors propose a residual mixture-of-experts policy for dexterous grasping, where the individual base policies are trained on different datasets. This is both a novel and a very interesting idea. In particular, the individual policies are trained on clusters of object geometries using only proprioceptive information, whereas the high level mixing policy is trained with state information.
2. The work further includes a curriculum of two reward functions: the first reward function encourages similarity to demonstrated grasps, whereas the second reward only encourages grasping success. This is a good trade-off between encouraging natural and optimal grasps.
3. The method is compared to prior work on the reinforcement learning of dexterous grasping and it is shown to achieve a higher zero-shot grasping success rate. Appropriate ablations for the various parts of the mixture policy are included.
summary: This work combines concepts from residual policy learning, mixture of experts and student-teacher distillation to train generalizable grasping policies with a dexterous hand in simulation. The proposed method ResDex has multiple stages of reinforcement learning in simulation, including the training of proprioception only policies on different types of objects, training of a residual mixture of experts policy and training of policies with a curriculum of reward functions. The resulting policies are shown to achieve a high performance for grasping unseen object instances and categories.
weaknesses: 1. The paper lacks any real-world experiments. Therefore, it is not clear if the specific design decisions made in this work, which increase the performance in the simulator, lead to a higher real-world grasping success rate. Further, real-world evaluation might be challenging because the Shadow Hand is very expensive. The work could be strengthened by also running experiments with the LEAP Hand, for example, which is more accessible.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - The actions from different base policies are summed together based on the predicted weights of the hyper policy. I am wondering about the rotation representation used in this summation as they lie in a different space than the Euclidean one. 

- Is it seemingly contradictory to first perform geometry-aware clustering and then learn a geometry-unware policy? In the end, the mixture of experts is geometry-aware. For the presentation part, it would be clearer to refine the texts for such differences. For the technical part, can they be merged into a single step in a more intelligent way?

- How is the part of grasp synthesis done? 

- It seems that the number of experts doesn't represent the specific grasp styles as the model performs the best with only 4 experts, which is counter-intuitive for a dataset with more than 3k objects.
soundness: 3
strengths: - This work tackles an important problem in learning-based grasping, i.e., how to learn a performant policy for grasp execution;

- The creative combination of existing ideas to address the problems in the previous work is well-motivated and sensible;

- The idea of using geometry-unware training to enhance the generalization of the base policy is interesting and meaningful;

- The presentation is easy to follow and possesses good readability; 

- Comprehensive comparison and ablation study in the experiments.
summary: For universal dexterous grasp execution, this work introduces a residual RL policy based on a mixture of experts for the base geometry-unaware policies.  The major motivation is to address the training inefficiency and limited generalization issues in the previous work. Technically, the authors propose to combine residual RL and the mixture of experts to tackle the gradient interference issues in training multi-task RL and the limited diversity of using a single base policy. The simulated benchmarking results and ablation study demonstrate superior generalization performance against the baseline.
weaknesses: - The definitions of different reward functions are scattered across several sub-sections. It would be more clear for the reader if they could be grouped and discussed together.

- It would be clearer to reframe the technical contributions in the draft so that the readers can grasp the key idea more conveniently. It's because the main technical contribution is to develop a novel combination of existing techniques and demonstrate its effectiveness in learning generalizable grasping policies. 

-  There is no specific comparison on this aspect. It would be nicer to also compare the training time with Unidexgrasp as the authors claim that the previous approach is inefficient, and this has been addressed by the proposed idea. 

- In the experiment part, for conciseness, the ablation of different numbers of experts in Tables 1 and 2 can be taken out and put into the ablation study subsection.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: - In line 064, what do you mean by “base policies that only observe … 3D positions of objects to infer the object location”? What’s the difference between the 3D positions of objects and the object location?
- When training the base policy, do you train it with randomized object positions? What about orientations?
- Tables 1 and 2 suggest that the proposed method’s performance peaks with four base policies. Why is it not the case that more base policies always yield better performance?
- What is the setup for the vision-based policy? How many cameras are used? How are the cameras placed? Are there any treatments for the observed point cloud before feeding it into the policy? Is the vision-based policy evaluated in simulation or on real robots?
- Around line 421 “… and we evaluate their performance on the training set”, does the training set refer to the training set of the ablation study (i.e., the six objects), or the training set of DexGraspNet?
soundness: 3
strengths: - This paper proposed a novel combination of residual learning and MoE for general dexterous grasping.
- The proposed method significantly outperforms SoTA grasping methods on a large-scale simulation benchmark.
- The proposed method avoids complex curriculum design and observes almost zero performance drop when generalizing to unseen objects and categories.
- The proposed method can be trained within 12 hours on a single 4090 GPU.
- Extensive experiments in simulation support the authors’ claims.
- Overall, the paper is well-organized and written.
summary: This paper proposes residual learning with an MoE method for generalized grasping in simulation. The proposed method includes a set of k geometry-unaware base policies and a hyper policy that learns the weights of each base policy. It also includes a residual action based on the geometry and position of the target object and the robot's proprioception. 

The proposed method avoids complex curriculum design and can be trained within 12 hours on a single 4090 GPU. Its performance peaks SOTA methods and shows no performance drop when generalized to unseen objects and categories. All claims are supported by solid experimental evidence from simulation.
weaknesses: - The authors did not discuss the proposed method's limitations and failure cases. It will be interesting to see and discuss what cases still challenge the proposed method.
- There is no real robot experiment to test if the learned policy adapts well to noises and challenges in the real world.
- In line 353, the authors wrote, “Increasing k leads to a slight performance gain.” This is not true, as the proposed method performs best when k=4 and the performance drops with k larger than 4. It would be better to discuss why the model performs best when k=4.
- When reading subsections 4.1 and 4.2, I am confused about whether the base policy is trained on a single object or multiple objects. The paper contains both descriptions. This confusion is quickly resolved when I discover MoE in subsection 4.3. I suggest specifying how the base policy is used early in subsection 4.1 to avoid this confusion in the future.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How is $g$ in equation 2 sampled? Will randomly sampling $g$ cause gradient interference?
2. In lines 160 and 237, are $q$ and $q_t$ hand joint positions or hand joint angle configurations?
3. One reason for using MoE is that "the base policy typically provides only a single grasping pose for its training object." Does this limitation arise due to the use of argmax for base actions (line 252)? Will other multimodal policy training methods also address this?
4. Could the authors provide more insights into how combining residual policy learning with MoE improves learning? Given that residual RL typically combines known, stable controllers with RL, what role does the MoE play? How does $\pi^H_{\phi}$ learn to weight $a^B_{t,i}$ dynamically without having $a^B_{t,i}$ as input? Could $\lambda_t$ collapse to a mean or one-hot value?
soundness: 3
strengths: 1. Thorough experiments: The experiments are comprehensive, including comparisons with baselines and ablation studies that validate the importance of the method's components.
2. Performance: ResDex demonstrates state-of-the-art success rates on the DexGraspNet dataset, achieving 88.8% success in grasping unseen objects.
3. Clarity: The method is well-explained, and the presentation is enhanced by figures and tables that clearly illustrate key components of the approach.
summary: In this work, the authors introduce ResDex, which integrates residual policy learning with a Mixture-of-Experts (MoE) framework for learning universal dexterous grasping policies. The method addresses drawbacks in conventional methods such as UniDexGrasp and UniDexGrasp++, including limited generalization and complex multi-task curriculum design, by leveraging geometry-unaware base policies. ResDex achieves efficient training and superior generalization, performing state-of-the-art on the DexGraspNet dataset.
weaknesses: 1. Complexity of Approach: While simpler than UniDexGrasp and UniDexGrasp++, the combination of multiple base policies and MoE adds complexity, which goes against the original spirit of residual RL to reduce exploration burden.
2. Training Efficiency: The claim of training efficiency is not substantiated through controlled experiments. Although training times are given in the appendix, there is no comparison to baselines using comparable parameter counts and hardware.
3. Generalizability: While generalization is a key claim, the evaluation is limited to simulation on DexGraspNet data. In contrast, both UniDexGrasp and UniDexGrasp++ evaluated generalizability in different experimental settings, providing stronger support for their claims.
4. Minor Writing Issues: There are some citation issues (e.g., misuse of \citep vs. \citet in lines 101-102, line 296). Section 4.4 would benefit from a \begin{algorithm}. Additionally, the term "geometry-unaware" could be more appropriately named "geometry-agnostic."

Meta-review: ResDex is a framework for dexterous grasping that combines residual policy learning and a mixture-of-experts (MoE) approach. Each base policy is trained on clusters of objects using only proprioceptive data, while a hyper policy fuses these base policies with a small residual adjustment. This design circumvents complex multi-task curricula, showing 88.8% success on a 3,200-object dataset (DexGraspNet), no performance gap on unseen objects, and efficient training (12 hours on a single GPU).

Strengths:
-- The method effectively tackles multi-object grasping by mitigating gradient interference and leveraging geometry-unaware base policies.

-- Demonstrates robust performance on unseen objects without additional fine-tuning.

-- Achieves state-of-the-art results within 12 hours, surpassing prior curriculum-heavy approaches.

Weaknesses:

-- The method’s transferability to physical robots remains unverified.

-- Insufficient discussion of cases where performance might degrade.

-- The multi-stage approach (MoE + residuals) still adds overhead despite aiming to streamline training.

After carefully reading the paper, the reviews and rebuttal discussions, the AC agrees with the reviewers on recommending to accept the paper.

Decision: Accept (Poster)