Learning Diverse Bimanual Dexterous Manipulation Skills from Human Demonstrations

## Abstract

Abstract Bimanual dexterous manipulation is a critical yet underexplored area in robotics. Its high-dimensional action space and inherent task complexity present significant challenges for policy learning, and the limited task diversity in existing benchmarks hinders general-purpose skill development.
Existing approaches largely depend on reinforcement learning, often constrained by intricately designed reward functions tailored to a narrow set of tasks. In this work, we present a novel approach for efficiently learning diverse bimanual dexterous skills from abundant human demonstrations. Specifically, we introduce BiDexHD , a framework that unifies task construction from existing bimanual datasets and employs teacher-student policy learning to address all tasks. The teacher learns state-based policies using a general two-stage reward function across tasks with shared behaviors, while the student distills the learned multi-task policies into a vision-based policy. With BiDexHD, scalable learning of numerous bimanual dexterous skills from auto-constructed tasks becomes feasible, offering promising advances toward universal bimanual dexterous manipulation. Our empirical evaluation on the TACO dataset, spanning 141 tasks across six categories, demonstrates a task fulfillment rate of 74.59% on trained tasks and 51.07% on unseen tasks, showcasing the effectiveness and competitive zero-shot generalization capabilities of BiDexHD. For videos and more information, visit our project page .

## 1 Introduction

Bimanual manipulation is crucial and beneficial.
Humans use both hands to do manipulations like using scissors, tying shoelaces, or operating kitchen utensils.
The ability to manipulate objects with two hands is fundamental for everyday tasks, because with both hands, we can not only do some “symmetry" collaborative tasks like carrying a heavy box with two hands, but also do “asymmetry” tasks (Liu et al., ) like twisting a bottle cap, which means one hand acts as an auxiliary hand for stabilizing objects and the other acts as an operator.

With the rapid development of embodied artificial intelligence, robotic bimanual dexterous manipulation is getting more and more important in manufacturing, healthcare, agriculture, construction, and tertiary industry (Zhang et al., ).
This emphasizes the effective use of tools or manipulations over objects that are deformable or of irregular shapes, overcoming the limitations of low-DOF end-effectors like grippers. Moreover, it addresses complicated human-like hand-object interaction and collaboration.
Despite its significance, achieving proficient bimanual manipulation remains a substantial challenge because it severely struggles with the high-dimensional action space.
While a line of previous work (Grannen et al., ; Yu et al., ; Kataoka et al., ; Liu et al., ) primarily focuses on bimanual manipulation with grippers, there is still much left to explore for bimanual manipulation with dexterous hands.
Previous attempts to solve bimanual dexterous manipulation tasks are mainly based on reinforcement learning (RL) (Lin et al., ; Huang et al., ; Zhang et al., ). However, they require intricate reward designs tailored to specific manually-designed tasks. Therefore, these approaches lack scalability and generalizability to a broader range of tasks. Recent research (Sindhupathiraja et al., ; Fu et al., ; He et al., ) has advanced robotic bimanual dexterous manipulation through teleoperation. Nevertheless, human intervention is inevitable.
We would ask a question:

“Can we learn diverse bimanual dexterous manipulation skills in a unified and scalable way?”

Our solution is to use human demonstrations. Compared to robotic demonstrations, human demonstrations are relatively easier to obtain with haptic gloves or MoCap devices rather than deploying a trained policy, and contain much more physically compliant and human-aligned behavior.
In this paper, we propose a novel approach to learn diverse bimanual dexterous manipulation skills from human demonstrations. Upon this setting, we propose BiDexHD, a unified and scalable framework to automatically turn a human bimanual manipulation dataset into a series of tasks in the simulation and conduct effective policy learning.

BiDexHD does not depend on manually-designed tasks or pre-defined tasks in existing benchmarks. Instead, it consistently constructs feasible tasks from any bimanual manipulation trajectory. Furthermore, we are not required to design specific rewards for each task but instead utilize a unified reward function for reinforcement learning followed by policy distillation.
In a word, BiDexHD breaks the bottleneck of limited tasks and manual designs, which is significant to the further development of general-purpose bimanual dexterous manipulation.
Though promising, several challenges must be addressed to fully realize this. It is essential to figure out how to accurately mimic fine-grained bimanual behaviors from human demonstrations and avoid collisions and disturbances while encouraging smooth trajectories and synchronous collaboration between both hands.
To address this, we carefully design a general two-stage reward function to assign curricula for RL training.

To sum up, our key contributions can be summarized as follows:

- •
We formalize the problem of learning bimanual dexterous skills from human demonstrations as a preliminary attempt towards universal bimanual skills.
- •
We propose BiDexHD, a unified and scalable reinforcement learning framework for learning diverse bimanual dexterous manipulation from human demonstrations, advancing the capabilities of robots in performing bimanual cooperative tasks.
- •
We evaluate BiDexHD across 141 auto-constructed tasks over 6 categories from the TACO (Liu et al., ) dataset and demonstrate the superior training performance and competitive generalization capabilities of BiDexHD.

## 2 Related Work

### 2.1 Bimanual Dexterous Manipulation

In recent years, the robotics community has increasingly focused on dexterous manipulation due to its remarkable flexibility and human-like dexterity. Researchers have developed methods using dexterous hands for tasks such as in-hand manipulation (Arunachalam et al., ; Yin et al., ; Handa et al., ; Qi et al., ; Chen et al., ; ), grasping (Xu et al., ; Wan et al., ; Qin et al., ; Ye et al., ; Qin et al., ), and manipulating deformable objects (Bai et al., ; Ficuciello et al., ; Li et al., ; Hou et al., ). However, most existing work focuses on a single dexterous hand, revealing the potential of bimanual dexterity. In fact, for humans, bimanual collaboration takes place frequently in daily life such as riding, carrying heavy objects, and using tools. There are heterogeneous research directions towards bimanual dexterous manipulation. Some researchers attempt to settle down to specific tasks via reinforcement learning. For example, recent work (Lin et al., ) investigates twisting lids with two multi-fingered hands, DynamicHandover (Huang et al., ) explores throwing and catching, and ArtiGrasp (Zhang et al., ) focuses on a few grasping and articulation tasks. Gbagbe et al. () leveraged large language models to design a system for bimanual robotic dexterous manipulation, while Wang et al. () proposed to solve bimanual grasping via physics-aware iterative learning and prediction of saliency maps. A recent work (Gao et al., ) adopts keypoints-based visual imitation learning to learn bimanual coordination strategies. Unlike existing work, in this paper, we offer a general solution to learn from bimanual demonstrations by designing a unified reward function to learn a state-based policy via reinforcement learning and distilling it into a vision-based policy.

### 2.2 Learning Dexterity From Human Demonstrations

As a sample-efficient data-driven way, learning from human demonstrations has been proven successful in robot learning (Jia et al., ; Mandlekar et al., ; Odesanmi et al., ).
Compared with learning dexterity via reinforcement learning which is notoriously challenging for policy learning due to the high degrees of freedom and the necessity of manually designing task-specific reward functions, learning complex dexterous behaviors from diverse accessible human demonstrations (Smith et al., ; Schmeckpeper et al., ; Shao et al., ) is a more stable and scalable approach.
A line of previous studies (Arunachalam et al., ; Mandikal & Grauman, ; Sivakumar et al., ; Qin et al., ; Mandikal & Grauman, ; Liu et al., ; Shaw et al., ; Chen et al., ) explicitly leverages human demonstrations to facilitate the acquisition of dexterous manipulation skills mainly by human-robot-arm-hand retargeting and imitation learning. However, these studies predominantly focus on single-hand manipulation and are often limited to tasks such as in-hand manipulation (Arunachalam et al., ) or video-conditioned teleoperation (Sivakumar et al., ).
With the recent advent of diverse and comprehensive human bimanual manipulation datasets (Zhan et al., ; Liu et al., ; Fan et al., ; Razali & Demiris, ) which naturally provide a rich resource for high-quality posture sequences of dual hands and bimanual interaction with diverse real objects, a lot of bimanual manipulation tasks can be automatically defined. Thus, in this work, we aim to address more challenging and general bimanual dexterous skill learning purely based on automatically constructed tasks from human demonstrations.

## 3 Preliminaries

### 3.1 Task Formulation

Dec-POMDP. We formulate each bimanual manipulation task as a decentralized partially observable Markov decision process (Dec-POMDP). The Dec-POMDP can be represented
by the tuple $Z=\left(\mathcal{N},\mathcal{M},S,\bm{O},\bm{A},P,R,\rho,\gamma\right)$. Dual hands with arms are separated as $\mathcal{N}$ agents, which is represented by set $\mathcal{M}$. The proprioception of robots and the information about objects are initialized at $s_{0}\in S$ according to the initial state distribution $\rho(s_{0})$. At each time step $t$, the state is represented by $s_{t}$, and the $i$-th agent receives an observation $o_{t}^{i}\in\bm{O}$ based on $s_{t}$. Subsequently, the policy of the $i$-th agent, $\pi_{i}\in\bm{\Pi}$, takes $o_{t}^{i}$ as input and outputs an action $a_{t}^{i}\in A^{i}$. The joint action of all agents is denoted by $\bm{a}_{t}\in\bm{A}$, where $\bm{A}=A^{1}\times A^{2}\times\cdots A^{\mathcal{N}}$.
The state transits to the next state according to the transition function $s_{t+1}\sim P(s_{t+1}|s_{t},\bm{a}_{t})$. After this, the $i$-th agent receives a reward $r_{t}^{i}$ based on the reward function $R(s_{t},\bm{a}_{t})$.
The objective is to find the optimal policy $\bm{\pi}$ that maximizes the expected sum of rewards $\mathbb{E}_{\bm{\pi}}[\sum_{t=0}^{T-1}\gamma^{t}\sum_{i=1}^{\mathcal{N}}r_{t}^
{i}]$ over an episode with $T$ time steps, where $\gamma$ is the discount factor.

Environment Setups. The leftmost subgraph in Figure 2 illustrates the setups for each bimanual manipulation task in IsaacGym (Makoviychuk et al., ). In general, there are a tool and a target object initialized on a table.
$\mathcal{N}=2$ robotic arms are installed in front of the table, with left LEAP Hand (Shaw et al., ) mounted on the left arm and right hand on the right arm. The right hand reaches for the tool, and the left hand targets the object. Both hands coordinate to simultaneously move, pick up, and manipulate the objects above the table. Note that our method applies to all dexterous hand embodiments.
The observation space $\bm{O}$ contains robot proprioception and object information.
The left and right policies both output 22 joint angles normalized to $[-1,1]$, and the robots are controlled via position control. See more details in Appendix B.4.

Dataset Preparation. A human bimanual manipulation dataset consists of $M$ trajectories $\mathcal{D}=\{\tau^{1},\tau^{2},\dots,\tau^{M}\}$, each of which describes a human using a tool with his right hand to manipulate a target object with his left hand. The behavior of each trajectory can be recapped with a triplet (action, tool, object). Any triplet belongs to a union $\mathcal{U}=\mathcal{V}\times\Omega\times\Omega$, where $\Omega$ denotes the set of all objects and tools, and $\mathcal{V}$ denotes the set of all human actions. According to different behaviors depicted in $\mathcal{V}$, we can split all the tasks into $|\mathcal{V}|$ categories.
Each trajectory $\tau^{i}=\{\mathbf{h}^{\text{tool}},\mathbf{h}^{\text{object}},\hat{\mathbf{x}
}_{t}^{\text{tool}},\hat{\mathbf{q}}_{t}^{\text{tool}},\hat{\mathbf{x}}_{t}^{
\text{object}},\hat{\mathbf{q}}_{t}^{\text{object}},\Theta^{\text{left}}_{t},
\Theta^{\text{right}}_{t}\}^{i}_{t:1..N}$ involves a pair of meshes of the tool and object from a object mesh set $\mathbf{h}^{\text{tool}},\mathbf{h}^{\text{object}}\in\mathcal{H}$, $N$-step position $\mathbf{x}\in\mathbb{R}^{3}$ and orientation $\mathbf{q}\in\mathbb{R}^{4}$ sequence of the tool and the object, and the pose sequence of hands described in MANO (Romero et al., ) parameters $\Theta$.

### 3.2 Teacher-Student Learning

It is well known (Chen et al., ; ) that directly learning a multi-task vision-based policy for dexterous hands is extremely challenging. A more popular and scalable approach is teacher-student learning (Wan et al., ), which not only simplifies the complexity of multi-DoF robot multi-task learning but also enhances the efficiency of point cloud-based policy learning. In the teacher learning phase, a single state-based policy is first trained via reinforcement learning, leveraging privileged information to solve multiple similar tasks. Once trained, multiple teacher policies can effectively tackle all tasks.
In the student learning phase, a vision-based student policy is distilled from the teacher policies. A key distinction between teacher and student observations is how object information is represented. While the teacher’s observation space includes precise details about an object’s position, orientation, and linear and angular velocities, the student’s observation relies on point clouds consisting of $P$ sampled points from the object’s surface mesh. In this way, the learned student policy is promising to be deployed in real world to deal with multiple tasks provided that the real point clouds can be constructed from real-time multi-view RGB-D camera system.

## 4 Learning Bimanual Dexterity From Human Demonstrations

### 4.1 Overview

As illustrated in Figure 1, we propose a scalable three-phase framework. In the first phase, we parallelize the construction of Dec-POMDP bimanual tasks from a human bimanual manipulation dataset within IsaacGym (Makoviychuk et al., ). After task initialization, the subsequent two phases adopt a teacher-student policy learning framework. Following the approach of Chen et al. (; ); Wan et al. (), we utilize Independent Proximal Policy Optimization (IPPO) (De Witt et al., ) during the second phase to independently train state-based teacher policies for constructed bimanual dexterous tasks in parallel. Each expert focuses on a subset of tasks that require similar behaviors. In the final phase, the teacher policies are distilled into a vision-based student policy, integrating skills across related tasks.

Figure: Figure 1: The three-phase framework, BiDexHD, unifies constructing and solving tasks from human bimanual datasets instead of existing benchmarks. In phase one, BiDexHD constructs each bimanual task from a human demonstration. In phase two, BiDexHD learns diverse state-based policies from a generally designed two-stage reward function via multi-task reinforcement learning. A group of learned policies are then distilled into a vision-based policy for inference in phase three.
Refer to caption: x1.png

### 4.2 Task Construction From Bimanual Dataset

In this work, we primarily focus on bimanual tool using tasks.
Recent datasets (Liu et al., ; Zhan et al., ; Fan et al., ; Razali & Demiris, )
capture a wide range of bimanual cooperative behaviors, involving the use of tools to manipulate objects via motion capture and 3D scanning. The rich data, including object pose trajectories and hand-object interaction postures, provides sufficient information to construct feasible bimanual tasks. The task construction process from the bimanual dataset involves data preprocessing and simulation initialization.

Data Preprocessing. We extract the wrist and fingertip pose of dual hands at each timestep $\{V^{\text{side}}_{t},J^{\text{side}}_{t}\}=\text{MANO}(\Theta^{\text{side}}_{
t}),\text{ side}\in\{\text{left, right\}}$ with MANO (Romero et al., ) parameters $\Theta=\{\alpha,\beta,\hat{\mathbf{x}}^{\text{w}}\}$, where $\alpha\in\mathbb{R}^{48},\beta\in\mathbb{R}^{10},\text{ and }\hat{\mathbf{x}}^
{\text{w}}\in\mathbb{R}^{3}$ represent hand pose, hand shape parameters, and wrist position respectively. $V\in\mathbb{R}^{778\times 3}$ and $J\in\mathbb{R}^{21\times 3}$ represent vertices and joints on a hand respectively. The quaternion of the wrist $\hat{\mathbf{q}}^{\text{w}}\in\mathbb{R}^{4}$ is translated from axis-angle $\beta_{0:3}$. Given that single LEAP Hand (Shaw et al., ) has only four fingers, we can easily filter the corresponding positions of these $m=4$ fingers in $J$, denoting them as $\mathbf{x}^{\text{ft}}\in\mathbb{R}^{m\times 3}$. In the following sections, $\tau^{i}$ is denoted as:

$$ $\tau^{i}=\{\hat{\mathbf{x}}_{t}^{\text{tool}},\hat{\mathbf{q}}_{t}^{\text{tool }},\hat{\mathbf{x}}_{t}^{\text{object}},\hat{\mathbf{q}}_{t}^{\text{object}}, \hat{\mathbf{x}}^{\text{lw}}_{t},\hat{\mathbf{q}}^{\text{lw}}_{t},\hat{\mathbf {x}}^{\text{rw}}_{t},\hat{\mathbf{q}}^{\text{rw}}_{t},\hat{\mathbf{x}}^{\text{ lft}}_{t},\hat{\mathbf{x}}^{\text{rft}}_{t}\}^{i}_{t:1..N}.$ (1) $$

Simulation Initialization. After data preprocessing, we can construct bimanual manipulation tasks $\varGamma=\{\mathcal{T}^{1},...,\mathcal{T}^{M}\}$ in Issac Gym in parallel. For each task $\mathcal{T}^{i}$, the mesh of a tool $\mathbf{h}^{\text{tool}}$ and a target object $\mathbf{h}^{\text{object}}$, along with two arms with hands are initialized with a fixed initial observation vector:

$$ $\displaystyle o_{0}^{\text{side}}=\{\left(\mathbf{j,v}\right)^{\text{side}}, \left(\mathbf{x,q}\right)^{\text{side,w}},\mathbf{x}^{\text{side,ft}},\left( \mathbf{x,q,v,w,}\text{id}\right)^{\text{obj}}\}_{0}^{\text{side}}$ (2) $\displaystyle\text{where \ }\text{side, obj}\in\{\left(\text{left, object} \right),\left(\text{right, tool}\right)\}.$ $$

The robot proprioception includes arm-hand joint angles and velocities, wrist poses, and fingertip positions, and object information includes object positions, orientations, linear and angular velocities, and a unique object identifier for multi-task learning. For all tasks, $(\mathbf{j},\mathbf{v})_{0}$ are all reset to zero. The initial states of wrist and fingertips are calculated with forward kinematics accordingly. Except identifiers, the initial observations for all tools and target objects keep unchanged.
It is worth noting that we assume the robot to be right-handed by default, i.e., the left hand handles the target object and the right hand handles the tool. For brevity, the repeated notation $\text{side, obj}\in\{\text{(left, object),(right, tool)\}}$ is omitted in subsequent sections.

To ensure the feasibility of each task, after initialization, we use retargeting optimizers (Qin et al., ) to map human hand motions to robot hand joint angles and solve inverse kinematics (IK) to determine the robot arm joint angles based on the robot’s palm base pose. By replaying all object-hand trajectories in the simulator, we can easily identify and remove invalid tasks to build up a complete task set $\varGamma$.

### 4.3 Multi-Task State-Based Policy Learning

In the second phase, we focus on learning a multi-task state-based policy for tasks that require similar behaviors. Broadly, these tasks can generally be divided into two stages: first, aligning the simulation state with initial $\tau^{i}_{0}$ of a trajectory, and second, following each step of the trajectory. During the alignment stage, both hands should prioritize approaching their objects as quickly as possible. The left hand learns to grasp or stabilize the target object, while the right hand learns to grasp the tool. Once simulation alignment is achieved, both hands are expected to maintain their hold and follow the pre-defined trajectory derived from the human demonstration dataset to perform the manipulations in sync. The pipeline is illustrated in Figure 2. We initialize objects and robots at stage zero, finish simulation alignment at stage one, and conduct trajectory tracking at stage two via IPPO to learn state-based policies $\pi_{\theta}^{\text{side}}(\mathbf{a}_{t}^{\text{side}}|o_{t}^{\text{side}},
\mathbf{a}_{t-1}^{\text{side}})$ conditioning on the current observation $o_{t}^{\text{side}}=\{\left(\mathbf{j,v}\right)^{\text{side}},\left(\mathbf{x,
q}\right)^{\text{side,w}},\mathbf{x}^{\text{side,ft}},\left(\mathbf{x,q,v,w,}
\text{id}\right)^{\text{obj}}\}_{t}^{\text{side}}$ and previously executed action $\mathbf{a}_{t-1}^{\text{side}}$ for dual hands.

Figure: Figure 2: General two-stage teacher learning. For each task $\mathcal{T}^{i}$, all joint poses are initialized at zero pose and a pair of tool-object are initialized at a fixed pose at stage zero. At stage one, approaching reward $r_{\text{appro}}$ encourages both hands to get close to their grasping centers $\hat{\mathbf{x}}_{\text{gc}}$, and lifting reward $r_{\text{lift}}$ along with extra bonus $r_{\text{bonus}}$ incentivizes moving both objects to thier reference poses respectively. After simulation alignment, dual hands will manipulate objects under the guidance of tracking reward $r_{\text{track}}$.
Refer to caption: x2.png

Stage 1: Simulation Alignment. The central goal of stage one is to align the state of simulation to the first step in a trajectory by moving the tool and target object from the fixed initial pose to $\tau_{0}$, which serves as an essential yet challenging prerequisite for subsequent trajectory tracking in stage two. Through experiments in Section 5.4, we find that it is not feasible to directly acquire dynamic skills from static poses through imitation. Instead, we adopt reinforcement learning to develop skills like grasping, twisting and pushing. Some previous work (Luo et al., ; Xu et al., ) on grasping prefers introducing additional pre-grasp poses by estimating grasping pose upon manipulated objects. We adopt a simpler but more generalizable approach by learning skills directly from the object poses provided in the dataset.
Specifically, we anchor the first timestep in the dataset as the reference timestep to establish a tool-object reference pose pair for each manipulation task. Stage one is considered complete once both the tool and the object reach the specified pose for a sustained $u$-step duration.
Rewards are carefully designed to encourage the object to be lifted above the table in reference to the filtered reference poses. The total reward consists of an approaching reward, a lifting reward, and a bonus reward.

The approaching reward, $r_{\text{appro}}$, encourages both dexterous hands
to approach and remain close to the object. In other words, the goal is to minimize the distance between the robot’s palm, fingertips, and the grasp center.
Since functional grasping is critical for tool using, we do not simply select the geometric center of the object. Instead, we pre-compute the grasping center $\hat{\mathbf{x}}_{\text{gc}}$ for each tool and object based on the dataset. Specifically, for each task, we use the human-demonstrated wrist and fingertip positions at the reference timestep–$\hat{\mathbf{x}}^{\text{lw}}_{0},\hat{\mathbf{x}}^{\text{rw}}_{0},\hat{\mathbf
{x}}^{\text{lft}}_{0},\hat{\mathbf{x}}^{\text{rft}}_{0}$–as anchor points. We then uniformly sample 1024 points from the surface of the object mesh $\mathbf{h}^{\text{tool}},\mathbf{h}^{\text{object}}$ to form a representative point set $\mathcal{P}$ and compute the average grasp center based on the top $L=50$ nearest points.
$r_{\text{appro}}$ penalizes the distance between the wrist, fingertips, and the grasp center, and is defined as

$$ $\begin{array}[]{l}r_{\text{appro}}^{\text{side}}=-\lVert\mathbf{x}_{t}^{\text{ side,w}}-\hat{\mathbf{x}}_{\text{gc}}^{\text{obj}}\rVert_{2}-w_{r}\sum^{m}{ \lVert\mathbf{x}_{t}^{\text{side,ft}}-\hat{\mathbf{x}}_{\text{gc}}^{\text{obj} }\rVert_{2}}\\ \text{where \ }\hat{\mathbf{x}}_{\text{gc}}^{\text{obj}}=\frac{1}{L}\sum{\text {NN}}\left(\mathcal{P},L,\frac{\hat{\mathbf{x}}_{0}^{\text{side,w}}+\sum^{m} \hat{\mathbf{x}}_{0}^{\text{side,ft}}}{m+1}\right).\end{array}$ (3) $$

The lifting reward $r_{\text{lift}}$ encourages holding objects tightly in hands and lifting to desired reference poses. As long as the lifting conditions are satisfied, the robots receive a lifting reward $r_{\text{lift}}$ composed of a non-negative linear position reward and a negative quaternion distance reward,

$$ $\begin{array}[]{l}r_{\text{lift}}^{\text{side}}=\left\{\begin{array}[]{l}r_{ \text{pos}}^{\text{side}}+w_{q}r_{\text{quat}}^{\text{side}}\quad\ \text{if\, \,}\mathbb{I}\left(\lVert\mathbf{x}_{t}^{\text{side,w}}-\hat{\mathbf{x}}_{ \text{gc}}^{\text{obj}}\rVert_{2}\leq\lambda_{\text{w}}\cap\sum^{m}{\lVert \mathbf{x}_{t}^{\text{side,ft}}-\hat{\mathbf{x}}_{\text{gc}}^{\text{obj}} \rVert_{2}\leq\lambda_{\text{ft}}}\right)\\ 0\qquad\qquad\qquad\ \ \text{otherwise}\\ \end{array}\right.\\ \text{where \ }r_{\text{pos}}^{\text{side}}=\max\left(1-\frac{\lVert\mathbf{x} _{t}^{\text{obj}}-\hat{\mathbf{x}}_{0}^{\text{obj}}\rVert_{2}}{\lVert\mathbf{x }_{0}^{\text{obj}}-\hat{\mathbf{x}}_{0}^{\text{obj}}\rVert_{2}},0\right),\ \ r _{\text{quat}}^{\text{side}}=-\mathbb{D}_{\text{quat}}\left(\mathbf{q}_{t}^{ \text{obj}},\hat{\mathbf{q}}_{0}^{\text{obj}}\right).\end{array}$ (4) $$

Here, $\mathbf{x}_{0}^{\text{object}}$ and $\mathbf{x}_{0}^{\text{tool}}$ respectively represent the initial positions of the target object and tool in the simulator, while $\hat{\mathbf{x}}_{0}$ denotes the first reference position in a human demonstration.

The bonus reward $r_{\text{bonus}}$ incentivizes the target object or the tool to reach and finally stay at their reference poses, which lays a foundation for the second manipulation stage. $r_{\text{bonus}}$ becomes positive only when the distance between an object’s current position and its reference position becomes lower than $\varepsilon_{\text{succ}}$. Stage one is considered successful only if both $r_{\text{bonus}}^{\text{left}}$ and $r_{\text{bonus}}^{\text{right}}$ are positive for at least $u$ consecutive steps. Thus, the bonus reward $r_{\text{bonus}}$ is defined as

$$ $\begin{array}[]{c}r_{\text{bonus}}^{\text{side}}=\left\{\begin{array}[]{l} \frac{1}{1+\lVert\mathbf{x}_{t}^{\text{obj}}-\hat{\mathbf{x}}_{0}^{\text{obj}} \rVert_{2}}\qquad\text{if\,\,}\mathbb{I}\left(\lVert\mathbf{x}_{t}^{\text{obj} }-\hat{\mathbf{x}}_{0}^{\text{obj}}\rVert_{2}\leq\varepsilon_{\text{succ}} \right)\\ 0\qquad\qquad\qquad\quad\text{otherwise}.\\ \end{array}\right.\end{array}$ (5) $$

The total alignment reward is the linear weighted sum of the three components.

$$ $r_{\text{align}}^{\text{side}}=w_{1}r_{\text{appro}}^{\text{side}}+w_{2}r_{ \text{lift}}^{\text{side}}+w_{3}r_{\text{bonus}}^{\text{side}}$ (6) $$

Stage 2: Trajectory Tracking.
Once stage one is completed, the left hand is securely holding the target object, and the right hand keeps grasping the tool at its desired reference pose. The next step is to maintain the grasp and follow a trajectory to perform the manipulation. To achieve this, we design a more fine-grained exponential reward, $r_{\text{track}}$, which encourages the dexterous hands to precisely track the desired positions at each timestep in a trajectory starting from the reference timestep.
Assuming that human hands are more flexible than robotic hands, we introduce a constant tracking frequency $f$, where $f$ simulation steps correspond to one step in the dataset. Let $\mathbf{\hat{x}}_{i}^{\text{obj}}$ represent the position of a object at $i$-th step in a $l$-step
human-demonstrated trajectory and $\mathbf{x}_{t_{i}}^{\text{obj}}$ represent the object’s position at the corresponding simulation step in IsaacGym. We have $i=\lceil{t_{i}}/{f}\rceil\in[0,l)$, and the tracking reward is defined as

$$ $r_{\text{track}}^{\text{side}}=\left\{\begin{array}[]{l}\exp\left(-w_{t}\lVert \mathbf{x}_{t_{i}}^{\text{obj}}-\hat{\mathbf{x}}_{i}^{\text{obj}}\rVert_{2} \right)\quad\,\,\,\text{ if stage 1 succeeds}\\ 0\qquad\qquad\qquad\qquad\qquad\qquad\text{otherwise}.\\ \end{array}\right.$ (7) $$

We adopt IPPO to learn a unified policy from the combination of all rewards for the two stages,

$$ $r_{\text{total}}^{\text{side}}=r_{\text{align}}^{\text{side}}+w_{4}r_{\text{ track}}^{\text{side}}.$ (8) $$

$r_{\text{total}}$ unifies two stages of bimanual dexterous manipulation, enabling scaling up to multi-task policy learning for a wide range of constructed bimanual tasks.

### 4.4 Vision-Based Policy distillation

We employ DAgger (Ross et al., ), an on-policy imitation learning algorithm, to develop a vision-based policy for each task category $\nu\in\mathcal{V}$, under the supervision of a group of state-based teacher policies. To enhance generalization capabilities for new objects or unseen tasks, we propose transforming the student policy into a trajectory-conditioned in-context policy, denoted as $\pi_{\phi}^{\text{side}}(\mathbf{a}_{t}^{\text{side}}|\mathbf{o}_{t}^{\text{
side}},\mathbf{p}^{\text{side}}_{t},\mathbf{a}_{t-1}^{\text{side}})$, where $\mathbf{o}_{t}=\{(\mathbf{j},\mathbf{v})^{\text{side}},(\mathbf{x},\mathbf{q})
^{\text{side,w}},\mathbf{x}^{\text{side,ft}},\text{pc}^{\text{obj}}\}_{t}$, $K$-step future pose $\mathbf{p}^{\text{side}}_{t}\in\mathbb{R}^{K\times 3}$, and $\text{pc}_{t}^{\text{obj}}\in\mathbb{R}^{P\times 3}$. Specifically, to get point clouds $\text{pc}_{t}^{\text{tool}}$ and $\text{pc}_{t}^{\text{object}}$, we pre-sample 4096 points from the surface of $\mathbf{h}^{\text{tool}}$ and $\mathbf{h}^{\text{object}}$ for each task during initialization. At each timestep $t$, a subset of points are sampled from the pre-sampled point clouds, transformed according to current object pose and added with Gaussian noise for robustness.
Besides, it is important to note that during DAgger distillation, we augment traditional vision-based policy $\pi_{\phi}^{\text{side}}(\mathbf{a}_{t}^{\text{side}}|\mathbf{o}_{t}^{\text{
side}},\mathbf{a}_{t-1}^{\text{side}})$ with next $K$ positions along the object’s trajectory as additional inputs.
This design allows the learned policy to utilize more information about the motion of objects, such as movement direction and speed in the near future, facilitating zero-shot transfer to unfamiliar tasks or objects. Notably, we can easily mask this additional input by setting $K=0$. We further investigate the influence of $K$ future steps in Section 5.4.
The whole teacher-student training process is summarized in Appendix A. More implementation details can be found in Appendix B.

## 5 experiments

### 5.1 Setups

Dataset. We evaluate the effectiveness of BiDexHD on the TACO (Liu et al., ) dataset, a large-scale bimanual manipulation dataset that encompasses diverse human demonstrations using tools to manipulate target objects in real-world scenarios.
BiDextHD converts 6 categories $\mathcal{V}=\{\text{Dust, Empty, Pour in some, Put out, Skim off, Smear}\}$ of total 141 human demonstrations in the TACO dataset to Dec-POMDP tasks (See Appendix D for task examples). Task diversity and abundance make BiDexHD easy to scale up. All tasks can be separated into 16 semantic groups, each of which gathers a number of similar demonstrations with the same action, the same tool-object category but different tool and object instances.
BiDextHD constructs a task from single demonstration, and thus each semantic group correspond to a semantic subtask. We adopt teacher-student learning to train 16 semantic sub-tasks and distill teacher policies with similar skills into 6 vision-based policies for each category eventually.

To evaluate the effectiveness of the framework as well as the generalizability of the learned policies, we split 80% tasks for training (Train) and the rest 20% unseen tasks for testing. Detailed descriptions of dataset split are provided in Appendix B.2. For each task in the testing set, if the object and tool both occur in the training set it is labeled as a kind of combinational task (Test Comb), and otherwise it is labeled as a new task (Test New).

Metrics. To measure the quality of our constructed tasks, we introduce two metrics $r_{1}$ and $r_{2}$.

- •
The first is the average success rate $r_{1}$ of stage one. For a number of $n$ episodes, $r_{1}=\frac{1}{n}\sum^{n}_{e=1}\mathbb{I}_{1}^{e}$ averages over the number of episodes that satisfys conditions $\mathbb{I}_{1}$ at stage one.
$\mathbb{I}_{1}:\quad\exists 0<t<T-u\quad\sum_{t}^{t+u}\mathbb{I}\left(\lVert
\mathbf{x}_{t}^{\text{object}}-\hat{\mathbf{x}}_{0}^{\text{object}}\rVert_{2}
\leq\varepsilon_{\text{succ}}\cap\lVert\mathbf{x}_{t}^{\text{tool}}-\hat{
\mathbf{x}}_{0}^{\text{tool}}\rVert_{2}\leq\varepsilon_{\text{succ}}\right)=u$
- •
The second is the average tracking rate $r_{2}$ of stage two. Each task corresponds to $l$-step human-demonstrated trajectory. For each episode, calculate the proportion of steps where two objects both effectively follows their desired poses. $r_{2}$ is the average tracking rate over $n$ episodes.
$r_{2}=\frac{1}{nl}\sum^{n}\sum^{l-1}_{i=0}\mathbb{I}\left(\lVert\mathbf{x}_{t_
{i}}^{\text{object}}-\mathbf{x}_{i}^{\text{object}}\rVert_{2}\leq\varepsilon_{
\text{track}}\cap\lVert\mathbf{x}_{t_{i}}^{\text{tool}}-\mathbf{x}_{i}^{\text{
tool}}\rVert_{2}\leq\varepsilon_{\text{track}}\right)$

It is important to note that $r_{2}$ serves as the primary metric for indicating task completion while $r_{1}$ is an intermediate metric for assessing task progression. Considering the choice of $\varepsilon_{\text{succ}}$ and $\varepsilon_{\text{track}}$ has a non-legligible impact for the reported results, we will discuss the sensitivity of these thresholds in Section 5.4. By default, we choose $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.1$ for evaluation.

### 5.2 Teacher Learning

Upon the framework of BiDexHD, different base RL algorithms can be incorporated. We mainly compare the performance of independent PPO (BiDexHD-IPPO) and centralized PPO (BiDexHD-PPO). For BiDexHD-IPPO, two agents possess their own observations and execute their own actions. For BiDexHD-PPO, a single policy takes as input both observations and is trained to output all actions that maximize the sum of all total rewards in an episode, which essentially transforms a Dec-POMDP task into a POMDP task.

**Table 1: The average success rate of stage 1 and tracking rate of stage 2 during training and evaluation across all tasks constructed from the TACO dataset under $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.1$.**
| Method | Train<br>$r_{1}$(%) | Train<br>$r_{2}$(%) | Test Comb<br>$r_{1}$(%) | Test Comb<br>$r_{2}$(%) | Test New<br>$r_{1}$(%) | Test New<br>$r_{2}$(%) |
| --- | --- | --- | --- | --- | --- | --- |
| BiDexHD-PPO | 90.55 | 53.88 | 78.74 | 36.99 | 81.42 | 26.24 |
| BiDexHD-IPPO (w/o stage-1) | 25.00 | 17.52 | 24.80 | 18.10 | 19.85 | 08.51 |
| BiDexHD-IPPO (w/o gc) | 90.53 | 66.39 | 91.47 | 52.11 | 77.03 | 22.63 |
| BiDexHD-IPPO (w/o bonus) | 97.67 | 66.65 | 98.01 | 59.76 | 77.96 | 17.52 |
| BiDexHD-IPPO | 98.71 | 78.18 | 98.37 | 59.94 | 75.48 | 21.34 |
| BC | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 |
| BiDexHD-PPO+DAgger | 95.35 | 55.82 | 76.75 | 30.42 | 86.34 | 30.00 |
| BiDexHD-IPPO+DAgger | 99.38 | 74.59 | 92.85 | 48.43 | 94.79 | 53.71 |

RL Results.
The first and last rows in the green section of Table 1 present the average performance across all auto-constructed bimanual tasks. For tasks with seen objects (Train and Test Comb), BiDexHD-IPPO nearly completes stage 1 by successfully reaching the reference poses and maintaining high-quality tracking during stage 2, which demonstrates its impressive scalability across diverse tasks in the TACO dataset. In contrast, BiDexHD-PPO underperforms compared to BiDexHD-IPPO, particularly on tasks with seen objects. This discrepancy arises because BiDexHD-IPPO is more efficient at acquiring robust skills within limited updates by independently learning left and right policies across a wide range of tasks with smaller observation and action spaces. Furthermore, two independent expert policies focusing solely on specific groups of target objects or tools adapt more easily to similar combinational tasks than a single policy that must attend to both. Consequently, we select IPPO as our base RL algorithm. See Appendix C.1 for detailed evaluation results.

When applied to tasks with new objects (Test New), both BiDexHD-IPPO and BiDexHD-PPO experience a noticeable performance decline. The primary reason for this drop is that these approaches incorporate one-hot object labels in observations during state-based training, leading the policy to heavily rely on this information. As a result, during evaluation, the introduction of new labels disrupts decision-making. Therefore, we remove one-hot object labels during policy distillation to enhance generalization.

### 5.3 Ablations on Teacher Learning

We conduct ablation studies focusing on the key designs at stage one during teacher learning.

Alignment Stage.
To demonstrate the necessity of the design of dataset-simulation alignment stage, we compare BiDexHD-IPPO with a more naive version, denoted as (w/o stage-1), which retains only $r_{\text{track}}$ in RL training at stage 2 and maintains a fixed number of free exploration steps at stage 1. The second line in the green section of Table 1 reveals a significant performance decline. We observe that only 30.5% of relatively easy tasks (See Appendix C.1 for details) achieve positive $r_{1}$ and $r_{2}$, while for the remaining tasks, the success rate of stage 1 and the tracking rate of stage 2 remain at zero. This emphasizes the importance of $r_{\text{align}}$ during stage 1.
Functional Grasping Center.
In BiDexHD, we pre-compute the grasping center $\hat{\mathbf{x}}_{\text{gc}}$ to calculate $r_{\text{appro}}$ in Equation 3. In this section, we explore replacing the grasping center with the geometric center of an object, denoted as (w/o gc). The results presented in the third line of Table 1 show a decrease in $r_{1}$ and $r_{2}$, particularly on tasks involving seen objects compared to BiDexHD-IPPO. To further figure out their discrepancy in behavior, we deploy both policies for inference and visualize their grasping poses for a typical task (dust, brush, pan) in Figure 3. BiDexHD-IPPO tends to align more closely with the calculated grasping centers (red points), exhibiting human-like grasping behavior. In contrast, BiDexHD-IPPO (w/o gc) with geometric centers (green points) struggles to find proper poses for using the brush or holding the pan. In fact, the geometric center of an object does not often fall within areas suitable for manipulation. These findings highlight the significance of incorporating a functional grasping center, particularly for objects that are thin, flat, or equipped with handles.
Figure 3: A comparison of grasping pose during policy deployment between BiDexHD-IPPO (w/o gc) and BiDexHD-IPPO.

Success Bonus.
The fourth line in the green section of Table 1 investigates whether removing reward $r_{\text{bonus}}$ defined in Equation 5 will affect performance. We observe a decline in $r_{2}$ on both the training set and unseen tasks involving new objects. We analyze the additional bonus in Equation 5 effectively signals the transition between the two stages, enhancing the policy’s awareness of task progression.

### 5.4 Student Learning

For the BiDexHD variants, several trained multi-task state-based teacher policies from one task category are distilled into a single vision-based policy, which is then tested on all tasks. We also introduce behavior cloning (BC) as our baseline. To directly learn bimanual skills from a dataset, we employ Dexpilot (Handa et al., ) to retarget human hand motions in the TACO dataset to joint angles for dexterous hands, solving inverse kinematics (IK) for arm joint angles. All joint angles are collected and replayed in IsaacGym (Makoviychuk et al., ) to gather observations. BC learns purely from this static observation-action dataset and is ultimately tested under the same configuration as BiDexHD.

DAgger Results.
The blue section of Table 1 displays the performance of the vision-based policies. Our BiDexHD-IPPO+DAgger significantly outperforms both PPO variant and BC, achieving a high task completion rate on the training set and an average $r_{2}=51.07\%$ across all unseen tasks (Test Comb and Test New). This evidence indicates the scalability and competitive generalization ability of BiDexHD framework. Among unseen tasks, we observe a slight decline in $r_{2}$ for combinational tasks, while tasks involving new objects show a sharp increase in $r_{2}$. This suggests that the vision-based policy relies more on information from the point clouds, such as shape and local features, rather than specific one-hot identifiers, enabling effective zero-shot generalization. Conversely, BC performs poorly due to the loss of true dynamics in the simulation, often getting confused by unfamiliar observations and stuck in stationary states. This also reflects the challenges associated with our constructed bimanual tasks. Our framework unifies bimanual skill learning through a combination of trial-and-error and distillation, providing a robust and scalable solution to diverse challenging bimanual manipulation tasks. See Appendix C.2 for detailed evaluation results.

**Table 2: The metrics of different $K$ future steps under $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.1$.**
| Metrics (%) | $K$ |  |  |  |
| --- | --- | --- | --- | --- |
| 0 | 1 | 2 | 5 |  |
| Train $r_{1}$ | 98.01 | 98.81 | 98.71 | 99.38 |
| Train $r_{2}$ | 72.09 | 75.40 | 75.01 | 74.59 |
| Test Comb $r_{1}$ | 94.36 | 92.11 | 93.26 | 92.85 |
| Test Comb $r_{2}$ | 46.64 | 49.02 | 48.60 | 48.43 |
| Test New $r_{1}$ | 93.96 | 94.67 | 94.38 | 94.79 |
| Test New $r_{2}$ | 49.27 | 51.00 | 50.39 | 53.71 |

Future Conditioned Steps.
We further examine the selection of $K\in\{0,1,2,5\}$ for future object positions. Specifically, when $K=0$, the vision-based policy relies exclusively on 3D information from object point clouds and the robot’s proprioception. As shown in Table 3, the performance across different values of $K$ does not vary significantly.
Even when future conditioned steps are masked $(K=0)$, $r_{2}$ only exhibits slight declines of 2.5% on trained tasks and an average of 3.1% on all unseen tasks compared to $K=5$.
This evidence suggests that after the multi-task RL training phase, the teachers have acquired diverse and robust skills, making pure imitation sufficient for a student to achieve acceptable performance. Nonetheless, $K$ future steps provide additional informative and fine-grained, albeit implicit, clues such as motion and intention for more precise tracking.

Discussion.
To investigate the impact of different thresholds on the metrics, we re-evaluate all tasks and report the performance of our BiDexHD-IPPO+DAgger under varying thresholds, $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=\varepsilon\in\{0.05,0.07
5,0.1\}$ in Table 3. Notably, stricter metrics have a more pronounced impact on the performance of unseen tasks compared to trained ones, underscoring the challenges of continuous spatial-temporal trajectory tracking in bimanual manipulation tasks. We will focus on addressing more precise behavior tracking in future work.

## 6 Conclusion & Limitations

In this paper, we introduce a novel approach to learning diverse bimanual dexterous manipulation skills that utilizes human demonstrations. Our framework, BiDexHD, automatically constructs bimanual manipulation tasks from existing datasets and employs a teacher-student learning approach for a vision-based policy that can tackle similar tasks. Our main technical contributions include designing a unified two-stage reward function for multi-task RL training and an in-context vision-based policy that enhances generalization capabilities. Experimental results demonstrate that BiDexHD facilitates robust RL training and policy distillation, successfully solves six categories of bimanual dexterous manipulation tasks, and effectively transfers to unseen tasks through zero-shot generalization.

Our work forwards a step toward universal bimanual manipulation skills, and some limitations need to be addressed in future research. Exploring strategies for achieving more precise spatial and temporal tracking is a valuable direction for future work. Additionally, incorporating a wider variety of real-world tasks–such as deformable object manipulation and bimanual handover–could reveal further potential in dynamic collaborative manipulation scenarios with bimanual dexterous hands.

## Appendix A Algorithm

Figure: Algorithm 1 BiDexHD framework.

## Appendix B Implementation Details

### B.1 DataSet Preprocessing

Reference Timestep.
Considering there are a number of useless preparation timesteps before grasping, the reference timestep in Section 4.2 is actually chosen based on the first sudden change of the distance between an object and a tool, because the distance between the tool and object almost stays unchanged before grasping.

More Details.
We further align the coordinates of human wrist to the coordinates of robot palm base to ensure the same dual-hand manipulation behavior.
Besides, due to the geometric discrepancy of objects, we found that the initial height of objects differ a lot in different tasks. Therefore, a translation offset in z-axis is added to all poses in the dataset to keep all the object at the same initial height on the same table.

### B.2 Constructed Tasks

Task Composition. Table 4 describes the detailed task categories, sub-task names, the split of training and testing set and the diversity of tools and target objects.

**Table 4: 141 constructed tasks across 6 categories for BiDexHD. “All” refers to the total number of a kind of sub-task. “Train” refers to the number of tasks in the training set. “Test Comb” and “Test New” refer to the number of tasks in two types of testing sets. “Tool” and “Object” refer to number of objects in the corresponding sub-tasks.**
| Action | Sub-Task Name | All | Train | Test Comb | Test New | Tool | Object |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Empty | (empty, bowl, bowl) | 10 | 7 | 1 | 2 | 5 | 5 |
|  | (empty, bowl, plate) | 34 | 26 | 1 | 7 | 8 | 9 |
|  | (empty, cup, plate) | 1 | 1 | 0 | 0 | 1 | 1 |
|  | (empty, teapot, plate) | 12 | 8 | 1 | 3 | 2 | 6 |
|  | (empty, teapot, teapot) | 3 | 3 | 0 | 0 | 2 | 2 |
| Pour in some | (pour in some, cup, cup) | 1 | 1 | 0 | 0 | 1 | 1 |
|  | (pour in some, cup, plate) | 2 | 2 | 0 | 0 | 2 | 2 |
|  | (pour in some, cup, teapot) | 1 | 1 | 0 | 0 | 1 | 1 |
|  | (pour in some, teapot, bowl) | 1 | 1 | 0 | 0 | 1 | 1 |
|  | (pour in some, teapot, cup) | 2 | 1 | 0 | 1 | 2 | 2 |
| Dust | (dust, brush, bowl) | 20 | 5 | 0 | 15 | 5 | 9 |
|  | (dust, brush, pan) | 9 | 6 | 3 | 0 | 4 | 3 |
| Put out | (put out, bowl, bowl) | 10 | 7 | 2 | 1 | 5 | 5 |
|  | (put out, bowl, plate) | 16 | 11 | 1 | 4 | 3 | 8 |
| Skim off | (skim off, bowl, plate) | 17 | 12 | 0 | 5 | 5 | 8 |
| Smear | (smear, glue gun, plate) | 2 | 2 | 0 | 0 | 1 | 2 |
| Total | – | 141 | 94 | 9 | 38 | – | – |

### B.3 Dexterous Hands

Currently we use LEAP Hands (Shaw et al., ). In future work we will introduce more kinds of dexterous hands.

### B.4 Simulation Setup

Two 6-DOF RealMan arms, spaced 0.68 meters apart, are placed in front of a 0.7m table. 16-DOF LEAP Hands are Shaw et al. () mounted on both the left and right arms, with an initial stretching pose. The tool and target object are spaced 0.4m apart horizontally, 0.5m distant from the robotic arm base.

### B.5 Training Details

DAgger Details. To make the student policy learn more efficiently especially at the early training stage, we mix a few imitation samples to DAgger buffer. Specifically, we choose to use the actions labelled by the experts with probability $p=0.05$ and otherwise actions output by the policy itself. This is often proven desirable in practice, as the naive policy may make more mistakes and visit states that are irrelevant at the early training stage with relatively few datapoints (Ross et al., ).

Hyperparameters.
Table 5 and 6 outlines the hyperparameters for IPPO and DAgger in BiDexHD respectively.

**Table 5: Hyperparameters of IPPO**
| Hyperparameter | Value |
| --- | --- |
| $w_{r}$ | 2.0 |
| $w_{t}$ | 15.0 |
| $w_{1}$ | 0.5 |
| $w_{2}$ | 1.0 |
| $w_{3}$ | 1.0 |
| $w_{4}$ | 1.0 |
| $\lambda_{\text{w}}$ | 0.12 |
| $\lambda_{\text{ft}}$ | 0.48 |
| Episode Length | 1000 |
| Parallel rollout steps per iteration | 8 |
| Training epochs per iteration | 5 |
| Discount factor | 0.96 |
| GAE lambda | 0.95 |
| Clip range | 0.2 |
| Optimizer | AdamW |
| Learning Rate | 3e-4 |
| Number of Environments | 15000 |
| Type of GPUs | A100, or Nvidia RTX 4090 Ti |

**Table 6: Hyperparameters of DAgger**
| Hyperparameter | Value |
| --- | --- |
| $P$ | 512 |
| $K$ | 5 |
| Parallel rollout steps per iteration | 8 |
| Training epochs per iteration | 5 |
| Optimizer | AdamW |
| Learning Rate | 3e-4 |
| Number of Environments | 5000 |
| Type of GPUs | A100, or Nvidia RTX 4090 Ti |

### B.6 Model Architecture

Our codebase for RL and DAgger is built upon UniDexGrasp++ (Wan et al., ). For each state-based policy, we employ five-layer multi-layer perceptrons (MLPs) for both the actor and the critic, featuring hidden layers with dimensions [1024, 1024, 512, 512] and using ELU activation functions. For the vision-based policy, we utilize a simplified PointNet (Qi et al., ) backbone that incorporates two 1D convolutional layers, a mixture of maximum and average pooling operations, and two MLP layers to process the object point cloud, resulting in an output dimension of 128. Both the actor and the critic share the output of this backbone.

### B.7 Computation Resources

we train a state-based IPPO policy for single sub-tasks for around two days, and distill teacher policies into a vision-based policy for each action category for around one day on single 40G A100 GPUs. All the evaluations are done on a 24G Nvidia RTX 4090 Ti GPU for about half an hour.

## Appendix C Evaluation Results

### C.1 Results of Teacher Learning

Tables below record the detailed evaluation results for each sub-task. ‘–’ represents the absence of testing tasks. The last row of each table shows the average results over all sub-tasks.

**Table 7: Detailed Metrics of BiDexHD-PPO for each sub-task under $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.1$.**
| Action | Sub-Task Name | Train<br>$r_{1}$(%) | Train<br>$r_{2}$(%) | Test Comb<br>$r_{1}$(%) | Test Comb<br>$r_{2}$(%) | Test New<br>$r_{1}$(%) | Test New<br>$r_{2}$(%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Empty | (empty, bowl, bowl) | 99.38 | 69.28 | 99.30 | 62.24 | 93.64 | 53.13 |
|  | (empty, bowl, plate) | 97.88 | 50.38 | 100.00 | 35.93 | 65.91 | 8.95 |
|  | (empty, cup, plate) | 77.03 | 33.41 | – | – | – | – |
|  | (empty, teapot, plate) | 100.00 | 82.09 | 71.83 | 23.62 | 96.84 | 28.04 |
|  | (empty, teapot, teapot) | 100.00 | 27.89 | 42.42 | 7.63 | – | – |
| Pour in some | (pour in some, cup, cup) | 25.50 | 22.71 | – | – | – | – |
|  | (pour in some, cup, plate) | 99.88 | 82.16 | – | – | – | – |
|  | (pour in some, cup, teapot) | 100.00 | 69.69 | – | – | – | – |
|  | (pour in some, teapot, bowl) | 99.56 | 5.85 | – | – | – | – |
|  | (pour in some, teapot, cup) | 85.48 | 31.26 | – | – | – | – |
| Dust | (dust, brush, bowl) | 99.61 | 77.70 | – | – | 78.57 | 17.54 |
|  | (dust, brush, pan) | 73.68 | 48.22 | 41.58 | 15.69 | – | – |
| Put out | (put out, bowl, bowl) | 91.54 | 39.80 | 100.00 | 46.25 | 67.74 | 5.48 |
|  | (put out, bowl, plate) | 99.57 | 69.75 | 96.08 | 67.54 | 81.51 | 34.51 |
| Skim off | (skim off, bowl, plate) | 99.68 | 74.59 | – | – | 85.76 | 36.01 |
| Smear | (smear, glue gun, plate) | 100.00 | 77.26 | – | – | – | – |
| Average | – | 90.55 | 53.88 | 78.74 | 36.99 | 81.42 | 26.24 |

**Table 8: Detailed Metrics of BiDexHD-IPPO for each sub-task under $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.1$.**
| Action | Sub-Task Name | Train<br>$r_{1}$(%) | Train<br>$r_{2}$(%) | Test Comb<br>$r_{1}$(%) | Test Comb<br>$r_{2}$(%) | Test New<br>$r_{1}$(%) | Test New<br>$r_{2}$(%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Empty | (empty, bowl, bowl) | 99.74 | 79.02 | 97.67 | 37.55 | 98.34 | 48.72 |
|  | (empty, bowl, plate) | 97.90 | 75.76 | 100.00 | 62.13 | 85.00 | 9.10 |
|  | (empty, cup, plate) | 99.84 | 79.47 | – | – | – | – |
|  | (empty, teapot, plate) | 100.00 | 84.29 | 90.91 | 16.84 | 100.00 | 25.80 |
|  | (empty, teapot, teapot) | 100.00 | 84.87 | 100.00 | 87.06 | – | – |
| Pour in some | (pour in some, cup, cup) | 100.00 | 99.70 | – | – | – | – |
|  | (pour in some, cup, plate) | 89.78 | 55.95 | – | – | – | – |
|  | (pour in some, cup, teapot) | 99.47 | 74.43 | – | – | – | – |
|  | (pour in some, teapot, bowl) | 100.00 | 75.48 | – | – | – | – |
|  | (pour in some, teapot, cup) | 95.76 | 57.23 | – | – | – | – |
| Dust | (dust, brush, bowl) | 100.00 | 91.24 | – | – | 84.34 | 32.10 |
|  | (dust, brush, pan) | 100.00 | 86.08 | 100.00 | 58.09 | – | – |
| Put out | (put out, bowl, bowl) | 100.00 | 75.09 | 100.00 | 72.92 | 81.25 | 17.87 |
|  | (put out, bowl, plate) | 96.98 | 77.97 | 100.00 | 84.97 | 29.41 | 7.03 |
| Skim off | (skim off, bowl, plate) | 100.00 | 73.11 | – | – | 50.00 | 8.74 |
| Smear | (smear, glue gun, plate) | 99.88 | 81.24 | – | – | – | – |
| Average | – | 98.71 | 78.18 | 98.37 | 59.94 | 75.48 | 21.34 |

**Table 9: Detailed Metrics of BiDexHD-IPPO(w.o. stage-1) for each sub-task under $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.1$.**
| Action | Sub-Task Name | Train<br>$r_{1}$(%) | Train<br>$r_{2}$(%) | Test Comb<br>$r_{1}$(%) | Test Comb<br>$r_{2}$(%) | Test New<br>$r_{1}$(%) | Test New<br>$r_{2}$(%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Empty | (empty, bowl, bowl) | 100.00 | 73.94 | 64.63 | 49.10 | 58.08 | 45.91 |
|  | (empty, bowl, plate) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
|  | (empty, cup, plate) | 0.00 | 0.00 | – | – | – | – |
|  | (empty, teapot, plate) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
|  | (empty, teapot, teapot) | 100.00 | 83.77 | 61.32 | 45.78 | – | – |
| Pour in some | (pour in some, cup, cup) | 0.00 | 0.00 | – | – | – | – |
|  | (pour in some, cup, plate) | 0.00 | 0.00 | – | – | – | – |
|  | (pour in some, cup, teapot) | 0.00 | 0.00 | – | – | – | – |
|  | (pour in some, teapot, bowl) | 0.00 | 0.00 | – | – | – | – |
|  | (pour in some, teapot, cup) | 0.00 | 0.00 | – | – | – | – |
| Dust | (dust, brush, bowl) | 0.00 | 0.00 | – | – | 0.00 | 0.00 |
|  | (dust, brush, pan) | 0.00 | 0.00 | 0.00 | 0.00 | – | – |
| Put out | (put out, bowl, bowl) | 100.00 | 71.32 | 47.65 | 31.80 | 11.02 | 6.79 |
|  | (put out, bowl, plate) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Skim off | (skim off, bowl, plate) | 100.00 | 51.37 | – | – | 69.88 | 6.86 |
| Smear | (smear, glue gun, plate) | 0.00 | 0.00 | – | – | – | – |
| Average | – | 25.00 | 17.52 | 24.80 | 18.10 | 19.85 | 8.51 |

**Table 10: Detailed Metrics of BiDexHD-IPPO(w.o. gc) for each sub-task under $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.1$.**
| Action | Sub-Task Name | Train<br>$r_{1}$(%) | Train<br>$r_{2}$(%) | Test Comb<br>$r_{1}$(%) | Test Comb<br>$r_{2}$(%) | Test New<br>$r_{1}$(%) | Test New<br>$r_{2}$(%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Empty | (empty, bowl, bowl) | 100.00 | 75.66 | 100.00 | 49.28 | 100.00 | 50.27 |
|  | (empty, bowl, plate) | 78.68 | 45.56 | 65.00 | 16.44 | 56.67 | 10.62 |
|  | (empty, cup, plate) | 52.53 | 19.67 | – | – | – | – |
|  | (empty, teapot, plate) | 69.73 | 47.65 | 91.23 | 14.89 | 83.00 | 35.43 |
|  | (empty, teapot, teapot) | 100.00 | 83.56 | 100.00 | 86.58 | – | – |
| Pour in some | (pour in some, cup, cup) | 100.00 | 99.71 | – | – | – | – |
|  | (pour in some, cup, plate) | 98.06 | 44.36 | – | – | – | – |
|  | (pour in some, cup, teapot) | 99.90 | 91.03 | – | – | – | – |
|  | (pour in some, teapot, bowl) | 100.00 | 74.71 | – | – | – | – |
|  | (pour in some, teapot, cup) | 94.41 | 56.88 | – | – | – | – |
| Dust | (dust, brush, bowl) | 99.78 | 88.39 | – | – | 80.15 | 27.45 |
|  | (dust, brush, pan) | 94.03 | 74.31 | 84.03 | 55.75 | – | – |
| Put out | (put out, bowl, bowl) | 99.95 | 62.07 | 100.00 | 61.41 | 84.51 | 11.88 |
|  | (put out, bowl, plate) | 84.15 | 66.27 | 100.00 | 80.39 | 79.59 | 18.73 |
| Skim off | (skim off, bowl, plate) | 91.40 | 70.22 | – | – | 55.26 | 4.06 |
| Smear | (smear, glue gun, plate) | 85.93 | 62.13 | – | – | – | – |
| Average | – | 90.53 | 66.39 | 91.47 | 52.11 | 77.03 | 22.63 |

**Table 11: Detailed Metrics of BiDexHD-IPPO(w.o. bonus) for each sub-task under $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.1$.**
| Action | Sub-Task Name | Train<br>$r_{1}$(%) | Train<br>$r_{2}$(%) | Test Comb<br>$r_{1}$(%) | Test Comb<br>$r_{2}$(%) | Test New<br>$r_{1}$(%) | Test New<br>$r_{2}$(%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Empty | (empty, bowl, bowl) | 100.00 | 52.77 | 98.91 | 44.44 | 96.49 | 24.22 |
|  | (empty, bowl, plate) | 98.95 | 66.06 | 99.51 | 63.93 | 84.87 | 8.59 |
|  | (empty, cup, plate) | 99.77 | 43.87 | – | – | – | – |
|  | (empty, teapot, plate) | 100.00 | 83.39 | 92.66 | 24.63 | 86.70 | 26.44 |
|  | (empty, teapot, teapot) | 100.00 | 84.03 | 99.61 | 85.76 | – | – |
| Pour in some | (pour in some, cup, cup) | 100.00 | 98.99 | – | – | – | – |
|  | (pour in some, cup, plate) | 89.91 | 31.76 | – | – | – | – |
|  | (pour in some, cup, teapot) | 100.00 | 86.55 | – | – | – | – |
|  | (pour in some, teapot, bowl) | 76.43 | 11.27 | – | – | – | – |
|  | (pour in some, teapot, cup) | 100.00 | 67.22 | – | – | – | – |
| Dust | (dust, brush, bowl) | 99.77 | 86.93 | – | – | 88.34 | 35.45 |
|  | (dust, brush, pan) | 99.44 | 72.85 | 95.41 | 46.95 | – | – |
| Put out | (put out, bowl, bowl) | 100.00 | 62.52 | 100.00 | 68.21 | 95.45 | 16.13 |
|  | (put out, bowl, plate) | 98.80 | 75.67 | 100.00 | 84.37 | 48.82 | 5.16 |
| Skim off | (skim off, bowl, plate) | 100.00 | 58.47 | – | – | 45.03 | 6.66 |
| Smear | (smear, glue gun, plate) | 99.70 | 84.10 | – | – | – | – |
| Average | – | 97.67 | 66.65 | 98.01 | 59.76 | 77.96 | 17.52 |

### C.2 Results of Student Learning

Tables below record the detailed evaluation results for each task category. The last row of each table shows the average results over all sub-tasks in all task categories. ‘–’ in the table represents the absence of testing tasks.

**Table 12: Detailed Metrics of BiDexHD-PPO+DAgger for each task category under $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.1$.**
| Action | Train<br>$r_{1}$(%) | Train<br>$r_{2}$(%) | Test Comb<br>$r_{1}$(%) | Test Comb<br>$r_{2}$(%) | Test New<br>$r_{1}$(%) | Test New<br>$r_{2}$(%) |
| --- | --- | --- | --- | --- | --- | --- |
| Dust (2) | 95.42 | 72.49 | 72.41 | 19.13 | 94.92 | 40.79 |
| Empty (5) | 96.61 | 56.95 | 70.18 | 24.58 | 90.91 | 27.63 |
| Put out (2) | 98.04 | 52.33 | 97.52 | 56.31 | 88.41 | 29.75 |
| Pour in some (5) | 91.73 | 42.96 | – | – | – | – |
| Skim off (1) | 97.60 | 71.54 | – | – | 42.19 | 20.79 |
| Smear (1) | 99.40 | 72.46 | – | – | – | – |
| Average | 95.35 | 55.82 | 76.75 | 30.42 | 86.34 | 30.00 |

**Table 13: Detailed Metrics of BiDexHD-IPPO+DAgger(K=5) for task category under $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.1$.**
| Action | Train<br>$r_{1}$(%) | Train<br>$r_{2}$(%) | Test Comb<br>$r_{1}$(%) | Test Comb<br>$r_{2}$(%) | Test New<br>$r_{1}$(%) | Test New<br>$r_{2}$(%) |
| --- | --- | --- | --- | --- | --- | --- |
| Dust (2) | 100.00 | 86.91 | 100.00 | 49.94 | 100.00 | 48.37 |
| Empty (5) | 99.04 | 73.20 | 87.13 | 36.62 | 96.61 | 61.96 |
| Put out (2) | 98.22 | 71.89 | 100.00 | 76.43 | 93.63 | 42.40 |
| Pour in some (5) | 100.00 | 72.31 | – | – | – | – |
| Skim off (1) | 98.78 | 71.59 | – | – | 77.58 | 45.71 |
| Smear (1) | 99.70 | 76.69 | – | – | – | – |
| Average | 99.38 | 74.59 | 92.85 | 48.43 | 94.79 | 53.71 |

**Table 14: Detailed Metrics of BiDexHD-IPPO+DAgger(K=5) for each task category under $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.075$.**
| Action | Train<br>$r_{1}$(%) | Train<br>$r_{2}$(%) | Test Comb<br>$r_{1}$(%) | Test Comb<br>$r_{2}$(%) | Test New<br>$r_{1}$(%) | Test New<br>$r_{2}$(%) |
| --- | --- | --- | --- | --- | --- | --- |
| Dust (2) | 99.64 | 75.79 | 97.06 | 22.34 | 96.97 | 28.81 |
| Empty (5) | 97.26 | 63.39 | 61.11 | 10.62 | 96.59 | 48.16 |
| Put out (2) | 94.44 | 60.38 | 100.00 | 61.64 | 62.50 | 20.29 |
| Pour in some (5) | 100.00 | 68.38 | – | – | – | – |
| Skim off (1) | 98.53 | 60.74 | – | – | 79.17 | 37.23 |
| Smear (1) | 99.40 | 67.15 | – | – | – | – |
| Average | 98.27 | 66.19 | 77.74 | 24.56 | 88.11 | 37.62 |

**Table 15: Detailed Metrics of BiDexHD-IPPO+DAgger(K=5) for each task category under $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.05$.**
| Action | Train<br>$r_{1}$(%) | Train<br>$r_{2}$(%) | Test Comb<br>$r_{1}$(%) | Test Comb<br>$r_{2}$(%) | Test New<br>$r_{1}$(%) | Test New<br>$r_{2}$(%) |
| --- | --- | --- | --- | --- | --- | --- |
| Dust (2) | 97.51 | 58.09 | 52.38 | 5.22 | 84.62 | 14.20 |
| Empty (5) | 94.52 | 50.47 | 27.78 | 6.65 | 88.89 | 21.71 |
| Put out (2) | 93.05 | 41.78 | 100.00 | 36.75 | 56.25 | 10.49 |
| Pour in some (5) | 100.00 | 59.37 | – | – | – | – |
| Skim off (1) | 98.75 | 42.11 | – | – | 69.39 | 13.94 |
| Smear (1) | 97.44 | 50.18 | – | – | – | – |
| Average | 96.87 | 52.58 | 49.30 | 13.02 | 79.56 | 17.19 |

**Table 16: Detailed Metrics of BiDexHD-IPPO+DAgger(K=2) for each task category under $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.1$.**
| Action | Train<br>$r_{1}$(%) | Train<br>$r_{2}$(%) | Test Comb<br>$r_{1}$(%) | Test Comb<br>$r_{2}$(%) | Test New<br>$r_{1}$(%) | Test New<br>$r_{2}$(%) |
| --- | --- | --- | --- | --- | --- | --- |
| Dust (2) | 99.81 | 86.47 | 98.33 | 48.19 | 99.79 | 52.57 |
| Empty (5) | 98.42 | 73.13 | 88.54 | 36.51 | 97.30 | 55.76 |
| Put out (2) | 98.37 | 70.56 | 100.00 | 79.25 | 91.41 | 39.62 |
| Pour in some (5) | 98.47 | 74.59 | – | – | – | – |
| Skim off (1) | 98.55 | 70.32 | – | – | 74.87 | 40.77 |
| Smear (1) | 100.00 | 77.14 | – | – | – | – |
| Average | 98.71 | 75.01 | 93.26 | 48.60 | 94.38 | 50.39 |

**Table 17: Detailed Metrics of BiDexHD-IPPO+DAgger(K=1) for each task category under $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.1$.**
| Action | Train<br>$r_{1}$(%) | Train<br>$r_{2}$(%) | Test Comb<br>$r_{1}$(%) | Test Comb<br>$r_{2}$(%) | Test New<br>$r_{1}$(%) | Test New<br>$r_{2}$(%) |
| --- | --- | --- | --- | --- | --- | --- |
| Dust (2) | 99.89 | 85.90 | 99.19 | 45.88 | 100.00 | 47.51 |
| Empty (5) | 98.55 | 73.04 | 86.13 | 39.56 | 98.44 | 59.65 |
| Put out (2) | 97.90 | 72.42 | 100.00 | 75.80 | 90.91 | 38.03 |
| Pour in some (5) | 98.78 | 74.90 | – | – | – | – |
| Skim off (1) | 98.94 | 71.45 | – | – | 72.65 | 40.66 |
| Smear (1) | 99.87 | 78.54 | – | – | – | – |
| Average | 98.81 | 75.40 | 92.11 | 49.02 | 94.67 | 51.00 |

**Table 18: Detailed Metrics of BiDexHD-IPPO+DAgger(K=0) for each task category under $\varepsilon_{\text{succ}}=\varepsilon_{\text{track}}=0.1$.**
| Action | Train<br>$r_{1}$(%) | Train<br>$r_{2}$(%) | Test Comb<br>$r_{1}$(%) | Test Comb<br>$r_{2}$(%) | Test New<br>$r_{1}$(%) | Test New<br>$r_{2}$(%) |
| --- | --- | --- | --- | --- | --- | --- |
| Dust (2) | 100.00 | 86.27 | 98.21 | 44.53 | 98.72 | 42.01 |
| Empty (5) | 95.77 | 64.37 | 90.57 | 35.44 | 96.20 | 59.15 |
| Put out (2) | 98.08 | 72.92 | 100.00 | 76.74 | 90.84 | 39.50 |
| Pour in some (5) | 99.01 | 73.35 | – | – | – | – |
| Skim off (1) | 98.39 | 69.76 | – | – | 79.44 | 33.94 |
| Smear (1) | 99.70 | 76.63 | – | – | – | – |
| Average | 98.01 | 72.09 | 94.36 | 46.64 | 93.96 | 49.27 |

## Appendix D Additional Visualizations

Figures below visualize samples of bimanual human demonstrations and policy deployment of constructed bimanual dexterous manipulation tasks.

Figure: Figure 4: Task visualization of (pour in some, cup, teapot).
Refer to caption: extracted/5898443/figs/vis1.png

Figure: Figure 5: Task visualization of (empty, bowl, bowl).
Refer to caption: extracted/5898443/figs/vis2.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The dataset used in this paper to generate tasks and provide trajectories is TACO: Benchmarking Generalizable Bimanual
Tool-ACtion-Object Understanding. In this dataset, there are 2.5k sequences of precise hand-object meshes and annotations. However, high-quality datasets such as TACO are costly to collect and difficult to scale as they require motion-tracking equipment and facilities. My questions are: how would reward engineering, depending on privileged state information in simulation tasks, that are constructed from motion-tracked trajectories, scale to infinitely diverse bimanual dexterous manipulation tasks? What are the limitations and bottlenecks?

2. Although I understand this work pertains to the study of bimanual dexterous manipulation framework in simulation, what would be some practical bottlenecks preventing this framework from being deployed onto a real bimanual dexterous robot?

3. For automatically constructed tasks in simulation, what are some limitations to the current framework preventing it from being "easily" scalable or applicable to other tasks? Or what quantifies as "easily"? (as the word "easily" is used in the paper)
soundness: 3
strengths: Originality:
- The framework described in the paper is novel in these perspectives:
1. While it is built on the widely used teacher-student framework, the authors designed two-staged reward functions that are universal to tool-use bimanual manipulation tasks constructed in simulation.
2. The idea of constructing many tasks from given bimanual trajectories is also a creative way of addressing the scalability challenge in simulation.

Quality:
- The authors performed several ablation experiments to demonstrate the effectiveness of each design choice, ranging from IPPO vs PPO, thresholds used in reward, each stage of the reward/training, functional grasping center, future conditioning steps, and baseline methods (BC). These efforts improved the technical soundness of the proposed method.

Clarity:
- I appreciate the clarity of the writing in terms of explaining the stages in this framework, the visual illustrations of each stage in figure 1 and figure 2, and the explanations for reward functions.

Significance:
- The problem of bimanual manipulation is becoming increasingly important for general-purpose robot intelligence. As the author pointed out, human-level performance on challenging bimanual dexterous manipulation skills is crucial for tasks involving coordination and tool usage. This work attempts to provide a unified and scalable framework that addresses several constraints in prior works.
summary: This paper introduces **BiDexHD**, a framework designed to automatically turn a human bimanual manipulation dataset into simulation tasks and learn diverse bimanual dexterous manipulation skills with teacher-student method (RL in sim with state-based observations + distill into a policy with point cloud observation via Dagger). It aims to address the complexity and lack of task diversity in existing approaches. The authors propose a novel frame that:

1. instead of manually designed or predefined tasks, it creates feasible tasks from given bimanual trajectories;
2. using a teacher-student learning framework leveraging unified two-staged reward functions and vision-based policy distillation
3. evaluated on the TACO dataset across 141 tasks with strong performance on both seen and unseen tasks

The central claim of this paper is centered around the notion of "a preliminary attempt towards universal bimanual skills", and a framework that is "unified and scalable".

As in its current form, I cannot argue for its acceptance, due to the questions and weaknesses enumerated below. There is not enough evidence to support the strong claims of this work.
weaknesses: 1. This work limits the training and testing to one dataset and one simulator for a method that aims to be universal and scalable. If it is feasible, a demonstration of the proposed framework's effectiveness in other bimanual manipulation benchmarks and simulation environments would provide more convincing signals.

2. While there are 141 tasks from 6 categories, they are limited to tool-usage-oriented tasks, and the reward functions are tailored for solving this flavor of tasks. However, bimanual manipulation tasks that do not fall into this genre would likely require separate reward function designs. For example, cloth folding, packing & unpacking, or assembling & disassembling tasks are harder to simulate and have well-defined reward functions. As is commonly known, designing and tuning a good reward function for RL requires human effort and knowledge, which are both demanding and challenging. Thus, further experimental designs and evaluations are needed to strengthen the scalable and universal claim.

2. It is confusing that the BC baseline trained using teleoperated data achieves 0 success even on trained tasks. Could authors include more details on BC data's quality + quantity, training, and evaluation setting? To the best of my knowledge, other works have shown non-zero success for BC methods (even as baselines) on bimanual dexterous manipulation tasks.[1]

[1] Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning, Chen et al.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Can you comment on how you think of sim-to-real transfer of the setup? From the videos it seems the motion is quite unstable and jittery at times, do you think some kind of regularization or reward shaping can fix it? Or do you envision some fundamental challenge in sim-to-real?

Can you comment on why BiDexHD-IPPO underperforms in Test New (Table 1) compared to BiDexHD-PPO?

What is the batch size used in IPPO/PPO? I don’t see it listed in the appendix. I am curious about the effect of batch size on training stability.

It would be good to discuss how the success rates vary among tasks (Appendix C) also in the main text.

I suggest using a different notation, e.g., M_1 and M_2, instead of r_1 and r_2, for the two metrics in the experiment section since readers might mistake them as rewards.
soundness: 3
strengths: I appreciate the authors carefully detailing the approach including the different tricks applied for setting up simulation environments based on human demos, and also the carefully designed reward functions. I don’t find major issues with the notations.

The overall approach is intuitive. It is clever to use human demos as an implicit representation of the task (despite not novel, see below), and then design RL training around it.

The experiment results are generally positive. The authors show the benefits of using IPPO instead of PPO in such a decentralized setup. The improvement over BC is also clear (despite the concern about the experiment setup).
summary: The authors propose BiDexHD for learning bimanual manipulation policies starting from human demonstrations. From human demonstrations, BiDexHD extracts human and object poses and defines the task based on them. During the policy training phase, BiDexHD first learns state-based policies that first aligns the hands and objects to the desired poses from human demos using RL and carefully designed reward functions, and then distills them into vision-based policies using DAgger. Experiments are based on the TACO dataset including six task categories. Ablation studies show the importance of aligning phase, and the main results show improvement over BC-only baseline.
weaknesses: My main concern of the paper is the lack of technical contribution over existing work. There is a similar work from Dasari et al., ICRA 2023 [1], which is not cited but proposes very similar ideas. In [3], the authors also use human demos and train RL policies to track the human trajectories. There is also an alignment phase by planning the hand to the object (not learned). Diverse tasks and objects are also considered. I think BiDexHD differs by (1) learning the alignment and (2) distilling into vision-based policies; (1) is new but (2) is well-studied in previous work as the authors also agree. I see the existing ideas from [3] also address the motivation of designing unified and scalable framework for learning bimanual dexterous tasks.

There are no details about how the BC baseline is designed and trained. I imagine with an expressive enough policy parameterization, e.g., diffusion, the BC baseline can achieve nonzero success rates. I urge the authors to carefully provide the details of the BC experiments.

I also find the writing of the introduction section can be improved. The current form reads rather vague —"unified and scalable” is emphasized multiple times, which I understand by reading the approach section, but the introduction does not explain at all why the approach is unified and scalable. I think the statement of contributions can be greatly improved to provide more details about the overall approach.

[1] Learning Dexterous Manipulation from Exemplar Object Trajectories and Pre-Grasps, Sudeep Dasari, Abhinav Gupta, Vikash Kumar, ICRA 2023

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: 1). The learned policy in the video includes a lot of jerking motion, is it possible to add some action penalty term in RL to smooth the motion?

2). Sec 4.2 mentioned identifying and removing invalid tasks to build up a complete task set. I am wondering how to identify invalid tasks after initialization.

3). Could you provide analysis on why BiDexHD-PPO outperforms BiDexHD-IPPO on teacher learning?

4). There has also been some work that learns bimanual dexterous policy from demonstration data. What is the difference between BiDexHD and other bimanual dexterous manipulation papers, for instance, DexCap?

5). Some discussion if the authors could address why existing trajectory-tracking methods ([1-4]) cannot be directly adapted for bimanual dexterous tasks.
soundness: 3
strengths: By utilizing human demonstration dataset, BiDexHD learns bimanual dexterous policy in a scalable way. Unlike some prior works that is only limited to learning a specific task, BiDexHD is capable of learning many bimanual dexterous skills, meanwhile avoiding the effort of complex reward shaping for individual tasks. The author compares BiDexHD against ablated baselines and demonstrates its high performance over the baselines and competitive generalization capabilities.
summary: This paper proposes an approach that learns bimanual dexterous manipulation from human demonstration dataset. The approach constructs corresponding tasks from existing bimanual dataset and applies teacher-student learning on the constructed tasks. The task construction part includes data preprocessing which converts human hand poses to LEAP hand pose and simulation initialization which initializes corresponding objects in the simulation environment according to the demonstration. Their approach also decomposes the teacher policy learning into two stages: 1) training the policy to matching the objects and hand poses with the first time step of the demonstration trajectory from initial pose and 2) tracking the reference object trajectory. The authors compare performance of different RL algorithms in teacher policy learning and different IL algorithms in vision-based policy distillation. They also ablate several design choices in teacher-student policy learning and demonstrate improved performance over other baselines.
weaknesses: 1). Although BiDexHD is able to learn many bimanual tasks, it is tailored to learning one category of tasks: one hand holding the tool and another hand holding the object. It seems the framework is not able to deal with tasks that require in hand manipulation of either hand, or bimanual manipulation of a single object. For example, hand over and open bottle cap.

2). Description of Fig. 2 mentioned the tool and object are initialized at a fixed pose, but in real world application, the manipulated objects are seldom initialized at a fixed pose. The policy are not trained on a randomized initial pose.

3). Learning bimanual dexterous skills is extremely challenging, hence it is still unclear if the learned policies are able to be deployed on real-world hardwares.

4). In Sec. 5.2, the metric $r_2$ might not be a complete metric for task completion, because the task might still be completed while the hands fail to track the demonstration. For instance, when the tool and object both move up the same distance, while their relative pose keeps constant, the task is still completed.

5). The use of object trajectory tracking in reinforcement learning for dexterous manipulation is not particularly novel, as evidenced by prior works such as [1][2][3][4]. It would add depth to the discussion if the authors could address why existing trajectory-tracking methods cannot be directly adapted for bimanual dexterous tasks. While I understand that bimanual tasks indeed expand the observation and action spaces significantly, it would be helpful to know if there are additional, perhaps more nuanced, challenges that prevent a straightforward adaptation of these methods.

[1] Han, Yunhai, et al. "Learning Prehensile Dexterity by Imitating and Emulating State-Only Observations." IEEE Robotics and Automation Letters (2024).
[2] Dasari, Sudeep, Abhinav Gupta, and Vikash Kumar. "Learning dexterous manipulation from exemplar object trajectories and pre-grasps." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.
[3] Guzey, Irmak, et al. "Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards." arXiv preprint arXiv:2410.23289 (2024).
[4] Chen, Yuanpei, et al. "Object-Centric Dexterous Manipulation from Human Motion Data." 8th Annual Conference on Robot Learning.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. There is one recent bimanual manipulation policies from point-cloud, DexCap[1] with imitation learning. That shows good results on learning tool use with bimanual manipulation. It would be interesting to see how their point-cloud based policy works in these tasks. 
2. It is a little surprising to see the IPPO has better results than PPO because some of bimanual manipulation tasks may need coordination. Could the authors provide more explanations about why IPPO behaves better than PPO.
3. The paper only compares PPO, IPPO and some ablation of different components of the IPPO. However, in the previous bimanual dexterous manipulation paper[2], they compare a variety of RL, MARL methods. It would make the paper stronger if the proposed method is also compared against MARL methods. 
4. What are the common failure modes of the baselines? It would be interesting to see the comparison between the videos of the  proposed methods and the baselines. Currently, the website only has videos of the proposed methods.
5. The paper already extracts the human poses from the bimanual demonstration dataset and can retarget from human poses to robot hand joint. Why not also use this data to add tracking reward not only to object pose but also to robot hand joint?
6. Adding the future object positions into the observations seems not very practical in real world. I am wondering if the authors have any insight about how to get future object positions to feed into the policy when deployed in the real world.

[1]Wang, Chen, et al. "Dexcap: Scalable and portable mocap data collection system for dexterous manipulation." In proceedings of Robotics: Science and System, 2024.

[2]Y. Chen et al., "Bi-DexHands: Towards Human-Level Bimanual Dexterous Manipulation," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 5, pp. 2804-2818, May 2024,
soundness: 3
strengths: 1. This paper proposes very detailed description of the pipeline and outlines each component and different stages
2. The learned multi-task policy can work on six categories of different bimanual tasks and outperforms the baselines. 
3. The paper also ablates the different components in the whole pipeline to highlight the importance of each component.
summary: This paper introduces an approach to learn multi-task bimanual manipulation policies through teacher-student training from human demonstrations. Before training, it uses existing bimanual dataset of human videos to extract human poses and other relevant information  for task construction. During training, It first trains a RL policy to grab the tool and object by two hands separately with multi-stage reward. Then it uses reward of tracking object poses to train a state-based RL teacher policy. After getting two-stage policies, it distills the policy to a point-cloud based policy with DAgger.
weaknesses: 1. The paper uses the metrics of difference between object pose  and the reference object pose along the trajectory as the success criteria.  The proposed method has better performance in this metrics than the baselines. However, from the video shown on the website. The policy looks very jerky and sometimes the hand is in touch of the table. Therefore, this reduces the plausibility of this learned policy to transfer to the real robot.  It might be helpful to use other metrics to evaluate the policy such as number of collisions, the smoothness of the policy, etc. to have a comprehensive review of the paper. 
2. The paper uses the teacher-student policy distillation to learn the bimanual policy which is commonly used for single dexterous hand manipulation. Extracting human poses from human videos are also common in prior work. Therefore, it is unclear to me what is the key innovation and contribution of this paper.
3. The paper introduces the behavior cloning (BC) baseline but doesn’t mention which algorithm they are using. Currently there are several popular BC methods including Diffusion Policy, BC Transformer, ACT, etc. The paper should include more details of the BC policy including the algorithm, the hyperparameters and the demonstration dataset size.

Meta-review: The submission introduces a framework for learning bimanual dexterous manipulation skills by leveraging human demonstrations. The reviewers acknowledge its novelty in task construction and the teacher-student framework, as well as its promising performance on the TACO dataset. However, they highlight several areas for improvement and concerns regarding its universality, scalability, and novelty compared to existing work.

Strengths:

The paper presents a novel approach to automatic task construction from human demonstrations and a unified reward function for reinforcement learning, addressing a significant challenge in bimanual dexterous manipulation. The empirical results are robust, showcasing competitive task fulfillment rates, including zero-shot generalization.

Weaknesses:

Several reviewers raised concerns about the limited scope of the experiments (one dataset and simulation environment), questioning claims of universality and scalability. Additionally, the BC baseline setup and performance metrics were criticized as unclear or inadequate. Concerns about novelty over existing methods like PGDM were also frequently mentioned.

While the paper presents promising results and tackles an important problem, the concerns about novelty, scalability, and universality outweigh its strengths in the current form. Therefore, I recommend rejection for this iteration.

Decision: Reject