DextrAH-G: Pixels-to-Action Dexterous Arm-Hand Grasping with Geometric Fabrics

Authors: tyler  ga
wei  lum
martin  matak
viktor  makoviychuk
nvidia  ankur
handa  nvidia
arthur  allshire
tucker  hermans
nathan d ratliff
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-16 03:36:04

Abstract: A pivotal challenge in robotics is achieving fast, safe, and robust dexterous grasping across a diverse range of objects, an important goal within industrial applications. However, existing methods often have very limited speed, dexterity, and generality, along with limited or no hardware safety guarantees. In this work, we introduce DextrAH-G, a depth-based dexterous grasping policy trained entirely in simulation that combines reinforcement learning, geometric fabrics, and teacher-student distillation. We address key challenges in joint arm-hand policy learning, such as high-dimensional observation and action spaces, the sim2real gap, collision avoidance, and hardware constraints. DextrAH-G enables a 23 motor arm-hand robot to safely and continuously grasp and transport a large variety of objects at high speed using multi-modal inputs including depth images, allowing generalization across object geometry. Videos at https://sites.google.com/view/dextrah-g.

Main paper: 
 introduction: 
 Grasping, a fundamental skill coarsely mastered by even two-year-olds [1] , still poses a challenge in robotics. Equipping machines with proficient grasping skill is paramount to automating aspects of logistics, manufacturing, space, and search-and-rescue among many other use cases. To date, robot grasping has emerged as a chief research problem with many advancements made. However, existing methods often have limited speed, dexterity, and reliability, especially with high-actuatorcount embodiments. Moreover, partial observability and succinct coordination over a large number of motors in dexterous platforms pose additional challenges in acquiring proficient grasping skill.Existing approaches in dexterous robot grasping (see Section 2) mostly focus on predicting grasp poses while relying on model-predictive control, path planning, or other motion generation tools to reach the grasp pose. While somewhat successful, this approach is not a continuously reacting strategy, typically does not plan through all degrees-of-freedom, and ignores grasp modulation and coordinated finger-arm control post-grasp. Generating high-frequency control predictions from all available data sources (e.g. fusing all sensory traces including proprioception and vision) is critical for reasoning through partially observable environments and improving skill performance.Encouragingly, reinforcement learning in simulation at scale has enabled significant progress in both legged locomotion and in-hand manipulation even with high-dimensional observations like point clouds, high-dimensional action spaces, and high-frequency action rates. The ability to efficiently scale experience with continued policy optimization and domain randomization has singularly enabled the creation of highly dynamic, complex control over dexterous robot platforms. A comparatively smaller but growing body of work also applies RL in simulation at scale to advance the state of dexterous robot grasping in the real world. However, the exact methods to date typically lead to some combination of the following qualities: slow robot motion, inadequate or missing hardware safety guarantees, unnatural movements and postures, and limited generalization.To address these challenges with real-world dexterous robot grasping, we propose DextrAH-G: a combined pixels-to-action fabric-guided policy (FGP) and geometric fabric controller that achieves state-of-the-art grasping performance in the real world. The main contributions of this work include: 1) a vectorized geometric fabric controller that creates an inductive bias for policy learning, avoids collision, upholds joint constraints, and shapes the behavior, 2) simulation-only RL training of a privileged FGP over vectorized geometric fabrics that enables high-performance grasping of many different objects, 3) depth-based, multi-modal FGP distillation of the privileged FGP that replicates the original behavior and enables object position predictions and 4) zero-shot sim2real transfer of DextrAH-G with new state-of-the-art dexterous grasping performance on many diverse novel objects in the real world, a significant progression towards grasp-anything skill in dexterous robotics.2 Related Work 
dexterous grasping: 
 Dexterous grasping is a research problem that has been extensively studied for decades. Traditional methods [2] use gradient-based or sampling-based optimizers to maximize analytical grasp metrics, such as differentiable approximations of form closure [3] , force closure [4] , the min-weight metric [5] , and the largest inscribed ball metric [6] . However, these works are typically limited to precision grasps using only the fingertips and require ground-truth object models.In recent years, data-driven learning-based methods have emerged as a promising approach for dexterous grasping of complex objects. Li et al. [7] and Wang et al. [3] create large-scale grasp datasets on diverse objects using approximate force closure optimization. Xu et al. [8] use these grasp datasets to learn a grasp proposal generation network to generate grasps conditioned on full-point clouds and then train a goal-conditioned grasp RL policy to perform these grasps. Wan et al. [9] build on this method with a geometry-aware curriculum and iterative generalist-specialist learning. However, their method requires full-point clouds of objects and has only been tested in simulation. Liu et al. [10] introduces a novel geometric and spatial hand-object interaction representation to capture dynamic object shape features and the spatial relations between hands and objects during grasping, and they use this representation to train a dexterous grasping policy using RL. However, when deployed in the real world, this representation requires ground-truth object models and requires registering these models to the measured partial point cloud, limiting its generality. Agarwal et al. [11] select a pre-grasp position on a target object by matching the DINO-ViT features of a previously annotated object, then use a blind (uses only proprioception) grasping RL policy with an eigengrasp action space that is trained entirely in simulation. Others trained a point-cloud-conditioned policy with RL in simulation using imagined hand point clouds as augmented inputs which was deployed in the real world on novel objects in the same category Qin et al. [12] . These two works are similar to ours except our policy uses depth images as input to jointly select actions for both the arm and hand, allowing for more agile, coordinated, and generalized behavior across novel objects. See Appendix A for an extended discussion on related works RL for robot control.Learned policies for robots typically issue actions to simple controllers like joint-level proportionalderivative (PD) control or operational space control (OSC). Consequently, controller simplicity shifts the burden of discovering and imparting well-controlled behavior entirely to the learned policy itself. Globally, well-controlled behavior has complex, multi-faceted priorities such as primary objective completion, various hardware constraints, collision avoidance, and other qualities like natural movements. Discovering this rich behavioral conglomerate via learning is very challenging due to optimization locality, approximation and generalization errors inherent to neural networks, behavior specification, exploration, and high-dimensional action spaces. This arduous path can be avoided by embedding many of these behaviors within more sophisticated control layers. Recently, fabric-guided policies (FGP) were trained over a geometric fabric via RL in simulation to achieve new state-of-the-art in-hand cube reorientation performance in the real world [13] . The geometric fabric itself handled joint hardware constraints, created an input action space for controlling the hand fingertips, and guided the fingertips towards making contact with the cube. Leveraging such a rich controller promoted a much simpler reward function predominantly geared towards primary task completion. Other works also mix sophisticated control and policy learning like Riemannian Motion Policies in [14] and QP control in [15] . Since geometric fabrics have been shown to outperform RMPs [16, 17] , DMPs [17] , Koopman Operator policies [18] , and derive from strong theoretical analyses [19, 16, 20] , we elect to use geometric fabrics within this work to construct DextrAH-G. 
dextrah-g: dexterous arm-hand grasping: 
 We present DextrAH-G, a combined pixels-to-action FGP and geometric fabric controller that achieves dynamic and reactive dexterous grasping in the real world. The instantiated geometric fabric for DextrAH-G is the most feature-rich design to date that shapes posture and grasping behavior, avoids environment and self-collision, imposes joint constraints, and exposes efficient action spaces. Our controller facilitates sim2real by: 1) efficiently scaling for vectorized RL training while maintaining real-time loop rates for real-world deployment, and 2) enabling safe real-world deployment even with delusional, hazardous policies. We now detail DextrAH-G by first describing the geometric fabric controller. Then, we discuss how we train our privileged teacher FGP using RL entirely in simulation. Finally, we describe how we distill this privileged FGP into a depth FGP that we deploy zero-shot on hardware. Figure 2 shows our proposed framework.Geometric fabrics generalize the behavior of classical mechanical systems and, thereby, can be used to model controllers with design flexibility, composability, and stability without the loss of modeling fidelity. Behavior expressed by a geometric fabric follows the formEQUATIONwhere M f ∈ R n×n is the positive-definite system metric (mass), which captures system prioritization, f f ∈ R n is a nominal path generating geometric force, and f π (a) ∈ R n is an additional driving force of some action a ∈ R m . q f , qf , qf ∈ R n are the position, velocity, and acceleration of the fabric. This fundamental equation produces an acceleration qf , which evolves the fabric state q f We train a privileged fabrics-guided policy (FGP) using RL (top), distill the privileged FGP into a depth FGP to predict the teacher's actions and object position (middle), and deploy DextrAH-G with a state machine in the real world for bin packing (bottom). See Table 3 for details. and qf over time through numerical integration. One can see that f π influences qf , and thereby, the fabric state. For in-depth discussion of geometric fabrics, we refer the reader to prior works [19, 16, 20, 13] . See Appendix B.1 for details on how a geometric fabric connects with a robot.We use a geometric fabric controller for four primary reasons: 1) avoiding undesirable collisions, 2) creating an inductive bias via the exposed action space that simultaneously guides policy exploration and favorably shapes the overall robot motion, 3) respecting joint constraints (see Appendix B.3), and 4) maintaining the robot posture to promote kinematic manipulability (see Appendix B.4).Collision Avoidance: Environmental and self-collision avoidance are handled through a geometric fabric term and a forcing fabric term. The geometric term is tuned to dominate the collision avoidance behavior with speed-invariant paths, while the forcing term prevents penetration near the collision boundary. First, we model the geometry of the robot as a collection of spheres and use the forward kinematics of the arm to map the robot configuration to the origin of every sphere attached to the robot body, x = ϕ f k (q) ∈ R 3 . We define ni = ri-x ∥ri-x∥ ∈ R 3 as the direction from the sphere point to the closest point on collision body i, r i ∈ R 3 (collision objects or other body spheres) andd i = max(d min , d i ) ∈ R +as a lower-bounded distance, where d min ∈ R + and d i ∈ R is the signed distance between the body sphere and collision body i (Figure 5 ∥M b ∥ is the normalized metric, which maintains its Eigenspectrum (directions of importance). s i = 1 2 tanh(-α 1 (v i -α 2 ) + 1) is a smooth velocity gate that goes high when this sphere is moving towards collision body i, where α 1 , α 2 ∈ R + are gains and v i = -ẋ • ni is the signed impact speed that is negative when moving towards collision body i (see Appendix B.2 for details).Action Space: During grasping, human finger motions exhibit highly correlated movements that do not fully exercise all the degrees of actuation afforded by our hands. These grasps can most broadly be categorized into power and precision grasps [21] . Absent intricate in-hand manipulation, grasping motion can be concisely captured by movement on a lower dimensional manifold. Constraining a policy's action space to an eigengrasp manifold has been shown to improve grasp policy learning [11] . We create such a manifold for the Allegro hand by retargeting human grasping motion data to the Allegro (see Appendix D for retargeting details) and apply principal component analysis to the motion dataset. Let A ∈ R 5×16 be the first five principal components from PCA and define A = [0, A] ∈ R 5×23 , then the taskmap from the full robot configuration space to PCA space is x = Aq ∈ R 5 . Within this taskmap, we define an attraction fabric term. The metric M(x) = mI is a constant isotropic mass, where m ∈ R + . ẍ = -k a tanh(α a ∥x-x pca,target ∥)x-xpca,target ∥x-xpca,target∥ -b ẋ,where k a , α a ∈ R + are gains and x pca,target ∈ R 5 is a target position in this space. We use x pca,target as a 5 dimensional action space for the hand (Figure 7 for example motions).To coordinate finger control with arm control, we create another action space that controls the pose of the palm. We accomplish this by creating a new taskmap that uses forward kinematics to map to 7 three-dimensional points attached to the palm, stacked into a 21 dimensional space. The attraction fabric in this space is the same as designed for the hand, but scales to the full 21 dimensions with the goal action position x g ∈ R 21 . We create a 6 dimensional action space for the arm that consists of target palm position x f,target ∈ R 3 and target palm orientation in Euler angles r f,target ∈ R 3 . These quantities are transformed to 3D point targets for all 7 palm-fixed points, i.e., x g , and issued to the fabric. Across the full robot, the action space is 11 dimensional. This action space resulted in faster training and more natural grasping behavior (see Appendix C).Next, we cast dexterous grasping as a reinforcement learning problem and train a privileged-state teacher policy in simulation to adeptly grasp 140 different objects (additional details in Appendix E). Since the geometric fabric action space ensures that the robot will execute safe and natural behaviors, our reward design centers entirely around fingertip-object contact and lifting the object to a goal. This simplifies reward engineering and reduces reward hacking behavior from the policy.Asymmetric Actor Critic: Although our control policies will not have access to privileged simulation state information when deployed in the real world, we can still use privileged information to accelerate training in simulation. We use Asymmetric Actor Critic training [22] , in which our critic V (s) is given all privileged state information s and our teacher policy π privileged (o privileged ) is provided an observation o privileged , which is a limited subset of this privileged state information. We believe this prevents the teacher policy from learning behavior that is heavily reliant on accurate privileged state information, which would not be learnable by the student policy with real-world perception inputs. However, the critic can still leverage this information to provide more accurate value estimates, improving the speed and quality of teacher policy training.Observation, State, and Action Space: We define the teacher policy's observation as o privileged = [o robot , x goal , o obj ]. o robot includes the cspace position q ∈ R Nq , cspace velocity q ∈ R Nq (N q = 23), positions of three points on the palm [x palm , x palm-x , x palm-y ] ∈ R 3×3 , positions of the N f ingers = 4 fingertips x f ingertips ∈ R N f ingers ×3 , and the fabric state [q f , qf , qf ] ∈ R 3×Nq .x goal ∈ R 3 is the goal object position. o obj includes the noisy object position x obj ∈ R 3 and quaternion q obj ∈ R 4 (more details below), and the object one-hot embedding e ∈ {0, 1} N objects , where N objects = 140 is the number of objects in the training dataset (see Appendix E.7 and E.8).We define the critic's input state as s = [o privileged , s privileged ]. s privileged contains privileged state information including robot joint forces f dof ∈ R Nq , fingertip contact forces f f ingers ∈ R N f ingers ×3 , true object position x obj ∈ R 3 , true object quaternion q obj ∈ R 4 , true object velocity v obj ∈ R 3 , and true angular velocity w obj ∈ R 3 . This additional information allows the critic to make more accurate value predictions, which helps the policy learn more quickly.We define the teacher policy's action a as inputs to the underlying geometric fabric, where a = [x f,target , r f,target , x pca,target ] ∈ R 11 , where x f,target ∈ R 3 is the target palm position, r f,target ∈ R 3 is the target palm orientation in Euler angles, and x pca,target ∈ R 5 is the target PCA position for the fingers. The fabric is integrated at 60 Hz and the simulation steps at 60 Hz. The teacher policy runs at 15 Hz, so actions are repeated for intermediate timesteps.Environment Modifications for Robust Grasping: RL policies often converge to unnatural solutions that work well in simulation, but fail to transfer to the real world (see Appendix E.13 for frail grasping behavior.) We learn robust grasping behavior by introducing the following environment modifications. Random Wrench Perturbations: We apply random wrenches that move and rotate the object in unpredictable ways (see Appendix E.3). This forces the policy to learn grasps that are robust to exogenous perturbations. Pose Noise: We add uncorrelated and correlated noise to the object pose observation (see Appendix E.4). This gives incentive to learn grasping behavior that opens the hand wider than typically needed when approaching the object to reduce unexpected contact and account for uncertainty in position and geometry. Friction Reduction: We reduce the default coefficient of friction of the object to µ = 0.7, mitigating grasping behavior that is overly reliant on friction. Domain Randomization: We employ domain randomization across simulation parameters to learn policies that are robust across a spectrum of dynamics (see Appendix E.5).We use the teacher-student framework and distill our expert to a student policy that can be deployed in the real world using an online version of Dagger [23] . This distillation results in a pixels-toaction policy that uses continuous image input at 15 Hz to perform reactive dynamic grasping in the real world. During distillation, the student π depth (o depth ) → (â, xobj ) receives an observation o depth = [o robot , x goal , I], where I ∈ [0.5, 1.5] 160×120 m is a raw depth image. It produces actions â ∈ R 11 and object position predictions xobj ∈ R 3 (x obj is used by the state machine during real-world deployment). The student is trained with a supervision loss L = L action + βL pos , where L action = ||â -a|| 2 and L pos = ||x obj -x obj || 2 where a are actions predicted by the teacher π privileged and x obj are ground-truth object positions from the simulator, and β = 0.1 (see Appendix F). Since the policy fuses depth and proprioceptive signals together, xobj is more accurate through occlusions and facilitates its usage in a state machine. We augment the simulated depth readings with noise (see Appendix F.1) to better match the distractors present in the real world.Meaning M f ∈ R n×n Positive-definite system metric (mass), which captures system prioritization.f f ∈ R nNominal path generating geometric force. fπ(a) ∈ R n Additional driving force of some action a ∈ R m . q f , qf , qf ∈ R n Position, velocity, and acceleration of the fabric (q f used as PD target to control robot). x = ϕ f k (q) ∈ R 3 Origin of each sphere used to model the robot geometry (computed through forward kinematics). r i ∈ R 3 Closest point on collision body i to a given robot body sphere. ni ∈ R 3 Direction from a given robot body sphere to the closest point on collision body i.d i ∈ RSigned distance between a given robot body sphere and collision body i.d i ∈ R +Lower-bounded distance between a given robot body sphere and collision body i. ẍb ∈ R 3 Base acceleration response per robot body sphere away from collision.M b ∈ R 3×3Base metric response per robot body sphere for collision avoidance.s i ∈ RSmooth velocity gate that goes high when this robot body sphere is moving towards collision body i. v i ∈ R Signed impact speed that is negative when moving towards collision body i.A ∈ R 5×16First five components from PCA on grasping motion data retargeted from human hand to Allegro hand.x f,target ∈ R 3 Target palm position. r f,target ∈ R 3Target palm orientation in Euler angles. xpca,target ∈ R 5 Target PCA position for the fingers. Noisy object position. q obj ∈ R 4 Noisy object quaternion. e ∈ {0, 1} N objects Object one-hot embedding. x palm , x palm-x , x palm-y ∈ R 3 Positions of three points on the palm.x f ingertips ∈ R N f ingers ×3Positions of the fingertips. s privileged ∈ R n Privileged state information provided only to the critic.f dof ∈ R Nq Robot joint forces. f f ingers ∈ R N f ingers ×3 Fingertip contact forces. x obj ∈ R 3True object position. q obj ∈ R 4 True object quaternion. v obj ∈ R 3True object velocity. w obj ∈ R 3True angular velocity. a ∈ R 11 Policy action, which is passed to the underlying geometric fabric. L PPO ∈ R PPO loss. While DextrAH-G achieves a 87% success rate at unseen test objects, we see that the leading causes of failure are from accidental pushes that push the object out of the graspable region (8%) and repeated grasp misses for more challenging objects (3%). The other failure modes were a loose grip resulting in the object being dropped before reaching the bin (1%) and poor grasp placement resulting in the object being dropped before reaching the bin (1%). We define "Grasp Success" as the robot's actions resulting in the object being grasped and lifted off the table more than 1 inch. We define "Transport Success" as the robot's actions resulting in the object grasped and lifted off the table, and then placed into the bin.In tandem with the results in Figure 8 , we find that the primary factors that affect policy performance include object size (small bottle), object slipperiness (green cup), object's propensity to easily roll (apple), transparent objects (sanitizer bottle), and geometric aspects of the object that increase the likelihood of the fingers "catching" or "snagging" on the object.Finally, we scope DextrAH-G's signals over a short horizon during grasping and plot the results in Figure 20 . As seen, the depth FGP can output quickly shifting actions while the geometric fabric produces smooth joint angle targets. Moreover, since the desired velocity is set to zero for the PD controller, the arm's physical position lags behind the targets by about 0.2 s. The hand's physical position lags behind the targets by about 0.1 s. Despite this phase lag, DextrAH-G operates well due to RL training for these lagged dynamics. We also see increased separation between desired and measured joint angle positions during the later phase of DextrAH-G execution because the robot is grasping the object. The increased tracking error induces contact forces between the robot and the object which facilitates grasping. Overall, the hand experiences greater joint angle swings during execution than the arm. This quality is desirable and indicates that the hand is doing much of the mechanical work required for grasping while minimally moving the arm. It is also the case that arm joint angle movements translate to much greater Cartesian palm movement so only minimal arm angular movement is required for effectively guiding the hand during grasping. We define successful grasp-and-transport as the robot taking actions that result in the object being grasped and lifted off the table, and then dropped into the bin. We define interventions as the human adjusting the position or orientation of the object after the robot has failed to grasp the object. We define failures as the robot taking actions that result in the object falling off the table or moving far away enough that the human decides to move onto another object instead of intervening.If the robot misses a grasp or drops the object, but then subsequently retries successfully without human intervention, this counts as one successful grasp-and-transport, zero interventions, and zero failures. Note that these rates are aggregate results of total cumulative count divided by total time, so this rate of successful grasp-and-transports is affected by the time taken for interventions and failures.Figure 18 : We define grasp-and-transport time as the time between successful grasp-and-transports. Note that with this method of computing times, if there is an intervention or failure between two successful grasp-and-transports, this results in a longer grasp-and-transport time, which makes these conservative estimates of grasp-and-transport time. We find the median grasp-and-transport time to be 8 seconds. Roughly 42% of grasp-and-transports took between 6.0 -7.3 seconds. DextrAH-G performed best on real-world objects such as the brick, spam, duck, mouse box, and container. It performed least well on real-world objects such as the small bottle, pot, and green cup. We define successful grasp-and-transport as the robot taking actions that result in the object being grasped and lifted off the table, and then dropped into the bin. We define intervention as the human adjusting the position or orientation of the object after the robot has failed to grasp the object. We define failure as the robot taking actions that result in the object falling off the table or moving far away enough that the human decides to move onto another object instead of intervening. If the robot misses a grasp or drops the object, but then subsequently retries successfully without human intervention, this counts as one successful grasp-and-transport, zero interventions, and zero failures.  
simulation: 
 We evaluate π depth on the 140 training objects and report 99% success rate on average per batch of finished environments, matching π privileged performance. Next, we evaluate π depth per object and report 80% success rate (versus 85% for π privileged ) on average per object with an average successful episode lasting 4 seconds (see Appendix F.4). The 19% gap stems from environments being reset after a successful grasp in the first experiment. This leads to easier objects being grasped more often, resulting in an increased success rate. Overall, π depth nearly matches the performance of π privileged , enabling sim2real transfer as discussed in the subsequent section.Hardware Setup: The physical setup consists of an Allegro Hand mounted to a Kuka LBR iiwa arm and one Intel Realsense D415 camera rigidly mounted to the table (see Fig. 3 ). This robot has 23 independent motors and a single camera stream for control policies. Both the arm and the hand have an underlying joint PD controller which operates at 1 kHz for the arm and 333 Hz for the hand. These operate in two different ROS 2 nodes and listen for joint commands. The geometric fabric runs at 60 Hz in another ROS 2 node, which receives actions and outputs joint commands at 60 Hz. Finally, π depth runs in yet another separate node which receives all the necessary inputs and outputs actions at 15 Hz. The separation in FGP and fabric nodes enables persistent controller command over the robot behavior regardless of the state of the FGP node or FGP model.Single Object Grasping Assessment: A popular assessment protocol for dexterous grasping is that of quantifying single-object success rates. We conduct this procedure across 11 objects that exist in standardized object sets as in [24] and adopted by others in grasping research. The procedure consists of placing an object in five different poses on the table and deploying the robot grasping behavior. The grasp success rate is calculated across five trials for each object. We run DextrAH-G continuously until the grasp succeeds or we experience an irrecoverable failure. This allows DextrAH-G to aggregate interaction data with its recurrent structure to improve and adapt its actions over time, resulting in new state-of-the-art grasping success rates as reported in Table 1 . This grasp assessment protocol is not a very complete assessment of grasping performance as it does not quantify grasping speed and does not consider grasping behavior within the context of a full pick-and-transport application. As such, we now propose a bin packing protocol.Bin Packing Assessment: The bin packing testing protocol quantifies the performance of continuously grasping and transporting a larger variety of objects. Unlike the single-object assessment, this test captures performance within a full application context. Specifically, the robot is charged with continuously grasping over 30 different objects one-at-a-time and transporting them to a bin placed to the side of the robot (see Figures 3, 4 ). A simple state machine governs the whole process and uses xobj to transition from grasping to transportation (see Appendix G). We propose three For additional performance analysis, see Appendix I. 
conclusion: 
 DextrAH-G is a high-performance dexterous grasping policy with depth inputs trained entirely in simulation and deployed in the real world. To achieve this, we combined RL, online distillation, and a vectorized, feature-rich geometric fabric controller. The fabric itself ensures hardware safety and exposes an action space with a strong inductive bias. RL leverages the geometric fabric to learn a privileged FGP for grasping many different objects which is then used to directly train a depth FGP via online distillation. DextrAH-G successfully grasps and transports a large variety of novel objects in the real world, bringing real-world application closer to reality. Critically, over the many hours of testing DextrAH-G (and a variety of ill-behaved FGPs), no hardware was damaged. 
 
 Appendix: Reinforcement learning is a data-driven approach to policy learning that excels in developing adaptive behaviors for complex, contact-rich environments, which are traditionally challenging to model with high accuracy. A prevalent method for using reinforcement learning on robots is to train a policy entirely in a simulated environment and then deploying these policies zero-shot in the realworld Wu et al. [29] . However, naively deploying these policies in the real world often results in poor performance. A common reason for this discrepancy can be from differences in sensing and actuation dynamics between the simulation and actual environments Jitosho et al. [30] , Tan et al. [31] , Boeing and Bräunl [32] , Koos et al. [33] . To mitigate this issue, domain randomization can be employed during training, enhancing the robustness of the policies against inaccuracies in simulation [34] . This technique has proven effective in various robotic applications, including legged locomotion [35, 36, 37, 38, 39] , in-hand manipulation [40, 41, 42] , and robotic grasping tasks [34, 43, 44] . Another challenge is that policies trained in simulation may require privileged information that is not accessible in the real world, such as object pose and velocity, which does not allow them to be deployed in the real world. Therefore, it is common to either train policies to use the real-world observations from scratch or use a teacher with privileged information and then distill it into a student policy that uses real-world observations. Qin et al. [12] train a point-cloudconditioned policy with RL in simulation, and they find that they need to use imagined hand point clouds as augmented inputs to overcome challenges with object-hand occlusion in the real world point cloud. Chen et al. [45] train a teacher RL policy for in-hand object reorientation with privileged information in simulation, use DAgger to distill this into a point-cloud conditioned student policy, and then deploy the system in the real world. Teacher-student distillation is also prevalent in legged locomotion, where privileged experts are distilled into policies that operate on corrupted elevation scans [37, 46] . In our work, we utilize domain randomization to improve robustness to a spectrum of dynamics, train a privileged teacher policy on compressed state information, and perform online distillation to train a student policy that operates on noisy depth images.Geometric fabrics is an artificial dynamical system that can be connected with real robots through a torque control law. Industrial, collaborative, and hobbyist robots commonly adopt torque laws that expose a joint position and velocity action space, e.g., joint-level PD control. These controllers track the joint position and velocity targets over time, resulting in closed-loop tracking control. The fabric state above can be used directly as these joint position and velocity targets. Thus, the two dynamical systems become coupled and positively correlated as discussed in [13] . In effect, the real robot's dynamics are shifted to follow the artificial dynamics closely. Note, in our implementation, the fabric is forward integrated with an approximate second-order Runge-Kutta scheme as in [13] at 60 Hz and we always set the velocity targets to 0.Next, we provide a more detailed explanation about the formulation of the collision avoidance base metric response per sphereM b = i si d i ni ⊗ ni .The outer product ni ⊗ ni creates a matrix with an eigenvector along ni . In essence, this matrix will give high priority to actions along ni since that is the direction of importance. Building the complete base metric as a weighted sum over outer products of all ni means that the resulting matrix will have eigenvectors along all ni , capturing all directions of importance (directions that point to prospective collision with collision body i).First, since a fabric is a second order controller, joint acceleration and jerk limits can be handled in closed-form. We adopt the same technique as covered in [13] by solving the quadratic program:EQUATIONwhere α ∈ R + . Since qf = -(M f + αI) -1 f f , we can see that as α → ∞, ||q f || → 0. Thus, a single α can be found that drives every joint acceleration under its limit, i.e., qi ∀ i, where qi is the i th joint acceleration limit. New joint acceleration limits, q, can be calculated that satisfies both the original acceleration limits and jerk limits, ... q , as q = min q, ∆t ...EQUATION)where ∆t is the integration timestep. Thus, at every evaluation of qf a single α can be calculated that simultaneously satisfies all acceleration and jerk limits of the robot.We also impose the robot's joint positional limits via the fabric by adopting the same joint repulsion term in [13] . Briefly, we construct an upper joint limit task space, x = qq, and lower joint limit task space, x = qq, where q and q are the upper and lower joint positional limits, respectively. The fabric term in these spaces consists of the metric, M(x) = diag max(-sgn( ẋ), 0) k b x , where k b ∈ R + is a constant gain, and acceleration ẍ = g -b ẋ, where g ∈ R n+ (n is the joint space dimensionality). Effectively, the system accelerates away from a joint limit with increasing priority as distance to the limit decreases.Since the fabric's exposed action space has fewer dimensions than controlled joints of the robot, we must resolve redundancy issues. This is easily accomplished by following the configuration space geometric attractor as covered in [13] . Specifically, we create a geometric attraction term in the full joint space of the arm with metric, M(x) = mI is a constant isotropic mass, where m ∈ R + . ẍ = -k a ∥ ẋ∥ 2 tanh(α a ∥x -x g ∥)x-xg ∥x-xg∥ , where k a ∈ R + is a constant attraction gain, α a is a constant sharpness parameter, and x g is a target state in this space. This is almost exactly the same design as the other two attraction terms, except the acceleration is homogeneous of degree two in velocity (HD2). This enables the fabric to guide the full robot movement towards x g in configuration space, but not prevent the convergence to x g in the PCA and pose taskmaps. The introduction of this term makes the fabric full rank, and furthermore, we set x g to an elbow-out, fingers-curled configuration. This specific target promotes kinematic manipulability by allowing the palm to move close to the table and robot. Without it, the overlying FGPs struggle in grasping objects closer to the robot base.Our initial design for DextrAH's geometric fabric (called a cspace fabric) allowed for issuing joint position targets as the action space instead of the PCA and palm position targets. However, with this geometric fabric design, we found that training times were very long even for single objects and multi-object training did not succeed at all. For example, Figure 6 (right) shows RL training curves for both geometric fabric designs and reveals that the fabric with joint position actions takes four times longer to achieve the same level of grasping performance. Moreover, due to the underspecified nature of the reward function and higher dimensional action space for the cspace fabric, awkward and unnatural grasping behaviors emerged. For example, Figure 6 (left) shows how the policy tries to grasp objects between the ring and middle finger and the thumb and palm. These unnatural strategies did not transfer well. Moreover, finger deranged finger arrangements also emerged, resulting in aesthetic degradation of the behavior. An important aspect for a controller is the action space it creates for overlying modules. Within the context of policy learning, the chosen action space can have significant impacts on learning performance and overall behavior. The action space for DextrAH-G enables finger-arm coordination, promotes natural grasping motions, and facilitates RL training for grasping many different objects in simulation at scale. We now focus our attention on part of the action space that involves the fingers. To create an efficient finger action space for reinforcement learning, we find a linear taskmap by running PCA on Allegro finger joint motions derived from retargeting human grasping data. To begin, we leverage a few demonstrations of grasping data from the DexYCB dataset in [47] . This dataset contains 3D point motion traces of human fingertips, joints, and palm of humans throughout object grasping trials. Since the Allegro hand is much bigger than a human hand and only has four fingers, the index, middle, ring, and thumb fingertip points, x h ∈ R 12 , were scaled (scaling factor α = 1.6) and aligned with the Allegro hand, x r ∈ R 12 (stacked fingertip points). We aligned the points by creating a palm-fixed coordinate system for both the human hand and Allegro hand by similarly placing origin and orthonormal axes as in [48] . Points were expressed in their respective palm-fixed coordinate system. We then optimize the following loss, L, that solves for Allegro joint angles, q r , for each data point x h (in sequence). For the first datapoint, we initialize q r to zeros. Optimization solves for an unconstrained q, which is then passed through a differentiable saturation function q r = 1 2 (tanh(q) + 1)(q -q) + q, where q and q are the upper and lower joint positional limits of the hand. The loss we optimize via ADAM isEQUATIONwherex c = [ x T , x T , x T ,x T ] T stacks x, where x is a single 3D point used for encouraging power or precision grasps. For instance, we place the point on the robot palm to encourage the fingers to fully curl, resulting in a power grasp. We place the point, centrally located among the fingertips to encourage precision grasping. x r are the stacked Allegro fingertip points by running forward kinematics on q r . γ = 1 -i+1 n is a blend factor that shrinks to 0 over a motion data trace (i is the index of the trace and n is the number of datapoints in a trace). Retargeting for the first data point in a motion trace places all the weight on optimizing the first term in (4) . Retargeting for the last data point in the motion trace places all the weight in optimizing the second term in (4). This effectively shifts the optimization objective from trying to generate a hand shape that mimics that in the dataset towards one that drives the fingertips towards a power or precision grasp. Finally, λ ∈ R + is a regularization weight of the retargeted angles towards some fixed, desired angles, q reg . For a precision grip, q reg = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.0, 0.75, 0, 0] (an opposed thumb, fingers straight configuration). For a power grip, q reg = [0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0.75, 0, 0] (an opposed thumb, fingers curled configuration). Overall, this retargeting process generates a jointlevel dataset of Allegro precision and power grasping motions. Please see video for demonstration of retargeted motions.Given the Allegro grasping motion dataset, we apply PCA to find a rectangular projection matrix with the most dominant eigenvectors. After several applications of PCA, we discover that the five most dominant eigenvectors explain 98% of the variance in the data, which is also consistent with [17] . This rectangular matrix serves as a taskmap within the fabrics framework and exposes a 5dimensional action space for a fabric-guided policy. Not only did this action space facilitate the learning of high-performance grasping behavior via RL, the grasping behavior itself looked natural as well without any additional reward shaping required. Please see video for fabric behavior that leverages this action space. Figure 7 shows a visual comparison of example grasps using PCA action spaces of different sizes.We define our reward as a weighted sum of individual reward terms r = i w i r i ∈ R, where r i ∈ R and w i ∈ R are the reward and weight associated with the i-th reward term, respectively. Let e ∈ R be an error term we want to minimize and e smallest ∈ R be the smallest the error term has been in this episode so far. We define a stateful function minimize(e) = max(e smallest -e, 0), which only gives a positive reward if the error term drops below the smallest it has been so far, otherwise it gets no reward. This means that when the reward is positive, e has dropped below e smallest , so we subsequently update e smallest so the teacher policy does not get additional reward for staying at the same error in subsequent timesteps. The 5D PCA action space enables more diverse grasping behavior while staying on the manifold of human-like grasping motions.We define 1(c) = 1 if c is true else 0, z(x) = x z , and lifted(x) = 1(z(x) > z lif ted ). We use the following reward terms: where z lif ted = z table + 0.2m, z table is the z posititon of the table, d success = 0.1m, T success = 15, T max = 150 is the max episode length, and T is the current timestep (with actions being taken at 15 Hz, this is 1 second for success and 10 seconds for max episode length). The associated weights are w to-obj = 5, w lif t = 50, w lif ted = 50, w to-goal = 1, 000, w reached = 40, w success = 100.r to-obj = minimize(||x f ingertips -x obj ||) r lif t = minimize(z lif ted -z(x obj )) × (1 -lifted(x obj ))We reset the environment if the object has fallen below the table, if the robot received the success reward r success (robot held the object at the goal position for T success consecutive timesteps), or if the episode time limit is reached:reset = any(z(x obj ) < z table , r success = 1, T > T max )We follow a reward structure similar to Petrenko et al. [49] . First, we only have positive rewards, so there is no incentive for the policy to end the episode prematurely to avoid punishment. Second, we use the stateful minimize(e) function to only reward the policy if it has reduced the error below the smallest it has been before. This ensures that the teacher policy only gets reward for getting closer to a target than it ever has in this episode, so there is no reward for just staying in place (unless it is at the goal). It is also easier to tune reward weights because the maximum cumulative reward from one of these terms is simply the initial value of e at the start of the episode. Third, we use a lifted(x obj ) term to turn on and off reward terms to prevent the robot from getting r to-goal rewards by pushing the object along the table without lifting it first.We tuned the weights in a very simple manner by ensuring that each new reward term contributes more to the total accumulated reward than the previous. At initialization, the object is roughly 0.5 meters away from the hand, so the total reward from r to-obj is about w to-obj ×0.5 = 2.5. The object is lifted by 0.2 m, so the total reward from r lif t is about w lif t × 0.2 = 10. The total reward from r lif ted is 50. The distance from the object to the goal position after being lifted is roughly 0.3 m, so the total reward from r to-goal is about w to-goal ×0.3 = 300. r reached is given at every timestep that the object has reached the goal, so this could be at most w reached ×(T max -T ) ≤ 40×150 = 6000. However, we want to ensure that there is sufficient incentive to acquire r success so that the policy holds the object at the goal position, or else the policy may purposely go into and out of the reached region to prevent early termination of the episode from succeeding. Thus, we set w success higher than w reached , so that the total reward from success w success × (T max -T ) ≤ 100 × 150 = 15000 will always be greater than the total reward from reached w reached × (T max -T ) ≤ 6000, as long as the robot can succeed. Thus, there will always be incentive and greater potential reward for getting r success over just getting r reached . Again, note that the cumulative reward from every reward term is larger than its previous term, and there is minimal opportunity for the policy to exploit reward hacking behavior.At the beginning of each episode, we sample an object pose and a robot configuration. The object position is uniformly sampled from [-0.18125m, 0.18125m]×[-0.29m, 0.29m]×[0.05m, 0.051m] with respect to the center of the table, which is half of the table length and width. For object orientation, we note that some objects in their default orientation are upright, so almost every sampled orientation will have the object topple over. Because we want our policy to handle both the upright and not upright case, we set the object orientation to be the upright orientation with probability of 0.5 and uniformly randomly sampled orientation with probability 0.5. For the robot's initial configuration, we set the robot's joint position to a predefined default position (incorporating elbow flare) with uniformly sampled noise with sampling bounds ±10% of the joint limit range. We also randomly sample small joint velocities in [-0.1, 0.1]. Fabric-based collision detection is then employed to conduct rejection sampling to ensure that the initial state is not in collision.At each timestep, we sample random unit vectors u f ∈ R 3 and u τ ∈ R 3 , then we apply random forces f perturb = f scale mu f and random torques τ perturb = τ scale Iu τ with probability p = 0.1, where m ∈ R is the object mass, I ∈ R 3×3 is the object inertia, f scale = 50 is a force scaling parameter, and τ scale = 100 is a torque scaling parameter.The policy observes x obj = x obj +n x,uncorr +n x,corr and q obj = q obj +n q,uncorr +n q,corr , where n x,uncorr ∼ N (0, σ xyz,uncorr ) and n q,uncorr ∼ N (0, σ rpy,uncorr ) are uncorrelated noise sampled once every simulation timestep, and n x,corr ∼ N (0, σ xyz,corr ) and n q,corr ∼ N (0, σ rpy,corr ) are correlated noise sampled once at the start of each simulation episode and kept the same for the duration of the episode, with σ xyz,uncorr = σ xyz,corr = 0.02m and σ rpy,uncorr = σ rpy,corr = 0.1rad. Because poses are represented as quaternions, we convert the roll-pitch-yaw noise to a quaternion then perform quaternion multiplication with the ground truth quaternion.We apply randomization to many parts of the system, including the robot's PD gains, gravity, object mass, and object friction, and we apply observation and action noise. The exact parameters are shown in Table 2 .Our network architecture is shown in Figure 2 . The critic uses a Multi-Layer Perceptron (MLP) network [50] because it has access to all privileged information, so it does not need to capture temporal dependencies. The teacher policy uses an MLP layer followed by a Long-Short Term Figure 8 : Dataset used in simulation. Objects enumeration starts top left and ends bottom right: e.g. object 15 would be indexed as [1, 1] in Python if this was an array data structure. Circles around the object indicate that they were easiest (green) or hardest (red) for the expert policy to grasp. Further information on per-object success analysis is provided in Appendix F. 4 Our geometric fabric allows our policy to have light contact with obstacles. However, it can also restrict exploration near these collision surfaces, which can result in slow reinforcement learning training. To overcome this, we train our teacher policy with a curriculum on cspace damping. Cspace damping is a geometric fabric parameter that influences how quickly the robot can move and how much collision contact is allowed.We start the training with cspace damping of 0, which allows for more environment contact and maximum exploration. From there, we increase the cspace damping by 0.1 when the average success rate of the policy goes above a threshold of 90%. We repeat this until cspace damping of 10 is reached, which allows minimal contact with the environment and can be safely deployed in the real world.We find that random seed plays a factor in the behavior of reinforcement learning policies. Different random seeds result in different high-level grasping behavior and different real-world performance. We believe this is because our reinforcement learning formulation is an under-constrained problem, as we only give task rewards, so there are many possible policies that could achieve high reward. Additionally, we train our policy completely from scratch and give no shaping rewards, so the randomness of exploration can result in very different learned behaviors. Our approach to this has been to train multiple policies with different random seeds, and then comparing simulation results and visually viewing policy behavior, and testing policies in the real world. This can sometimes result in a "random seed shootout", where we simply test several policies out in the real world and choose the best one (this strategy is the same as in [52, 13] . These different policies often have different strengths and weaknesses (e.g. some better at low-profile objects, some better at large objects). We have found that simulation performance can give some indication about real-world performance, but there is no replacement for real-world testing.We also find that for a given random seed, the policy typically sticks to the same general strategy over the training run, with slight improvements to handle different objects and adjust to curriculum updates. It generally has a similar approach to grasping for most objects. We hypothesize that we could enable more diverse grasping behavior by training multiple teacher policies and critics, each with different behavior. Then during policy distillation, we can supervise the student policy to match the action of any of the teachers or the teacher policy with the highest predicted value from the critic.We train our policy using Isaac Gym [54] , a high-performance simulator that leverages GPU parallelization, which allows us to simulate 8192 robots simultaneously per GPU. We train using four NVIDIA V100 GPUs, each with 32 GB of VRAM. This compute allows us to run simulation at about 20k FPS, where each frame is one action step with a control timestep of 66.7 ms (15 Hz) and is broken up into 4 simulation timesteps of 16.7 ms (60 Hz). We train for about 4.7B frames (about 9000 policy updates), which takes about 68 hours (wall clock time). This amounts to about 10 years of simulated training time (4.7B / 15 / 3600 / 24 / 365). We train our teacher policy using Proximal Policy Optimization (PPO) [55] using a highly-optimized GPU implementation called rl games [56] , which uses vectorized observations and actions for faster training.We train our teacher policy with a learning rate of 5e-4, a discount factor γ of 0.998, entropy coefficient of 0, and a PPO clipping interval ϵ clip of 0.2. We also normalize the observations, values, and advantages, and we train the policy with 5 mini-epochs per policy update. Using a horizon length of 16 (number of timesteps between updates for each robot, with all robots running in parallel), and 8192 simulated robots. We use a critic MLP of size [512, 512, 256, 128], a policy MLP of size [512, 512], and a policy LSTM of size 1024.As seen in Figure 9 , the privileged FGP trains well within 20 hours of wall-clock time with the vast majority of objects successfully being lifted from the table, brought to a target position, and held there. The remaining training time residually improved policy reward with correspondingly residual improvements in policy performance across all scoped metrics. Overall, the required training time for this complex task is favorable and in line with other related manipulation works [52, 13] .To encourage policy robustness to noise, disturbances, and discrepancies in physics, we apply random force-torque perturbations to the object during grasping, lower contact friction, object pose noise, and significant domain randomization. Like prior works [52, 13] , these additions are necessary for strong sim2real transfer for robot manipulation problems. Without it, unnatural and frail strategies emerge as seen in Figures 10, 11 , where policies attempt to pick up objects without proper force-and form-closed fingers arrangements. Attempts to grasp objects between the middle and ring fingers only or between the index and ring fingers only were observed. With the above additions, policy strategies become much more robust as seen in Figures 12 and ultimately facilitated DextrAH-G's high real-world performance.We simulation-rendered depth images are entirely free of noise as shown in Figure 13 . To address the visual sim2real gap, we add a variety of augmentations to the rendered depth image. Specifically, the following augmentations are added: 1) a pixel value would be set to 0 with probability p dropout = 0.003, 2) a pixel value would be set to a random value ∈ (-0.5, -1.3) with probability p randu = 0.003, 3) a linear segment of pixels of up to 18 pixels length and 3 pixels width would appear with probability of p stick = 0.0025 to mimic robot wires and other artifacts, and 4) the uncorrelated and correlated depth noise models and their parameters are used exactly as reported in [57] . We also stochastically perturbed the depth camera placement in simulation during distillation so that the distilled FGP gained robustness to calibration errors in the real world. The effect of these augmentations can be seen in Figure 13 . See our video to better visualize the depth image noise in both simulation and the real-world. To compute these metrics, we keep a running average over the 100 most recently finished episodes. "lifted" is a boolean that is 1 if the object has been lifted at least 20 cm above the table surface during the episode. "reached success region" is a boolean that is 1 if the object has been less than 10 cm away from the goal position during the episode. "has enough consecutive successes to end" is a boolean that is 1 if the object has been held within 10 cm from the goal position for 1 simulated second (15 consecutive timesteps). "reward" is the total accumulated reward at the end of the episode. "num drops metric" is the number of times that the object has been lifted above 20 cm and then subsequently brought below 10 cm (typically when the object has been dropped).Figure 10 : Unstable grasps emerge with naive environment implementation without random force and torque perturbations, lower friction, object pose noise, and domain randomization. The policy is able to find a grasping strategy that works very reliably in simulation, but fails in the real world due to real world perception and control errors. During distillation, we run n = 6 steps before running backpropagation through all n steps (BPTT [58] ) and updating π depth weights. We believe that a similar effect could have been achieved by using the last n steps (concatenated) as input to a stateless model. This number of steps corresponds to a time window of 0.4s. Additionally, we update weights every time an environment is done -the object falls off the table, 10 seconds expires, or the robot grasps the object.Typically, the time needed for the student to learn to match the teacher is about 12 hours (wall clock), or 140 rollouts across 480 parallel environments on a single NVIDIA GeForce RTX 3090. This compute allows us to run simulation at about 140 FPS, where each frame is one action step with a control timestep of 66.7 ms (15 Hz) and is broken up into 4 simulation timesteps of 16.7 ms (60 Hz). This amounts to about 6.05M frames, which is about 4.67 simulated days (6.05M / 15 / 3600 / 24).The student architecture consists of simple encoders for different modalities and a state-based network that processes encodings. Our architecture is shown in Figure 2 . o robot and x goal are encoded using 3-layer MLPs (512,256,128) with elu activation, while I is encoded using three convolutional layers with increasing depth (16, 32, 64) , kernel size 3, stride 1, padding 1, followed by max-pooling operations (kernel size 2, stride 2) with ReLU activations, and followed by a 2-layer MLP (128,128) also with ReLU activation. Encodings of all modalities are concatenated and passed through a GRU layer (1 hidden layer, 1024 units) that predicts â. The whole architecture is learned from scratch. We opt for this rather simple architecture due to GPU memory constraints since the whole distillation process is executed on a single GPU (RTX 3090). We believe the student architecture would benefit from a better perception backbone (e.g. ResNet).We designed the student π depth to be a state-based model (GRU) because the teacher π teacher is also a state-based model (LSTM), and we found that a state-based teacher achieves higher success rates than stateless teacher (MLP). Further, we believe having a state allows π depth to reason about the object's position during occlusion since the model has seen the object before the robot got so close to the object to occlude it. This means any model architecture that takes history in some way in theory has the ability required to mimic the expert. We opted for GRU, but we believe any form of a state-based model (RNN, LSTM, GRU) or a stateless model (MLP) with history as concatenated input should work.We evaluate our teacher and student policy in the simulator across 140 different objects from the training dataset. The objects are visualized in Figure 8 . The policy has a 10 second window to grasp the object. If the object is lifted to the target position in the air, the attempt is labeled as successful. Otherwise, it's a failed attempt. Results are shown in Figure 14 .Since the teacher policy has access to privileged information, it achieves a slightly higher success rate compared to the student. The average time needed to grasp an object is about 4 seconds, which matches our real-world observations. We believe this type of per-object analysis could be leveraged during training to explicitly train more on the objects that are harder to grasp.To complete a bin-packing application, we leverage DextrAH-G and a simple state machine. The bin packing program initializes by immediately engaging DextrAH-G. If the z-coordinate (height off the table) of the predicted object position is sufficiently high, then we freeze the last issued PCA action and give the fabric a goal pose over the bin for a fixed amount of time. Afterwards, a finger-opening PCA command is issued to the fabric to release the object for a fixed amount of time. Finally, a nominal PCA and pose target are issued to the fabric for a fixed amount of time to bring the robot back to a nominal pose. DextrAH-G's high performance and auxiliary object position prediction makes application programming extremely simple. The only additional complexity is that if the palm height ever swings too high (indicating policy collapse), then DextrAH-G is deactivated and Figure 14 : Success rates per object (10s allowed per attempt). Figure 8 visualizes the objects used we issue the geometric fabric a single command to bring the robot back to a nominal configuration for a fixed amount of time. Afterwards, we re-engage DextrAH-G.We summarize our nomenclature in Table 3 .We create several additional plots to further capture DextrAH-G's performance with greater granularity. Figure 15 characterizes DextrAH-G's success and failure modes. Overall, DextrAH-G has a 89% grasp success rate and an 87% transport success rate across the many tested objects. The slight drop between these two covers cases where the object falls out of the hand during transportation. The principal failure mode for grasping is the robot pushing the object out of the allowable work area as delineated by the geometric fabric. The less frequent failure mode occurs when DextrAH-G repeatedly attempts to grasp an object, but consistently fails. In this case, the object is reset by a human.A time-series plot of CS is shown in Figure 16 . Overall, DextrAH-G consistently grasps and transports several objects in a row before experiencing a failure. Several times, DextrAH-G moved more than 20 objects in a row before experiencing a failure. DextrAH-G did produce failures within 5 CS several times due to the previously mentioned failures modes. Figure 19 shows the success rates per object for all tested objects in the continuous run trials. Overall, most objects were handled well, but a few like the pot, small bottle, and green cup caused a greater level of handling errors. Training DextrAH-G on an even greater variety of objects in simulation could help close the handling gap on these objects and further improve DextrAH-G's performance more broadly.8th Conference on Robot Learning (CoRL 2024), Munich, Germany.LimitationsLimitations of DextrAH-G are as follows. First, the overlying FGPs issue goals in the PCA taskmap for controlling the fingers. While this was intentionally chosen to focus grasping behavior, it does limit the kinematic dexterity of the robot. Second, some amount of obstacle avoidance behavior should ideally be learned based on sensory inputs to reduce dependency on model-based behaviors. Additionally, RL still struggles to explore near high-cost regions. For example, the fabric's obstacle avoidance keeps the robot from significant collision with the table, but this behavior also makes effective exploration in these regions more difficult resulting in reduced performance for low-profile objects. Improving exploration strategies, the RL algorithm itself, or learning collision avoidance via a curriculum are possible avenues forward. Finally, DextrAH-G can only handle one object in the scene at a time and likely needs changes (e.g. segmentation) to work effectively with clutter. 


Decision: This paper proposes to use teacher-student distillation with a geometric fabric controller for dexterous grasping. It trains the controller in simulation and transfer it to the real-world. Reviewers appreciated the proposed method and experimental design. Reviewers raised concerns about relationships with past work, some claims that are without empirical evidence, some missing experiments, and clarity of presentation in certain sections and figures. The author response addressed reviewer concerns and all reviewers continue to vote for the acceptance of the paper.

Reviews: review: title: title: Impressive empirical results but also needs more evaluation
paper_summary: paper_summary: This paper proposes to use teacher-student distillation with a geometric fabric controller for dexterous grasping. They train the controller in simulation and transfer it to the real-world.
main_review: main_review: Strength:

The trained controller can be transferred to the real-world and achieves very smooth and dexterous grasping behavior on many different objects.

It is a good demonstration to push the boundary of sim-to-real via teacher-student framework.

Weakness:

The proposed DextrAH-G system consists of two parts: FGP and a geometric fabric controller. If I understand correctly, both components were used in a previously published work [13]. Given this, I think the authors should highlight the differences between this work and the previous work. I also want to emphasize I’m not criticizing the novelty as I value the system work. And from my perspective, there are at least three differences: 1) different tasks; 2) modeling and avoiding environmental collisions; 3) action space. However, I think a clearer explanation on the differences instead of saying FGP and geometric fabric controller is a contribution (line 43 - line 47) would make this paper more focused.

Some claimed contributions do not have empirical evidence to support them. For example, the authors propose using object position prediction as a contribution (line 47). However, there are no experiments showing the benefit of this. Further ablation experiments on system with/without position prediction are needed.

Another missing experiment would be to study an alternative for collision avoidance. For example, one could add a penalty term in reinforcement learning training to penalize self-collision and distance between the object and environment.

The ablation study of action space design is only presented in the supplementary material (Figure 5). This should be one of the important contributions and should be moved to the main paper.

I hope the authors can make section 3.1 clearer, either using graphic illustrations or providing more high-level summary of motivation for each mathematical equation. It also does not fully explain how to avoid collision with environments.

The author claims the proposed method “enables safe real-world deployment even with delusional and hazardous policies”. I think this claim should be studied. For example, if a policy is not fully trained or with worse reward formulation, what is the performance with / without geometric fabric controller?

The success rate is measured by “until the grasp succeeds or we experience an irrecoverable failure.” I think the authors should also evaluate each single grasping success. From the supplementary video, it is common for the policy to fail to grasp and for the object to be moved to another place. Although this demonstrates the robustness of the policy, claiming a 100% success rate in Table 1 could be misleading.

From the video, it seems the smaller objects are harder to grasp. How does the scale of the object affect the policy?

I hope the authors could elaborate more on the student policy evaluation in simulation. The student policy is only trained on 480 environments, therefore the type of objects and randomization coverage is far less than the setting in teacher policy training. As a result, comparing the success rate between teacher/student during training may be unfair.

Minor Comments:
- The term multi-object grasping (line 45) refers to grasping multiple objects in-hand. This work apparently only deals with single-object grasping. It should revise this term.
- No explanation on AH-G in the name.
questions: questions: About object position prediction: What is the system's performance with and without position prediction?

Are there alternative ways to do collision avoidance? For example, one could add a penalty term in reinforcement learning training to penalize self-collision and the distance between the object and the environment.

Can the proposed controller improve the performance or safety of a worse policy?

What is the single grasp success rate for the policy?

Is there an evaluation of the relationship between performance and object scale?

It would be helpful to evaluate the student policy with thousands of parallel environments. One can achieve this using multiple GPU parallelization.
limitations: limitations: 3
review_summary: review_summary: I would recommend acceptance for now due to the strong hardware results. However, I want to emphasize that several experiments and clarifications are needed during the revision period.
title: title: Review of Submission 22
paper_summary: paper_summary: This paper introduces a novel depth-based dexterous grasping policy trained entirely in simulation. The proposed method, DextrAH-G, combines reinforcement learning, geometric fabrics, and teacher-student distillation to enable a 23-motor arm-hand robot to safely and continuously grasp and transport a variety of objects.
main_review: main_review: Review:
Strengths
1. The combination of geometric fabrics, reinforcement learning, and policy distillation proves to be an effective approach for transferring policies trained in simulation directly to real-world hardware without additional tuning. It also demonstrates smooth and agile motion during grasping, thanks to geometric fabrics.
2. The experiments cover both simulation and real-world scenarios, demonstrating the robustness and versatility of DextrAH-G.
The creation of grasping manifold by retargeting human grasp helps regularize the action space and make sure the policy rollouts reasonable hand actions.
3. The presentation of this paper is clear and easy to follow.

Weakness:
The experiments mainly show grasping middle-sized rigid objects. It would be interesting to see if RL can be used to generate behaviors that better leverage the dexterity of the hand to grasp objects. For example, a thin, large object like a book, which is not directly graspable with parallel jaw gripper. Could this behavior be learned by the proposed sim-to-real method?
questions: questions: I would like the authors to answer the question in the weakness part.
limitations: limitations: 3
review_summary: review_summary: The paper proposes a simple but effective method for dexterous grasping, although showing relatively simple grasping behavior.
title: title: Review
paper_summary: paper_summary: This paper presents an RL approach to learn closed-loop, robust visuomotor grasping policies through a combination of underlying geometric fabric controller and teacher-student distillation policy.
main_review: main_review: I think this paper presents a very interesting solution to a problem with impressive execution.  Overall, I think the result is very convincing, and the methodology and the pipeline all make sense to me. 

That being said, I think there is great room for improvement despite all the impressive results: 

1. The pipeline drawn in Figure 2 is very hard to follow, even though the overall structure makes sense to me. If the authors want a broader audience, I strongly recommend that they redo the figures (the arrows are extremely hard to follow). 
2. For details of 3.2 and 3.3, quite a lot of symbols are used. I think it will help the readability of the manuscript if the authors can have a table of math symbols that explains which subset of symbols are used in which step of the method. It would be even better if the most important notations (e.g., the math notations for the geometric fabric target) were highlighted and visualized in Figure 2.

I would like to give this submission a very good recommendation, conditional on the authors carefully addressing all the comments, especially about the paper presentation part.
questions: questions: 1. Is there any concrete interpretation of the 5 PCA dimensions of fingers? 
2. If authors need to extend this work to other types of hand-object interaction, is the PCA approach sufficient? Or should we use the full-hand finger joints?
3. Address the concerns about paper writing in the Review section above, especially revising figure2.
4. Does the location of the point clouds need to be transformed to a fixed workspace assuming access to a calibrated camera extrinsic? Or it
limitations: limitations: 3
review_summary: review_summary: See my review.