DexCatch:  Learning to Catch Arbitrary Objects with Dexterous Hands

Authors: fengbo  lan
shengjie  wang
yunzhe  zhang
haotian  xu
oluwatosin  oseni
ziye  zhang
yang  gao
tao  zhang
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 13:33:11

Abstract: Achieving human-like dexterous manipulation remains a crucial area of research in robotics. Current research focuses on improving the success rate of pick-and-place tasks. Compared with pick-and-place, throwing-catching behavior has the potential to increase the speed of transporting objects to their destination. However, dynamic dexterous manipulation poses a major challenge for stable control due to a large number of dynamic contacts. In this paper, we propose a Learning-based framework for Throwing-Catching tasks using dexterous hands (LTC). Our method, LTC, achieves a 73% success rate across 45 scenarios (diverse hand poses and objects), and the learned policies demonstrate strong zero-shot transfer performance on unseen objects. Additionally, in tasks where the object in hand faces sideways, an extremely unstable scenario due to the lack of support from the palm, all baselines fail, while our method still achieves a success rate of over 60%.

Main paper: 
 introduction: 
 The study of dexterous hands is a widely pursued research area aimed at matching the intricate capabilities of human hands. Researchers employ these hands for delicate manipulations [1, 2, 3, 4] . Recently, researchers have applied reinforcement learning (RL) techniques to various tasks involving dexterous hands, leading to diverse outcomes [5, 6, 7] . They have tackled a series of manipulation tasks, including in-hand reorientation [6, 8] , opening a door [5] , assembling LEGO blocks [9] , and even solving a Rubik's cube [7] . These tasks can be summarized as common behaviors in daily living, where hands pick up an object, reorient it in hand, and then either place it or use it as a tool [8] .While pick-and-place can fulfill the requirements of many daily tasks, dynamic throwing-catching holds the potential for more efficient task execution [10, 11, 12] . For example, two robots can employ throwing and catching to efficiently transfer objects instead of pick-and-place, as shown in Fig. 1 . Meanwhile, it can extend the working space of robots, which avoids the potential collisions for robot interactions [10] . However, precisely catching arbitrary objects proves to be highly challenging [13] . 1) First, the dynamic contact with strong force makes the object fall from the hand. In previous work [13] , model-free reinforcement learning methods achieved nearly 0% success rate in the throwing-catching task within 30 million environment steps. 2) Secondly, varying object-centric properties (e.g. mass distribution, friction, shape) require the catching behavior robust enough, especially for objects of daily life. To target this problem, previous methods must learn the dynamics of a moving object [14] . The additional module requires expert demonstrations and increases the system's complexity. 3) Thirdly, when faced with flying objects coming from random directions, the catching hand must adopt different postures, enabling a larger working space. However, the challenge arises in catching objects with hands facing sideways, where objects easily fall due to the lack of any supporting surface, as illustrated in Fig. 1 . Due to this challenge, the task of catching objects with hands facing sideways has not been explored previously.To address the aforementioned challenges, we propose a Learning-based framework for Throwing-Catching tasks using dexterous hands (LTC). The simple but effective framework can achieve superior performance across diverse daily objects, in various throwing and catching scenarios where the hands face upward or sideward. The main contributions of this work are as follows.• We propose a simple and effective framework using a model-free reinforcement learning algorithm, to solve the throwing-catching tasks. Particularly, it can catch the flying objects even on the most challenging catching task with the hands facing sideward.• We propose a hybrid advantage estimation method, which considers Lyapunov stability. While the effectiveness of the stability condition has primarily been verified for robustness, this is the first time it has been observed to benefit the generation of stable catching behavior for flying objects, thereby improving the accuracy of throwing-catching.• By using compressed point cloud features of objects as input and applying domain randomization during training, our method excels in generalizing to daily objects with diverse properties (e.g., mass distribution, friction, shape, and initial pose), and performs effectively even on previously unseen objects. 
related work: 
 The versatile dexterous hand plays a crucial role in performing various tasks across different humanfocused settings. However, effectively managing dexterous hand systems poses a challenge due to their complex actions and intricate contact interactions [15] . Traditionally, controllers for manipulation tasks heavily rely on dynamic models and trajectory optimization techniques [1, 2, 3, 4] . The paper "Catching Objects in Flight" is highly regarded in this field [14] . To enhance model accuracy, their method learns the dynamics of a moving object and develops a reachable and graspable model using concise demonstrations. Furthermore, Williams et al. [16] achieved success using the Model Prediction Path Integration Control (MPPI) method to skillfully manipulate a cube. Charles Voss et al. [13] improved the MPPI method to manage task handovers between two hands. However, they focused only on a single object due to the complexities involved in model learning.In recent times, RL-based approaches simplify the controller design process, and model-agnostic techniques have gained substantial popularity within the control domain. In the realm of dexterous manipulation, numerous endeavors have showcased remarkable advancements compared to conven-tional controllers [17, 18, 19] . Entities like OpenAI have harnessed reinforcement learning-based controllers to successfully rearrange blocks and solve Rubik's cubes [7] . To expedite training, a blend of reinforcement learning and imitation learning has been employed to facilitate dexterous hands in acquiring diverse skills such as repositioning objects, utilizing tools, opening doors, and more [5] . Acknowledging the existing method's limited ability to generalize, Chen et al. [8] introduced an effective system capable of learning how to reorient a wide array of objects. For the cooperation between two hands, Zakka et al. [20] and Chen et al. [19] proposed a series of challenging tasks for bi-manual dexterity, like mastering the piano and lifting the pot. Furthermore, a recent work introduced the Population-Based Training (PBT) algorithm in RL learning, leading to solving the two-handed reorientation task [18] . However, most recent works [18, 21] only focus on a single object or a kind of posture of hands. Our method successfully achieves the throwing and catching of diverse objects, especially for the task with hands facing sideward.3 Preliminaries 
reinforcement learning: 
 The throwing-catching tasks of dexterous hands can be formalized as an infinite horizon Markov Decision Process (MDP), which is defined by a tuple (S, A, p, R, ρ 0 , γ). Among them, S ∈ R n is the state space, A ∈ R m is the action space, p :S × A × S → [0, 1] is the transition probability distribution, r : S × A → R is the reward, γ ∈ [0, 1)is the discount factor, and ρ 0 : S → [0, 1] is the initial state distribution. The goal of RL is to find an optimal cumulative return of the MDP with the policy π :S × A → [0, 1].The problem is formulated as maximizing the expected discounted cumulative return:E s0∼ρ0,at∼π(•|st),st+1∼p(•|st,at) [ ∞ t=0 γ t r(s t , a t )].The PPO algorithm is an on-policy reinforcement learning approach. It excels in robotics tasks featuring high-dimensional action spaces. The algorithm contains an actor and a critic network, where θ and ϕ denote the parameters of the actor network π θ (a t |s t ) and the critic network V ϕ (s t ), respectively. For the policy function π θ (a t |s t ), the output corresponds to the mean and standard deviation of the Gaussian distribution. On the other hand, the value function V ϕ (s t ) yields the current state's value. Building upon the policy gradient method, the parameter updating formula for the actor-network takes the following form:EQUATIONwhere ϵ is a hyperparameter,A π θ k (s t , a t ) = G π θ k t -V π θ k ϕ (s t ), is the advantage function. G π θ k tis cumulative returns. For parameters ϕ, the update is expressed as:ϕ k+1 = arg min ϕ E st,at∼π θ k [(G π θ k t -V π θ k ϕ (s t )) 2 ](2)However, because the calculation of G π θ k t value needs the whole data of a finished episode, we approximate G π θ k t to r t + γV π θ k ϕ (s t+1 ) through TD learning method. 
dexterous catching tasks: 
 Our research centers on the challenges of throwing-catching tasks using dexterous shadow hands. Our aim is for these shadow hands to adeptly handle diverse objects in various positions and execute agile catches using different gestures. To address this objective, our design comprises three key components.Figure 2 : A: The method takes as input the environmental observation and the point cloud feature of the object. Then, it learns the catching policy for dexterous hands through an Actor-Critic structure. B: The Lyapunov function, the policy function and the value function are estimated using neural networks. Their network structures are similar, and the only difference is the dimension of the output. For example, the Lyapunov function contains a linear layer, a batch normalization layer, and a penultimate normalization layer, and the last linear layer takes a scalar as output.Environment Setting The shadow hand models, which emulate human skeletons, possess 24 degrees of freedom (DOFs) and are operated using 20 pairs of agonist-antagonist tendons [8] . The presence of force sensors on the ten fingertips facilitates the acquisition of force-related data during catching and throwing maneuvers. In our paper, we investigate five catching scenarios, namely Overarm Catch, Overarm2Abreast Catch, Under2Overarm Catch, Abreast Catch, Underarm Catch and which are shown in Fig. 4 respectively. In these five tasks, one hand should throw an object to the other hand, and the other one should hold it in the palm later. Some tasks with vertical hands are challenging. It is because the object in the hands is not stable without the support from the palm.Object-generalized Setting Our design objective is to generalize across diverse objects by introducing point cloud features, which are obtained from a pre-trained model, PointClipV2 [22] . Additionally, we apply Principal Component Analysis (PCA) to compress the features. The experiments illustrate that the shapes of different objects can be distinguished using the compressed features. More details can be found in Appendix B.State, Action and Reward 1) State: The state space consists of 424 elements. Among these, the 422-dimensional state incorporates crucial information: hand position, hand orientation, angular and linear velocities of the hands, position and speed of each hand's degree of freedom, force-torque sensor readings of the hands, target object's position and orientation, as well as the current object's position and orientation. The above observation corresponds to the setting in [19] . Additionally, a two-dimensional state features the input derived from the point cloud data of the object. It serves as the perceptual part for the agent regarding the object. 2) Action: The action space comprises 52 dimensions, encompassing 20 pairs of agonist-antagonist tendons for both hands, along with three dimensions each for the required for ce and torque of each hand. 3) Reward: The reward design comprises three components: the difference in position and posture between the object and the target, the penalty for the action scale of the shadow hand joints, and a combination of task completion bonuses and failure costs. We explain the details of the reward in Appendix C. 
method: 
 In this section, we introduce the LTC algorithm in detail. LTC is an end-to-end framework based on the Proximal Policy Optimization (PPO) [23] algorithm. Since the throwing-catching task is a dynamic and agile behaviour, the basic PPO algorithm encounters two key challenges: How to achieve efficient throwing and stable catching without falling? We make some algorithmic modifications to the basic PPO algorithm. First, to accelerate the learning of throwing, we design an intrinsic advan-tage and add penultimate normalization layers in the network. Secondly, a stable catching behaviour is holding the object in the palm without falling the object. As depicted in Fig. 3 , the maximum sum reward (optimality) can not guarantee the generation of the stable catching behaviour. To encourage more stable catching, we include the Lyapunov stability condition in policy learning. The specific formulation can be found in Section 5.2.1.Figure 3 : To enhance the success rate, our approach prioritizes optimizing rewards while ensuring the stability of the grasping gesture. By integrating Lyapunov stability alongside reward optimization, we mitigate the risk of completing tasks in unstable poses. The size of the stable region varies based on task difficulty, with simpler tasks offering a larger stable region.The architecture is illustrated in Fig. 2 . Here's a breakdown of its components:1) Point Cloud Feature Extraction: The initial step involves extracting point cloud features using the Point-ClipV2 [22] . Subsequently, Principal Component Analysis (PCA) is employed to reduce dimensionality, yielding a 2-dimensional feature output. This output serves as the perception state for the shadow hands system; 2) Observation Input: The agent takes both the current observation of objects and the shadow hands and the compressed features from point clouds as input; 3) Actor-Critic Framework: The actor and critic are represented as two neural networks, which is the same as PPO algorithm. The network architecture for the Lyapunov function is depicted on the right side of Fig. 2 . This structure comprises three linear layers, with a Batch normalization layer and an ELU activation function appended to the output of each layer. Following these, there is a penultimate normalization layer [24] , succeeded by a linear layer to regulate the output dimension. The network structure for the policy and value function are similar to that of the Lyapunov value network.The hybrid advantage A all consists of 3 parts: standard advantage estimation based on PPO, stability advantage estimation, and intrinsic advantage estimation. The formula for the hybrid advantage is expressed as follows:EQUATIONEQUATION) where A π θ (s t , a t ) refers to the time differential advantage function of the PPO algorithm, A L (s t , a t ) represents the Lyapunov stability advantage function, A I (s t , s t+1 ) represents the intrinsic advantage function, and β 1 , β 2 , β 3 ∈ [0, 1] are hyperparameters. As follows, we introduce the Lyapunov stability advantage and intrinsic advantage in detail.The Lyapunov function serves as a potent tool for assessing system stability. One approach, known as the self-learning Lyapunov stability method [25] , is utilized in our paper. First, we introduce a Lyapunov stability critic network, which represents the candidate Lyapunov function V L using neural networks. Subsequently, the empirical Lyapunov risk R(Λ) [25] is defined as per Eq. ( 5). The three parts in Eq. ( 5) represent three requirements for the Lyapunov function [25] . The parameter update is then executed by minimizing the loss function associated with the Lyapunov risk R(Λ), as part of achieving the desired system stability.EQUATIONFigure 4 : Video snapshots of throwing-catching tasks performed by our method. There exist five tasks with different postures of hands, in order from difficult to easy: Overarm Catch, Over-arm2Abreast Catch, Under2Overarm Catch, Abreast Catch, and Underarm Catch. The most challenging task is Overarm Catch because the hands are both oriented vertically. Notably, our method can catch diverse objects in five tasks.where f is the dynamical system, s t is the state at step t. The empirical Lyapunov risk is nonnegative, we approximate the Lie derivative L f V L (s t ) by finite difference of the system's sampling trajectory because of the complexity of modeling this system's dynamics:EQUATION)where ∆t is the time difference between them. Since the Lyapunov function is considered to be another value function, the Lie derivative term L f V L (s t ) is the only term in the Lyapunov risk affected by the policy. Therefore, an additional goal of ensuring L f V L (s t ) < 0 is added to the policy optimization, to encourage the policy's update with the requirement of stabilization. However, the classical constraint L f V L (s t ) < 0 makes the value of the Lyapunov function likely to be stuck in a minimal number close to 0. This drawback hinders the process of policy improvement due to the lack of an explicit gradient.Therefore, we design a more strict constraint L fπ θ V L (s t ) < -kV α L (s t ), where α and k are constants ranging from 0 to 1. Note that kV α L (s t ) is non-negative. When kV α L (s t ) is larger, an intuitive phenomenon is that V L approaches 0 faster. In other words, the system can reach a stable region in fewer steps, which benefits the generation of stable catching behaviors. More details can be found in Appendix D. Hereafter, we design the part of the Lyapunov stability advantage function based on the clipped Lie derivative term:EQUATIONBased on the advantage function of the PPO algorithm, in order to improve the ability of the system efficiently, we design the intrinsic advantage function with reference to the intrinsic reward mechanism [26] . For a Markov decision process trajectory, different values V ϕ (s t ) will affect the success of the final result of this trajectory. To provide a specific example, let's consider the fact that the next states s t+1 exhibit varying conditions, leading to different values V ϕ (s t+1 ). It's worth noting that in a successful process, the value V ϕ (s t+1 ) tends to be higher. Consequently, within a Markov decision process, we can further refine the advantage function A(s, a) using empirical measures. If an action is effective, it should result in a higher value, as represented byV ϕ (s t+1 ) -V ϕ (s t ) > 0.This signifies that action a t transitions to a state with a higher success rate. Conversely, an unfavorable action can lead to a decrease in value, reflected as V ϕ (s t+1 ) -V ϕ (s t ) < 0. In order to reduce failure in the early stage of the throwing process, we consider the intrinsic advantage A I (s t , s t+1 ) as part of the total advantage function to accelerate the RL training, shown as follows:EQUATIONwhere k is a positive constant.  
experiments: 
 To test the algorithm's ability to generalize, we train and test it with objects from the YCB Pybullet dataset [27] . The results show that our algorithm outperforms baseline methods. Four essential components of our algorithm improve its performance. Additionally, our method showcases the ability to generalize different gestures across five skilful throwing-catching tasks, effectively allowing for the successful manipulation of various objects. Importantly, our approach extends its capabilities to catch unseen objects successfully.In this section, we conduct a comparative analysis of our algorithm against mainstream reinforcement learning (RL) algorithms in dexterous hands' throwing-catching tasks. These algorithms include TRPO [28] , SAC [29] , DDPG [30] , and PPO [23] . The results, as illustrated in Fig. 5 , are quite telling. In the most challenging task, the Overarm Catch, our algorithm significantly outperforms the baseline methods. For the Over2abreast Catch, Under2Overarm Catch, and Abreast Catch tasks, our approach demonstrates superior learning efficiency in acquiring throwing and catching skills. Furthermore, even in the relatively easier task, Underarm Catch, our method exhibits slightly improved learning efficiency and performance compared to the baseline algorithms. However, the variance of the training curve is larger than those baselines in Underarm Catch. This is because simpler tasks have more gestures to complete tasks (see Fig. 3 ), leading to numerous feasible solutions. Thus, the strong exploration of our method results in a high variance of the sum of rewards. Conversely, complex tasks feature limited gestures, resulting in minimal reward variance post-completion. Video snapshots illustrating the performance in five tasks can be seen in Fig. 4 . It's worth emphasizing that we train the model using nine objects with diverse shapes and mass distributions. This showcases the algorithm's robust generalization ability, as it can adapt to various objects and perform tasks efficiently and reliably.We evaluate the impact of four crucial components in our algorithm. To clarify, here are their definitions: a. indicates the addition of a Lyapunov stability component. b. indicates the addition of an intrinsic advantage component. c. indicates network structure parameter normalization, and d. indicates that the compressed point cloud feature of the object is added as a prior to the state. The effect of these components is shown in the reward curve in Fig. 5 . It can be seen from the figure that each component of our method benefits the performance compared with the PPO algorithm.The inclusion of Lyapunov stability (a.) in the baseline PPO notably enhances the hand's catching ability, particularly accentuated during complex tasks like the Overarm Catch. There exists a more detailed explanation of ablation experiments in Appendix E.We evaluate our method in the throwing-catching task using various objects, quantifying the success rate as a key metric. We run the testing task in 10k episodes and use the average reward threshold to determine whether it is successful. Since our algorithm pays more attention to the efficiency and stability of generalization ability learning, we no longer use the distance threshold of the final state from the target position as the success rate. The success rate of different objects is illustrated in the Table 1 . Among them, we use 11 unseen objects (test set) to verify the generalization ability of our method. Specifically, we adopt two methods to test objects in the training and test sets: i. Test the success rate of multiple objects in parallel. ii. Test the success rate of each object individually. We take the most challenging task, Overarm Catch as an example. The average success rate for the training set objects reaches 60.30%, and the highest of individual tests on each object was pen up to 81.23%. In addition, the average success rate of the test set object is close to 67.58%. Most importantly, our method obtains a good success rate, 56.83%, in testing scissors, which is a new but difficult object. Furthermore, we can observe that our method can provide a robust catching strategy because there is not a significant gap between the success rates of training and test objects. The results provide strong evidence of our algorithm's robust generalization capability for catching diverse objects. We perform a more detailed analysis of the generalization ability in Appendix E. 
conclusion: 
 Summary: In this study, we introduce an end-to-end framework (LTC) based on model-free reinforcement learning to tackle throwing-catching tasks with dexterous hands. Remarkably, our method demonstrates exceptional generalization ability, even for unseen objects, and effectively handles scenarios where the hands face sideways or other postures. This research underscores the potential of RL-based approaches in improving object transportation efficiency through dynamic manipulation with dexterous hands.Limitation: Addressing the randomness for more complex objects or the initial position and posture of the hands is a key challenge for our method. To enhance robustness and success rate, we plan to iteratively refine task randomness by combining curriculum learning methods. Furthermore, we intend to optimize the motion and dynamic control parameters through adaptive parameter learning methods. Additionally, we've constructed a robot hardware platform comprising a mobile chassis, a 7-DoF robotic arm, and an allegro hand. Through the meticulous design of domain randomization terms, we hope to achieve zero-shot simulation-to-real transfer and execute throwing-catching tasks in real-world settings. 
 
 Appendix: In our experiments, we conduct training and testing on a platform equipped with an RTX A6000 GPU and an AMD CPU. We utilize nine objects for training across five different task types and test the algorithm with four objects. Our approach also leverages meta-learning for multi-object parallel training. The objects used in our experiments are sourced from the YCB dataset [27] . Due to the limitations of the platform, we employ 64 parallel environments per object for the Abreast Catch task and 256 parallel environments for others. The policy updates every 8 environment steps, and the length of each episode is set to 100 steps. Furthermore, every experiment is built using early reset functions mentioned in Appendix B. To determine the reward threshold, we randomly sampled 50 experimental environments during the testing phase. By visualizing and analyzing the corresponding reward values after successfully catching the objects, we calculated the average and rounded it to establish the reward threshold.B Object-generalized design The 3D point cloud feature of an object represents its shape, created by densely gathered points in 3D space. We adopt point clouds to approximate object features as prior information. Our approach involves feature extraction via a point clouds processing network utilizing PointClipV2 [22] . This algorithm facilitates zero-shot classification segmentation to identify features. To reduce computational complexity, the extracted feature map degrades to two dimensions using Principal Component Analysis (PCA). Our visual analysis has substantiated that these 2-dimensional features adequately capture the distinct attributes of various objects, as demonstrated in Fig. 6 . The distribution of various objects exhibits three distinct directions, leading to their classification into three categories. Elongated objects, like bottles, fall into the first category. Flat objects, such as keyboards, form the second category. Regular objects, like cups, which do not fit into the previous categories, are classified under the third category.Furthermore, to enhance the algorithm's ability to generalize across various physical properties of objects, we broaden the training dataset by randomly generating mass and inertia values within a reasonable range for the same object. This approach increases the diversity of physical properties encountered during training. In addition, we reset the environment when the object falls under a threshold or gets out of some bounds. The process enhances the diversity of effective data in the early stage of training.During the training, we leverage the IsaacGym simulator [31] , which enables parallel training, allowing the system to learn the skill of catching various objects simultaneously.The reward is composed of five elements: 1) r p quantifies the distance between the position of the goal object p g and the current object p c . 2) r o represents the angular discrepancy between the target object's orientation and the current object's orientation, utilizing quaternions for accurate angular measurement. 3) r a evaluates the smoothness of joint movement through the current actions a c . 4) A constant reward c 1 is given for successfully completing tasks r succ . 5) A constant penalty c 2 is imposed for task failure r f all . Specifically, the smoothness of joint movement is controlled by negatively rewarding the sum of squares of action quantities as per the strategy. Table 2 : Reward Desgin 1 The reward is composed of five parts, involving the position, posture of the object, task success or failure, and smooth actions.r p ||p g -p c || 2 r o 2 arcsin( ||q d || 2 -||q d || min ||q d || max -||q d || min ) r a -a 2 c r succ c 1 r f all c 2 r total r p + r o + r a + r succ -r f allThe overall reward is expressed r total in Table 2 , where target position distance is calculated using Euler distance, angular distance is determined using quaternions, joint smoothness is penalized using action quantities' sum of squares, and success and failure rewards are assigned as positive and negative constants, as outlined in Table 2 .We design a stricter constraint:L fπ θ V L (s t ) < -kV α L (s t ), where α and k are constants ranging from 0 to 1. Note that kV α L (s t ) is non-negative. This constraint avoids the issue of the Lyapunov function having very small values, thus speeding up the convergence of the stability part in policy improvement.Our idea is inspired by a classical theorem on finite-time stability convergence in a continuous-time system [32] . Lemma D.1 (Finite-Time Stability Convergence). In a continuous-time system, the system can be stable within a finite time T ≤EQUATIONNote that s 0 is the initial state, VL is the Lie derivative L f V L , k and α are constants ranging from 0 to 1.Proof. First, we perform an integral operation on both sides of the condition. VLEQUATIONThen, we can obtainEQUATIONwhere s 0 is the initial state. Finally, the convergence time T can be represented as:EQUATIONCompared with VL ≤ 0, the following lemma accelerates the convergence time from infinity to finite time. We can provide an intuitive explanation. Equation 9 guarantees that the derivative of the Lyapunov function V L is constantly less than a large negative value (-kV α L ) instead of 0. Thus, we can similarly use the condition of Equation 9 in our method. This process not only facilitates the policy in finding behaviors with better stability properties but also avoids the values of the Lyapunov function being close to 0 in the entire state space. For the choice of α and k, we set α to 0.7 and increase k linearly from 0.1 to 1 according to the training steps. The small k in the early stage can alleviate the collapse of the model.The inclusion of Lyapunov stability (a.) in the baseline PPO notably enhances the hand's catching ability, particularly accentuated during complex tasks like the Overarm Catch. By guiding dexterous hands to avoid unstable gestures and focusing on learning stable catching motions, we achieve higher success rates and rewards. It is worth noting that in the simplest task, the Underarm Catch, the effect of Lyapunov stability is limited. As seen in Fig. 3 , the task's stability region is large, thereby making it hard to balance between stability and rewards. Compared with PPO, it is more challenging to explore optimal solutions for success rate. The impact of increasing intrinsic advantage (b.) is observed primarily during the early stages of learning. While its effect may not be substantial, it demonstrates the capability to expedite early learning and convergence across multiple tasks. Parameter normalization (c.) significantly enhanced the ability of the throwing process. The normalization layer accelerates the early convergence of the model due to a more stable gradient. The stable gradient update benefits searching the optimal point, particularly in tasks with relatively simple throwing processes. It performs well in tasks where the success of the complete throwing-catching task depends heavily on the throwing process, such as Under2Overarm Catch, Underarm Catch, and Abreast Catch. The incorporation of the object's point cloud feature (d.) simultaneously considers both the throwing and catching processes, offering valuable cues for each. This enhancement notably boosts the algorithm's performance by improving convergence speed and task success rates. Given that the object's shape affects the catching, this feature significantly improves tasks with challenging catching behavior like Overarm Catch. Because the throwing-catching behavior is insensitive to the object's shape in Underarm Catch, adding point cloud to extend state space harms the performance slightly. For diverse objects: In this section, we delve into a detailed analysis of our method's generalization capabilities. It demonstrates the ability to execute distinct catching gestures based on the positions of the shadow hands and the objects being caught. Through an analysis of the high-dimensional point cloud features, we categorize most objects into three primary classes. Remarkably, our method exhibits over three different catching gestures for each of these three object types, as depicted in Fig. 7 . For objects falling into the regular, flat, and elongated categories, the model generalizes three distinct gestures in each of the five tasks. For instance, when dealing with regular objects like apples, it employs either a half-grip or pinching gesture. Slender objects like bananas are handled with a full-grip position. Flat objects, such as scissors and pokers, are directly held and secured between the fingers and palms. This versatility in catching gestures reflects the algorithm's remarkable adaptability to different object types and positions.For different initial postures: Furthermore, given that the initial position of the object is randomly selected in each episode, the object's motion follows different trajectories due to aerodynamic processes. Impressively, our method excels at solving dynamic catching tasks by generating diverse catching gestures, as illustrated in Fig. 8 . For example, when capturing a pen, we observe that the catching hand can adapt to grasp the pen with varying postures. In practical scenarios, catching a randomly rotating pen in mid-air, especially due to its anisotropic shape, would be challenging. However, our results demonstrate the algorithm's ability to catch randomly rotating objects in the air, even when dealing with challenging objects like pens.To ensure the successful transfer from the simulation environment to the real world, we address key challenges associated with sim-to-real transfer and offer specific solutions to overcome them.Challenge 1: Can the observations obtained during the simulation process be effectively acquired in the real world? Solution 1: In the following, we introduce how to obtain each component of the observation in the real world.• Position, rotation, velocity, target position, rotation of the object. We have validated the use of RGB image input from the main viewing angle, utilizing GPT-4V or other multimodal large models in combination with object detection and segmentation technologies. This approach enables us to obtain the initial pixel position of both the current object and the target. Real-time 6D pose estimation allows us to track the object's pose and pixel position during its flight. By integrating data from a binocular or depth camera, we can extract the object's three-dimensional world coordinates, and by leveraging the video stream frame rate, we can calculate the object's motion speed. We test using the FAIRINO ROBOT 3 robotic arm and the RealSense D435 camera on our platform. The execution frequency for detection, segmentation, and 6D pose estimation reached 10Hz, demonstrating that our system meets the initial performance requirements for executing dynamic tasks.• Point cloud prior information of the object. The object point cloud feature serves as prior information, capturing the shape characteristics after standardizing the object's size. This feature does not need to be acquired in real-time during the process. When acquisition is not feasible, the point cloud feature can be substituted with the representative geometry of the three types of objects detailed in Appendix B.• The position, rotation, and speed of shadow hands' wrists and joints. These can be obtained from the robot itself.• The force of the fingertips. These can be obtained through a fingertip force sensor.• The object's physical properties and the hand as well as the release velocity of the object being thrown. The terminal motion velocity can indeed be derived from the joint angular velocity, specifically the wrist movement velocity. The initial velocity of the object can be estimated through visual recognition, and subsequently corrected by combining the terminal motion velocity with six-dimensional force/torque data from the wrist sensors.Challenge 2: How can we address the observation errors that occur in the real-world observations?Solution 2: To account for potential errors in real-world observation acquisition, we introduce a pose perturbation of ±5°on each of the three axes of the current hand posture and a perturbation of ±5 cm in the distance between the hands. This approach is designed to test the policy's tolerance for variations in hand positions and postures. Additionally, we incorporate an observation error of ±5 cm and a pose perturbation of ±20°on each axis for the position and pose measurements of the object and target, respectively, along with an observation error of ±20% for the object's velocity measurement. The success rate measured after applying all these errors is compared with the original success rate, as presented in Table 3 . Challenge 3: Can the action outputs from the simulation process be implemented for control in the real world?Solution 3: To ensure that the actions generated in the simulation can be effectively controlled in the real world, our algorithm framework utilizes most position-based control, and minimizes reliance on force-based control. In our approach, the finger joints of the shadow hands are position-controlled, while only the force and torque of the wrists are controlled. When the robotic arm is connected to the wrist and transitioned to the real-world setting, the planning and control of the robotic arm can be handled by an impedance controller after getting the force and torque of the wrists. This design facilitates future research and verification tasks involving the connection of the wrists to the robotic arm.Challenge 4: Can our method generalize to different hand poses with significant variations?Solution 4: Although we believe that the five hand postures designed in this work already include the main combinations of the three axes of the two-hand xyz, and the other hand postures can be varied from these hand postures, in order to ensure the effectiveness of our policy, we added three more general postures: overarmout45, abreastin45, and overarmrightleft45, and achieved a good meta-success rate on the test set and the training set after training, as shown in Table 4 . A schematic diagram of the task is shown in Figure 9 .Finally, regarding future work, we will support further research on the combination of dexterous hands with robotic arms.8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 


Decision: Update: I have read the author s rebuttal  and my concern still stands that this framework would not be realizable on a real robot. The authors argue that we can use a segmentation algorithm to track the object being thrown and use an impedance controller to implement the movement on an arm with a Shawdow Hand as its end-effector. However  designing such a controller is a very difficult problem  because the impedance must depend on the object s physical properties and the hand  as well as the release velocity of the object being thrown. It is unclear how these quantities can be obtained in the real world. Furthermore  tracking an object in the air using perception foundation models (e.g. FoundationPose) seems unreasonable -- it would only work in a setup where the object is moving at a speed where such models can make inferences at a matching frequency. These models however are generally large and slow.   On the bright side  the paper tackles an important problem of dynamic catching of objects and has an interesting approach. Further experiments addressing concerns around real-world implementation in the camera ready version will strengthen the paper. 

===========================================  Reviewers appreciate that the paper tackles the problem of dynamic catching  which is an important yet understudied problem in robotics. They found the proposed method clever and interesting. However  reviewers are concerned whether this can be realized in the real world -- especially with the unrealistic motions it generates. A real-world experiment would resolve these concerns.

Reviews: review: title: title: A Potential Framework for Learning to Solve Throwing-Catching Tasks in Simulation
paper_summary: paper_summary: The authors present a framework (LTC) based on model-free reinforcement learning for throwing-catching tasks using dexterous hands in simulation. In the proposed framework, an intrinsic advantage, the normalization layers and the Lyapunov stability condition are added to the basic PPO algorithm to train better policies. The learned policies can generalize to unseen objects and effectively handle unstable scenarios.
main_review: main_review: The authors propose a reasonable approach to learning throwing-catching behavior. Tackling dynamic dexterous manipulation tasks is challenging yet important. It’s good to see the authors’ effort in this. 
Strength: 
- The use of hybrid advantage to successfully train the shadow hand to throw and catch the object in simulation is an important and clever way.
- It is impressive to see the Lyapunov stability benefit the generation of stable catching behavior for flying objects.
Weakness: 
- Further discussion is needed regarding the effectiveness of the trained policies in more general hand posture tasks (beyond those hand postures included in the 5 throwing-catching scenarios mentioned in Line111).
- The discussion is insufficient regarding sim2real transfer and challenges that might be encountered. As the state space consists of the information (such as the position of the object, velocities of the object, etc.) that is easily available in the simulator but not in the real world, the method’s strength won’t be adequately established until evaluated against a full transfer to the real word.
questions: questions: It is stated in Line 250 that “…use the average reward threshold to determine whether it is successful.” However, in Line415 and Line416, one of the components of the reward function is a constant reward r_succ which is given for successfully completing tasks. It seems that there is a loop that how to determine whether it is successful is based on the reward while the reward is related to whether it is successful. Please explain. And I would like to know whether r_succ is determined by the distance threshold of the final object position from the target position. If it were, what’s the meaning of using the average reward threshold instead of the distance threshold of the final state from the target position as the success rate (mentioned in Lines 251-253). Since the reward is related to the distance threshold, it seems that there is a redundancy as I’m not sure whether or not the average reward threshold provides more information about the task completion than the distance threshold. Please explain.
limitations: limitations: 2
review_summary: review_summary: I think the paper presents a nice framework for throwing and catching tasks in simulation. I appreciate the authors’ effort but I feel discussion and clarification regarding the mentioned weaknesses are necessary before acceptance.
title: title: Learning to catch with dexterous hand: novel framework, more experiments recommended
paper_summary: paper_summary: The authors propose a learning-based framework for throwing-caching tasks using dexterous hands. It is an end-to-end framework that takes as input the environmental observation and the point cloud feature of the object and then learns the policy through an actor-critic structure. The framework employs Proximal Policy Optimization (PPO) algorithm, enhanced with modifications, to address the challenges of dynamic manipulation. Specifically, it integrates standard PPO advantage estimation with Lyapunov stability and intrinsic advantage estimations to improve stability and efficiency in learning. The proposed method is evaluated on various tasks using two shadow hands in simulation, based on training efficiency and success rate of throwing and catching different objects.
main_review: main_review: Overall, the paper is structured and readable. The proposed framework is novel and relevant to the CoRL audience. The relevant work seems adequate. The proposed framework is clearly presented with details on the advantage functions, making it easier for potential reproductions. The problem formulation and policy training method are generalizable to other dynamic manipulation tasks. The evaluation can be strengthened by comparison of success rate and generalization ability test in domains other than objects and initial hand poses.  

Strengths:
- The integration of Lyapunov stability is significant to enhance the performance of reinforcement learning in dynamic tasks.
- The zero-shot transfer to unseen objects, facilitated by compressed point cloud object features and domain randomization during training, demonstrates the robustness of the framework.
- The simulation results show that the proposed framework outperforms the baselines regarding training efficiency and achieves a relatively high success rate.
- The ablation experiments indicate that all components of the proposed method are crucial to the performance of the system.  

Weaknesses:
- The generalization ability is not tested regarding the distance between the two hands.
- Dynamic tasks are susceptible to external factors in the real world, such as air drag. The robustness of the proposed framework in real-world tasks is unclear.
questions: questions: 1. While the experiment results clearly show that the proposed method has better training efficiency compared to the baselines, there does not seem to be a quantified comparison between the success rates of different methods.
2. More settings in the simulation can be changed during experiments to test generalization ability, such as the distance between the hands.
limitations: limitations: 2
review_summary: review_summary: The authors present a framework that is relevant and novel, they provide detailed information that helps the readers to understand the entire pipeline, from design of the inputs to the implementation of advantage functions during policy training. The framework is well grounded and tested on various objects in the simulation. The authors might further strengthen this paper by running more experiments to prove that the framework is robust and generalizable.
title: title: Review
paper_summary: paper_summary: This paper introduces LTC, a learning-based framework for dexterous throwing-catching tasks. The method incorporates Lyapunov stability for improved catching accuracy and stability. Using compressed point cloud features and domain randomization, LTC generalizes well to diverse objects, including highly challenging scenarios.
main_review: main_review: Strength:
1. The throwing and catching task is dynamic and challenging, and the authors categorize the tasks into 5 different settings, which is clear.
1. The stability function idea is useful for learning stable policies in the RL framework.
1. The paper is clear and the experiments are well-designed.

Weakness:
1. The novelty of the method is limited. It looks like a mix of PPO and Lyapunov functions. 
1. There are no real-world results, which I believe matter a lot for RL papers in simulation since anything can happen in simulation and RL can always figure something out. The sim2real gap is large for such dynamic and contact-rich tasks. It would be good if you can show sim2real results or provide a more exhaustive list of sim2real challenges and plans to address them. 
1. The quality of the figures and the video can be improved. For example, the text in the figure is too small and the black background makes it hard to tell the robot hand and the object.
1. The throwing motion looks unrealistic, e.g., a 720-degree rotation before releasing the object. Hard for me to imagine this working on a real robot. I understand the necessity of such motion to create momentum, but maybe it's more realistic to connect the hand to a robot arm and use the arm motion to create momentum. Connecting the hand to a robot arm in the simulation also adds additional kinematic constraints to the policy to ensure realistic motions.
1. The success rate for the training set objects is unsatisfying, although you can argue that the task is very difficult.
1. Regarding the plan for real-world experiments, I’m a bit concerned about the embodiment gap between the allegro hand and the shadow hand.
questions: questions: 1. Does the action space include the wrist pose for both hands? 
1. What’s the current and target object pose on line 123? How do you get the target object pose?
1. In Section 6.1, using the mean episode reward as the only metric seems not very satisfying. Can you include e.g. the success rate of the policy trained by different networks like in Table 1?
1. How do you decide the reward threshold for the success rate? “Lines 252-253 in Section 6.3 need more explanation. 
1. What does w.d. w.c. w.b. w.a. mean in Figure 5?
1. What would be the main challenges for sim2real transfer? Some things I can think of: How do you acquire the point cloud? Depth cameras in the real world will probably suffer from occluded surfaces, blurred motion, noisy points positions, and features. Will the policy be responsive enough given that it looks like a colored point cloud is needed for the input? Which information is directly available in your state space? How do you estimate the state information for those not directly available?
1. Contact-rich and dynamic tasks like this usually have very different physics in the real world. Given that you’re using IsaacGym, whose physics I’m not very confident about, why do you think you can do zero-shot transfer?
limitations: limitations: 3
review_summary: review_summary: I recommend a weak reject for this paper. While it presents a clear and well-designed framework for the challenging task of dynamic throwing and catching with promising simulation results, the novelty of the method is limited and lacks real-world validation. The paper also faces issues with unrealistic motions, and lower success rates for training set objects, highlighting concerns about the sim-to-real gap.