DexCatch:  Learning to Catch Arbitrary Objects with Dexterous Hands
Authors: Fengbo Lan
Shengjie Wang
Yunzhe Zhang
Haotian Xu
Oluwatosin OluwaPelumi Oseni
Ziye Zhang
Yang Gao
Tao Zhang
Venue: CoRL 2024 (2024)
Keywords: Reinforcement Learning
Dexterous Manipulation
System Stability
TL;DR: We propose a Learning-based framework for Throwing-Catching tasks using dexterous hands (LTC).

Abstract: Achieving human-like dexterous manipulation remains a crucial area of research in robotics. Current research focuses on improving the success rate of pick-and-place tasks. Compared with pick-and-place, throwing-catching behavior has the potential to increase the speed of transporting objects to their destination. However, dynamic dexterous manipulation poses a major challenge for stable control due to a large number of dynamic contacts. In this paper, we propose a Learning-based framework for Throwing-Catching tasks using dexterous hands (LTC). Our method, LTC, achieves a 73% success rate across 45 scenarios (diverse hand poses and objects), and the learned policies demonstrate strong zero-shot transfer performance on unseen objects. Additionally, in tasks where the object in hand faces sideways, an extremely unstable scenario due to the lack of support from the palm, all baselines fail, while our method still achieves a success rate of over 60%.

DexCatch: Learning to Catch Arbitrary Objects with Dexterous Hands
Fengho Lan
Shengjie Wang 1,,,3,+
Yunzhe Zhang'
Haotian Xu
Oluwatosin Oseni
Ziye Zhang'
Yang 1.23,1
Tao Zhang" 1.1
Tsinghua University SShangaii Artificial Intelligence Laboratory
Shanghai Qi Zhi Institute
Colorado School Mines
Equal contribution
t Corresponding author
Achieving human-like dexterous manipulation remains a crucial area of research in roboties. Current research focuses on improving the success rate of pick-and-place tasks. Compared with pick-and-plas throwing-catchinn be- havior has the potential increase the speed of transporting objects their des- tination. However, dynamic dexterous manipulation poses major challenge for stable control due to a large number of dynamic contacts. In this paper, we pro- framework for Throwing-Catching Catching tasks using dexterous
Abstract:
pose a Learning-basee hands (LTC). Our method, LTC, achieves 73% success rate across 45 scenar- ios (diverse hand poses and objects), and the learned policies demonstrate strong zero-shot transfer performance unseen objects. Additionally, in tasks where the object in hand faces sideways, an extremely unstable scenario due to the lack of support from the palm, all baselines fail, while our method still achieves success rate of over 60%. Video demonstrations of learned behaviors and the code can be found on the supplementary website.
Keywords: Reinforcement Learning, Dexterous Manipulation, System Stability
Daily life
Throwing and Catching
Figure 1: Generalization tasks for throwing and catching objects with shadow hands. Shadow hands can throw and catch a variety objects with different shapes, masses, and other properties when the hands face sideward.
Introduction
The study of dexterous hands is a widely pursued research area aimed at matching the intricate capabilities of human hands. Researchers employ these hands for delicate manipulations [1, 2, 3, 4]. Recently, researchers have applied reinforcement learning (RL) techniques various tasks involving dexterous hands, leading to diverse outcomes [5, 6, 7]. They have tackled a series of manipulation
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

tasks, including in-hand reorientation [6, 8), opening door [5], assembling LEGO blocks [9], and even solving a Rubik's ss cube [7]. These tasks can be summarized as common behaviors daily living, where hands pick up an object, reorient in hand, and then either place or use it as a tool [8].
While pick-and-place can fulfill the requirements of many daily tasks, dynamic throwing- holds the potential for more efficient task execution [10, 11, 12]. For example, two robots can em- play throwing and catching to efficiently transfer objects instead of pick-and-plas
as shown in Fig.1. Meanwhile, can extend the working space of robots, which avoids the potential collisions for robot interactions [10]. However, precisely catching arbitrary objects proves to be highly chal- lenging [13]. 1) First, the dynamic contact with strong force makes the object fall from the hand. In previous work [13], model-free reinforcement learning methods achieved nearly 0% success rate in the chrowing-catching task within 30 million environment steps. 2) Secondly, varying object-centric properties (e.g. mass distribution, friction, shape) require the catching behavior robust enough, es- pecially for objects of daily life. To target this problem, previous methods must learn the dynamics of a moving object [14]. The additional module requires expert demonstrations and increases the system's 's complexity. 3) Thirdly, when faced with flying objects coming from random directions, the catching hand must adopt different postures, enabling larger working space. However, the challenge arises in catching objects with hands facing sideways, where objects easily fall due the lack of any supporting surface, as illustrated in Fig. 1. Due to this challenge, the task of catching objects with hands facing sideways has not been explored previously.
To address the aforementioned challenges, we propose Learning- based framework for Throwing- Catching tasks using dexterous hands (LTC). The simple but effective framework can achieve supe- rior performance across diverse daily objects, in various throwing and catching scenarios where the hands face upward sideward. The main contributions of this work are follows.
We propose simple and effective framework using a model-free reinforcement learning algorithm, solve the strowing-catchinn tasks. Particularly, it can catch the flying objects even on the most challenging catching task with the hands facing sideward.
We propose hybrid advantage estimation method, which considers Lyapunov stability. While the effectiveness of the stability condition has primarily been verified for robustness, this is the first time it has been observed to benefit the generation of stable catching behavior for flying objects, thereby improving the accuracy of throwing-catchine
By using compressed point cloud features of objects as input and applying domain ran- domization during training. our method excels in generalizing to daily objects with diverse properties (e.g., mass distribution, friction, shape, and initial pose), and performs effec- tively even on previously unseen objects.
2
Related Work
The versatile dexterous hand plays a crucial role in performing various tasks across different human- focused settings. However, effectively managing dexterous hand systems poses a challenge due to their complex actions and intricate contact interactions [15]. Traditionally, controllers for manipu- lation tasks heavily rely on dynamic models and trajectory optimization techniques [1, 2, 3, 4]. The paper "Catching Objects in Flight" is highly regarded in this field [14]. To enhance model accuracy, their method learns the dynamics of moving object and develops reachable and graspable model using concise demonstrations. Furthermore, Williams et al. [16] achieved success using the Model Prediction Path Integration Control (MPPI) method to skillfully manipulate a cube. Charles Voss er al. [13] improved the MPPI method to manage task handovers between two hands. However, they focused only on single object due to the complexities involved in model learning.
In recent times, RL-based approaches simplify the controller design process, and model-agnostic techniques have gained substantial popularity within the control domain. In the realm of dexterous manipulation, numerous endeavors have showcased remarkable advancements compared to conven-

---

tional controllers [17, 18, 19]. Entities like OpenAl have harnessed reinforcement learning-baa -asee controllers to successfully rearrange blocks and solve Rubik's cubes [7]. To expedite training, a blend of reinforcement learning and imitation learning has been employed to facilitate dexterous hands in acquiring diverse skills such as repositioning objects, utilizing tools, opening doors, and more [5]. Acknowledging the existing method's limited ability to generalize, Chen et al. [8] in- troduced effective system capable of learning how to reorient wide array of objects. For the cooperation between two hands, Zakka et al. [20] and Chen al. [19] proposed series of chal- lenging tasks for bi-manual dexterity, like mastering the piano and lifting the pot. Furthermore, recent work introduced the Population- Bassd Training (PBT) algorithm in RL learning, leading to solving the two-handed reorientation task [18]. However, most recent works [18, 21] only focus on a single object or a kind of posture of hands. Our method successfully achieves the throwing and catching of diverse objects, especially for the task with hands facing sideward.
3 Preliminaries
3.1 Reinforcement Learning
The throwing-catching tasks of dexterous hands can be formalized as an infinite horizon Markov Decision Process (MDP), which is defined by tuple (S, P, R, po: 7). Among them, S E I is the state space, A E RT is the action space, p S x A S - [0, 1] is the transition probability distribution, S x A - IR is the reward, 7 E [0, 1) is the discount factor, and Po S - [0, 1] is the initial state distribution. The goal of RL is to find an optimal cumulative return of the MDP with the policy π S A + [0, 1]. The problem is formulated as maximizing the expected discounted cumulative return: (x)--po.annn((()))))::=:
(Er yrr(s,, at)].
3.2 Proximal Policy Optimization(PPOO)
The PPO algorithm an on-policy reinforcement learning approach. It excels in robotics tasks featuring high-dimensional action spaces. The algorithm contains actor and critic network, where 9 and ø denote the parameters of the actor network <e(()s))) and the critic network Vş(s;), respectively. For the policy function To (ar((s)). the output corresponds to the mean and standard deviation of the Gaussian distribution. On the other hand, the value function (o(s:) yields the current state's value. Building upon the policy gradient method, the parameter updating formula for the actor-network takes the following form: 0u+1 = arg max E
[L(s,, at, 0k,0)],
8t Sc,aertT
To, (pr(s)) L(-) = min { To (a(a((8)) T at), clip
(a(a)
A ATH (s), (sn,
- e,l+
(1)
To, To, (ap)s:)
where is hyperparameter, T (St, ar) = G Téx
viTeh (st), is the advantage function. G, is cumulative returns. For parameters o, the update is expressed as:
Øk+1 arg min E 3Lo0
[(G
Te
To (st))
(2)
However, because the calculation of G Top value needs the whole data of finished episode, we approximate G, to ri + -V (8t+1) through TD learning method.
4 Dexterous Catching Tasks
Our research centers on the challenges of dtrowing-catchiin tasks using dexterous shadow hands. Our aim is for these shadow hands adeptly handle diverse objects in various positions and execute agile catches using different gestures. To address this objective, our design comprises three key components.
3

---

532-d festure
function
FCA
butaa
4p4 Oisevirers
Point
Laneer layat
Folry
ELU
Crlx
Butter
Paru himane Formall bation layer
Vill
Lindar
Figure 2: A: The method takes as input the environmental observation and the point cloud feature of the object. Then, it learns the catching policy for dexterous hands through Actor-Critic structure. B: The Lyapunov function, the policy function and the value function are estimated using neural networks. Their network structures output. For example, the Lyapunov function contains a linear layer, a batch normalization layer, and penultimate normalization layer, and the last linear layer takes scalar output.
similar, and the only difference is the dimension of the
Environment Setting The shadow hand models, which emulate human skeletons, possess 24 de- grees of freedom (DOFs) and operated using 20 pairs of agonist-antagonisl tendons [8]. The presence of force sensors the ten fingertips facilitates the acquisition of force-related data dur- ing catching and throwing maneuvers. In our paper, investigate five catching scenarios, namely Overarm Catch, Overarm2Abreass Catch, Under20verarm Catch, Abreast Catch, Underarm Catch and which are shown in Fig. 4 respectively. In these five tasks, one hand should throw an object to the other hand, and the other one should hold it in the palm later. Some tasks with vertical hands are challenging. is because the object the hands is not stable without the support from the palm.
Object-generalized Setting Our design objective to generalize across diverse objects by in- troducing point cloud features, which are obtained from pre-trained model, PointClipV2 [22]. Additionally, apply Principal Component Analysis (PCA) to compress the features. The ex- periments illustrate that the shapes of different objects can be distinguished using the compressed features. More details can be found in Appendix B.
State, Action and Reward 1) State: The state space consists of 424 elements. Among these, the 422-4iinnaeeee 222-dimensional state incorporates crucial information: hand position, hand orientation, angular and linear velocities of the hands, position and speed of each hand's degree of freedom, force-torque sensor readings of the hands, target object's position and orientation, as well as the current object's position and orientation. The above observation corresponds the setting in [19]. Additionally, two-dimensional state features the input derived from the point cloud data of the object. It serves as the perceptual part for the agent regarding the object. 2) Action: The action space comprises 52 dimensions, encompassing 20 pairs of agonist-antagonis tendons for both hands, along with three dimensions each for the required for ce and torque of each hand. 3) Reward: The reward design comprises three components: the difference in position and posture between the object and the target, the penalty for the action scale of the shadow hand joints, and combination of task completion bonuses and failure costs. We explain the details of the reward in Appendix C.
Method
In this section, we introduce the LTC algorithm in detail. LTC is an end-to-end framework based on the Proximal Policy Optimization (PPO) [23] algorithm. Since the dtrowing-catching task is dy- namic and agile behaviour, the basic PPO algorithm encounters two key challenges: How achieve efficient throwing and stable catching without falling? We make some algorithmic modifications to the basic PPO algorithm. First, to accelerate the learning of throwing, we design an intrinsic advan-

---

tage and add penultimate normalization layers in the network. Secondly, stable catching behaviour holding the object the palm without falling the object. As depicted in Fig. 3, the maximum sum reward (optimality) can not guarantee the generation of the stable catching behaviour. To encourage more stable catching, we include the Lyapunov stability condition policy learning. The specific formulation can be found in Section 5.2.1.
5.1 Framework
The architecture is illustrated in Fig. 2. Here's a breakdown of its components:1) Point Cloud Feature Extraction: The initial step involves ex- tracting point cloud features using the Point- ClipV2 [22]. Subsequently, Principal Com- ponent Analysis (PCA) is employed to reduce dimensionalityy, yielding 2-dimensional fea- ture outpur. This output serves the per- ception state for the shadow hands system; 2) Observation Input: The agent takes both the current observation of objects and the shadow hands and the compressed features from point clouds input; 3) Actor-Critic Framework: The actor and critic are represented two neu- ral networks, which is the same as PPO algo- rithm. The network architecture for the Lya- punov function is depicted on the right side of Fig. 2.
Saaiiiyy
Exn Task
Negilne
tank Ruden
Hiverd
Negadeo
Sdakas
Paliyy Spure
To enhance the success rate, approach prioritizes optimizing rewards while ensuring the sta- bility the grasping gesture. By integrating Lyapunov stability alongside reward optimization, we mitigate the risk of completing tasks in unstable poses. The size of the stable region varies based task difficulty, with simpler tasks offering a larger stable region.
Figure 3:
This structure comprises three linear layers, with a Batch normalization layer and an ELU activation function appended the output of each layer. Following these, there is penul- timate normalization layer [24], succeeded by linear layer to regulate the output dimension. The network structure for the policy and value function are similar to that of the Lyapunov value network.
5.2 Hybrid Advantage Estimation
The hybrid advantage Aatl consists of 3 parts: standard advantage estimation based on PPO, stability advantage estimation, and intrinsic advantage estimation. The formula for the hybrid advantage is expressed follows:
Aau(s ae S1+1) 3T (st, az) 32AL (St. 8441) (s,,
(3)
B1 + B2 + 3/ =1
(4)
where Th (st, az) refers the time differential advantage function of the PPO algorithm, (Se, (s,, at) represents the Lyapunov stability advantage function, A (st, 81+1) represents the intrinsic advantage function, and B1, _2, ,3 E [0, 1] are hyperparameters. As follows, we introduce the Lyapunov stability advantage and intrinsic advantage in detail.
5.2.1 Lyapunov Stability
The Lyapunov function serves as potent tool for assessing system stability. One approach, known as the self-learning Lyapunov stability method [25], utilized in our paper. First, we introduce Lyapunov stability critic network, which represents the candidate Lyapunov function V,, using neural networks. Subsequently, the empirical Lyapunov risk R(A) [25] is defined as per Eq. (5). The three parts in Eq. (5) represent three requirements for the Lyapunov function [25]. The parameter update is then executed by minimizing the loss function associated with the Lyapunov risk R(A), as part of achieving the desired system stability.
R(A) E max( Manax(VVz(ss Vz()s), 0) Lax(OO,,VV(ss))) V22(0)
(5)
5

---

Overarm
Overarm2Abrenst
Under20verarm Catch
Abreust Catch
Underarm Catch
Catch
Catch
Figure 4: Video snapshots of throwing-catching tasks performed by method. There exist five tasks with different postures of hands, in order from difficult to easy: Overarm Catch, Over- arm2Abreast Catch, Under2Overarm Catch, Abreast Catch, and Underarm Catch. The most chal- lenging task is Overarm Catch because the hands are both oriented vertically, Notably, our method catch diverse objects in five tasks.
where is the dynamical system, S, is the state at step t. The empirical Lyapunov risk is non- negative, we approximate the Lie derivative VyVi(si) (s,) by finite difference of the system's 's sampling trajectory because of the complexity modeling this system's is dynamics:
(s) = (VZ(514)) Vz():)
(6)
where At is the time difference between them. Since the Lyapunov function is considered to be another value function, the Lie derivative term (s,) is the only term in the Lyapunov risk affected by the policy. Therefore, an additional goal of ensuring I. (sa) < 0 is added to the policy optimization, to encourage the policy's update with the requirement of stabilization. However, the classical constraint L, (s:) < makes the value of the Lyapunov function likely to be stuck in a minimal number close to O. This drawback hinders the process of policy improvement due to the lack of an explicit gradient.
Therefore, design more strict constraint Lsx, (s.) < -kV, where and k are constants ranging from 0 to 1. Note that kV (s:) is non-negative. When kV/ (st) is larger, an intuitive phenomenon is that Vz approaches faster. In other words, the system can reach a stable region in fewer steps, which benefits the generation of stable catching behaviors. More details can be found in Appendix Hereafter, we design the part of the Lyapunov stability advantage function based on the clipped Lie derivative term:
Ar(s,, ar) min(kV2(st). - -L (s()))
5.2.2 Intrinsic Advantage
Based on the advantage function of the PPO algorithm, in order to improve the ability of the system efficiently, we design the intrinsic advantage function with reference to the intrinsic reward mecha- nism [26]. For a Markov decision process trajectory, different values V*(*)) will affect the success of the final result this trajectory. To provide a specific example, let's consider the fact that the next states S1+1 exhibit varying conditions, leading to different values Va (ot=i)). It's worth noting that in successful process, the value Vo($++1) tends be higher. Consequently, within Markov further refine the advantage function A(s, a) using empirical measures. If an action is effective, it should result in higher value, represented by Vo($++1) Vo(s)) 0. This signifies that action a, transitions to state with a higher success rate. Conversely, an unfavor- able action can lead to a decrease in value, reflected as Vo(5++1) failure in the early stage of the throwing process, we consider the intrinsic advantage (s $1+1) as part of the total advantage function to accelerate the RL training, shown as follows:
decision process,
(s((s) < 0. In order to reduce
Ar S141) = min(k (V) Vs(s))),
(8)
where ls is positive constant.

---

Overarm
Overarm2Abreast Catch
Under2Overarm Catch
Abrenst Catch
Underarm Catch
Catch
Ablation Experiments Compare Expertment
110
PPO
BM
TEPO
10
ETC
FP0 v.
F00
mP
Figure 5: The average reward curve for training multiple objects for five different tasks. The first row is the comparison experiment reward graph. The second row is an ablation experiment reward graph. (Curves smoothed report the mean times, and shadow areas show the variance.)
Experiments
To test the algorithm's ability to generalize, we train and test it with objects from the YCB Pybullet dataset [27]. The results show that our algorithm outperforms baseline methods. Four essential com- ponents of our algorithm improve its performance. Additionally, our method showcases the ability to generalize different gestures across five skilful throwing-catching tasks, effectively allowing for the successful manipulation of various objects. Importantly, our approach extends its capabilities to catch unseen objects successfully.
6.1 Comparison Experiments
In this section, we conduct comparative analysis of our algorithm against mainstream reinforce- ment learning (RL) algorithms in dexterous hands' throwing-catching tasks. These algorithms in- clude TRPO [28], SAC [29], DDPG [30], and PPO [23]. The results, illustrated in Fig. 5, are quite telling. In the most challenging task, the Overarm Catch, our algorithm significantly outperforms the baseline methods. For the Over2abreast Catch, Under2Overarm Catch, and Abreast Catch tasks, our approach demonstrates superior learning efficiency in acquiring throwing and catching skills. Fur- thermore, even in the relatively easier task, Underarm Catch, our method exhibits slightly improved learning efficiency and performance compared the baseline algorithms. However, the variance of the training curve larger than those baselines in Underarm Catch. This is because simpler tasks have more gestures to complete tasks (see Fig. 3), leading to numerous feasible solutions. Thus, the strong exploration of our method results in high variance of the sum of rewards. Conversely, complex tasks feature limited gestures, resulting in minimal reward variance post-completion. Video snapshots illustrating the performance in five tasks can be seen in Fig. 4. It's worth emphasizing that we train the model using nine objects with diverse shapes and mass distributions. This showcases the algorithm's robust generalization ability, as can adapt to various objects and perform tasks efficiently and reliably.
6.2 Ablation Experiments
We evaluate the impact of four crucial components in our algorithm. To clarify, here are their definitions: indicates the addition of a Lyapunov stability component. b. indicates the addition of intrinsic advantage component. e. indicates network structure parameter normalization, and d. indicates that the compressed point cloud feature of the object is added prior the state. The effect of these components is shown in the reward curve in Fig. 5. It can be seen from the figure
7

---

Table 1: Success rate of Task
Success Rate
Task
Train (9 objects)
Test (11 objects)
META
Appletregular)
Bananatelongated)
Poker(flat)
META
Stapler
Scissor
Overarm
60.30
63.62
73,51
73.53
67.58
69.71
56.83
76.00
77.34
Overarm2Abreast
82.89
Under20verarm
73.95
75.35
82.21
78.79
73.41
76.85
68.46
57.34
81.88
67.41
88.65
57.52
Underamm
88,99
95.87
86.50
90.57
80.47
84.10
63.63
63.10
83.23
rale tests( times) singlo objects multiplo objects. Specifically, we only display the success rate three typical objects within the training set and four typical objects within the rest set. We utilize reward threshold in ascertain the sucoess rate. In the Iveeerm Carch mskk, success is determined if the surpasses bocause the longer execution process,
dividiig
training
sets, perform distinct
exoceding indicates success.
that each component of our method benefits the performance compared with the PPO algorithm. The inclusion of Lyapunov stability (a.) in the baseline PPO notably enhances the hand's catching ability, particularly accentuated during complex tasks like the Overarm Catch. There exists a more detailed explanation of ablation experiments in Appendix E.
6.3 Generalization Ability
We evaluate our method in the bbrowing-catching task using various objects, quantifying the success rate key metric. We run the testing task in 10k episodes and use the average reward threshold to determine whether it is successful. Since algorithm pays more attention the efficiency and stability of generalization ability learning, from the target position as the success rate. The success rate of different objects is illustrated in the Table 1. Among them, we use 11 unseen objects (test set) to verify the generalization ability of our method. Specifically, we adopt two methods to test objects in the training and test sets: i. Test the success rate of multiple objects in parallel. ii. Test the success rate of each object individually. We take the most challenging task, Overarm Catch as an example. The average success rate for the training set objects reaches 60.30%, and the highest of individual tests each object was pen up to 81.23% In addition, the average success rate of the test set object is close to 67.58%. Most importantly, our method obtains good success rate, 56.83%, in testing scissors, which is new but difficult object. Furthermore, we can observe that our method can provide a robust catching strategy because there is not significant gap between the success rates of training and test objects. The results provide strong evidence of our algorithm's 's robust generalization capability for catching diverse objects. We perform more detailed analysis of the generalization ability in Appendix E.
longer the distance threshold of the final state
Conclusion
Summary: this study, we introduce an end-to-end framework (LTC) based on model-free rein- forcement learning tackle throwing-catching tasks with dexterous hands. Remarkably, our method demonstrates exceptional generalization ability, even for unseen objects, and effectively handles sce- narios where the hands face sideways other postures. This research underscores the potential of RL-based approaches in improving object transportation efficiency through dynamic manipulation with dexterous hands.
Limitation: Addressing the randomness for more complex objects or the initial position and posture of the hands is key challenge for method. To enhance robustness and success rate, we plan to iteratively refine task randomness by combining curriculum learning methods. Furthermore, we intend to optimize the motion and dynamic control parameters through adaptive parameter learning methods. Additionally, we' ve constructed robot hardware platform comprising a mobile chassis, a 7-DoF robotic arm, and an allegro hand. Through the meticulous design of domain randomization terms, we hope to achieve zero-shot simulation-to-real transfer and execute throwing-catching tasks in real-worke settings.

---

Acknowledgments
Thank you all the authors and those who put in the effort in this work. This work was supported by the National Natural Science Foundation of China Igrant numbers U21B6002). This work is also supported by the Ministry of Science and Technology of the People 's Republic of China, the 2030 Innovation Megaprojects "Program on New Generation Artificial Intelligence" (Grant No. 2021 AA00500000000 and the National Key RD Program of China (20227D0161700),
References
[1] T. Asfour, P. Azad, N. Vahrenkamp, K. Regenstein, A. Bierbaum, K. Welke, J. Schroeder, and R. Dillmann. Toward humanoid manipulation in human-centred environments. Robotics and Autonomous Systems, 56(1):54-65, 2008.
[2] U. Kim, Jung, Jeong, J. Park, H.-M. Jung, Cheong, H. R. Choi, H. Do. and C. Park. Integrated linkage-driven dexterous anthropomorphie robotic hand. Nature communications, 12(1):7177, 2021.
[3] Y. Bai and C. K. Liu. Dexterous manipulation using both palm and fingers. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 1560-1565. IEEE, 2014.
[4] V. Kumar, E. Todorov, and S. Levine. Optimal control with learned local models: Application to dexterous manipulation. In 2016 IEEE International Conference on Robotics and Automa- tion (ICRA), pages 378.383. IEEE, 2016.
[5] A.
Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. Robotics:
Science and Systems 2018.
[6] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3 2020.
[7] Akkaya, M. Andrychowicz. M. Chociej, Litwin, B. McGrew, A. Petron, A. Paino,
M. Plappert, G. Powell, R. Ribas, al. Solving rubik's cube with robot hand. arXiv preprint 1rX1v:191..07113. 2019.
[8] Chen, J. Xu, and P. Agrawal. A system for general in-hand object re-orientation. In Confer- ence Robot Learning, pages 297-307. PMLR, 2022.
[9] R. Jeong, T. Springenberg. J. Kay, D. Zheng, Y. Zhou, A. Galashov, N. Heess, and F. Nori. Learning dexterous manipulation from suboptimal experts. arXiv preprint arXiv:2010.08387, 2020.
[10] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 3674):1307-1399, 2020.
[11] B. Bäuml, T. Wimböck, and G. Hirzinger. Kinematically optimal catching flying ball with a hand-arm-system. In 2010 JEEE/RS/ International Conference on Intelligent Robots and Systems, pages 2592-2599, IEEE, 2010.
[12] A. Namiki, Y. Imai, M. Ishikawa, and M. Kancko. Development of a high-speed multifingered hand system and its application to catching. In Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) No. 03CH37453), volume 3, pages 2666-2671. IEEE, 2003.
[13] H. Charlesworth and G. Montana. Solving challenging dexterous manipulation tasks with trajectory optimisation and reinforcement learning. In International Conference Machine Learning, pages 1496-1506. PMLR, 2021.

---

[14] S. Kim, A. Shukla, and A. Billard. Catching objects in flight. IEEE Transactions on Robotics, 1045):104--1055, 2014.
[15] A. M. Okamura, N. Smaby, and M. Cutkosky. An overview of dexterous manipulation. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37063), volume pages 255-262.
IEFF, 2000.
[16] G. Williams, A. Aldrich, and E. Theodorou. Model predictive path integral control using covariance variable importance sampling. arXiv preprint /5X.:: 149, 2015.
[17] W. Huang. L. Mordatch, P. Abbeel, and D. Pathak. Generalization in dexterous manipulation via geometry-awan multi-task learning. arXiv preprint arXiv:211 2021.
[18] A. Petrenko, A. Allshire, G. State, A. Handa, and V. Makoviychuk. Dexpbt: Scaling up dexterous manipulation for hand-an arm systems with population based training. arXiv preprint arXiv:2305. 02.00///2222 /2127, 2023.
[19] Y. Chen, T. Wu, S. Wang, X. Feng, J. Jiang, Z. Lu, S. McAleer, H. Dong, S.-C. Zhu, and Y. Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Systems, 35:5150-5163, 2022.
[20] K. Zakka, L. Smith, N. Gileadi, T. Howell, X. B. Peng, S. Singh, Y. Tassa, P. Florence, A. Zeng, and P. Abbeel. Robopianist: A benchmark for high-dimensional robot control. arXiv preprint arXiv:2304.044 150 2023.
[21] B. Huang, Y. Chen, Wang. Y. Qin, Y. Yang, Atanasov, and Wang. Dynamic handover: Throw and catch with bimanual hands. arXiv preprint arXiv:2309.088 05655, 2023.
[22] X. Zhu, R. Zhang, B. He, Z. Zeng, S. Zhang, and P. Gao. Pointelip v2: Adapting clip for powerful 3d open-world learning. arXiv preprint arXiv:2211.11682, 2022.
[23] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06333 2017.
[24] J. Bjorck, C. P. Gomes, and K. Q. Weinberger. Is high variance unavoidable in rI? case study in continuous control. In International Conference on Learning Representations, 2021.
[25] Y.-C. Chang and S. Gao. Stabilizing neural control using self-learned almost Ιуарunov critics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1803- 1809. IEEE, 2021.
[26] Y. Li, T. Gao, J. Yang, H. Xu, and Y. Wu. Phasic self-imitative reduction for sparse-reward goal-conditioned reinforcement learning. International Conference on Machine Learning, pages 12765 12781. PMLR, 2022.
[27] B. Calli, A. Singh, J. Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M. Dol- lar. Yale-cmu- berkeey dataset for robotic manipulation research. The International Journal of Robotics Research, 36(3):261-268 2017.
[28] Schulman, S. Levine, Abbeel, M. Jordan, and Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[29] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[30] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (0XX::155990277111 2015.
10

---

[31] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physies simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
[32] S. Yu, X. Yu, and R. Stonier. Continuous finite-time control for robotic manipulators with terminal sliding modes. In Information Fusion, 2003. Proceedings of the Sixth International Conference of, 2003.

---

A Implementation detail
In experiments, we conduct training and testing platform equipped with RTX A6000 GPU and an AMD CPU. We utilize nine objects for training across five different task types and test the algorithm with four objects. Our approach also leverages meta-learning for multi-object parallel training. The objects used in our experiments are sourced from the YCB dataset [27]. Due to the limitations of the platform, we employ 64 parallel environments per object for the Abreast Catch task and 256 parallel environments for others. The policy updates every environment steps, 100 steps. Furthermore, every experiment is built using early reset functions mentioned in Appendix B. To determine the reward threshold, randomly sampled 50 experimental environments during the testing phase. By visualizing and analyzing the corresponding reward values after successfully catching the objects, we calculated the average and rounded it to establish the reward threshold.
and the length of each episode
B Object-generalized design
The 3D point cloud feature of an object rep- resents its shape, created by densely gathered points in 3D space. We adopt point clouds approximate object features prior infor- mation. Our approach involves feature extrac- tion via a point clouds processing network uti- lizing PointClipV2 [22]. This algorithm fa- cilitates zero-shot classification segmentation to identify features. To reduce computational complexity, the extracted feature map degrades two dimensions using Principal Component Analysis (PCA). Our visual analysis has sub- stantiated that these 2-dimensional features ad- equately capture the distinct attributes of vari- ous objects, as demonstrated in Fig.6. The dis- tribution of various objects exhibits three dis- tinct directions, leading to their classification into three categories. Elongated objects, like bottles, fall into the first category. Flat objects, such keyboards, form the second category. Regular objects, like cups, which do not fit into the previous categories, are classified under the third category.
Akplane
Bottle
Bowl
Cone
Cup
Glass bot
Keyboard
Radio
Vase
Figure 6: The features of different point clouds are ex- tracted by PointClipV2 [22], then reduced by PCA, and finally visualized in two dimensions. The Clustering Using Representatives(CUn
method shows that any object is approximately divided into three categories ac- cording to different shapes, extending in three direc- tions, respectively, elongated, flat, and regular. Sample of objects in each category bottles, keyboards, and cups, respectively.
Furthermore, to enhance the algorithm'ss 's ability to generalize across various physical properties of objects, we broaden the training dataset by randomly generating mass and inertia values within reasonable range for the same object. This approach increases the diversity of physical properties encountered during training. In addition, reset the environment when the object falls under threshold or gets out of some bounds. The process enhances the diversity of effective data in the early stage of training.
During the training, we leverage the IsaacGym simulator [31], which enables parallel training, al- lowing the system to learn the skill of catching various objects simultaneously.
C Reward design
The reward is composed of five elements: 1) rp quantifies the distance between the position of the goal object Pg and the current object Pe- 2) To represents the angular discrepancy between the target object's orientation and the current object's orientation, utilizing quaternions for accu- rate angular measurement. 3) evaluates the smoothness of joint movement through the current
12

---

actions 4) A constant reward 01 is given for successfully completing tasks ssuce- 5) A con- stant penalty 02 is imposed for task failure fful. Specifically, the smoothness of joint movement controlled by negatively rewarding the sum of squares of action quantities as per the strategy. The overall reward is expressed Ttotal in Ta- ble 2, where target position distance cal-
Table 2: Reward Desgin
Tp
1g
Pc 2
culated using Euler distance, angular distance is determined using quaternions, joint smooth- ness is penalized using action quantities' sum of squares, and success and failure rewards are assigned as positive and negative constants, as outlined in Table 2.
To
arcsin
dd
mas
Txnen F'alll
Ttotal
Ppt To
Fa
Tsuce
r fall
roward composed posture of the object, task success or failure, and smooth ac- bons.
involvng position,
D Details of Lyapunov Stability
We design a stricter constraint: I,,, Vz( (&V) where and constants ranging from 1. Note that (s)) is non-negative. This constraint avoids the issue of the Lyapunov function having very small values, thus speeding up the convergence of the stability part in policy improvement.
Our idea inspired by a classical theorem on finite-time stability convergence in continuous-time system [32].
Lemma D.1 (Finite-Time Stability Convergence). In continuuu--imme system, the system can be Vxo)
stable within finite time T S k(1-a) if the following condition holds.
V -VV
(9)
Note that So is the initial state, VL is the Lie derivative LVVL, k and are constants ranging from 0 to 1.
Proof. First, we perform an integral operation on both sides of the condition.
VL
dt -
kdtt
(10)
Vp.
Then, obtain
-1 T - =kt
(V) (V=-"(())) 5 KT
(11)
V4-a (80)
1 2 KT
where so is the initial state. Finally, the convergence time T can be represented as:
T Vi-0 (sp) R(1 a)
(12)
L
Compared with V.. O, the following lemma accelerates the convergence time from infinity to finite time. We can provide an intuitive explanation. Equation 9 guarantees that the derivative of the Lyapunov function V,, is constantly less than a large negative value (-IV2) instead of 0. Thus, we can similarly use the condition of Equation 9 in our method. This process not only facilitates the policy in finding behaviors with better stability properties but also avoids the values of the Lyapunov function being close to 0 in the entire state space. For the choice of a and we a 0.7 and increase & linearly from 0.1 to according to the training steps. The small & in the early stage can alleviate the collapse of the model.
13

---

E Ablation Experiment Analysis
The inclusion of Lyapunov stability (a.) in the baseline PPO notably enhances the hand's catching ability,, particularly accentuated during complex tasks like the Overarm Catch. By guiding dexterous hands to avoid unstable gestures and focusing on learning stable catching motions, we achieve higher success rates and rewards. It worth noting that in the simplest task, the Underarm Catch, the effect of Lyapunov stability is limited. As seen in Fig.3, the task's s stability region large, thereby making it hard to balance between stability and rewards. Compared with PPO, it is more challenging to explore optimal solutions for success rate.
The impact of increasing intrinsic advantage (b.) is observed primarily during the early stages of learning. While its effect may not be substantial, demonstrates the capability to expedite early learning and convergence across multiple tasks. Parameter normalization (c.) significantly enhanced the ability of the throw- ing process. The normalization layer acceler- ates the early convergence of the model due to a more stable gradient. The stable gradient up- date benefits searching the optimal point, par- ticularly tasks with relatively simple throw- ing processes. It performs well tasks where the success of the complete throwing-catching task depends heavily on the throwing pro- cess, such as Under2Overarm Catch, Under- arm Catch, and Abreast Catch. The incorpo- ration of the object's point cloud feature (d.) si- multaneously considers both the throwing and catching processes, offering valuable cues for each. This enhancement notably boosts the al- gorithm's 's performance by improving convergence speed and task success rates. Given that the ob- ject's shape affects the catching, this feature significantly improves tasks with challenging catching behavior like Overarm Catch. Because the grrowing-catching behavior is insensitive to the object's 's shape in Underarm Catch, adding point cloud to extend state space harms the performance slightly.
Figure 7: For the regular, flat, and elongated classes of objects, the model generalizes three different sta- ble catching strategics for unseen objects in cach of the three tasks. For regular objects such as apples and strawherries, it uses half-grip (3) or pinching gestures (2). A full-grip gesture (i) is used for slender objects such as bananas. For flat objects such as scissors and pokers, the hand hold them directly in the palm (4)5).
Generalization Ability
For diverse objects: In this section, we delve into detailed analysis of our method's general- ization capabilities. It demonstrates the ability to execute distinct catching gestures based on the positions the shadow hands and the ob- jects being caught. Through an analysis of the high-dimensional point cloud features, cat- egorize most objects into three primary classes. Remarkably, our method exhibits over three different catching gestures for each of these three object types, as depicted in Fig. For ob- jects falling into the regular, flat, and elongated categories, the model generalizes three distinct gestures in each of the five tasks, For instance, when dealing with regular objects like apples, it em- ploys either half-gun position. Flat objects, such as scissors and pokers, are directly held and secured between the fingers
Figure 8: When the pen oriented obliquely or tum- bling because the randomness in the initial posture, the shadow hand can generalize various grasping ges- tures successfully complete the tasks.
pinching gesture. Slender objects like bananas handled with full- gill-grip
14

---

and palms. This versatility in catching gestures reflects the algorithm's remarkable adaptability to different object types and positions.
Furthermore, given that the initial position of the object is randomly selected in each episode, the object's motion follows different trajectories due aerodynamic pro- cesses. Impressively, catching gestures, as illustrated in Fig. 8. For example, when capturing a pen, we observe that the catching hand can adapt to grasp the pen with varying postures. In practical scenarios, catching a randomly rotating pen in mid-air, especially due to its anisotropic shape, would be challenging. However, our results demonstrate the algorithm's 's ability to catch randomly rotating objects in the air, even when dealing with challenging objects like pens.
For different initial postures:
method excels at solving dynamic catching tasks by generating diverse
G Simulation to Real World Design
To ensure the successful transfer from the simulation environment to the real world, we address key challenges associated with sim-to-real transfer and offer specific solutions to overcome them.
Challenge I: Can the observations obtained during the simulation process be effectively acquired in the real world?
Solution 1: In the following, we introduce how to obtain each component of the observation in the real world.
Position, rotation, velocity, target position, rotation of the object. We have validated the use of RGB image input from the main viewing angle, utilizing GPT-4V or other multimodal large models in combination with object detection and segmentation technologies. This approach enables us to obtain the initial pixel position of both the current object and the target. Real-time 6D pose estimation allows us to track the object's pose and pixel position during its flight. By integrating data from binocular or depth camera, we can extract the object's
world coordinates, and by leveraging the video stream frame rate, we can calculate the object's motion speed. We test using the FAIRINO ROBOT robotic arm and the RealSense D435 camera on our platform. The execution frequency for detection, segmentation, and 6D pose estimation reached 10Hz, demonstrating that our system meets the initial performance requirements for executing dynamic tasks.
Point cloud prior information the object. The object point cloud feature serves as prior information, capturing the shape characteristics after standardizing the object's size. This feature does not need to be acquired in real-time during the process. When acquisition is not feasible, the point cloud feature can be substituted with the representative geometry of the three types of objects detailed in Appendix B.
The position, rotation, and speed of shadow hands' wrists and joints. These can be obtained from the robot itself.
The force of the fingertips. These can be obtained through fingertip force sensor.
The object's physical properties and the hand well as the release velocity of the object being thrown. The terminal motion velocity can indeed be derived from the joint angu- lar velocity, specifically the wrist movement velocity. The initial velocity of the object can be estimated through visual recognition, and subsequently corrected by combining the terminal motion velocity with six-dimensional force/torque data from the wrist sensors.
Challenge 2: How can we address the observation errors that occur in the real- observations?
Solution 2: To account for potential errors in real-world observation acquisition, we introduce a pose perturbation of +5 on each of the three axes of the current hand posture and perturbation of +5 cm in the distance between the hands. This approach is designed to test the policy's 's tolerance for variations in hand positions and postures. Additionally, we incorporate observation error of +5 em and pose perturbation of +20 each axis for the position and pose measurements of
15

---

the object and target, respectively, along with an observation error of +20% for the object's velocity measurement. The success rate measured after applying all these errors compared with the original success rate, as presented in Table 3.
Overarm0ur45 Catch
AbreastOur45 Catch
Overarmm Catch
Y
X
Figure 9: Video snapshots of catching tasks performed by our method. The description of tasks can be found in Appendix G.I.
Table 3: Success rate of Task with disturbance
Success Rate (%0
Task
Orignal(11 object)
Add disturbunce (6 object)
META(Train)
META(Test)
META
Overarm
60.30
69.97
Overam22Abreast
68.77
79.88
66.75
Uverr2Overarn
73.95
71.06
70.09
Abreast
72.95
79.97
68.84
Underarm
88.99
73.51
83.65
Challenge 3: Can the action outputs from the simulation process be implemented for control the real world?
Solution 3: To ensure that the actions generated in the simulation can be effectively controlled in the real world, our algorithm framework utilizes most position- control, and minimizes reliance on force-based control. In our approach, the finger joints of the shadow hands are position-controllld while only the force and torque of the wrists are controlled. When the robotic arm is connected to the wrist and transitioned the real-world setting, the planning and control of the robotic arm can be handled by an impedance controller after getting the force and torque of the wrists. This design facilitates future research and verification tasks involving the connection of the wrists the robotic arm.
Challenge 4: Can our method generalize to different hand poses with significant variations?
Solution 4: Although believe that the five hand postures designed in this work already include the main combinations of the three axes of the two-hand xyz, and the other hand postures can be varied from these hand postures, in order to ensure the effectiveness of our policy, we added three more general postures: overarmout45, abreastin45, and overarmrighttef44,, and achieved a good meta-success rate on the test set and the training set after training, as shown in Table 4. schematic diagram of the task is shown in Figure 9,
Finally, regarding future work, we will support further research on the combination of dexterous hands with robotic arms.
16

---

Table 4: Success rate of New Task
Success Rate (%)
Task
META
Overarmout45
61.44
Abreastin45
68.77
Overarmrightlef15
85.41
G.1 Description of Added Tasks
The new tasks are depicted as follows.
The hands are oriented vertically. Then, the throwing hand rotates around the x-axis the reference coordinate system (shown on the right side Fig. 9) by -45 degrees, while the catching hand rotates around the x-axis by 45 degrees.
Overarmout45 Catch:
Abreastin45 Catch: The setting of hands built upon the Abreast Catch task. Then, the throwing hand rotates around the 7-axis of the reference coordinate system (shown on the right side of Fig. by 45 degrees, while the catching hand rotates around the z-axis by -45 degrees.
Overarmrighttef445 Catch:
The hands are oriented vertically. Hereafter, the throwing hand rotates around the y-axis of the reference coordinate system (shown the right side of Fig. 9) by 45 degrees, while the catching hand rotates around the y-axis by -45 degrees.
H Comparison Experiments Additional Notes
To further demonstrate the effectiveness of the proposed method compared to baseline and other reinforcement learning approaches, we conducted an additional comparative experiment, focusing on the success rate, as shown in Table The results clearly indicate that our approach consistently outperforms both the baseline and other methods across all metrics.
Table 5: Success rate of Task with Other Algorithm
Succem Ban CS
LBC
PPO
TRPO
SAC
DIPG
MBTA//rate
METS/Do:
META/Train)
MBTAATrale) SHCTA/Tbet
METX/eed)
METX/Trabi
METAA/eet)
METT//Te)
Ovarm
69.30
09.17
5.16
8.96
6.00
0.50
666
OoramaZAbeac
65.77
26.25
0.54
8.10
Unter20ivaam
71.86
Aleco:
72.55
70.97
43.88
$3.76
47.51
44.55
8.00
0.00
17


Reviews:
[{"review_id": "pbHUX1UQ6X", "reviewer": "Reviewer_JPgA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present a framework (LTC) based on model-free reinforcement learning for throwing-catching tasks using dexterous hands in simulation. In the proposed framework, an intrinsic advantage, the normalization layers and the Lyapunov stability condition are added to the basic PPO algorithm to train better policies. The learned policies can generalize to unseen objects and effectively handle unstable scenarios.", "summary_of_recommendation": "I think the paper presents a nice framework for throwing and catching tasks in simulation. I appreciate the authors’ effort but I feel discussion and clarification regarding the mentioned weaknesses are necessary before acceptance.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "It is stated in Line 250 that “…use the average reward threshold to determine whether it is successful.” However, in Line415 and Line416, one of the components of the reward function is a constant reward r_succ which is given for successfully completing tasks. It seems that there is a loop that how to determine whether it is successful is based on the reward while the reward is related to whether it is successful. Please explain. And I would like to know whether r_succ is determined by the distance threshold of the final object position from the target position. If it were, what’s the meaning of using the average reward threshold instead of the distance threshold of the final state from the target position as the success rate (mentioned in Lines 251-253). Since the reward is related to the distance threshold, it seems that there is a redundancy as I’m not sure whether or not the average reward threshold provides more information about the task completion than the distance threshold. Please explain.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DsFkOvuYsW", "reviewer": "Reviewer_2Jip", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a learning-based framework for throwing-caching tasks using dexterous hands. It is an end-to-end framework that takes as input the environmental observation and the point cloud feature of the object and then learns the policy through an actor-critic structure. The framework employs Proximal Policy Optimization (PPO) algorithm, enhanced with modifications, to address the challenges of dynamic manipulation. Specifically, it integrates standard PPO advantage estimation with Lyapunov stability and intrinsic advantage estimations to improve stability and efficiency in learning. The proposed method is evaluated on various tasks using two shadow hands in simulation, based on training efficiency and success rate of throwing and catching different objects.", "summary_of_recommendation": "The authors present a framework that is relevant and novel, they provide detailed information that helps the readers to understand the entire pipeline, from design of the inputs to the implementation of advantage functions during policy training. The framework is well grounded and tested on various objects in the simulation. The authors might further strengthen this paper by running more experiments to prove that the framework is robust and generalizable.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "1. While the experiment results clearly show that the proposed method has better training efficiency compared to the baselines, there does not seem to be a quantified comparison between the success rates of different methods.\n2. More settings in the simulation can be changed during experiments to test generalization ability, such as the distance between the hands.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mtxaKGJgbg", "reviewer": "Reviewer_RD6F", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces LTC, a learning-based framework for dexterous throwing-catching tasks. The method incorporates Lyapunov stability for improved catching accuracy and stability. Using compressed point cloud features and domain randomization, LTC generalizes well to diverse objects, including highly challenging scenarios.", "summary_of_recommendation": "I recommend a weak reject for this paper. While it presents a clear and well-designed framework for the challenging task of dynamic throwing and catching with promising simulation results, the novelty of the method is limited and lacks real-world validation. The paper also faces issues with unrealistic motions, and lower success rates for training set objects, highlighting concerns about the sim-to-real gap.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Does the action space include the wrist pose for both hands? \n1. What’s the current and target object pose on line 123? How do you get the target object pose?\n1. In Section 6.1, using the mean episode reward as the only metric seems not very satisfying. Can you include e.g. the success rate of the policy trained by different networks like in Table 1?\n1. How do you decide the reward threshold for the success rate? “Lines 252-253 in Section 6.3 need more explanation. \n1. What does w.d. w.c. w.b. w.a. mean in Figure 5?\n1. What would be the main challenges for sim2real transfer? Some things I can think of: How do you acquire the point cloud? Depth cameras in the real world will probably suffer from occluded surfaces, blurred motion, noisy points positions, and features. Will the policy be responsive enough given that it looks like a colored point cloud is needed for the input? Which information is directly available in your state space? How do you estimate the state information for those not directly available?\n1. Contact-rich and dynamic tasks like this usually have very different physics in the real world. Given that you’re using IsaacGym, whose physics I’m not very confident about, why do you think you can do zero-shot transfer?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Update: I have read the author s rebuttal  and my concern still stands that this framework would not be realizable on a real robot. The authors argue that we can use a segmentation algorithm to track the object being thrown and use an impedance controller to implement the movement on an arm with a Shawdow Hand as its end-effector. However  designing such a controller is a very difficult problem  because the impedance must depend on the object s physical properties and the hand  as well as the release velocity of the object being thrown. It is unclear how these quantities can be obtained in the real world. Furthermore  tracking an object in the air using perception foundation models (e.g. FoundationPose) seems unreasonable -- it would only work in a setup where the object is moving at a speed where such models can make inferences at a matching frequency. These models however are generally large and slow.   On the bright side  the paper tackles an important problem of dynamic catching of objects and has an interesting approach. Further experiments addressing concerns around real-world implementation in the camera ready version will strengthen the paper. 

===========================================  Reviewers appreciate that the paper tackles the problem of dynamic catching  which is an important yet understudied problem in robotics. They found the proposed method clever and interesting. However  reviewers are concerned whether this can be realized in the real world -- especially with the unrealistic motions it generates. A real-world experiment would resolve these concerns.
Decision: Accept