Dynamic Handover: Throw and Catch with Bimanual Hands
Authors: Binghao Huang
Yuanpei Chen
Tianyu Wang
Yuzhe Qin
Yaodong Yang
Nikolay Atanasov
Xiaolong Wang
Venue: CoRL 2023 Poster (2023)
Keywords: Bimanual Dexterous Manipulation
Sim-to-Real Transfer

Abstract: Humans throw and catch objects all the time. However, such a seemingly common skill introduces a lot of challenges for robots to achieve: The robots need to operate such dynamic actions at high-speed, collaborate precisely, and interact with diverse objects. In this paper, we design a system with two multi-finger hands attached to robot arms to solve this problem. We train our system using Multi-Agent Reinforcement Learning in simulation and perform Sim2Real transfer to deploy on the real robots. To overcome the Sim2Real gap, we provide multiple novel algorithm designs including learning a trajectory prediction model for the object. Such a model can help the robot catcher has a real-time estimation of where the object will be heading, and then react accordingly. We conduct our experiments with multiple objects in the real-world system, and show significant improvements over multiple baselines. Our project page is available at https://binghao-huang.github.io/dynamic_handover/

Dynamic Handover: Throw and Catch with Bimanual Hands
Binghao Huang 0.1
Yuanpei Chen
Tianyu Wang!
Yuzhe Qin'
Yaodong Yang²
Nikolay Atanasov
Xiaolong Wang'
UC San Diego'
Peking University²
Abstract: Humans throw and catch objects all the time. However, such seemingly common skill introduces a lot of challenges for robots to achieve: The robots need to operate such dynamic actions at high-speed, collaborate precisely, and interact with diverse objects. In this paper, we design a system with two multi-finger hands attached to robot arms to solve this problem. We train our system using Multi-Agent Reinforcement Learning simulation and perform Sim2Real transfer deploy on the real robots. To overcome the Sim2Real gap, we provide multiple novel algorithm designs including learning trajectory prediction model for the object. Such a model can help the robot catcher has real-time estimation of where the object will be heading, and then react accordingly. We conduct our experiments with multiple objects in the real-world system, and show significant improvements over multiple baselines. Our project page is available at https: / //binghao-hann -uungg. github. io
Keywords: Bimanual Dexterous Manipulation, Sim-to-Real Transfer
Training in Simulation
Test in Real Robot System
We propose Dynamic Handover. a new bimanual dexterous hands system designed for throwing catching tasks. The system consists of two Allegro Hands, each individually attached to separate XArm robot, arranged in a facing configuration. Using multi-agent reinforcement learning, we train policies in simulation environment and subsequently transfer them the real world.
Figure 1:
Indicates equal contribution.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Introduction
Human's 's ability to throw and catch objects exists commonly in both sports (e.g., baseball) as well as casually passing through items in daily life. In the context of handing over objects, enabling the robots to obtain such bimanual manipulation skills not only improves efficiency, but also extends the workspace the robots by a large margin outside the robots' kinematic range, Importantl, provides a contact- and safer solution for robot interactions, since direct handing objects over can cause collisions between two manipulators, which brings damage. To enable robust manipulation of diverse objects, we propose to use two multi-finger robot hands as the thrower and the catcher.
Learning such skills brings multiple challenges: (i) Precise, high-speed dynamic action: Both robot hands will need to obtain dynamic extrinsic dexterity at high-speed and operate precisely at the same time. small error or mismatch of coordination will easily lead to failure. (ii) Diverse objects: To train policy that is applicable diverse objects makes the problem even more challenging. Traditional approach with planning based on full object state estimation can introduce new errors and is hard to generalize to unseen objects with limited training data; (iii) High-ddimessional action space: Learning bimanual manipulation with two multi-finger hands introduce a high- nigh-dimensional state and action space, which increases the difficulty for optimization.
In this paper, we propose a new system and a new method for throwing an object with one hand and catching it with the other hand in a coordinated manner. Our hardware system includes two xArm robots attached with two Allegro Robot Hands (each with 16 degrees of freedom) in the end separately as shown in Fig. 2. The two hands will need to operate collaboratively at the same time. To solve this problem, we propose an approach leveraging Reinforcement Learning to train a policy in simulation and then perform Rim2Real transfer on the real robots. Specifically, to solve the above challenges, we introduce three key technical contributions:
Multi-Agent Reinforcement Learning (MARL). Dexterous manipulation with RL has been extensively studied for learning robust and generalizable skills [L. 2, 3]. However, has rarely been explored for dynamic manipulation with bimanual hands. We propose to tackle this problem multi-agent problem, with each hand being one agent. This helps improve the coordination of two hands during manipulation which allows better Sim2Real transfer. Mcanwhile, we train the policy with multiple diverse objects at the same time, which allows generalization to unseen objects. Dynamic dexterity. Instead of explicitly estimating the dynamic properties and modeling them, using RL with randomization allows the network to learn to handle diverse dynamics. We apply various physical property randomization during training, including randomization in friction, inertia, the object's s center of mass, and contact force. To enhance the stability of the thrower, we also investigate how the quality of the initial grasp of the object can affect the performance.
Object trajectory prediction. While we can perform various randomization approaches for training, there will always be a physics gap when transferring to the real robot. To further increase the robustness for catching, we introduce a model to predict the object's future trajectory and destination ahead in real time. Instead of pre-defining the object destination and asking both hands to follow, the catch policy will take the predicted object's destination position as input in close-loop manner. This allows flexible real-time adjustment of the catch policy.
We conduct both simulation and real-world experiments. The results demonstrate that our method enables the hands to successfully throw and catch the object, and surpass basclines by a large margin. We observe applying MARL in training effectively reduces the sim2real gap, as using partial observations for each agent improves the robustness of each policy. Our prediction module offers extra flexibility to help the robot hand to adjust real-time and accurately eatch the object. To our knowledge, this is the first work that enables bimanual dynamic multi-finger hands manipulation.
2 Related Work
Multi-Agent Reinforcement Learning. Multi-Agent Reinforcement Learning (MARL) is powerful approach for complex tasks involving multiple interacting agents. In MARL, agents learn decision- making based on interactions with others and the environment [4, 5, 6, 7, 8]. Multi-Agent Proximal

---

Policy Optimization (MAPPO) [9] extends policy gradient algorithms into multi-agent scenarios by addressing challenges such as non-stationarity, coordination, and credit assignment. In robotics, MARL has been applied to cooperative navigation [10, 11], UAV swarm [12, 13], and distributed manipulation [14, 15J. Similar to us, Zhang et al. [16] employ disentangled attention within MARL for bimanual tasks, while Li et al. [17] use symmetry-ane
actor-critic for efficient handover. Both studies were conducted exclusively in simulation without complex dynamics, allowing for end2end MARL training. In contrast, propose multi-stage training pipeline for MARL to overcome sim2real transfer challenges.
Dynamic Manipulation. Different from common manipulation tasks like grasping and opening drawers, dynamic manipulation abandons the quasi-static assumption of interaction. It leverages object dynamics, such as inertia and momentum, handle tasks requiring high-speed actions and extended robot workspace, like throwing and catching [18, 19, 20, 21]. Traditional systems for such tasks often rely on handcrafted models of system dynamics, which may fall short when dealing with difficult-to tollltttt-ettatee parameters or new objects. To address this issue, recent works [22, 23, 24, 25, 26] employ data-driven approaches to optimize control commands using partial dynamics models. For example, Chi et al. [25] propose an iterative residual policy to solve tasks with complex dynamics; Zeng el al. [24] use end-to-end training to learn stable grasps that generate predictable throws. However, these works focus on low-DoF manipulators rather than high-DoF dexterous hands, which introduce additional complexities due intricate hand-object interactions. [27] uses Population Based Training (PBT) to train the throws and catches using dual hands. However, while our methodologies primarily focus on sim2real transfer, [27] does not include the real-world experiments. In this study, we use learning-based approach to tackle dynamic problems, specifically throwing and catching, with multi-finger dexterous hands. We explore the impact of initial dexterous grasps on throwing performance and investigate strategies to bridge the sim2real gap in the context of dynamic manipulation.
Bimanual Dexterous Manipulation. In recent years, the robotics community has increasingly focused on dexterous manipulation due to its great flexibility and human-like dexterity. Researchers have developed methods using dexterous hands for tasks such as grasping [28, 29, 30, 32), in- hand rotating [33, 34, 35, 36, 37], and manipulating deformable objects [38, 39, 40, 41J. Similar to us, DexPoint [42] achieves generalizable manipulation for grasping and door opening by training on multiple objects with Allegro hand. However, complex tasks like throwing and catching objects require bimanual robot system achieve human-level manipulation skills. Researchers have investigated bimanual manipulation through task planning [43, 44, 45], and reinforcement learning [46, 47, 48]. However, most previous work focused on using two parallel jaw grippers for quasi-static interaction, leaving dexterous bimanual manipulation largely unexplored. Few studies have delved into this area, mainly in simulation without real robot validation [49, 50, 51]. In this paper, we take a step forward by developing a bimanual dexterous manipulation system capable of throwing and catching various objects. Our work also demonstrates that simulation training without real-world data can still tackle this challenging task even with great sim2real gap.
3 System Setup
Task Description. In this work, we focus on the bimanual Catching and Throwing task with two dexterous robot hand. This task involves two robot agents: (i) a thrower robot agent (Figure 2 right) that needs to execute swift movements to toss the grasped object towards the other side, and (ii) catcher agent (Figure 2 left) that needs to react dynamically catch the airborne object. This task is important because it enables the catcher robot to access objects beyond its kinematic range by leveraging the object's momentum imparted by the thrower. It also serves as a good test- bed for evaluating the coordination and performance of bimanual systems in high-speed, real-time scenarios.
Real-world Setup. We construct a bimanual system for executing our throwing and catching task, as depicted in Figure 2. The system includes two arm-hand subsystems and RealSense D435 camera. Each arm- hand subsystem features 6- 6-DoF XAmm-6 robot arm paired with 16-DoF Allegro Hand, culminating in 44-DoF system. To create a closed-loop policy, we use a RealSense camera to
3

---

Object's Position
(Gim=3)
Action (dim=22)
Action (dim=22)
T
Joint Position'
01
Og
Joint Position
(dim-22)
(dim=22)
Catcher
Predicted Goal
Target Goal
Thrower
(dim-3)
Policy(MAPPO)
(ditw3)
Object's Position
(dims))
Figure 2: Real Robot System: We employ two Allegro Hands, each individually mounted on separate XArm-6 robots, arranged in face-to-face configuration. We incorporate RealSense D435 camera for real-time object position tracking, which is oriented towards the working space. We usc prior states in observation.
capture the real-time position of objects within the catcher robot's frame of reference. For observation, distinct feedback mechanisms are provided for each agent, depicted at the bottom of Figure The thrower (Figure 2: right) depends exclusively
its proprioceptive data, while the catcher (Figure 2: left) obtains feedback from not only its proprioception but also the object's real-time positions estimated by the camera. In other words, the thrower operates based on its current state, whereas the catcher necessitates both proprioceptive and visual input to dynamically and interactively perform catching actions. Further information regarding this system's 's implementation can be found the Appendix.
Simulation Setup. In this work, throwing and catching task. The simulation setup is shown the left side of Figure 1. The simulation frequency is set at 120Hz while the control frequency is 20Hz. We train the end-to-ent -o--to-ed reinforcement learning policy in the simulated environment and then transfer the policy to the real world. Further information regarding the details about the simulation setup (e.g. policy architecture) can found in the Appendix.
the IsaacGym physical simulator [52] for training
Action Space. The policy outputs 22- 12-dimensional PD control target, with the first six dimensions corresponding to X XArm-6 and the remaining 16 dimensions corresponding to Allegro hand. For the XArm-6, we employ delta joint positions as the control target, while for the Allegro hand, we utilize absolute joint positions as the control target. This design choice is made to avoid jerky motion of robot arm for safety reasons, while still allowing the hand to swiftly react and release its grasp to throw the object. In our experiments, we find that controlling only the second and third joints of the robot arm and keeping the other four joints fixed results Therefore, the action space consists of an 18-dimensional target for the thrower and 22-dimensional target for the catcher.
effective and safer policy.
Learning Bimanual Dexterous Hands Policy
Catching an object in mid-air poses significant difficulties due to the high-speed requirement. First, object's 's real-time velocity and anticipated trajectory must be taken into account in order for the catcher to determine its movement. Second, even though the thrower can consistently toss the object toward the pre-defined target goal in simulated environment, the policy transfer to the real world is imperfect due the substantial dynamics gap between simulation and real physical. Consequently, there is discrepancy between the pre-defined throwing goal and the object's actual destination. In light of these two challenges, goal estimator becomes crucial for predicting the thrower's actual

---

in Stago
Throw Goal: G.
Stage and
Throw Goal: Gg
AP
Thrower Propriaception
Thrower
Pollcy
BMC
Catch Goal: &g
00
A
Predicted Catch Goal:
Catcher
Goal Estimstor
Proprioception
Catcher
Object
IsaacGym
Position
Figure 3: Joint End2End Learning: The two agents receive input from both their own observations and the catcher agent additionally receives the predicted catching position. The goal estimator takes past 20 frames of the object's positions input and predicts the catch goal for each time step. We use violet ball to represent the pre-defined goal for the throwing. The orange represents the predicted goal for catcher to catch during the throwing task. The blue ball represents the object that is currently been manipulated.
destination instead of the predetermined target goal. This allows the catcher to move based on the forecasted object destination and successfully catch it.
To achieve this, introduce novel three-stage training pipeline for learning bimanual throwing and catching. (i) In the first stage, we train a base policy using Multi-Agent RL to tackle the task, with the catcher observing the pre-defined throwing goal. The policy trained during this stage is expected to perform well within the simulator but may hardly transfer to the real world. (ii) Next, we freeze the base policy and train a goal estimator through supervised learning, using the rollout trajectory of the base policy as training data. (iii) Finally, we replace the pre-defined throwing goal the catcher's observation with the estimated goal and unfreeze the policy for fine-tuning both the base policy and the goal estimator in end-to-ert fashion. The refined policy is expected to bridge the dynamics gap between simulation and reality with the predicted object's future trajectory.
4.1 Stagel: Multi-Agent Reinforcement Learning.
We employ the Multi-Agent Proximal Policy Optimization (MAPPO) [9] in non non- parameter sharing way to train the thrower and the catcher to obtain basic policies in the first stage. MAPPO is an application of the PPO algorithm to multi-agent settings. It leverages centralized training with decentralized execution, allowing each robot agents to efficiently accomplish the cooperative task using partial observations.
As shown at the bottom of Figure 2, the observations for the thrower and catcher in MAPPO training is not identical. The thrower's policy, denoted as To, receives its proprioception and pre-defined target position for throwing. In contrast, the catcher's policy, denoted as πı, takes as input its proprioception, the pre-defined goal position and the current position of the object for catching. To satisfy MAPPO's requirement for equal input dimensions across agents, we pad zeros to the thrower's input for dimension alignment. Besides, we include observations from past frames as input for both policies provide temporal information. In our implementation, we set 2.
Given the object position Pt, target goal position Gr, the velocity of the object v, the unit direction vector from thrower to catcher û, and robot joint torque we design the reward function using three components: (i) distance between object and throwing goal; (ii) object velocity projected in the direction from thrower to catcher; (iii) robot joint torque. The final reward can be computed
as r = Fdis + Tlinre! + Ttorques where Tdis = exp(-20 * (p: G.)) represents the distance,
Tlineel clamp(v  -0.1, 0.1) denotes the object's velocity towards the catcher, and Ttorque -0.003 | corresponds the torque penalty.
4.2 Stage2: Goal Estimator Learning.
After training the basic policies, the next step involves freezing the basic policies and training goal estimator (Orange Block in Figure 3) predict the goal for the catcher based on the trajectory of
5

---

the object. This is a crucial step due to the sim2real gap, which implies that although the thrower may consistently hit the goal in simulation, it is unlikely to achieve the same level of accuracy in the real world. Therefore, predicting the actual goal based on the object's 's trajectory becomes essential for improving sim-to-real transfer. In this stage, we utilize the historical positions of the object over span of & frames as imput to the goal estimator. The oulput of the goal estimator the predicted 3D position of the goal, which provides crucial information for the catcher anticipate the intended catching point and enhance the coordination between the two hands. We use Adam to optimize the 1.2 distance between the position of the predicted goal and the thrower's goal until convergence:
[(w) || 1
where pl is the position of the object from ( -- k to frames, 9 is the thrower's goal, and the and represent the thrower and catcher respectively. It is important to highlight that the previous training stage, the object's landing point was primarily influenced by the thrower's goal. This is because the thrower, operating in the simulation environment, had the ability to consistently hit the specified goal in the simulation.
4.3 Stage3: End2End Joint Learning.
In stage 1, the catcher's base policy uses a pre-defined throw goal in its observation. In this stage, we replace the pre-defined goal with the predicted one from the goal estimator. However, the distribution shift brought by this replacement can result in compounding errors. To address this issue, we jointly fine-tune the goal estimator and the policy network in this stage, as visualized in Figure 3, allowing the catcher to adapt to the goal estimator. For example, when the goal estimator is inaccurate, the policy will not depend solely on it for decision-making. This joint training approach helps reduce compounding errors when integrating the goal estimator with the policy.
5 Experiments
Evaluation Criterion. To evaluate the performance of the trained policies for throwing and catching, we consider several metrics as follow:
(i) Success Rate(SR): It is calculated as the ratio of successful throws and catches to the total attempts. better policy will lead to higher Success Rate.
(ii) Hit Rate(HH): This metric is defined as the proportion of objects that successfully hit the hand palm of catcher. A better policy and goal estimator will lead to a higher Hit Rate.
Training and Dataset. During the training process, we utlize three different objects: a ball, a cube, and a rod, which are three typical geometries for robot manipulation. At the beginning of each episode, we randomly select one object for training. For simulation experiments, we expand the object set to include additional objects to evaluate the generalizability of our policy to novel objects. The objects used in the simulation experiments intentionally incorporate a rod during the training procedure, we find this leads more robust policy in the training. Specifically, we observed that the simultancous movement of all fingers when throwing an object led to more robust results, whereas when using balls/square objects, policy often used only a few fingers. Therefore, we designed the shape of the rod in training. the policy wants to throw the rope stably, must learn to use all of its fingers throwing, so can make our policy more robust. Further discussion about using the rope for training can be found in the Appendix. In the real- world experiments, employ sandbags in three different shapes for throwing, shown in Figure 5(c): ball, a cylinder, and triangle prism.
depicted in Figure 5(a) and Figure 5(b). We
Baselines. In this paper, we compare our method with the following baselines:
(i) Open-Loop Policy: We employ a pre-defined trajectory for the bimanual hand-arm system to execute the throwing and catching task. The trajectories are collected on the real robot using kinesthetic teaching and replayed later without considering feedback or adjustment during execution.

---

Settings
Known Obj.
Novel Obj.
(a)
w/o Multi-Agent
0.89 1 0.07
0.24 0.05
w/o Goal Estimation
0.88 + 0.04
0.22 0.04
w/o Both
0.93 + 0.07
0.12 + 0.06
Ours
0.95 0.07
0.37 0.04
Table I: Ablation Study in Simulation: Success Rate of throwing and catching task on different objects in simulation. We 11 trained objects and 14 novel objects. The results are averaged on 5 seeds, each seed has 100 trails.
(b)
1.0
Ours
Average Episode Success Rate
W.O. Both
W.O. GE
0.8
W.O. MA
0.6
0.4
(c)
0.2
0.0
200
400 Environment Steps
600
800
Figure 5: Objects Sets. (a) Training objects. (b) Additional objects evaluation. (c) Real-world objects.
Figure 4: Training Curves. The plot shows multi- object training curves our method and 3 baselines.
(ii) Without Multi-Agent: We train our policy using PPO instead of using MAPPO. However, we still incorporate goal estimation during the learning process. Under this setup, both agents share the same observation.
(iii) Without Goal Estimation: We restrict our learning process to the first stage in Section 4 with MАPPO algorithm and evaluate the policies without goal estimation.
(iv) Without Both: We restrict our learning process to the first stage in Section 4 with PPO algorithm and evaluate the policies without goal estimation.
5.1 Results in Simulation
We conduct an analysis our method with three baselines in the simulation environment. Figure 4 shows the training curve of four methods. Table 1 presents the success rates for categories: Known Objects and Novel Objects. Our findings can be summarized as follows.
First, for the test on known objects experiment, we observe that the baseline (Without Both) outper- forms the other three methods, including ours. One possible reason for this is that the policy without multi-agent coordination has access to full observations of both hands and the arm system, well as the ground-trut truth object position. As result, the policy can easily overfit to a specific point and successfully solve the task. However, it important note that this baseline policy may have lower generalization capabilities when it comes to novel objects uncertain parameters. In the real world, we encounter noise and uncertainties, and obtaining the ground-truth object position is not feasible. Therefore, the policy with full observations demonstrates lower transferability from simulation to the real world, as we validate Section 5.3.
Secondly, in the novel objects experiment, although all methods show significant drop in perfor- mance, method outperforms the baselines. The utilization of multi-agent reinforcement learning and goal estimation proves beneficial for accomplishing the throwing and catching task. Both the
7

---

Settings
Ball
Cylinder
Triangle
HR
SR
SR
Open-Loop
0.00 0.12
0.13
0.12
0.47
2.12
0.13
0.27
= 0.12
0.07
0.12
w/o Multi-Agent
0.73 1: 0.31
0.40
0.20
0.53
20550::: 0.31
0.20
0.00
0.47
0.12
0.20
0.20
w/o Goal Extimation
0.60
0.20
0.33
0.23
0.67
0.31
0.40
0.34
0.40
+ 0.20
0.13 4: 0.23
w/o Both
0.47 + 0.13
0.12
0.12
0.40
0.20
0.07
+ 0.12
0.20
+ 0.20
0.00
0.00
Ours
0.93 0.12
0.60
0.20
0.80
0.20
0.53
0.12
0.86
0.12
0.33
0.12
Table 2: Ablation Study in Real- World: Performance of throwing and catching task on 3 different unknown
objects in real robot platform. Objects made of sandbags with the are averaged on seeds with trails for each. The two terms stand for:() Hit Rate(HR): This metric is defined as the proportion objects that successfully hit the hand palm of catcher. A better policy and goal estimator will lead to higher Hit Rate. (ii) Success Ratc(SR): It is calculated as the ratio of successful throws and catches the total attempts. A better policy will lead to higher Success Rate.
but different shapes. The results
thrower and catcher agents receive observations from their respective perspectives, enabling them to perform their tasks cohesively. Furthermore, goal estimation assists the catcher in predicting the landing point of the objects based on historical positions. This feature helps mitigate the impact of unpredictable parameters during the manipulation process, such as friction, unexpected collisions, and other dynamic factors.
5.2
Why MAPPO Outperform PPO in the Dynamic Handover?
Commonly, the single-agent setting should be strictly easier than the multi-agent setting, since both thrower and catcher have full access to the states of both sides. However, our result suggests otherwise which seems rather counter- counter-intuitive So why MAPPO outperform PPO in our system? We agree that single-agent RL might be easier than the multi-agent RL training, so the performance of the single-agent RL and the multi-agent RL the trained object are similar. But single-agent RL can access more information, they are more likely overfit into the environment, so they will be harder to generalize than multi-agent, which is why multi-agen agget performs well in novel objects in simulation and the sim2real transfer.
5.3 Results in Real World
We perform sim-to-real experiments to assess the performance of our method and two baselines on real-robot platform. As depicted in Figure 2, we deploy multi-agent reinforcement learning policies on the real robot agents, with both agents controlled by the same host. The task execution sequence is visualized on the second and third columns in Figure 1. Further implementation details and communication methods can be found in the Appendix.
The results of our world evaluation presented in Table 2. We successfully transfer multi- agent reinforcement policy to the real robot system with reasonable success rate after performing system identification to align the PD controllers between the simulation and the real robot. Our method outperforms the baseline methods, indicating the effectiveness of multi-agent reinforcement learning (MARL) and goal estimation real robot experiments. These components provide benefits in dealing with various unpredictable factors encountered in the real-world setting, leading to improved performance and robustness. We also notice that the success rates achieved in real-world experiments lower than the hit rate. This is primarily attributed occasional challenges encountered during the grasping phase of the catcher. In some cases, the catcher may fail to firmly grasp the object before it bounces back, leading to the object slipping off the robot's s hand.
6 Conclusion and Limitation
Limitation In our study, we acknowledge that the use of objects with low restitution may not fully capture the challenges faced in real-life scenarios where objects often have higher restitution. Objects with higher restitution tend to bounce back upon impact, making it more difficult catch them smoothly without collisions.
Conclusion We present Dynamic Handover, a system capable of throwing and catching with bimanual hands. Through the use of multi-agent reinforcement learning and goal estimation, our system demonstrates the ability achieve successful throw and catch in both simulation and real-world environments. We find that the goal estimation aids in mitigating the effects of unpredictable parameters and enhances the overall stability to bridge the large dynamics gap between sim and real.
8

---

Acknowledgment. This work was supported, in part, by the Qualcomm Innovation Fellowship. and the Technology Innovation Program (200181 12, Development autonomous manipulation and gripping technology using imitation learning based on visual and tactile sensing) funded by the Ministry of Trade, Industry Energy (MOTIE), Korea.
References
[1] H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 International Conference on Robotics and Automation (ICRA), pages 3651-3657. IEEE, 2019.
[2] T. Chen, J. Xu, and P. Agrawal. A system for general in-hand object re-orientation. In Conference on Robot Learning, pages 297-307. PMLR, 2022.
[3] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The Interna- tional Journal of Robotics Research, 39(1):3-20, 2020.
[4] J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016.
[5] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[6] H. He, Boyd-Graber, K. Kwok, and H. Daumé III. Opponent modeling in deep reinforcement learning. In International conference machine learning, pages 1804-1813. PMLR, 2016.
[7] Wang, Dong, V. Lesser, and C. Zhang. Roma: Multi-agent reinforcement learning with emergent roles. arXiv preprint arXiv:2003.08039, 2020.
[8] K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pages 5887-5896. PMLR, 2019.
[9] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611 1-2424 2022.
[10] A. Malus, D. Kozjek, et al. Real-time order dispatching for fleet of autonomous mobile robots using multi-agent reinforcement learning. CIRP annals, 69(1)3397-400, 2020.
[11] Y. Jin, Y. Zhang, J. Yuan, and X. Zhang. Efficient multi-agent cooperative navigation in unknown environments with interlaced deep reinforcement learning. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2897-2901. IEEE, 2019.
[12] Z. Xia, J. Du, J. Wang, C. Jiang, Y. Ren, G. Li, and Z. Han. Multi-agent reinforcement learning aided intelligent uav swarm for target tracking. IEEE Transactions on Vehicular Technology, 71 (1):931-945, 2021.
[13] Y.J. Chen, D.-K. Chang, and C. Zhang. Autonomous tracking using swarm of uavs: A constrained multi-agent reinforcement learning approach. IEEE Transactions on Vehicular Technology, 69(1 11111))3312137777 13717, 2020.
[14] G. Ding, J. Koh, K. Merckaert, B. Vanderborght, M. M. Nicotra, C. Heckman, A. Roncone, and L. Chen. Distributed reinforcement learning for cooperative multi-robon object manipulation. arXiv preprint arXiv:2003.09544 2020.

---

[15] O. Nachum, M. Ahn, H. Ponte, S. Gu, and V. Kumar. Multi-agent manipulation via locomotion using hierarchical sim2real. arXiv preprint /908, 05224, 2019.
[16] M. Zhang, Jian, Y. Wu, H. Xu, and X. Wang. Disentangled attention as intrinsic regularization for bimanual multi-object manipulation. CoRR, 2021.
[17] Y. Li, C. Pan, H. Xu, X. Wang, and Y. Wu. Efficient bimanual handover and rearrangement via symmetry-award actor-critic learning. 2023 IEEE International Conference Roboties and Automation (ICRA), 2023.
[18] M. Mason and K. Lynch. Dynamic manipulation. In Proceedings of (IROS) JEEE/RSJ International Conference on Intelligent Robots and Systems, volume 1, pages 132 159, July 1993.
[19] Y. Gai, Y. Kobayashi, Y. Hoshino, and T.. Emaru. Motion control of a ball throwing robot with flexible robotic arm. World Academy of Science, Engineering and Technology, International Journal of Computer, Electrical, Automation, Control and Information Engineering, 1.937-945, 2013.
[20] T. Senoo, A. Namiki, and M. Ishikawa. High-speed throwing motion based on kinetic chain approach. 2008 IEEE/RS International Conference on Intelligent Robots and Systems, pages 3206-3211, 2008.
[21] J. Kober, M. Glisson, and M. Mistry. Playing catch and juggling with humanoid robot. In 2012 12th IEEE-RAS International Conference Humanoid Robots (Humanoids 2012), pages 875-881, IEEE, 2012.
[22] J. Kober, E. Öztop, and J. Peters. Reinforcement learning to adjust robot movements to new situations. In International Joint Conference on Artificial Intelligence, 2010.
[23] A. Ghadirzadeh, A. Maki, D. Kragic, and M. Björkman. Deep predictive policy training using reinforcement learning. In 2017 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 2351-2358 IEEE, 2017.
[24] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. 2019.
[25] C. Chi, B. Burchfiel, E. Cousineau, S. Feng, and Song. Iterative residual policy: for goal- conditioned dynamic manipulation of deformable objects. arXiv preprint arXiv.2203.00663, 2022.
[26] C. Wang, S. Wang, B. Romero, F. Veiga, and E. H. Adelson. Swingbot: Learning physical features from in-hand tactile exploration for dynamic swing-up manipulation. In IROS, pages 5633-5640. IEEF, 2020.
[27] A. Petrenko, A. Allshire, G. State, A. Handa, and V. Makoviychuk, Dexpht: Scaling up dexterous manipulation for hand-arm systems with population based training. In Robotics: Science and Systems, 2023.
[28] J. Ye, J. Wang, B. Huang, Y. Qin, and X. Wang. Learning continuous grasping function with dexterous hand from human demonstrations. IEEE Robotics and Automation Letters, 2023,
[29] J. Bohg, A. Morales, Asfour, and D. Kragic. Data-driven grasp synthesis survey. IEEE Transactions on robotics, 30(2):289-309, 2013.
[30] A. Gupta, C. Eppner, Levine, and Abbeel. Learning dexterous manipulation for soft robotic hand from human demonstrations. In 2016 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 3786-3793. IEEE, 2016.
10

---

[31] M. Ciocarlie, C. Goldfeder, and P. Allen. Dimensionality reduction for hand- hand-independent dexterous robotic grasping. In 2007 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3270-3275. IEEE, 2007.
[32] Y. Qin, H. Su, and X. Wang. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camers teleoperation. IEEE Robotics and Automation Letters, 7(4): 10873-10881, 2022.
[33] Z.-H. Yin, B. Huang, Y. Qin, Q. Chen, and X. Wang. Rotating without seeing: Towards in-hand dexterity through touch. arXiv preprint arXiv:2303. 10880, 2023.
[34] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk,
K. Van Wyk, A. Zhurkevich, B. Sundaralingam, Y. Narang, J.-F. Lafleche, D. Fox, and G. State. Dextreme: Transfer of agile in-hand manipulation from simulation to reality, arXiv, 2022.
[35] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik, In-Hand Object Rotation via Rapid Motor Adaptation. In Conference on Robot Learning (CoRL), 2022.
[36] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's 's cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019.
[37] T. Chen, M. Tippur, S. Wu, V. Kumar, E. Adelson, and P. Agrawal. Visual dexterity: In-hand dexterous manipulation from depth. arXiv preprint arXiv:2211.1174, 2022.
[38] Y. Bai, W. Yu, and C. K. Liu. Dexterous manipulation of cloth. In Computer Gruphics Forum, volume 35, pages 523-532. Wiley Online Library, 2016.
[39] F. Ficuciello, A. Migliozzi, E. Coevoet, A. Petit, and C. Duriez. Fem-based deformation control for dexterous manipulation of 3d soft objects. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 4007-4013. IEEE, 2018.
[40] Li, Z. Huang, T. Chen, T. Du, H. Su, B. Tenenbaum, and C. Gan. Dexdeform: Dexterous deformable object manipulation with human demonstrations and differentiable physics. arXiv preprint arXiv:2304.03323, 2023.
[41] Y. Hou, K. S. M. Sahari, and D. N. T. How. review on modeling of flexible deformable object for dexterous robotic manipulation. International Journal of Advanced Robotic Systems, 14199988844,88894,, 2019.
[42] Y. Qin, B. Huang, Z.-H. Yin, Su, and X. Wang. Dexpoint: Generalizable point cloud rein- forcement learning for sim-to-real dexterous manipulation. In Conference on Robot Learning, pages 594-605. PMLR, 2023.
[43] N. Vahrenkamp, M. Przybylski. Asfour, and R. Dillmann. Bimanual grasp planning. In 201 I 11th IEEE-RAS International Conference on Humanoid Robots, pages 493-499, IEEE, 2011.
[44] R. Zollner, T. Asfour, and R. Dillmann. Programming by demonstration: dual-arm manipulation tasks for humanoid robots. 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)IEEE Cat. No. 04CH37566),, volume pages 484. IEEE, 2004.
[45] N. Xi, T.-J. Tarn, and A. K. Bejczy. Intelligent planning and control for multirobot coordination: An event- based approach. JEEE transactions on robotics and automation, 12(3):439 452, 1996.
[46] R. Chitnis, S. Tulsiani, S. Gupta, and A. Gupta. Efficient bimanual manipulation using learned task schemas. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1149-1155. IEEE, 2020.
[47] F. Amadio, A. Colomé, and C. Torras. Exploiting symmetries in reinforcement learning of bimanual robotic tasks. IEEE Robotics and Automation Letters, 4(2):1838-1 2019.
11

---

[48] S. Kataoka, S. K. S. Ghasemipour, D. Freeman, and Mordatch. Bi-manual manipulation and attachment via sim-to-real reinforcement learning. arXiv preprint arXiv:2203.08277, 2022.
[49] A. M. Castro, F. N. Permenter, and X. Han. An unconstrained convex formulation of compliant contact. IEEE Transactions Robotics, 2022.
[50] Y. Chen, Yang. T. Wu, Wang. X. Feng, J. Jiang. Lu, S. M. McAleer, H. Dong, and S.-C. Zhu. Towards human-level bimanual dexterous manipulation with reinforcement learning. In Thirty-sixth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL htttps://openreeeeeee http://ww..ettooommmmdd-0000000.
[51] K. Zakka, L. Smith, N. Gileadi, Howell, Peng. S. Singh, Tassa, P. Florence, A. Zeng, and P. Abbeel. RoboPianist: A Benchmark for High-Dimensional Robot Control, 2023. URL
[52] V. Makoviychuk, L. Wawrzyniak, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.104 10470, 2021.
12

---

A Detailed Implementation of Real Robot System
Bimanual Hands System. For our system, we have developed a ROS-based pipeline that operates at a control frequeney of 20Hz. This pipeline serves as foundation for controlling our setup, enabling efficient communication and coordination between the different components. In our configuration, the Arm-Hand subsystems are controlled by single policy utilizing multiple agents. This unified policy governs the actions of both subsystems, promoting synchronized and collaborative behavior in our setup. To achieve this, we control the motion of the robotic arms through Modbus TCP (Transmission Control Protocol) using an AC/DC Control Box. The control boxes of the two robotic arms are connected to a router via Ethernet cables, and the router is then connected to the host computer. Additionally, the two robot hands are directly connected to the same computer using RS-485 serial communication.
Object Tracking. Real-time object tracking is performed with Intel RealSense D435 stereo camera. Since the object has high color contrast from its background, we first use simple color detector on the RGB image to find the pixel location of the object. The color range for detecting blue object is constrained between [80, 200, 0] and [120, 255, 0] in HSV color space. Next, the 3D position of that pixel obtained from querying the corresponding depth value on the depth image, where post-proc post-proeessing filters including disparity, spatial and temporal, are applied reduce depth noise. Finally, we get the 3D object position in robot frame from image frame with calibrated camera extrinsics parameters.
Domain Randomization Isaac Gym offers several domain randomization functions for reinforcement learning training. We apply randomization to the task, as indicated in Table. 3 for each environment. generate new randomizations every 1000 simulation steps.
Parameter
Type
Distribution
Initial Runge
Robot
Scaling
uniform
[0.5,
Friction
Scaling
unifomm
[0.7. 1.3]
Juint Lower Limit
Scaling
loguniform
[0.0.0.01]
Joint Upper Limit
Scaling
logeniform
[0.0, 0.01]
Stiffness
Scaling
loguniform
Joint Damping
Scaling
logunifurm
[0.0.001]
Object
[0.3,
Priction
Scaling
uaiform
10.5, 1.5]
Scale
Scaling
uniform
[0.95, 1.05]
Observation
Obs Correlated, Noise Obs Uncurrelated Noise
Addiive
gaussian
[0.0, 0000]
Addicve
gassiaa
[0.0, 0.002]
Actinn Correlased Noise Action Uncorrelated Noise
Addiiie
gusssia gaussina
[0.0, 0.033]
Additve
(0.0, 0035
Rnvironment
Gravity
Addiave
normal
(0) 04]
Table 3: Domain randomization parameters.
B Sim2Real Transfer
System Identification. To achieve successful sim-to-real transfer, utilize system identification techniques to align the behavior of the PD ciopoottional-eeriattiie controller of the arm and hand in simulation with that in the real world. This involves tuning the PD coefficients of the controllers to ensure that their responses to impulse and sinusoidal inputs are aligned. This step is crucial in ensuring that the control actions generated in simulation can be effectively applied to the real robot setup, enabling reliable sim-to-real transfer our system.
13

---

Using Two Fingers Together
Using Only One Finger
Figure 6: Ablation study Left: policy with rod in Training. Right: policy without Using rod in Training.
C Why is There a rod Used in Training Objects?
We find this leads to more robust policy in the training. Specifically, we observed that the simultaneous movement of all fingers when throwing a ball led to more robust results, whereas when using balls/square objects, policy often used only a few fingers. Therefore, we designed the shape of the rod training. If the policy wants throw the rod stably, must learn ablation experiment, the results of which
all its fingers in throwing,
it can make our policy more robust. We conducted presented in Table.4. In terms of novel objects' success rate, observed that the policy's success rate without using a rod is lower compared to when the rod is used. We have included videos on our website that showcases the hand's s motion when training with and without the rod, as shown in Figure. 6. The video demonstrates that when the rod is excluded from training, the hand fails to effectively use of its fingers during throwing. leading to reduction in generalization ability.
Ablation
Known obj
Novel obj
Ours (w/o rod)
0.94
0.26#0.06
Ours
0.9540.03
0.37+0.04
Table 4: Ablation study in training w/w.o. rod
D How Pre-throw Conditions Impact Throwing Performance?
In this experiment, we examine the repeatability of the thrower. Our intuition that a thrower policy capable of generating consistent object trajectories often results in higher success rate for the catcher. We find that the robots' and ob- jects' initial positions have significant impact on the stability of the throwing motion. To ad- dress this, we investigate three different initial conditions (depicted in Figure 7) in real-world. For each initial position, we train a MARI. pol- icy and conduct 10 trials with the thrower robot. We compute the variance of the landing point on the table to evaluate the repeatability under different initial conditions. A smaller variance indicates better repeatability and stability. The results are summarized in Table 5. We observe that condition (c) demonstrates the smallest vari- ance and enables more stable throws towards the target compared to conditions (a) and (b). This suggests that initial firm grasp is advan- tageous for subsequent throwing behavior.
Target
Initial Position
Figure 7: Throwing Stability Test of different initial settings: (a) simply placing object on an open robot hand, (b) gripping the object with the robot hand, re- sembling parallel gripper, and (c) firmly grasping the object with the robot hand.
14

---

Settings
Pose
Pose
Pose C
Std(x)
0.051
0.072
0.024
Std(y)
0.087
0.048
0.043
Table 5: Comparison for Pre-throw Conditions: We calculate the standard deviation of landing points the table in the and y directions, based on 10 runs for each pose. The units are meters.
E Hyperparameters of the RL algorithms
Table. 6 and 7 are the hyperparameters of the RI. algorithms.
Hyperpanmmtees
Thrw and Catch
Num muni batches
I
Num opt-spochs
-pisade-knngth
Hyperpemeetess
Canch
Hiddon sine
[1024,
1024, .12)]
Num min: batches
Use
True
Num opt-spactts
value
Num episode length
Use proper time limils
False True
1024,
1054,
Use huber less
Clip mago
0.2
Huber delua
10
Max grad norm
Clip range
Max, grad nurm
10
Disonunt (t)
Lenriing mile
Si--
GAE lanbhda (A)
0.95
Discount (ry)
Opt-eps
Sie-4
Init noise
0.8
GAE lumbda
Sud x coef
0.95
1
Ent-coef
Sul coef
0.5
Table 7: Hyperparameters of PPO.
Table 6: Hyperparameters of MAPPO.
F
Reward design
The reward of our system can be computed as - Tdis + Tlinvel + Ttorgue- In the design of our reward, Fais is the reward that mainly responds to throwing objects to the target position. Piianel is reward that encourages throwers to release the ball from hand. Ttorque is a penalty item for robots that torque is too big. In our reward function, if rais missing, the object will be thrown the exact position, but will only be thrown forward vigorously. Without Ftianel, would often fall into sub-optimal where the thrower holds the ball in its hand and doesn' 't release. Ttorque common reward term that allows robots to avoid jitter and large dangerous movements.
G Perturbation Test in Simulation
We have conducted experiments such perturbation for the test set and show the video results in simulation (see project website, under the Perturbation Test section). We add three environmental conditions, e.g., wind flow opposing the thrower, along the thrower, orthogonal to the throw, as we show our website, as shown in Figure. 8. We show the strength the wind our videos (1st column under Perturbation Test. Our policy can still perform well in these scenarios, indicating that we have learned robust policy.
15

---

Perturbation Test
Orthogonal Perturbation
Test in Orthogonal Perturbation
Test in Orthogonal Perturbation
Opposing Perturbation
Opposing Perturbation
Opposing Perturbation
Along Perturbation
Test in Along Perturbation
Test in Along Perturbation
Figure 8: Perturbation test simulation
16


Reviews:
[{"review_id": "xLdwVGW-zDL", "reviewer": "Reviewer_VRgh", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a novel approach for throwing and catching objects with multi-fingered hands attached to robot arms using multi-agent reinforcement learning (MARL). In the proposed approach, thrower and catcher policies are trained using MARL in simulation and then deployed on the robot using Sim2Real transfer. A trajectory prediction model is also proposed to reduce the sim2Real gap and increase the chances of catching the object successfully. Experiments with multiple objects in a real-world setting demonstrate the superior performance of the proposed approach in comparison to baseline methods.", "summary_of_recommendation": "The paper is well-written and the core ideas are presented comprehensively. I enjoyed reading the detailed discussions on the experimental results. However, I am a bit concerned about the generalizability of the approach beyond the particular setup. It is unclear how well the learned policies will generalize to different settings, such as different embodiment or environmental conditions. If the authors can establish the generalization capabilities of the approach during the rebuttal period, I would feel more confident in recommending the paper for acceptance.\n\n\n**Post-rebuttal update:**\nThe authors have addressed my concerns satisfactorily. I appreciate the additional experiments conducted by the authors to demonstrate the effectiveness of the proposed method. I have updated my recommendation accordingly.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- It is stated in L162 that \"..include observation from past k frames as input for both policies to provide temporal information..\". While it is evident that adding last k frames should help the catcher to predict its goal position, it is unclear what benefits does it provide to the thrower policy. Please explain.\n- I would like to see the generalization capabilities of the trained policies. I would be curious to know how the learned policies will perform if we swap the thrower and catcher robots.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well-written and the core ideas are presented comprehensively. Experimental results are impressive and I appreciate the detailed discussions on them. However, I believe that the manuscript could be improved by including/expanding on the following points:\n- In Figure 1, left column, please explain why there are two blue balls in the scene.\n- In Figure 3, please explain what is the violet ball in the first subfigure in the top row, what are the ball trajectory start and end points in the first figure in the bottom row, and what the red and blue balls indicate in the remaining figures. Please add explanations for these in the caption as well.\n- Add details for the policy architecture used in the main text as well.\n- Table 2, please specify the units used. Also, briefly describe the main take-aways.\n- Table 3, in the caption, please specify again what HR and SR stand for for better readability. Also, briefly describe the main take-aways.\n- Please explain how the ball velocity is estimated using the current setup.\n- Please explain how the hyperparameters for the reward function were chosen.\n- A comparison to the policies learned using kinesthetic teaching and motion primitives (DMPs/ProMPs) with goal estimation will help establish the proposed method strongly.\n- It would be interesting to see how robust the learned policies are to environmental conditions, e.g., wind flow opposing the thrower, along the thrower, orthogonal to the throw, etc.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "Y2sV7v61u-n", "reviewer": "Reviewer_PiJr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper addresses the problem of throwing and catching objects from one hand + arm system to another. The robot behaviors are learned via multi agent RL in simulation and transferred to the real robot. Key contributions are the system itself (perhaps the first demo of sim2real for bimanual throwing/catching) and a potentially useful learning pipeline (training policies with a fixed final object state, training an estimator of the actual final object state, fine tuning the estimator and the policies).\n\nFrom the methods side the use of multi agent RL is not novel (the key algorithm is MAPPO, which has been described before) and the use of multi agent RL for bimanual dexterous manipulation is also not novel (references [16] and [17] in the paper). However it seems that multi agent RL for dexterous manipulation + sim2real has not been done before.\n\nSuccess rates are about 90% in simulation for known objects and 25-30% for unknown objects, and 33-60% in real for unknown objects.", "summary_of_recommendation": "The paper's core value is in demonstrating a system (maybe the first one) that uses sim2real for bimanual throwing and catching. In that sense the paper should be evaluated as a systems paper. The weaknesses on the systems side are the throwing/catching only happens once, there are only 3 real world objects, the success rates are low (33-60%), and although the real objects are technically unseen they are a bit easier to catch because they are bean bags. The video clips are nice to see since they demonstrate simultaneous coordination of many joints and some anticipatory and reactive behavior of the catching robot.\n\nThe paper's methodology is based on MAPPO and training an estimator of the final state of the thrown object. MAPPO is not new but the application to this context may be new. The estimator could potentially be claimed as contribution but I believe that the evaluations of its usefulness are weak in both simulation (unclear result for known objects) and real (limited ablation).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- What is meant by dynamic extrinsic dexterity?\n- Can you please provide a comparison of methods and results between your work and DexPBT? DexPBT is simulation only but it would still be very useful to see a description of the pros/cons of your two very different methodologies as well as your simulation based results.\n- How is the object tracked?\n- The action space is said to be 22 dim per arm + hand but then it's said on line 131 that the system performs better when the robot joints are fixed -- please clarify what is the actual size of the action space then?\n- How was the reward constructed and what did/didn't work well (see previous comment on reward)? I would also like to see an evaluation of the performance of the different reward formulations that were considered\n- Please define the notation in equation 1. Related questions -- how is the predicted goal defined? Is it position after fixed number of timesteps? And is a neural net actually necessary to predict something as simple as goal location after a toss?\n- It's not too clear to me why in phase 3 you simultaneously fine tuned both the estimator and the policy. I would like to request an ablation study for phase 3 -- simultaneous fine tuning of both the estimator and the policy versus just fine tuning the policy\n- Can you train your simulated policies over more seeds and also for more timesteps to see if the trends from Table 1 (especially the surprising results for known objects) are still true?\n- \"after performing system identification\" -- this seems to be a critical step. Can you describe your procedure and provide a comparison of performance with and without this procedure?\n- It would be nice if there were 4 categories in Table 3 (the same ones as in Table 1) so that the individual effects of MAPPO and goal estimation can be seen\n- Please provide videos of failure cases\n- Is the pre throw configuration manually specified during training in simulation and testing on the real robot?\n- What code do you plan to publicly release? The code to train the policies in simulation? The code to execute the policies on the hardware? Both?\n- The policies appear to be all underhand throws and catches -- could you comment on this?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The problem is interesting, potentially useful (extending workspace), and difficult (two arms and hands, large action space, dynamic, sim2real)\n- The experimental setup does not require a mocap system -- it uses a RealSense\n- Training code is provided (although I did not run it)\n\nMajor weaknesses:\n- I think multi agent RL for dexterous manipulation should not be listed as a contribution because of references [16] and [17]\n- Dynamics randomization is not a contribution, since it's been a standard approach since about 2018 -- would recommend removing this from the contributions list\n- The authors may have missed an important reference -- DexPBT by Petrenko et al. The work is simulation only but shows very impressive dynamic manipulation including between two hand + arm systems (see the videos on their project website)\n- The real world objects are sandbags -- this does make them much easier to catch. This is acknowledged in the limitations section (low restitution).\n- The simulation results are not particularly strong. There is a confusing result that the agent w/o MAPPO and goal estimation does better on known objects. (The training curves look noisy so I think it's worth trying this over more seeds and also training for more steps). Generalization to novel objects also gets a max success rate of 28%\n- \"this feature helps mitigate the impact of unpredictable parameters\" -- does not seem to be a well supported statement. Why not vary these parameters directly in simulation to see if this is true?\n- Success rates in the real world are quite low -- between 33-60%. I think this should be mentioned as early as possible in the paper (abstract and/or intro) so that readers are aware that this work is just a first step and doesn't reliably solve the problem\n- No summary video of the paper is provided\n\nMinor weaknesses:\n- \"Bimanual hands\" is redundant -- bimanual means using both hands\n- There are non standard phrases used and grammar errors throughout the paper -- would recommend closely revising the writing\n- The thrower agent is fixed, the catcher agent is fixed and a throw + catch is executed once (not back and forth) -- only a minor weakness because just a single throw + catch is a good contribution\n- Should mention that you use k prior states in figures 2 and 3\n- Reward shaping is often a critical process -- would be good to get more insight on how you constructed the reward, what other rewards you tried, how sensitive the reward was to coefficients like 0.003 etc\n- There is not much justification provided for the expanded set of simulated objects in figure 4b. Why these objects?\n- MAPPO is critical to the paper and ablations but not much description of it (and differences with standard PPO) is provided\n- Code uses outdated version of Isaac Gym\n- Code is not documented properly (missing docs)", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "NJV5c49a6U", "reviewer": "Reviewer_wGRc", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "- The paper aims to solve a dynamic and dexterous bi-manual handover problem, where one thrower robot is tasked to throw an object to a catcher robot, and the catcher should catch the object.\n- Both robots have a multi-fingered hand mounted on a fixed base arm.\n- A sim-to-real approach is adopted, where a policy is first trained in a simulated environment and then transferred to the real world.\n- The task is treated as a two agent problem and the policy is trained with a multi-agent RL algorithm.\n- The key idea is to use a goal estimator aside from the policy network to estimate the goal position of the thrown object for the catcher.\n- To tackle the sim-to-real challenge, a three stage training pipeline is proposed, which trains the policy networks and goal estimator to be robust to variations in the target throwing position.\n- The proposed method is compared against baselines in both sim and real, on 11 shape varying objects (in sim) and 3 sandbag objects (in real), respectively.", "summary_of_recommendation": "Overall, I think this is a solid work with good execution and impressive results. Some improvements can be made on the presentation of three-stage training and on the sim-to-real analysis. But this is already a strong submission regardless.\n\n### Post Rebuttal\nMy initial review has been positive and the rebuttal does not change that view. I'm happy to keep my position and recommend a strong accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See the review above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Strengths\n- The result video is quite impressive and visually appealing. Setting up a complex real system like this one and getting sim-to-real to work as demonstrated should be a major achievement and not something commonly seen.\n- The method is technically sound and well presented. The use of the goal position predictor seems sensible. For a dynamic and real-time task like this one, the model requires some kind of predictive capability for fast reaction.\n- Overall, the paper is well written and a pleasant read. Graphics and illustrations are well used and helpful for understanding the content.\n\n### Weaknesses\n- It is a bit counter-intuitive how the proposed three-stage training pipeline can be expected to bridge the dynamics gap (L143-151). The goal estimator is still somewhat overfitting to the trajectories from the simulator, so how is that expected to improve the transfer? \n- Following the last point, you fine-tune all the components with end-to-end training in stage 3. Will this further break the decoupling you deliberately established earlier between policy and goal estimation? Are there still some constraints for the goal estimator in stage 3?\n- While the sim-to-real transfer looks impressive from the video, a thorough sim-to-real analysis is lacking. For example:\n  - RL training has high variance, and you do multiple runs for each training. For each baseline, which run do you pick for real world evaluation?\n  - Does the one performing well in sim (e.g., in terms of return) also perform well in real?\n  - Can two models with similar training performance in sim have different qualitative performances in real and different sim-to-real transfer outcome?\n  - For each baseline, do you test multiple models (e.g., trained from different runs) in real?\n  - What are other sources of sim-to-real gap besides dynamics, e.g., perception? And how can you combat them?\n\n### Questions and Suggestions\n- [L23] \"bring multiple challenges\" -> \"brings multiple challenges\"\n- [L103] \"There are two robot agents in the this task: This task involves two robot agents:\" -> Duplicated sentences.\n- [Fig. 2] Why is object position 3 dimensional? Is it just (x, y, z) coordinates of the center?\n- [Fig. 2] Since the thrower's policy also takes in target position, I suppose you are also randomizing the target goal position (otherwise you don't need that input)? However, I don't see this randomization mentioned anywhere. The throw goal seems to be pre-defined in stage 1 (L145: \" the catcher observing the pre-defined throwing goal\"). Then is it being randomized in stage 3?\n- [L127] \"For the XArm-6, we employ delta joint positions as the control target, while for the Allegro hand, we utilize absolute joint positions as the control target. This design choice is made to avoid jerky motion of robot arm for safety reasons, ...\" -> Any reference or empirical evidence for avoiding jerky motion with this design?\n- [L146] \"(ii) Next, we freeze the base policy and train a goal estimator through supervised learning, using the rollout trajectory of the base policy as training data.\" -> Isn't the base thrower policy from (i) always consistently throwing to a pre-defined goal (see L136)? If so, how do you generate diverse trajectories to train the goal estimator?\n- [L146] Following the last point, is the dynamics gap between sim and real coming from hand-object interaction, or also from the flying phase where the object is traversing though air passively subjecting to gravity force?\n- [L148] \"(iii) Finally, we replace the pre-defined throwing goal in the catcher’s observation with the estimated goal and unfreeze the policy for fine-tuning both the base policy and the goal estimator in an end-to-end fashion.\" -> Is there still a loss for goal prediction in stage 3? If no, will the goal estimator diverge from goal prediction with fine-tuning, i.e., will it learn to predict some arbitrary output as long as the policy can accomplish the final catching task?\n- [L160] \"..., the catcher's policy, ..., takes as input its proprioception, the pre-defined goal position ...\" -> Should it be the \"predicted goal position\" instead of \"pre-defined goal position\"?\n- [L165] \"... we design the reward function using three components: (i) distance between object and throwing goal; (ii) object velocity projected in the direction from thrower to catcher; (iii) robot joint torque.\" -> Why is there no reward for catching?\n- [Eq. 1] While easy to infer, please still explicitly explain what $\\omega$ is.\n- [Eq. 1] What is the meaning of superscript $1$ and $0$ here?\n- [Tab. 2] \"We calculate the standard deviation of ... We calculate the standard deviation in ...\" -> Can be trimmed.\n- [Fig. 5] For \"Ours\", is the training curve from stage 3?\n- [L234] \"... as we validate in Section 5.1\" -> 5.2?\n- [L236] \"The utilization of multi-agent reinforcement learning and goal estimation proves beneficial for accomplishing the throwing and catching task.\" -> Why should \"w/ MAPPO\" outperform \"w/o MAPPO\"? The single agent setting should be strictly easier than the multi-agent setting, since both thrower and catcher have full access to the states of both sides. Therefore, the single agent setting should perform strictly better than the multi-agent setting? However, this result suggests otherwise which seems rather counter-intuitive.\n- [L275] \"table  2\" -> \"Table 2\"\n- [Supp L13] Object tracking is the real world may be noisy, while it is always noise-less in sim. Is the sim-to-real gap on that front taken into account? Do you need to add noise to object position in sim? Does this gap affect the goal estimator's performance in the real world since it takes the tracking output?\n- [Supp L13] Following the last point, is there any bottleneck (i.e., frame rate) from the vision sensor? And how fast is the object tracking running? Is it faster than the control rate of the policy (i.e., in training)?", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "4ToAOZ69xw", "reviewer": "Reviewer_GCGh", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method to learn a bimanual policy for a dynamic manipulation task, namely catch-and-throw. This paper formulates the task as a multi-agent problem and uses multi-agent RL to learn policies. The authors also designed a sim-to-real pipeline so that the RL policies can be cheaply learned in simulators and the policies can be deployed on the physical hardware. Both qualitative and quantitative results in simulation and on real robots are provided, and the learned policies are demonstrated to generalize on different objects that are never seen during policy training.", "summary_of_recommendation": "I think the paper is well-written, and the experiments are reasonably done. However, several questions need to be addressed (which I believe can be done during the rebuttal phase). I'll adjust the score based on the quality of the authors' replies (They don't need to be lengthy. I want to see the replies that are straight to the point.)", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- About Experiments: Why is there a rod (capsule object) used while none of the testing objects are similar to this object in geometry?\n- Which point in the object is considered as its tracking point to specify its positions? \n- My understanding is that the authors did not leverage the geometry information of objects. In this case, to what extent (in terms of size or shape variance) do we expect a learned policy to generalize?\n- How do failure modes look like? Please include the videos. \n- Why are the numbers in real robot experiments on unseen object generalization significantly higher than in simulation?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall comments: I think the authors have proposed an interesting problem in the bimanual manipulation and focus on the dynamic manipulation tasks that are less explored in prior works. The execution of the experiments (especially real robots) are well designed. \n\n\nA high-level comment on the control and task design: Why is only delta positions enough? To my understanding, since only PD control is used here, the robot is only following delta positions. That being said, the control has nothing to do with the end effector velocities or acceleration, which are important to the dynamic manipulation authors focus on. While it might seem reasonable to do tasks like catch/throw in the paper, there are many dynamic manipulations that might be able to do with the current setup. I think this is a shortcoming in the proposed system which is supposed to solve a wide range of dynamic manipulation tasks. Following this point, there are two comments:\n- Ln92: Please remove “Opening cap” example if this paper did not cover this task. It sounds a bit overclaimed.\n- Ln 93: I think doing this requires “contact modes” that need to be achieved by more than one arm. Just having two arms do not constitute as necessary component for human-level skills. I think this claim should be rewritten in a more rigorous way.\n\nAnother shortcoming in this paper is its mismatch in the reported numbers between simulation and real-robot experiments --- Why does the simulation only have 28% success rate while all SRs in real-robot experiments are higher than 28%? I read that the simulation trail is evaluated over 100 while the real robots are evaluated over 5 trials. I think the statistical evaluations need to be improved.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a system for dynamic handover using multi-agent RL and sim2real transfer. The reviewers agree that the presented task is interesting, useful, and challenging. Therefore, the result achieved by the final system is impressive.  In the rebuttal, the author provides additional experiment results that the reviewer finds informative, and we encourage the author to consider including this information in the final version of the paper.
Decision: Accept (Poster)