Twisting Lids Off with Two Hands
Authors: Toru Lin
Zhao-Heng Yin
Haozhi Qi
Pieter Abbeel
Jitendra Malik
Venue: CoRL 2024 (2024)
Keywords: Bimanual Manipulation
Sim-to-Real
Reinforcement Learning

Abstract: Manipulating objects with two multi-fingered hands has been a long-standing challenge in robotics, due to the contact-rich nature of many manipulation tasks and the complexity inherent in coordinating a high-dimensional bimanual system. In this work, we share novel insights into physical modeling, real-time perception, and reward design that enable policies trained in simulation using deep reinforcement learning (RL) to be effectively and efficiently transferred to the real world. Specifically, we consider the problem of twisting lids of various bottle-like objects with two hands, demonstrating policies with generalization capabilities across a diverse set of unseen objects as well as dynamic and dexterous behaviors. To the best of our knowledge, this is the first sim-to-real RL system that enables such capabilities on bimanual multi-fingered hands.

Twisting Lids Off with Two Hands
Toru Lin
Zhao-Heng Yin
Haozhi Qi
Pieter Abbeel
Jitendra Malik
University of California, Berkeley
Hardware Setup
Twist off Lids in the Real-Worid
Allegro Hands
Camera
Holding Objects Stably
Coordinsting Twist
Twisting Lide
Generalization over Diverse Household Objects with Different Physical Properties and Shapes
12.5 om, 72 129g
9.5 cm, 7 F.mm, 40
7.0 on d.cm,, 71g
12.8 cm, 65
10.4 un 7.9 t5
10 cm. 8.6 on, 44g
42
146
125g
We train two anthropomorphic robot hands to twist (off) lids various articulated ohjects. The control policy is first trained in simulation with deep reinforcement learning, then zero-shot transferred to a real-world setup. We show that single policy trained manipulate simplistic, simulated bottle-like objects generalize to real-world objects that have drastically different physical properties (e.g. shape, size, color, material, mass). The length, diameter (or diagonal length), and mass each object are annotated bottom of individual subfigures. More results can be found in our video and and project website.
Figure
Abstract: Manipulating objects with two multi-fingered hands has been long- standing challenge in robotics, due to the contact-rich nature of many manipula- tion tasks and the complexity inherent in coordinating a high-dimensional biman- ual system. In this work, we share novel insights into physical modeling, real-time perception, and reward design that enable policies trained in simulation using deep reinforcement learning (RL) to be effectively and efficiently transferred to the real world. Specifically, we consider the problem of twisting lids of various bottle-like objects with hands, demonstrating policies with generalization capabilities across diverse set of unseen objects well as dynamic and dexterous behaviors. To the best of our knowledge, this is the first sim-to-real RL system that enables such capabilities on bimanual multi-fingered hands.
Keywords: Bimanual Manipulation, Sim-to-Real, Reinforcement Learning
Introduction
Achieving dexterous bimanual manipulation with two anthropomorphic robot hands has been excep- tionally challenging, due to the inherently contact-rich nature of many manipulation tasks and the complexity of coordinating high-dimensional bimanual system. This work takes a step towards this grand goal, by demonstrating the feasibility of learning highly dexterous and dynamic bimanual
"Correspondence to (toru, zhachhengyin/eessekeeee.. edu.. The first two authors contribute equally.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

manipulation policy purely in simulation and zero-shot transferring it to the real world. Specifically, we study the task of twisting or removing lids with two multi-fingered robot hands. This task is both practically important and profoundly interesting: for one, the ability to twist or remove lids from containers a crucial motor skill that toddlers acquire during their early developmental stages [1, 2]; for another, the manipulation skills required for this task, such as the coordination of fingers to ma- nipulate multi-part object, can be generally useful across a large collection of practical tasks.
Since collecting human expert demonstration data to solve contact-rich tasks via imitation learning is highly challenging and expensive [3], we aim at training a generalizable policy through sim-to-real reinforcement learning without using any expert data (Figure 1). Our method does not require pre- cise modeling of any individual object, or hardcoding prior knowledge on object properties; instead, stable and natural bimanual finger behaviors emerge through large-scale reinforcement learning (RI. training. Below, we share the novel insights that enable us develop such system.
Physical Modeling. Our work features a novel class of objects for in-hand manipulation: articu- lated objects defined as two rigid bodies connected via revolute joint with threaded structure. Accurately modeling friction and contact with revolute joints and threaded structures has tradition- ally been a hard challenge in robotic simulation [4]. To address this, we introduce a brake-based design to model the interaction between the lid and body of bottle-like objects. This design is fast to simulate while maintaining high fidelity to real-world physical dynamics, enabling efficient policy learning and successful sim-to-real transfer.
Perception. We initially hypothesize that fine-grained. contact-rich manipulation task like lid- twisting must require precise perceptual information object states and shapes. To our surprise, two-point sparse object representation, extracted from off-the-shelf object segmentation and tracking tools, is sufficient to solve the perception problem. With simple domain randomization techniques, we train policies that are robust against occlusion and camera noise. This discovery suggests that a minimal amount of perception information can be adequate for complicated bimanual manipulation tasks.
Reward Design. Previously performant reward designs for tasks like in-hand reorientation [5. 6, 7] cannot be straightforwardly applied our task, since those tasks focus on manipulating single-part rigid bodies with one hand rather than multi-part articulated bodies with two hands. Solving this task is more challenging since it involves more complex and precise contact (e.g. using two hands to hold a lid). In addressing this challenge, we discover a simple keypoint- based contact reward that yields natural lid-twisting behavior on the robot fingers.
We conduct several controlled experiments in both simulation and the real world. Through empirical analysis, we verify that our simulation modeling, perception module, and reward design can reliably lead to the desired behavior of lid-twisting. Our final successful policy manifests natural behavior across test objects with various physical properties such as shapes, sizes, and masses in simulation. Moreover, the learned policy can be zero-shot transferred to a wide range of novel houschold objects whose lids can be removed (Figure 1), and it is robust against perturbations.
2 Background
For decades, bimanual manipulation has remained an unsolved challenge in robotics [8, 9, 10, 11, 12, 13]. While multi-fingered robot hands seem be a natural choice for bimanual robot systems in theory, designing controllers for high-dimenssioal action spaces remains an open problem. Classical approach has made significant progress but usually assume known object or physics information [14, 15] and the generalizability remains unknown. In recent years, bimanual manipulation has been actively studied with learning-based methods, as a result of progress in learning algorithms and compute infrastructure. These learning-bact based approaches can be categorized into two types: 1) learning from real-world data; 2) learning in simulation, then transferring to the real world (sim-to- real).
Learning from Real-World Data. Rapid progress has been made in RL in the real world. Zhang et al. [16] learn to chain motor primitives for vegetable cutting, with relatively simple motion prim- itives; much of the task difficulty is bypassed via the use of specialized end-effectors [17, 18]. Chiu

---

Revoluto Joint
Custom-Mact Bottles
A-B
(within Training Distribution except the Square Bottle)
Prismati
Real World Household Objects (Outtof-Ditriiutioo)
Lid Link
B
Brake Link
A
Training Objects Simulation
MAcCA
nutelld
B
Figure 2: Our bottle model and used bottles in simulation the real world. A: Simulated bottle URDE. B: Training bottle objects in simulation. C: Custom-made bottles (in-distribution except for the rightmost square bottle). D: Household object bottles (out-of-din) )
et al. [19] learn precise needle manipulation with two grippers by integrating RL with a sampling- based planner. While impressive, these works cannot easily scale to higher dimensional action space due to their sample inefficiency or the need to define heuristic- based action primitives.
Most recent successes in bimanual manipulation achieved by learning from demonstrations [20, 21, 22, 23, 24, 25]. However, successes so far are largely limited to simple end-efffectoss like parallel jaw grippers due to the lack of high-quality demonstration data from multi-fingered robot hands [26]. Although several works aiming to improve demonstration data collection with two arms [27, 28, 29] or multi-fingered hands [21, 30, 31, 32, 33], their latency and retargeting errors limit their practical applicability and scalability. Lin et al. [20] proposes scalable hands-arms teleoperation system that learns smooth bimanual policies, but the system compromises on dexterity. Similar systems that offer more dexterous control [34, 35], on the other hand, suffer from drawbacks ranging from jittery control high costs. Our method uses RL in simulation and is thus not limited by the hardware and data collection infrastructure problems faced by learning from demonstration approaches.
Sim-to-Real. There has been growing interest in sim-to-real approaches for robotics i.e. learn- ing policies in simulation and transferring them to the real world - stimulated by several notable successes recent years ranging from locomotion [36, 37, 38] to manipulation [5, 7, 6, 39]. Existing works in manipulation, however, are mostly done with either a single multi filti-ffigeed hand [3, 40, 41, 42, 43, 44, 45, 46, 47], or two arms with simpler end-effectors [48, 49, 50). While Chen al. [51] and Zakka al. [52] feature bimanual tasks with dexterous hands, only simulation results shown. The work most related to ours is Huang al. [53], where the thors demonstrate throwing and catching objects using two dexterous hands. However, our task is significantly more contact-rich and requires substantially more challenging bimanual coordination to maintain object stability at all times. To our best knowledge, there is no learning-base method directly comparable to ours on the proposed task.
3 Learning to Twist Lids with Two Hands
We focus on the challenging task of lid-twisting for container objects, since it is a complex in-hand manipulation process that requires dynamic dexterity of multiple fingers and precise coordination between two hands. The goal this task is to twist the lid about the object's axis rotation in one direction as much as possible; during this process, the object should always stay in hand. Achieving this involves a sequence of delicate movements: 1) after initialization, the robot hand should firmly grasp and slightly rotate the bottle suitable pose; 2) the hand that closer the object lid should place its finger around the lid initiate twisting motion; 3) the two hands should coordinate avoid dropping the object while one hand twists the lid. Motor skills that arise from this task could serve

---

Finger Contact Reward
Rotation Reward
Object Pose
Reward
Left Hand Ref. Contact Paints
Right Hand Ref. Contact Points
Figure 3: Left:: Real-time perception system. Top: overview. Bottom: segment and track object parts from the RGB frames (left), take mask centers object part centers (middle), and estimate 3D object keypoints using noisy depth information from camera (right). Right: Illustration reward design. Our task-specific reward contains finger contact reward (yellow arrows), twisting reward (white arrow), and pose reward (bluc arrow). In particular, our keypoint- based finger contact reward is crucial for Icarning the desired behavior.
as generic abstractions for skills necessary to manipulate many other household objects, especially those with revolute joints such as Rubik's cubes, light bulbs, and jars.
3.1 Object Simulation
A central challenge in simulating the lid-twisting task is how to model friction between the bottle body and the lid properly, particularly static friction. Simulating this type of physical force has been a long-standing problem in robotics and graphics [4]. We design simple modeling approximation that strikes balance between fidelity and speed during physical simulation; our bottle-like object model is illustrated in Figure 2(A). Our design features a special Brake Link that constantly presses against the bottle lid via prismatic joint. This artificially generates frictional forces between the bottle body (Base Link) and the lid (Lid Link), preventing relative rotation between them - similar to a bottle with its lid screwed on. We replicate these bottles in the real world, as shown in Figure 2(B).
We note that naively tuning static friction properties between two revolute-joint-comee connecce bodies realistic enough with our simulator. Such brake-based is the only way find that simulate the static friction well.
3.2 Task Initialization
To better benchmark the bimanual twisting capability, we consider a class articulated bottles with lids that can be twisted infinitely (see Figure 2(A) and Section 3.1 for more details). Each object of interest consists of two rigid, near-cylindrical parts (a "body" and "lid"); the two parts are connected via continuous revolute joint, allowing them to rotate about each other. At the beginning of each episode, the two robotic hands initialized in static pose with faward-facing palms, and a bottle-like object is gently dropped or placed onto the fingers. The initial pose of the object is randomized both in translation and rotation to fixed default pose; the initial joint positions of the hands are randomized about a canonical pose by adding Gaussian noise. Note that since we do not assume a stable grasp configuration at task initialization, the control policy needs to learn in- in-grasp reorientation to place the object stable location perform successive manipulation.
3.3 Policy Learning
Bimanual in-hand dexterous manipulation involves highly complex hand-object contacts, and re- mains challenging to solve with traditional methods. In this work, we address the control challenge through RL. We formulate our control problem as a partially observable Markov Decision Process.
Observation Space. At each time step t, the control policy observes the following information from the environment: the proprioceptive hand joint positions (t, the estimated center-of-mass 31) positions of the bottle base and lid, and previously commanded target joint positions q..
Action Space. We use PD controller drive the robot hand. The control policy produces relative target joint position the action a,, which is added to the current target joint position 9e to produce the next target: f:11 = qe + ()FMM(a;)) 17 is scaling factor. Note that we smooth the action with

---

Single Ciject (40)
S2ng Object
Sniw OZNES (TT))
Sigle Diject
choblec (012
519
dbublic
240
0.23
reduced 50%
1030
433
mouee 50%
400
0.033
2010
102
+20
0.000
3.506
0.003
295
200
0.082
339
340
0.3
0.5
0.0
0.5
1.0
0.0
0.5
3.0
3.0
0.3
3.0
(TTT)
Diject
chaclic (05)
WD viiin
593
dbublec (0%)
0.0399
(20%
00150
431
1230761
10025
0.000
132
3.0075
302
240
00853
330
120
3.009
3.
0.5
3.5
1.5
00 35
3.5
3.5 2.5
0.0
0.5
3.0
3.5
3.5
0.0
35
1.5
35
2.0
Figure 4: Training curves different settings. Top: Single-object training results (evaluated on sinele-object
multi-object setup). Left half: Comparisons of different reward setups. Right half: Ablations on the use of vision. The results are averaged on 5 seeds. The shaded area shows the standard deviation. The AD score is averaged by the total execution steps.
setup). Bottom: Multi-object training results (evaluated
its exponential moving average (EMA) to produce smooth motion. The next target position is sent to the PD controller to generate torque on each joint.
Reward. While one way approach hard exploration problems is to add intrinsic rewards [54, 55), we introduce the following fine-grained reward terms to shape the hand behavior (Figure 3 right). (1). Twisting Reward. We define the twisting reward as Ttwisting = A = 1+1 Bottie which the rotation angle of the lid during one-step execution. This reward term encourages the hand to twist the lid. (2). Finger Contact Reward. We find crucial to use a set of reference contact points to guide effective contact between fingertips and the bottle. We define two set of points X E RRax and X m3x3 attached on the bottle base and lid respectively. Then, we define the finger contact reward as Toontact = Σi I+ad(X', FFjj + (Fad(X",FF5) where F E R4x3 and F R2x3 are the position of left and right fingertips, or is a scaling hyperparameter, and d is distance function defined as d(A,x) = min; A|A x1|2. Therefore, we require each fingertip to stay as close one of the reference contact points as possible. As will see later, this term is necessary for eliciting desired behavior and task success. (3). Pose Reward. We also introduce pose matching reward term to encourage the bottle main axis Xaris aligned with a predefined direction v. This term is defined as posee = arccos((Xazis. v)). (4). Regularizations. Besides the three task-specific rewards, we also introduce another few regularization terms as in previous works [44], including work penalty and action penalty to penalize large, jerky motions. We leave the details of the definition to the appendix.
Reset Strategy. There exist many possible hand-object interaction modes. Among these, most modes lead to failures such as getting the object stuck between fingers, and exploring those modes rarely provides good learning signals. To circumvent the high dimensionality of our exploration problem, we introduce two early termination criteria. First, we reset an episode if the robot hands fail to rotate the bottle into desired pose for bimanual twisting within a short time limit. Additionally, we reset when the bottle's 's 2- z-position is below a certain threshold, as the fingertips of the two hands can pinch the bottle at low position without being able to reposition into the palm.
Training. We use PPO [56] with asymmetric critic observation [57] to train our policy, and introduce various domain randomizations to make the policy transferable the real world. We apply both both physical and non-physical randomizations. The detailed training setup can be found in the appendix.
3.4 World Perception.
Figure 3 shows an overview of our real-world perception pipeline. To make our RL policy more transferable, use bottle and lid center points instead of pixels as vision input. We extract these keypoints from real-time images through object segmentation and tracking in the real world. Specif-
5

---

Force
Dan 100% Contect
Foce
Force
Coumany
Dur 55%
Figure 5: Left: Behavior of different reward functions. Top: Our full reward function achieves stable grasp. as well as smooth, natural, human-like twisting motion. Middle: naive gait constraint reward without any contact hints leads to erratic finger motion and unnatural grasps. Bottom: A reduced contact reward yields somewhat natural hehavior, but the grasp is loose compared to the full contact reward case. Right: Perturhing a learned policy with random external force. Our policy is resilient these external forces and able recover.
ically, we utilize the Segment Anything model [58] generate two separate masks for the bottle body and the lid on the first frame of each trajectory sequence, and XMem [59] to track the masks throughout all remaining frames. To approximate the 3D center-of-mase coordinates of the bottle body and lid, we calculate the center position of their masks in the image plane, then obtain noisy depth readings from depth camera to recover a corresponding 3D position. The perception pipeline runs at 10 Hz to match the neural network policy's s control frequency.
4 Simulation Experiments
To test how to enable the emergence of natural and robust manipulation behaviors, we first conduct several experiments in simulation. Specifically, we study the following questions: How important the keypoint-based reward for eliciting desired twisting behavior in bimanual manipulation? How important is visual information for solving this task? Is a sparse keypoint representation enough for learning generalizable policy?
4.1 Setup
Object Set. In simulated experiments, we use collection of simulated cylindrical bottles with varying aspect ratios for both training and evaluation. Some samples are visualized in Figure 2(B). We consider two setups in simulation: 1) multi-object, in which all the objects are used, and 2) single-object, in which single medium-sized bottle that represents the mean of the dataset used.
Evaluation Metric. We introduce the following metrics for evaluating the performance: 1) Angular Displacement (AD) the total number of degrees through which the lid has been twisted; 2) Time- to-Fail (TTF) the period measured from the moment the bottle is held to the point when it either slips from the hand becomes lodged; Velocity (Vel) AD divided by TTF, reflecting the speed of twisting motion.
Baselines. We compare our policy with the following baselines. Policy without Vision. This is a neural network policy without object state information. We use this to evaluate the importance of vision. 2) Policy with Reduced Contact Reward. In training this policy, we reduce the intensity of our proposed finger contact reward. We use this to study the role of our contact reward in policy learning and shaping the policy's behavior. 3) Policy with Gait Constraint Reward. training this policy, we replace our contact reward with gait constraint reward function similar to ones used for in-hand reorientation tasks [39]. This baseline is only used for qualitative analysis.
4.2 Results
Reward Design. We first compare our approach with the reduced finger reward baseline (Figure 4). After decreasing the scale of finger contact reward, learned policies fail to master the desired lid-twisting skill and have low performance in general. We hypothesize that this is because the

---

Table 1: Comparison with baselines on real setup. For each method, we deploy 3 policies trained on 3 different seeds and average the results. Each deployment trial is conducted for 30 seconds.
BlueBottle
WoodBottle
RedBottle
GoldBontl
SquureBottle
Method
AD1 f T
Vel
AD
TTF
Vel +
AD f TTF 1 Vel
AD f TIF
Vel f
AD 1 TTF 1 Vel
Replay
128.33
7.67
11.68
2.67
7.67
0.22
15.00
4.67
1,50
28.33
7.67
2.99
29.67 =5.02
10.00
2.97
2217.66
24.93
119.50
24.02
13.35
20.38
120.98
#4.02
$2.00
243.04
24.04
24.17
20.00
20.55
No-Vis
1,33
21.67
0.04
1.07
14.67
0.27
1.90
8.33
0.27
0.67
16.33
0.04
5.00
20.33
0.18
#2.31
214.45
20:08
$1.85
215.1
20.36
23.29
26.11
20.41
21.15
213.05
20.06
20.34
111.2
20.20
No-Asym
18.67
30.00
0.62
0.67
19.33
0.03
8,33
13:00
0.28
3.67
0.54
0.00
2.33
0.00
+7.51
Large
2,00
22.33
0.14
0.00
24,00
0.00
2.33
9.33
0.47
2.67
22.33
0.09
1.67
30.00
0.06
#2.00
213 28
20.15
20:00
210.30
20.00
22.33
$4.75
20.08
23.00
213 26
20.30
$2.89
20.00
20.10
Ours
946.33
23,67
41.26
499.50
30,0
16.65
150.67
30,00
5.02
98.67
30,00
3.29
43,00
30.00
1.43
10.997
4.36
537833
3000
::
19:27
113447
20.00
3377
66.91
20.00
2:223
1:2.22
=0.00
=0.40
motion of lid-twisting requires a very specific pose pattern for holding the object; without explicitly encouraging such a pose pattern (e.g., via its contact modes), RL exploration becomes so hard that unsolvable within the available training time. We also observe positive correlation between the intensity of finger contact reward and both 1) sample efficiency during learning and 2) performance of learned policies (as reflected by Figure and qualitative observations in Figure 5 (left)).
Vision vs. No Vision. We also study the importance of vision modality. Existing works show that certain rotation behaviors can be achieved through implicit tactile sensing (via proprioception) [39]. However, our empirical results show that, in both single and multi-object setups, the no- vision base- line performs substantially worse than our full method. This suggests that knowledge of the position of bottle keypoints is essential for successful fid-twisting.
Single Object vs Multi Object. We run RL training with two object settings: 1) using a single bottle-like object; 2) using multiple bottle-like objects with more variation in the ratio between the bottle base and lid. For results shown in Figure 4, all multi-object training runs are evaluated on multi-object setup and all single-object training runs are evaluated on single-object setup. The two settings pose trade-off between specialization and gemeralization: in the single-object scenario, the policy might learn successful behaviors more easily but find it harder to generalize to unseen objects, and vice versa. To our surprise, we observe that multi-object training yields slightly better performance compared to single-object training. We hypothesize that multi-object makes exploring lid-twisting behavior an easier process by introducing an object curriculum that covers both easy and hard object instances during training.
Real-world Experiments
5.1 Experiment Setup
Hardware Setup. We use two 16-DoF Allegro Hands from Wonik Robotics for our experiments. Each Allegro Hand is mounted on a fixed URSe arm. We employ a single RealSense D435 depth camera to provide visual information, from which we extract object state information. We send control commands the robot at frequency of 10 Hz. via a Linux workstation.
Object Set. For quantitative evaluation, we evaluate the sim-to-real transfer capabilities of our poli- cies on five different articulated bottle objects (Figure 2(C)). Among them, four are in-distribution (round-body bottles) and one is outside of the training distribution (square-body body bottle).
Evaluation Metric. We measure both AD and TTF in 20 trials, with each trial lasting for maxi- mum of 30 seconds. For each evaluated method, select the three best policies out of ten policies trained on ten different random seeds. We end trial if the bottle falls off the palm.
Baselines. We compare our final policy with the following baselines to study the effect of several key design choices. 1) Open-loop Replay Policy (Replay). We record successful trials of our learned policy in the simulation and randomly select a trajectory to replay on the real robot. This baseline is used to evaluate whether the task can be solved by a deterministic motion pattern. 2) Policy withour Vision (No-Vis) This baseline policy only takes proprioceptive state information as input, without information about the object state. 3) Policy without Asymmetric Training (No-Asym). We compare with baseline where policy is trained without asymmetric PPO, and evaluate whether introducing additional privielged information into the value network will affect the transfer performance. 4)
7

---

Larger Policy Network Size (Large). We increase the size of our actor- network and train large-size policy. We use this to evaluate whether over-parameeeriiztion harms policy performance.
5.2 Twisting Lids in the Real World
We show quantitative results comparing our policy with bascline policies in Table 1. For both metrics, our policy outperforms all baselines across all evaluated objects. Our method can perform stable grasp on all the objects, and can rotate 3 out of 5 objects at reasonable speed. In particular, for the blue bottle, one of the deployed policies can achieve full turns (360 degrees) in 30 seconds on average. contrast, almost all the baselines fail to achieve any effective rotations, either getting stuck or dropping the bottle to the ground. We find that the open-loop policy has the lowest TTF score. Replaying a successful trajectory will not lead to stable grasp for most of the time, and the bottle will directly roll on the fingers and then drop off the palm. This suggests that the considered task involves very fine-grained contacts and requires the policy to act very precisely according to the object state. Another interesting observation is that the large policy does not transfer to the real world, although we confirm that it can achieve similar performance to our full policy in simulation. This suggests that some overfitting occurs, and controlling the size of the policy network very important for the successful sim-to-real transfer of our considered contact-rich task.
5.3 Robustness against Perturbation
Finally, we also evaluate our policy's 's robustness against force perturbation. Specifically, we perturb the object during deployment at random times by poking or pushing it along random directions using a picker tool (see the right of Figure 5). We find that our policy can reorient and move the object back to a stable pose for continuous manipulation, indicating that has some robustness against external forces and can adapt to these unexpected changes. Note that use marker-basee object detection system in this experiment to disentangle the visual occlusion effect.
5.4 Exploration of Twisting Lids Off
In the above section, we mainly study whether the twisting behavior can naturally emerge and be transferred the real world. Next, explore the limit of approach by testing it 10 novel household objects (Figure 2(D)). These objects differ substantially from our training objects in terms of shape, size, mass, material, color, and mechanical design. While the lids of the synthetic bottles that we use for both simulation training and real-world testing can be twisted infinitely, the lids of these household objects cannot. To evaluate our policy's 's ability to generalize the fid-twisting skill to these novel objects, we use the success rate on novel yet adjacent lid-removal task as the criterion. We define lid-removal as the object's lid being completely detached from the object body (c.g.. when the lids fall from the robot's hands in Figure 1.
We find that our policy continues to achieve stable and natural twisting behaviors on these novel objects. Furthermore, while we only train the policy for lid-twisting, we also find our policy capable of removing lids. For HairMask and FiberGunmies, our policy showcases lid-removal rates of more than 50%. For particularly challenging objects that require many turns to remove the lid, such as PeanulButter and EmptyNutella, our policy only achieves 10% lid-removal rates; however, its twist- ing behavior steady and robust to perturbation (see video supplementary for visualization).
6 Conclusion
We present an RL-based sim-to-real system for twisting or removing lids of bottle-like objects with two hands. We propose several techniques to handle the challenges that arise: a novel reward design, a sparse object representation for real-time perception, and an efficient yet high-fidelity method simulate twisting bottle caps. We conduct experiments in both simulation and real world to demonstrate the effectiveness of our approach. Our real-world results show generalization across wide range of seen and unseen objects.
8

---

Acknowledgments
We thank Chen Wang and Yuzhe Qin for helpful discussions on hardware setup and simulation of the Allegro Hand. TL is supported by fellowships from the National Science Foundation and UC Berkeley. ZY is supported by funding from InnoHK Centre for Logisties Robotics and ONR MURI N00014-22-1-2773. HQ is supported by the DARPA Machine Common Sense and ONR MURI N00014-21-1-2801.
References
[1] A. A. O. Pediatrics. Caring for Your Baby and Young Child: Birth to Age 5, American Academy Of Pediatrics, 2019.
[2] R. Watling, Peabody Developmental Motor Scales. Springer, 2013.
[3] Chen, J. Xu, and Agrawal. system for general in-hand object re-orientation. In CoRL, 2021.
[4] Y. Narang, K. Storey, I, Akinola, M. Macklin, P. Reist, L.. Wawrzyniak, Y. Guo, A. Moravan- szky, G. State, M. Lu, A. Handa, and D. Fox. Factory: Fast contact for robotic assembly. In RSS, 2022.
[5] T. Chen, M. Tippur, S. Wu, V. Kumar, E. Adelson, and P. Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes. Science Robotics, 2023.
[6] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej. M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welin- der, L. Weng, Q Yuan, W. Zaremba, and L. Zhang. Solving rubik's cube with a robot hand.
11X1v:1910.07111,, 2019.
[7] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, Y. Narang, J.-F. Lafleche, D. Fox, and G. State. Dextreme: Transfer of agile in hand manipulation from simulation to reality. In ICRA, 2023.
[8] F. Krebs and T. Asfour. A bimanual manipulation taxonomy. RA-L, 2022.
[9] C. Smith, Karayiannidis, Nalpantidis, X. Gratal, P. Qi, D. V. Dimarogonas, and D. Kragic. Dual arm manipulation- a survey. RA-L,, 2012.
[10] N. Vahrenkamp, M. Przybylski, T. Asfour, and R. Dillmann. Bimanual grasp planning. In Humanoids, 2011
[11] K. Chatzilygeroudis, B. Fichera, I. Lauzana, F. Bu, K. Yao, F. Khadivar, and A. Billard. Bench- mark for bimanual robotic manipulation of
objects. RA-I, 2020.
[12] N. Sommer, M. Li, and A. Billard. Bimanual compliant tactile exploration for grasping un- known objects. In ICRA, 2014.
[13] R. Platt, A. H. Fagg, and R. A. Grupen. Manipulation Gaits: Sequences of Grasp Control Tasks. In ICRA, 2004.
[14] C. Ott, O. Eiberger,
W. Friedl, B. Bauml, U. Hillenbrand, C. Borst, A. Albu-Schaffer, B. Brun- ner, H. Hirschmuller, S. Kielhofer, R. Konietschke, M. Suppa, T. Wimbock, Zacharias, and G. Hirzinger. humanoid two-arm system for dexterous manipulation. In Humanoids, 2006.
[15] T. Wimbock, C. Ott, and G. Hirzinger. Impedance behaviors for two-handed manipulation: Design and experiments. In ICRA, 2007.
[16] K. Zhang, M. Sharma, M. Veloso, and O. Kroemer. Leveraging multimodal haptic sensory data for robust cutting. In Humanoids, 2019.

---

[17] A. Amice, P. Werner, and R. Tedrake. Certifying bimanual rrt motion plans in second. 10 16603, 2023.
[18] Cohn, S. Shaw, M. Simehowitz, and R. Tedrake. Constrained bimanual planning with ana- lytic inverse kinematics. arXiv:2309.08700 2023.
[19] Z.-Y. Chiu, F. Richter, E. K. Funk, R. K. Orosco, and M. C. Yip. Bimanual regrasping for suture needles using reinforcement learning for rapid motion planning. In ICRA, 2021.
[20] T. Lin, Y. Zhang, Q. Li, H. Qi, B. Yi, S. Levine, and Malik. Learning visuotactile skills with two multifingered hands. arXiv:2404. 16823, 2024.
[21] C. Wang. Shi, W. Wang, R. Zhang, Fei-Fei, and C. K. Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint 2437772403,07788 2024.
[22] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching without in tn-the-wild robots. arXiv preprint WxXv:22402.10329 2024.
[23] L. Shi, A. Sharma, T. Z. Zhao, and C. Finn. Waypoint- imitation learning for robotic manipulation. In CoRL, 2023.
[24] S. Stepputtis, M. Bandari, S. Schaal, and H. B. Amor. A system for imitation learning of contact-rich bimanual manipulation policies. IROS, 2022.
[25] T. Z. Zhao, V. Kumar, Levine, and Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In RSS, 2023.
[26] F. Krebs, A. Meixner, I. Patzer, and T. Asfour. The kit bimanual manipulation dataset. In Humanoids, 2021.
[27] Z. Li, B. Huang, A. Ajoudani, C. Yang, C.-Y. Su, and A. Bicchi. Asymmetric bimanual control of dual-arm exoskeletons for human-cooperative manipulations. Transactions on Robotics, 2017.
[28] M. Laghi, M. Maimeri, M. Marchand, C. Leparoux, M. Catalano, A. Ajoudani, and A. Bicchi. control for intuitive bimanual tele-manipulation In Humanoids, 2018.
[29] H. Fang, H.-S. Fang, Y. Wang, J. Ren, J. Chen, R. Zhang, W. Wang, and C. Lu. Low-cost exoskeletons for learning whole-arm manipulation the wild. arXiv:2309. /4975, 2023.
[30] A. Handa, K. Van Wyk, W. Yang, J. Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox. Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system. In ICRA, 2020.
[31] S. P. Arunachalam, Güzey, S. Chintala, and Pinto. Holo-dex: Teaching dexterity with immersive mixed reality. In ICRA, 2023.
[32] S. P. Arunachalam, S. Silwal, B. Evans, and L. Pinto. Dexterous imitation made easy: A learning-based framework for efficient dexterous manipulation. In ICRA, 2023.
[33] Y. Qin, W. Yang, B. Huang, K. Van Wyk, H. Su, X. Wang, Y.W. Chao, and D. Fox. Anyteleop:
A general vision-based dexterous robot arm-hand teleoperation system. In RSS, 2023.
[34] X. Cheng, Li, S. Yang, G. Yang, and X. Wang. Open-television Teleoperation with immer- sive active visual feedback. arXiv preprint arXiv22407.0151222000 2024.
[35] Jun 2024. URL
10

---

[36] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged rohots. Science Robotics, 2019.
[37] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, 2022.
[38] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots, In RSS, 2021.
[39] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik. In-hand object rotation via rapid motor adaptation. In CoRL. 2022.
[40] G. Khandate, S. Shang, E. T. Chang, T. L. Saidi, J. Adams, and M. Ciocarlie. Sampling-base exploration for reinforcement learning of dexterous manipulation. In RSS, 2023.
[41] J. Pitz, L. Rostel, L. Sievers, and B. Bauml. Dextrous tactile in-hand manipulation using modular reinforcement learning architecture. In ICRA, 2023.
[42] H. Qi, B. Yi, Y. Ma. S. Suresh, M. Lambeta, R. Calandra, and Malik. General in-hand object rotation with vision and touch. In CoRL, 2023.
[43] L. Rostel, Pitz, L. Sievers, and Bauml. Estimator-coupled reinforcement learning for robust purely tactile in-hand manipulation. In Humanoids, 2023.
[44] Z. Yin, B. Huang, Y. Qin, Q. Chen, and X. Wang. Rotating without seeing: Towards in-hand dexterity through touch. In RSS, 2023.
[45] Y. Yuan, H. Che, Y. Qin, B. Huang, Z.-H. Yin, K.-W. Lee, Y. Wu, S.-C. Lim, and X. Wang. Robot synesthesia: In-hand manipulation with visuotactile sensing. In ICRA, 2024.
[46] Y. Chen, Wang, L. Fei-Fei, and C. K. Liu. Sequential dexterity: Chaining dexterous policies for long-horizon manipulation. In CoRL., 2023.
[47] S. Suresh, H. Qi, T. Wu, Т. Fan, L. Pineda, M. Lambeta, J. Malik, M. Kalakrishnan, R. Ca- landra, M. Kaess, Ortiz, and M. Mukadam. Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation. arXiv:2312. 2023.
[48] Y. Lin, A. Church, M. Yang. H. Li, J. Lloyd, D. Zhang, and N. Lepora. Bi-touch: Bimanual tactile manipulation with sim-to-real deep reinforcement learning. RA-L, 2023.
[49] Y. Li, C. Pan, H. Xu, X. Wang, and Y. Wu. Efficient bimanual handover and rearrangement via symmetry-aware actor-critic learning. ICRA, 2023.
[50] S. Kataoka, S. K. S. Ghasemipour, D. Freeman, and Mordatch. Bi-manual manipulation and attachment via sim-to-real reinforcement learning. arXiv:2203.08277 2022.
[51] Y. Chen, T. Wu, S. Wang, X. Feng. Jiang. Z. Lu, S. McAleer, H. Dong, S.-C. Zhu, and Y. Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. In NeurIPS, 2022.
[52] K. Zakka, P. Wu, Smith, N. Gileadi, Howell, X. B. Peng, S. Singh, Tassa, Florence, A. Zeng, al. Robopianist: Dexterous piano playing with deep reinforcement learning. In CoRL, 2023.
[53] B. Huang, Chen, Wang, Y. Qin, Y. Yang, Atanasov, and X. Wang. Dynamic handover: Throw and catch with bimanual hands. In CoRL,, 2023.
[54] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. In ICLR, 2019.
[55] T. Lin and A. Jabri Mimex: Intrinsic rewards from masked input modeling. In NeurIPS, 2023.

---

[56] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv: 177.v.7707.06377, 2017.
[57] I.. Pinto, M. Andrychowicz, Welinder, W. Zaremba, and P. Abbeel. Asymmetric actor critic for image-based robot learning. In RS5, 2018.
[58] A. Kirillov, E. Mintun, N. Ravi, Mao, Rolland, Gustafson, Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo,, P. Dollár, and R. Girshick. Segment anything. In ICCV, 2023.
[59] H. K. Cheng and A. G. Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. ECCV, 2022.
[60] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simula- tion for robot learning. arXiv:2108.1442 2021.
[61] Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. wrXiv:1506.02438, 2015.
[62] D.-A. Clevert, T. Unterthiner, and Hochreiter. Fast and accurate deep network learning by exponential linear units. arXiv:/511.0728) 2015.
12

---

7 Object Details
Simulated Bottles. We use Isaac Gym [60] model the simulated learning environments. For the multi-object environment, we use bottles whose bodies range from 82cm to 86cm in diameter and 55cm to 67cm in height, and whose caps range from 62cm to 70cm in diameter and 20cm to 33em in height. For the single-object environment, we use bottle whose body is 84cm in diameter and 60cm height, and whose cap is 67cm in diameter and 26cm in height.
Real-Wer World Bottles. We show details of our real-world bottle design Figure 6.
Figure 6: Real-world bottle design. Each bottle is consisted of three parts: the cap (top left), the pin (top right), and the body (bottom left). The parts be 3D printed and assembled by inserting the pin into the cap and fixing the cap onto the body. With the pin holding the cap and body in place, the cap can be infinitely twisted about the body. Examples of printed bottles shown in Figure 2(C).
8 World Experiment Details
Camera Calibration. We use a novel marker-based approach to calibrate the extrinsics matrix of our camera. Specifically, we add the marker tag used our real-world setup into the simulation environment, such that pair coordinates of the marker corners be obtained easily in both the camera frame and the world frame (Figure 7). We then use the paired coordinates to solve for camera extrinsics. Doing so greatly reduces the manual labor required by other camera calibration approaches, such as capturing checkerboard images and solving for multiple extrinsic matrices.
Figure 7: Modeling real-world marker tag in simulation for easy camera calibration.
Hardware Communication. To keep control loop running reliably at 10Hz, use ZeroMQ to manage communication between robot hands, camera, and Linux workstation.
13

---

Figure 8: Real-world task initialization. At the beginning of cach task sequence, initialize the robot hands about a canonical position with upward-facing palms. Then, we lightly place an object onto the fingers.
Task Initialization. We illustrate details of how initialize the task sequence in the real world in Figure 8. The canonical joint positions of each finger is documented in Table 2.
Table 2: Initial joint positions of both robot hands.
Finger
Initial Joint Positions
Index
[-0.0080, 0.9478, 0.6420, -0.0330]
Middle
[0.0530, 0.7163, 0.9609, 0.0000] [0.0000, 0.7811, 1, 0.7868, 0.3454]
Ring
Thumb
[1.0670, 1.1670, 0.7500, 0.4500]
Training Details
RL implementation. We use the proximal policy optimization (PPO) algorithm to learn RL poli- cies. We use an advantage clipping coefficient = 0.2; horizon length of 16, with y 0.99, and generalized advantage estimator (GAE) [61] coefficient = 0.95. The policy network is a three- layer MLP with ELU [62] activation, whose hidden layer is [256, 256, 128]. The policy network outputs a Gaussian distribution with a learnable state-independent standard deviation. The value network is also an MLP with ELU activation, whose hidden layer [512, 512, 512]. We use an adaptive learning rate with KL threshold of 0.016 [36]. During training, we normalize the state in- put, value, and advantage. The gradient norm is set 1.0 and the minibatch size is set to 8192. We use asymmetric observation [57] for the policy and value network, adding privileged information to the value network inputs. This privileged information is not accessible by the policy network.
Asymmetric States. In addition to the policy inputs, we provide the following privilege state inputs to the value network of asymmetric PPO: hand joint velocities, all fingertip positions, all contact keypoint positions, object orientation, object velocity, object angular velocity, random forces applied to object, object brake torque, object mass randomization scale, object friction randomization scale, and object shape randomization scale.
Action Hyperparameters. To generate action commands, we clip neural network policy output to [-1, 1] range. We then apply action scale of 0.1 and moving average parameter of 0.75 to the actions.
Reward Hyperparameters. We use a1 2.5, 02 500.0, = 20, 04 -0.001, and os -1.0 as reward weights.
Domain Randomization Setup We apply a wide range of domain randomizations to ensure zero- shot sim-to-real transfer, including both physical and non-physical randomizations. Physical ran- domizations include the randomization of object friction, mass, and scale. We also apply random forces to the object to simulate the physical effects that are not implemented by the simulator. Non- physical randomizations model the noise in observation (e.g. joint position measurement and de- tected object positions) and action. A summary of our randomization attributes and parameters is shown in Table 3.
14

---

Table 3: Domain Randomization Setup.
Object: Mass (kg)
[0.03, 0.1]
Object: Friction
[0.5, 1.5]
Object: Shape
xlA(0.95, 1.05)
Object: Initial Position (cm)
+0(000 0.02)
Object: Initial z-orientation
(70.75,, 0.75)
Hand: Friction
[0.5, 1.5]
PD Controller: Gain
xM((08, 1.1)
PD Controller: D Gain
xlI (0.7, 1.2)
Random Force: Scale
2.0
Random Force: Probability
0.2
Random Force: Decay Coeff. and Interval
0.99 every 0.1s
Bottle Pos Observation: Noise
0.02
Joint Observation Noise.
+. 0.4)
Action Noise.
+0(00,.1)
Frame Lag Probability Action Lag Probability
0.1
0.1
9.1 Generalization Experiment Details
In Table 4, we provide per-object quantitative results and further analysis on the lid-removal success rate mentioned Section 5.4. These results show that overall low lid-removal success rate mostly comes from "hard" objects, i.c. objects that require more turns and/or are more out-of- distribution in shape. We also note that, if we define "number of turns needed to remove lids" of in-distribution objects to be 1, the success rates are consistently 100% for the in-distribution objects.
Table 4: Real-world objects differ greatly in the design of lids. Each object requires different number of turns to remove the lids, and the difficulty of manipulating different object shapes also varies. In this table, objects are sorted by number of turns needed to remove lids (high to low) and then lid-removal success rate (low to high). Note that lid-removal is a novel task rather than the training task.
Object
No. of Turns
Lid- Removal
PeanutButter
5
10
EmptyNutella
5
10
Nutella
5
40
FiberGummies
5
Earplugs
3
20
OilCapsules
40
StressGummics
40
HairMask
60
Overall
33.75
9.2 Generalization to a Vertical Task Setup
To showcase the generalizability of our approach, we train policies with novel vertical setup (i.e. the agent opens lids bottles held vertically). Other than changing the initialization setup (see Figure 9 and Table 5) and turning off the perception system, no change is made to the system components proposed in our work. We additionally note that the horizontal setup in our main text is more challenging than the vertical setup. The vertical setup prevents the most common failure case lack of stabilization and dropping objects by design.
15

---

Figure 9: successful lid-twisting policy in a novel vertical setup. The leftmost image shows task initialization of the vertical setup. The remaining three images show action trajectory of successful lid-twisting policy being deployed. Additional results can be found the video supplementary materials.
Table 5: Initial joint positions of robot hands for vertical task setup.
Finger
Initial Joint Positions
Left: Index
[-0.0080, 0.0772, 1.6655, 0.2697]
Left: Middle
[0.0530. 0.0031, 1.7090, 0.0000]
Left: Ring
[0.0000, -0.0617, 1.5400, 0.3454]
Left: Thumb
[0.6670, 1.1670, 1.0000, 0.8800]
Right: Index
[-0.0080, 0.9478, 0.6420, -0.0330]
Right: Middle
[0.0530, 0.7163, 0.9609, 0,0000)
Right: Ring
[0.0000, 0.7811, 0.7868, 0.3454]
Right: Thumb
[0.6670, 1.1670, 0.7500, 0.4500]
9.3 Additional Details on Domain Randomization
During the process of hyperparameter tuning, note that the highest policy variances are intro- duced by the following parameters: Bottle Position Observation Noise, Joint Observation Noise, Action Noise. This suggests that noise parameters relevant to action space and observation space might be the most important for domain randomization for successful sim-to-real transfer.
16


Reviews:
[{"review_id": "SphFhJv2Mp", "reviewer": "Reviewer_qvJe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method for training a bi-manual anthropomorphic robot to twist lids off containers. The authors train a PPO RL policy using asymmetric actor-critic in simulation. To model the contact dynamics stably, particularly static friction, they use a simplified physical model without threads but with a novel link pushing the lid against the container. In addition, they rely on simple off-the-shelf segmentation models to estimate the noisy positions of the container and the lid. They use reward shaping and domain randomization. The authors show that their simplified model and training techniques successfully transfer to the real world and generalize to unseen geometries. The same policy network can twist lids off different containers with varying degrees of success. For seen container lids, up to 946 degrees (~2.6 turns) and an unseen square bottle’s lid by 43 degrees (refer to table 1). Their policy can remove lids off containers with a 10-60% success rate depending on difficulty, e.g. no. of turns required to remove the lid (refer to table 4 in the appendix).", "summary_of_recommendation": "I think the paper has good convincing results but needs some analyses and clearer presentation of results", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Apart from the weaknesses highlighted above,\n\n1. How was the inference delay discrepancy between sim and real (due to additional state estimation, communication delay, etc) handled? It is known to significantly impact sim-to-real performance https://arxiv.org/pdf/2301.12587\n2. In line 214, “We hypothesize that multi object makes exploring lid-twisting behavior an easier process by introducing an object curriculum 216 that covers both easy and hard object instances during training.” This hypothesis can be easily validated by (1) evaluating intermediate ckpts of multi-object training on objects of varying difficulty (2) training only on the easiest object\n3. In line 143, what are the values of m and n. Is there a special reason to only measure the distance to the nearest point from a set instead of anywhere around a circumference?\n4. What do the authors think would take to train a policy that can indefinitely twist a lid?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "coysHdg45t", "reviewer": "Reviewer_DcnM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Summary: This paper presents a sim2real RL approach to learn bimanual multi-finger control policies for the task of twisting lids of bottles with two hands.", "summary_of_recommendation": "Limited novelty, but good contribution that shows sim2real can work well for bi-hand dexterity.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- The observation space seems too impoverished to lead to a robust policy. The bottle+lids are distilled into 3D positions of the center of mass losing significant information about the object shape.  Why not build on point cloud or implicit representations? \n\n- It would seem that lid removal and determining number of twists needed would require some force feedback. If a compliant controller is running under the hood, one may need to adjust the gains based on friction. Why is the lid removal success low and how can it be improved.\n\n- The paper uses highly engineered rewards. Is the performance sensitive to its choice?\n\n- Angular Displacement (AD) - how is this measured accurately in practice on the real objects if there is slippage?\n\n- Baselines feel more like ablations, but there is hardly any comparable work on a task like this. If so, please clarify in related works section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "niaWYyRsKS", "reviewer": "Reviewer_qU69", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses the challenge of learning cooperative behaviour for two robot hands to twist the lid off containers. The authors develop a novel coarse-grain simulation to model the complex interaction between the bottle and lid. Their approach achieves zero-shot transfer, and the learned policies generalise to objects with varying physical attributes that were not seen during simulation without an adaptation phase. The coordination of fingers, a particularly challenging task, is effectively managed through deep reinforcement learning combined with sim-to-real transfer, showcasing dynamic and dexterous behaviours across diverse unseen objects.", "summary_of_recommendation": "The paper presents a well-executed study with clear experiments, innovative simulation techniques, successful zero-shot sim-to-real transfer, and robust real-world application validated through thorough ablation studies. I recommend accepting the work.", "technical_quality": 5, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- While the results and the videos are impressive, from the videos, it appears that in many containers (e.g., the Nutella 01:39, StressGummies 01:45), the lid is quite loose even before the twist starts. Maybe one way to mitigate that perception from a casual observer is to simply turn the bottle upside down (e.g., through manual control) before the twisting action starts to show that twisting is essential to open and it wouldn’t open with random jerks of the hands.\n\n- In line 31: you mention “articulated objects defined as two rigid bodies connected via a revolute joint with a threaded structure.” In the Isaac Gym simulation, do you have a threaded structure or do you assume that the prismatic joint A-C in Fig 2A can mimic the interaction of the components at the thread?\n\n- Have you tested (or have arguments against) having a firmer fixed grasp on the base-bottle hand and the RL restricted to the other hand? Wouldn’t that result in a stable outcome, similar to how a human would perform the same manoeuvre?\n\n- Line 60: “Classical …. ”, A word seems to be missing in the beginning.\n\n- Line 175: “To test how to enable natural and robust manipulation behavior emerge”: Grammatical error in line.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Pre-rebuttal**

This paper receives three positive reviews. Overall the reviewers appreciate the clarity of the presentation and the impressive zero-shot sim-to-real transfer.

Mentioned strengths:
- Relevance to humanoid (**DcnM**).
- Innovation of the simulation technique for the task (**qU69**).
- Thoroughness of ablation studies and model evaluation (**qU69**, **qvJe**).

Mentioned weaknesses:
- Limited algorithmic novelty (**DcnM**).
- Soundness of the TFF metric (**DcnM**, **qvJe**).
- Task remains largely unsolved (i.e. <10% removal rate on unseen objects) even with a fairly narrow task focus (**DcnM**).
- Lid is always lose at the start; unclear whether it can work on fully closed real bottle (**qU69**).
- Presentation of experiments and results (**qvJe**).

---
**Post-rebuttal**

This paper initially received three weak accepts. All the reviewers remain positive after the rebuttal.

Overall the paper has delivered a solid simulation and training framework for an interesting task and demonstrated impressive sim-to-real transfer. AC agrees with the assessments of the reviewers and thereby recommends accept.
Decision: Accept