Twisting Lids Off with Two Hands

Authors: toru  lin
zhao-heng  yin
haozhi  qi
pieter  abbeel
jitendra  malik
allegro  hands
rgbd  camera
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-02 02:20:05

Abstract: Manipulating objects with two multi-fingered hands has been a long-standing challenge in robotics, due to the contact-rich nature of many manipulation tasks and the complexity inherent in coordinating a high-dimensional bimanual system. In this work, we share novel insights into physical modeling, real-time perception, and reward design that enable policies trained in simulation using deep reinforcement learning (RL) to be effectively and efficiently transferred to the real world. Specifically, we consider the problem of twisting lids of various bottle-like objects with two hands, demonstrating policies with generalization capabilities across a diverse set of unseen objects as well as dynamic and dexterous behaviors. To the best of our knowledge, this is the first sim-to-real RL system that enables such capabilities on bimanual multi-fingered hands.

Main paper: 
 introduction: 
 Achieving dexterous bimanual manipulation with two anthropomorphic robot hands has been exceptionally challenging, due to the inherently contact-rich nature of many manipulation tasks and the complexity of coordinating a high-dimensional bimanual system. This work takes a step towards this grand goal, by demonstrating the feasibility of learning a highly dexterous and dynamic bimanual manipulation policy purely in simulation and zero-shot transferring it to the real world. Specifically, we study the task of twisting or removing lids with two multi-fingered robot hands. This task is both practically important and profoundly interesting: for one, the ability to twist or remove lids from containers is a crucial motor skill that toddlers acquire during their early developmental stages [1, 2] ; for another, the manipulation skills required for this task, such as the coordination of fingers to manipulate a multi-part object, can be generally useful across a large collection of practical tasks.Since collecting human expert demonstration data to solve contact-rich tasks via imitation learning is highly challenging and expensive [3] , we aim at training a generalizable policy through sim-to-real reinforcement learning without using any expert data (Figure 1 ). Our method does not require precise modeling of any individual object, or hardcoding prior knowledge on object properties; instead, stable and natural bimanual finger behaviors emerge through large-scale reinforcement learning (RL) training. Below, we share the novel insights that enable us to develop such a system. Physical Modeling. Our work features a novel class of objects for in-hand manipulation: articulated objects defined as two rigid bodies connected via a revolute joint with a threaded structure. Accurately modeling friction and contact with revolute joints and threaded structures has traditionally been a hard challenge in robotic simulation [4] . To address this, we introduce a brake-based design to model the interaction between the lid and body of bottle-like objects. This design is fast to simulate while maintaining high fidelity to real-world physical dynamics, enabling efficient policy learning and successful sim-to-real transfer.Perception. We initially hypothesize that a fine-grained, contact-rich manipulation task like lidtwisting must require precise perceptual information on object states and shapes. To our surprise, a two-point sparse object representation, extracted from off-the-shelf object segmentation and tracking tools, is sufficient to solve the perception problem. With simple domain randomization techniques, we train policies that are robust against occlusion and camera noise. This discovery suggests that a minimal amount of perception information can be adequate for complicated bimanual manipulation tasks.Reward Design. Previously performant reward designs for tasks like in-hand reorientation [5, 6, 7] cannot be straightforwardly applied to our task, since those tasks focus on manipulating single-part rigid bodies with one hand rather than multi-part articulated bodies with two hands. Solving this task is more challenging since it involves more complex and precise contact (e.g. using two hands to hold a lid). In addressing this challenge, we discover a simple keypoint-based contact reward that yields natural lid-twisting behavior on the robot fingers.We conduct several controlled experiments in both simulation and the real world. Through empirical analysis, we verify that our simulation modeling, perception module, and reward design can reliably lead to the desired behavior of lid-twisting. Our final successful policy manifests natural behavior across test objects with various physical properties such as shapes, sizes, and masses in simulation. Moreover, the learned policy can be zero-shot transferred to a wide range of novel household objects whose lids can be removed (Figure 1 ), and it is robust against perturbations. 
background: 
 For decades, bimanual manipulation has remained an unsolved challenge in robotics [8, 9, 10, 11, 12, 13] . While multi-fingered robot hands seem to be a natural choice for bimanual robot systems in theory, designing controllers for high-dimensional action spaces remains an open problem. Classical approach has made significant progress but usually assume known object or physics information [14, 15] and the generalizability remains unknown. In recent years, bimanual manipulation has been actively studied with learning-based methods, as a result of progress in learning algorithms and compute infrastructure. These learning-based approaches can be categorized into two types: 1) learning from real-world data; 2) learning in simulation, then transferring to the real world (sim-toreal).Learning from Real-World Data. Rapid progress has been made in RL in the real world. Zhang et al. [16] learn to chain motor primitives for vegetable cutting, with relatively simple motion primitives; much of the task difficulty is bypassed via the use of specialized end-effectors [17, 18] . Chiu et al. [19] learn precise needle manipulation with two grippers by integrating RL with a samplingbased planner. While impressive, these works cannot easily scale to higher dimensional action space due to their sample inefficiency or the need to define heuristic-based action primitives.Most recent successes in bimanual manipulation are achieved by learning from demonstrations [20, 21, 22, 23, 24, 25] . However, successes so far are largely limited to simple end-effectors like parallel jaw grippers due to the lack of high-quality demonstration data from multi-fingered robot hands [26] .Although several works aiming to improve demonstration data collection with two arms [27, 28, 29] or multi-fingered hands [21, 30, 31, 32, 33] , their latency and retargeting errors limit their practical applicability and scalability. Lin et al. [20] proposes a scalable hands-arms teleoperation system that learns smooth bimanual policies, but the system compromises on dexterity. Similar systems that offer more dexterous control [34, 35] , on the other hand, suffer from drawbacks ranging from jittery control to high costs. Our method uses RL in simulation and is thus not limited by the hardware and data collection infrastructure problems faced by learning from demonstration approaches.Sim-to-Real. There has been growing interest in sim-to-real approaches for robotics -i.e. learning policies in simulation and transferring them to the real world -stimulated by several notable successes in recent years ranging from locomotion [36, 37, 38] to manipulation [5, 7, 6, 39] . Existing works in manipulation, however, are mostly done with either a single multi-fingered hand [3, 40, 41, 42, 43, 44, 45, 46, 47] , or two arms with simpler end-effectors [48, 49, 50] . While Chen et al. [51] and Zakka et al. [52] feature bimanual tasks with dexterous hands, only simulation results are shown. The work most related to ours is Huang et al. [53] , where the authors demonstrate throwing and catching objects using two dexterous hands. However, our task is significantly more contact-rich and requires substantially more challenging bimanual coordination to maintain object stability at all times. To our best knowledge, there is no learning-based method directly comparable to ours on the proposed task. 
learning to twist lids with two hands: 
 We focus on the challenging task of lid-twisting for container objects, since it is a complex in-hand manipulation process that requires dynamic dexterity of multiple fingers and precise coordination between two hands. The goal of this task is to twist the lid about the object's axis of rotation in one direction as much as possible; during this process, the object should always stay in hand. Achieving this involves a sequence of delicate movements: 1) after initialization, the robot hand should firmly grasp and slightly rotate the bottle to a suitable pose; 2) the hand that is closer to the object lid should place its finger around the lid to initiate twisting motion; 3) the two hands should coordinate to avoid dropping the object while one hand twists the lid. Motor skills that arise from this task could serve as generic abstractions for skills necessary to manipulate many other household objects, especially those with revolute joints such as Rubik's cubes, light bulbs, and jars.A central challenge in simulating the lid-twisting task is how to model friction between the bottle body and the lid properly, particularly static friction. Simulating this type of physical force has been a long-standing problem in robotics and graphics [4] . We design a simple modeling approximation that strikes a balance between fidelity and speed during physical simulation; our bottle-like object model is illustrated in Figure 2 (A). Our design features a special Brake Link that constantly presses against the bottle lid via a prismatic joint. This artificially generates frictional forces between the bottle body (Base Link) and the lid (Lid Link), preventing relative rotation between them -similar to a bottle with its lid screwed on. We replicate these bottles in the real world, as shown in Figure 2 (B).We note that naively tuning static friction properties between two revolute-joint-connected bodies is not realistic enough with our simulator. Such a brake-based is the only way we find that can simulate the static friction well.To better benchmark the bimanual twisting capability, we consider a class of articulated bottles with lids that can be twisted infinitely (see Figure 2 (A) and Section 3.1 for more details). Each object of interest consists of two rigid, near-cylindrical parts (a "body" and a "lid"); the two parts are connected via a continuous revolute joint, allowing them to rotate about each other. At the beginning of each episode, the two robotic hands are initialized in a static pose with upward-facing palms, and a bottle-like object is gently dropped or placed onto the fingers. The initial pose of the object is randomized both in translation and rotation to a fixed default pose; the initial joint positions of the hands are randomized about a canonical pose by adding Gaussian noise. Note that since we do not assume a stable grasp configuration at task initialization, the control policy needs to learn in-grasp reorientation to place the object in a stable location to perform successive manipulation.Bimanual in-hand dexterous manipulation involves highly complex hand-object contacts, and remains challenging to solve with traditional methods. In this work, we address the control challenge through RL. We formulate our control problem as a partially observable Markov Decision Process.Observation Space. At each time step t, the control policy observes the following information from the environment: the proprioceptive hand joint positions q t , the estimated center-of-mass 3D positions of the bottle base and lid, and previously commanded target joint positions qt .Action Space. We use a PD controller to drive the robot hand. The control policy produces a relative target joint position as the action a t , which is added to the current target joint position q t to produce the next target: qt+1 = qt + ηEMA(a t ). η is a scaling factor. Note that we smooth the action with its exponential moving average (EMA) to produce smooth motion. The next target position is sent to the PD controller to generate torque on each joint.Reward. While one way to approach hard exploration problems is to add intrinsic rewards [54, 55] , we introduce the following fine-grained reward terms to shape the hand behavior (Figure 3 right).(1). Twisting Reward. We define the twisting reward as r twisting = ∆θ = q t+1 bottle -q t bottle which is the rotation angle of the lid during one-step execution. This reward term encourages the hand to twist the lid. (2) . Finger Contact Reward. We find it crucial to use a set of reference contact points to guide effective contact between fingertips and the bottle. We define two set of points X L ∈ R n×3 and X R ∈ R m×3 attached on the bottle base and lid respectively. Then, we define the finger contact reward asr contact = i 1 1+αd(X L ,F L i ) + 1 1+αd(X R ,F R i ), where F L ∈ R 4×3 and F R ∈ R 4×3 are the position of left and right fingertips, α is a scaling hyperparameter, and d is a distance function defined as d(A, x) = min i ∥A i -x∥ 2 . Therefore, we require each fingertip to stay as close to one of the reference contact points as possible. As we will see later, this term is necessary for eliciting desired behavior and task success. (3) . Pose Reward. We also introduce a pose matching reward term to encourage the bottle main axis x axis aligned with a predefined direction v. This term is defined as r pose = -arccos(⟨x axis , v⟩). (4). Regularizations. Besides the three task-specific rewards, we also introduce another few regularization terms as in previous works [44] , including work penalty and action penalty to penalize large, jerky motions. We leave the details of the definition to the appendix.Reset Strategy. There exist many possible hand-object interaction modes. Among these, most modes lead to failures such as getting the object stuck between fingers, and exploring those modes rarely provides good learning signals. To circumvent the high dimensionality of our exploration problem, we introduce two early termination criteria. First, we reset an episode if the robot hands fail to rotate the bottle into a desired pose for bimanual twisting within a short time limit. Additionally, we reset when the bottle's z-position is below a certain threshold, as the fingertips of the two hands can pinch the bottle at a low position without being able to reposition it into the palm.Training. We use PPO [56] with asymmetric critic observation [57] to train our policy, and introduce various domain randomizations to make the policy transferable to the real world. We apply both both physical and non-physical randomizations. The detailed training setup can be found in the appendix.Figure 3 shows an overview of our real-world perception pipeline. To make our RL policy more transferable, we use bottle and lid center points instead of pixels as vision input. We extract these keypoints from real-time images through object segmentation and tracking in the real world. Specif- as well as a smooth, natural, and human-like twisting motion. Middle: A naive gait constraint reward without any contact hints leads to erratic finger motion and unnatural grasps. Bottom: A reduced contact reward yields somewhat natural behavior, but the grasp is loose compared to the full contact reward case. Right: Perturbing a learned policy with random external force. Our policy is resilient to these external forces and able to recover. ically, we utilize the Segment Anything model [58] to generate two separate masks for the bottle body and the lid on the first frame of each trajectory sequence, and XMem [59] to track the masks throughout all remaining frames. To approximate the 3D center-of-mass coordinates of the bottle body and lid, we calculate the center position of their masks in the image plane, then obtain noisy depth readings from a depth camera to recover a corresponding 3D position. The perception pipeline runs at 10 Hz to match the neural network policy's control frequency. 
simulation experiments: 
 To test how to enable the emergence of natural and robust manipulation behaviors, we first conduct several experiments in simulation. Specifically, we study the following questions: How important is the keypoint-based reward for eliciting desired twisting behavior in bimanual manipulation? How important is visual information for solving this task? Is a sparse keypoint representation enough for learning a generalizable policy?Object Set. In simulated experiments, we use a collection of simulated cylindrical bottles with varying aspect ratios for both training and evaluation. Some samples are visualized in Figure 2(B) . We consider two setups in simulation: 1) multi-object, in which all the objects are used, and 2) single-object, in which a single medium-sized bottle that represents the mean of the dataset is used.Evaluation Metric. We introduce the following metrics for evaluating the performance: 1) Angular Displacement (AD) is the total number of degrees through which the lid has been twisted; 2) Timeto-Fail (TTF) is the period measured from the moment the bottle is held to the point when it either slips from the hand or becomes lodged; 3) Velocity (Vel) is AD divided by TTF, reflecting the speed of twisting motion.Baselines. We compare our policy with the following baselines. 1) Policy without Vision. This is a neural network policy without object state information. We use this to evaluate the importance of vision. 2) Policy with Reduced Contact Reward. In training this policy, we reduce the intensity of our proposed finger contact reward. We use this to study the role of our contact reward in policy learning and shaping the policy's behavior. 3) Policy with Gait Constraint Reward. In training this policy, we replace our contact reward with a gait constraint reward function similar to ones used for in-hand reorientation tasks [39] . This baseline is only used for qualitative analysis.Reward Design. We first compare our approach with the reduced finger reward baseline (Figure 4 ). After decreasing the scale of finger contact reward, learned policies fail to master the desired lid-twisting skill and have low performance in general. We hypothesize that this is because the motion of lid-twisting requires a very specific pose pattern for holding the object; without explicitly encouraging such a pose pattern (e.g., via its contact modes), RL exploration becomes so hard that it is unsolvable within the available training time. We also observe a positive correlation between the intensity of finger contact reward and both 1) sample efficiency during learning and 2) performance of learned policies (as reflected by Figure 4 and qualitative observations in Figure 5 (left)).Vision vs. No Vision. We also study the importance of vision modality. Existing works show that certain rotation behaviors can be achieved through implicit tactile sensing (via proprioception) [39] . However, our empirical results show that, in both single and multi-object setups, the no-vision baseline performs substantially worse than our full method. This suggests that knowledge of the position of bottle keypoints is essential for successful lid-twisting.Single Object vs Multi Object. We run RL training with two object settings: 1) using a single bottle-like object; 2) using multiple bottle-like objects with more variation in the ratio between the bottle base and lid. For results shown in Figure 4 , all multi-object training runs are evaluated on multi-object setup and all single-object training runs are evaluated on single-object setup. The two settings pose a trade-off between specialization and generalization: in the single-object scenario, the policy might learn successful behaviors more easily but find it harder to generalize to unseen objects, and vice versa. To our surprise, we observe that multi-object training yields slightly better performance compared to single-object training. We hypothesize that multi-object makes exploring lid-twisting behavior an easier process by introducing an object curriculum that covers both easy and hard object instances during training.5 Real-world Experiments 
experiment setup: 
 Hardware Setup. We use two 16-DoF Allegro Hands from Wonik Robotics for our experiments. Each Allegro Hand is mounted on a fixed UR5e arm. We employ a single RealSense D435 depth camera to provide visual information, from which we extract object state information. We send control commands to the robot at a frequency of 10 Hz via a Linux workstation.Object Set. For quantitative evaluation, we evaluate the sim-to-real transfer capabilities of our policies on five different articulated bottle objects (Figure 2 (C)). Among them, four are in-distribution (round-body bottles) and one is outside of the training distribution (square-body bottle).Evaluation Metric. We measure both AD and TTF in 20 trials, with each trial lasting for a maximum of 30 seconds. For each evaluated method, we select the three best policies out of ten policies trained on ten different random seeds. We end a trial if the bottle falls off the palm.Baselines. We compare our final policy with the following baselines to study the effect of several key design choices. 1) Open-loop Replay Policy (Replay). We record successful trials of our learned policy in the simulation and randomly select a trajectory to replay on the real robot. This baseline is used to evaluate whether the task can be solved by a deterministic motion pattern. 2) Policy without Vision (No-Vis). This baseline policy only takes proprioceptive state information as input, without information about the object state. 3) Policy without Asymmetric Training (No-Asym). We compare with a baseline where policy is trained without asymmetric PPO, and evaluate whether introducing additional privielged information into the value network will affect the transfer performance. 4)Larger Policy Network Size (Large). We increase the size of our actor-network and train a large-size policy. We use this to evaluate whether over-parameterization harms policy performance.We show quantitative results comparing our policy with baseline policies in Table 1 . For both metrics, our policy outperforms all baselines across all evaluated objects. Our method can perform stable grasp on all the objects, and can rotate 3 out of 5 objects at a reasonable speed. In particular, for the blue bottle, one of the deployed policies can achieve 4 full turns (360 degrees) in 30 seconds on average. In contrast, almost all the baselines fail to achieve any effective rotations, either getting stuck or dropping the bottle to the ground. We find that the open-loop policy has the lowest TTF score. Replaying a successful trajectory will not lead to a stable grasp for most of the time, and the bottle will directly roll on the fingers and then drop off the palm. This suggests that the considered task involves very fine-grained contacts and requires the policy to act very precisely according to the object state. Another interesting observation is that the large policy does not transfer to the real world, although we confirm that it can achieve similar performance to our full policy in simulation. This suggests that some overfitting occurs, and controlling the size of the policy network is very important for the successful sim-to-real transfer of our considered contact-rich task.Finally, we also evaluate our policy's robustness against force perturbation. Specifically, we perturb the object during deployment at random times by poking or pushing it along random directions using a picker tool (see the right of Figure 5 ). We find that our policy can reorient and move the object back to a stable pose for continuous manipulation, indicating that it has some robustness against external forces and can adapt to these unexpected changes. Note that we use a marker-based object detection system in this experiment to disentangle the visual occlusion effect.In the above section, we mainly study whether the twisting behavior can naturally emerge and be transferred to the real world. Next, we explore the limit of our approach by testing it on 10 novel household objects (Figure 2 (D)). These objects differ substantially from our training objects in terms of shape, size, mass, material, color, and mechanical design. While the lids of the synthetic bottles that we use for both simulation training and real-world testing can be twisted infinitely, the lids of these household objects cannot. To evaluate our policy's ability to generalize the lid-twisting skill to these novel objects, we use the success rate on a novel yet adjacent lid-removal task as the criterion.We define lid-removal as the object's lid being completely detached from the object body (e.g., when the lids fall from the robot's hands in Figure 1 .We find that our policy continues to achieve stable and natural twisting behaviors on these novel objects. Furthermore, while we only train the policy for lid-twisting, we also find our policy capable of removing lids. For HairMask and FiberGummies, our policy showcases lid-removal rates of more than 50%. For particularly challenging objects that require many turns to remove the lid, such as PeanutButter and EmptyNutella, our policy only achieves 10% lid-removal rates; however, its twisting behavior is steady and robust to perturbation (see our video supplementary for visualization). 
conclusion: 
 We present an RL-based sim-to-real system for twisting or removing lids of bottle-like objects with two hands. We propose several techniques to handle the challenges that arise: a novel reward design, a sparse object representation for real-time perception, and an efficient yet high-fidelity method to simulate twisting bottle caps. We conduct experiments in both simulation and real world to demonstrate the effectiveness of our approach. Our real-world results show generalization across a wide range of seen and unseen objects.7 Object Details Simulated Bottles. We use Isaac Gym [60] to model the simulated learning environments. For the multi-object environment, we use bottles whose bodies range from 82cm to 86cm in diameter and 55cm to 67cm in height, and whose caps range from 62cm to 70cm in diameter and 20cm to 33cm in height. For the single-object environment, we use a bottle whose body is 84cm in diameter and 60cm in height, and whose cap is 67cm in diameter and 26cm in height.Real-World Bottles. We show details of our real-world bottle design in Figure 6 . 8 Real-World Experiment Details Camera Calibration. We use a novel marker-based approach to calibrate the extrinsics matrix of our camera. Specifically, we add the marker tag used in our real-world setup into the simulation environment, such that pair coordinates of the marker corners can be obtained easily in both the camera frame and the world frame (Figure 7 ). We then use the paired coordinates to solve for camera extrinsics. Doing so greatly reduces the manual labor required by other camera calibration approaches, such as capturing checkerboard images and solving for multiple extrinsic matrices. Task Initialization. We illustrate details of how we initialize the task sequence in the real world in Figure 8 . The canonical joint positions of each finger is documented in Table 2 . 9 Training Details RL implementation. We use the proximal policy optimization (PPO) algorithm to learn RL policies. We use an advantage clipping coefficient ϵ = 0.2; a horizon length of 16, with γ = 0.99, and generalized advantage estimator (GAE) [61] coefficient τ = 0.95. The policy network is a threelayer MLP with ELU [62] activation, whose hidden layer is [256, 256, 128] . The policy network outputs a Gaussian distribution with a learnable state-independent standard deviation. The value network is also an MLP with ELU activation, whose hidden layer is [512, 512, 512] . We use an adaptive learning rate with KL threshold of 0.016 [36] . During training, we normalize the state input, value, and advantage. The gradient norm is set to 1.0 and the minibatch size is set to 8192. We use asymmetric observation [57] for the policy and value network, adding privileged information to the value network inputs. This privileged information is not accessible by the policy network.Asymmetric States. In addition to the policy inputs, we provide the following privilege state inputs to the value network of asymmetric PPO: hand joint velocities, all fingertip positions, all contact keypoint positions, object orientation, object velocity, object angular velocity, random forces applied to object, object brake torque, object mass randomization scale, object friction randomization scale, and object shape randomization scale.Action Hyperparameters. To generate action commands, we clip neural network policy output to [-1, 1] range. We then apply an action scale of 0.1 and a moving average parameter of 0.75 to the actions.Reward Hyperparameters. We use α 1 = 2.5, α 2 = 500.0, α 3 = 20, α 4 = -0.001, and α 5 = -1.0 as reward weights. 
additional details on domain randomization: 
 During the process of hyperparameter tuning, we note that the highest policy variances are introduced by the following parameters: Bottle Position Observation Noise, Joint Observation Noise, Action Noise. This suggests that noise parameters relevant to action space and observation space might be the most important for domain randomization for successful sim-to-real transfer. 
: 
 Figure 1 : We train two anthropomorphic robot hands to twist (off) lids of various articulated objects. The control policy is first trained in simulation with deep reinforcement learning, then zero-shot transferred to a real-world setup. We show that a single policy trained to manipulate simplistic, simulated bottle-like objects can generalize to real-world objects that have drastically different physical properties (e.g. shape, size, color, material, mass). The length, diameter (or diagonal length), and mass of each object are annotated at the bottom of individual subfigures. More results can be found in our video and and project website.Abstract: Manipulating objects with two multi-fingered hands has been a longstanding challenge in robotics, due to the contact-rich nature of many manipulation tasks and the complexity inherent in coordinating a high-dimensional bimanual system. In this work, we share novel insights into physical modeling, real-time perception, and reward design that enable policies trained in simulation using deep reinforcement learning (RL) to be effectively and efficiently transferred to the real world. Specifically, we consider the problem of twisting lids of various bottle-like objects with two hands, demonstrating policies with generalization capabilities across a diverse set of unseen objects as well as dynamic and dexterous behaviors. To the best of our knowledge, this is the first sim-to-real RL system that enables such capabilities on bimanual multi-fingered hands. 
 
 Appendix: We apply a wide range of domain randomizations to ensure zeroshot sim-to-real transfer, including both physical and non-physical randomizations. Physical randomizations include the randomization of object friction, mass, and scale. We also apply random forces to the object to simulate the physical effects that are not implemented by the simulator. Nonphysical randomizations model the noise in observation (e.g. joint position measurement and detected object positions) and action. A summary of our randomization attributes and parameters is shown in Table 3 .  


Decision: **Pre-rebuttal**

This paper receives three positive reviews. Overall the reviewers appreciate the clarity of the presentation and the impressive zero-shot sim-to-real transfer.

Mentioned strengths:
- Relevance to humanoid (**DcnM**).
- Innovation of the simulation technique for the task (**qU69**).
- Thoroughness of ablation studies and model evaluation (**qU69**, **qvJe**).

Mentioned weaknesses:
- Limited algorithmic novelty (**DcnM**).
- Soundness of the TFF metric (**DcnM**, **qvJe**).
- Task remains largely unsolved (i.e. <10% removal rate on unseen objects) even with a fairly narrow task focus (**DcnM**).
- Lid is always lose at the start; unclear whether it can work on fully closed real bottle (**qU69**).
- Presentation of experiments and results (**qvJe**).

---
**Post-rebuttal**

This paper initially received three weak accepts. All the reviewers remain positive after the rebuttal.

Overall the paper has delivered a solid simulation and training framework for an interesting task and demonstrated impressive sim-to-real transfer. AC agrees with the assessments of the reviewers and thereby recommends accept.

Reviews: review: title: title: Good sim to real robot results
paper_summary: paper_summary: This paper presents a method for training a bi-manual anthropomorphic robot to twist lids off containers. The authors train a PPO RL policy using asymmetric actor-critic in simulation. To model the contact dynamics stably, particularly static friction, they use a simplified physical model without threads but with a novel link pushing the lid against the container. In addition, they rely on simple off-the-shelf segmentation models to estimate the noisy positions of the container and the lid. They use reward shaping and domain randomization. The authors show that their simplified model and training techniques successfully transfer to the real world and generalize to unseen geometries. The same policy network can twist lids off different containers with varying degrees of success. For seen container lids, up to 946 degrees (~2.6 turns) and an unseen square bottle’s lid by 43 degrees (refer to table 1). Their policy can remove lids off containers with a 10-60% success rate depending on difficulty, e.g. no. of turns required to remove the lid (refer to table 4 in the appendix).
main_review: main_review: **Strength**:

1. The paper had good real robot results showing successful sim-to-real transfer
2. Works on many different containers and generalizes to unseen ones
3. For the most part, the paper is up-front about the scope of results which I appreciate. The title rightly says “Twisting Lids (0ff)” instead of just “Twisting Lids” since the policy cannot put a lid on a container or tighten a lid
4. Extensively tested across training seeds - Experiments with 10 seeds, of which the top 3 are evaluated on a real robot

**Weakness**:

I like the results in the paper. All weaknesses mentioned below are in my understanding addressable and are meant to improve the quality of the paper.

Sim experiments need a clearer representation

1. A single table should show scores of ablation, just like Table 1 shows real robot results. Ideally, the same table should compare sim and real scores so we have a clear picture in one place. 
2. Fig. 4 is difficult to interpret. AD and TFF are interleaved and it’s difficult to go back and forth between the figures and understand what worked best. Could we have multiple line plots on the same figure so it’s easier to compare? Also feel TFF plots are not necessary. Removing them will reduce confusion and create space for the table mentioned above.
3. Line 213 says multi-object makes exploring easier. Does it improve the score even if the evaluation is on a single object? Both Fig. 4 and the text should make it clear what the training and evaluations are on. For a fair comparison, for multi-object training, single-object evaluation scores need to be presented.

Evaluation criteria:

1. TFF (Time-to-Fail): A higher value is claimed to be better. However, a slow policy could do the same or less no. of turns and yet achieve a higher TFF. Table 1 shows such an example where for the Blue Bottle, No-Asym achieves lower AD but higher TFF than the proposed approach. Yet, the score of the proposed approach is highlighted since AD is higher. I don’t see the value in the TFF metric given its limitations. I think a better metric would be AD/TFF which will indicate how fast the rotations are.

Novelty claim:

1. Line 267 claims twisting lids off to be a novel task/criterion which I don’t feel is right. The task is still to turn the lid, which falls off once it’s turned. Is there a strong reason to call it a novel?

Miscellaneous:
1. Line 237 is confusing “Larger Policy Network Size (Large). We increase the size of our actor-network and train a reduced size policy”. Actor network and policy are the same thing. Also, how much larger is it?
2. There exist teleoperation systems for smooth bimanual operation for anthropomorphic hands, e.g. teleoperation on shadow hand https://youtu.be/J-3CcyF4Emw?si=HZ1mkT79rM1pDutB. I think this should be referenced around line 77
3. In Fig. 1 it would be good to add the simulated scenario as well
4. Typos in line 59. Should it be “Classical methods” instead of “Classical”?
questions: questions: Apart from the weaknesses highlighted above,

1. How was the inference delay discrepancy between sim and real (due to additional state estimation, communication delay, etc) handled? It is known to significantly impact sim-to-real performance https://arxiv.org/pdf/2301.12587
2. In line 214, “We hypothesize that multi object makes exploring lid-twisting behavior an easier process by introducing an object curriculum 216 that covers both easy and hard object instances during training.” This hypothesis can be easily validated by (1) evaluating intermediate ckpts of multi-object training on objects of varying difficulty (2) training only on the easiest object
3. In line 143, what are the values of m and n. Is there a special reason to only measure the distance to the nearest point from a set instead of anywhere around a circumference?
4. What do the authors think would take to train a policy that can indefinitely twist a lid?
limitations: limitations: 3
review_summary: review_summary: I think the paper has good convincing results but needs some analyses and clearer presentation of results
title: title: Bimanual multi-finger control
paper_summary: paper_summary: Summary: This paper presents a sim2real RL approach to learn bimanual multi-finger control policies for the task of twisting lids of bottles with two hands.
main_review: main_review: Strengths:

- the paper is fairly well written and clear in most respects. The problem is well motivated, and the Sim2RL approach seems natural and straightforward

- with increasing interest in humanoid like form factors, bimanual multi-finger control is critical for success. The paper provides an interesting case study in the power of sim2real approaches when paired with simple perception heuristics and reward design. 

- zero-shot sim2real transfer is an encouraging finding for biarm dexterity. 

Weaknesses

- Algorithmic novelty is somewhat low paired with a fairly narrow task focus

- the rewards are hand-engineered, a textbook RL approach is used. Preliminarily, the value of this work is in demonstrating that high-dof dexterous tasks are still within the reach of current methods provided the physical modeling and sim environment is carefully prepared for learning. 

- Task success is measured by soft metrics like lid twisting degrees and time-to-fall. When benchmarked on a harder metric like lid removal rates, the paper reports 50% on seen bottles and 10% on more challenging instances. This would suggest that there is a long way to go.
questions: questions: - The observation space seems too impoverished to lead to a robust policy. The bottle+lids are distilled into 3D positions of the center of mass losing significant information about the object shape.  Why not build on point cloud or implicit representations? 

- It would seem that lid removal and determining number of twists needed would require some force feedback. If a compliant controller is running under the hood, one may need to adjust the gains based on friction. Why is the lid removal success low and how can it be improved.

- The paper uses highly engineered rewards. Is the performance sensitive to its choice?

- Angular Displacement (AD) - how is this measured accurately in practice on the real objects if there is slippage?

- Baselines feel more like ablations, but there is hardly any comparable work on a task like this. If so, please clarify in related works section.
limitations: limitations: 3
review_summary: review_summary: Limited novelty, but good contribution that shows sim2real can work well for bi-hand dexterity.
title: title: 
paper_summary: paper_summary: This paper addresses the challenge of learning cooperative behaviour for two robot hands to twist the lid off containers. The authors develop a novel coarse-grain simulation to model the complex interaction between the bottle and lid. Their approach achieves zero-shot transfer, and the learned policies generalise to objects with varying physical attributes that were not seen during simulation without an adaptation phase. The coordination of fingers, a particularly challenging task, is effectively managed through deep reinforcement learning combined with sim-to-real transfer, showcasing dynamic and dexterous behaviours across diverse unseen objects.
main_review: main_review: ### Strength:

- The paper is well-written, with clear and detailed experimental setups.
- The novel coarse-grained simulation effectively mimics twistable lids by combining revolute and prismatic joints, ingeniously simulating frictional forces.
- The authors achieve zero-shot sim-to-real transfer without requiring an adaptation phase.
- The work demonstrates a robust real-world implementation to problems with variable physical properties using two Allegro Hands.
- The reward engineering is elegant, incorporating a twist reward, finger contact reward, and pose reward, with comprehensive ablation studies to validate their contributions.

### Weakness:

- While the authors successfully demonstrate the bi-manual twist motion, it is not clear from the videos if the motion is sufficient to overcome the resistance at the thread in a fully closed real bottle.
- One of the drawbacks of using a prismatic joint to mimic friction is that it is hard to train the agent to hold the lid after it has come off. That probably explains the videos where the lids fall off to the floor after the twisting motion is complete. However, I acknowledge that is not part of the problem definition.
questions: questions: - While the results and the videos are impressive, from the videos, it appears that in many containers (e.g., the Nutella 01:39, StressGummies 01:45), the lid is quite loose even before the twist starts. Maybe one way to mitigate that perception from a casual observer is to simply turn the bottle upside down (e.g., through manual control) before the twisting action starts to show that twisting is essential to open and it wouldn’t open with random jerks of the hands.

- In line 31: you mention “articulated objects defined as two rigid bodies connected via a revolute joint with a threaded structure.” In the Isaac Gym simulation, do you have a threaded structure or do you assume that the prismatic joint A-C in Fig 2A can mimic the interaction of the components at the thread?

- Have you tested (or have arguments against) having a firmer fixed grasp on the base-bottle hand and the RL restricted to the other hand? Wouldn’t that result in a stable outcome, similar to how a human would perform the same manoeuvre?

- Line 60: “Classical …. ”, A word seems to be missing in the beginning.

- Line 175: “To test how to enable natural and robust manipulation behavior emerge”: Grammatical error in line.
limitations: limitations: 3
review_summary: review_summary: The paper presents a well-executed study with clear experiments, innovative simulation techniques, successful zero-shot sim-to-real transfer, and robust real-world application validated through thorough ablation studies. I recommend accepting the work.