Lessons from Learning to Spin “Pens”

Authors: jun  wang
ying  yuan
haichuan  che
haozhi  qi
yi  ma
jitendra  malik
xiaolong  wang
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-03 06:19:09

Abstract: In-hand manipulation of pen-like objects is a most basic and important skill in our daily lives, as many tools such as hammers and screwdrivers are similarly shaped. However, current learning-based methods struggle with this task due to a lack of high-quality demonstrations and the significant gap between simulation and the real world. In this work, we push the boundaries of learning-based in-hand manipulation systems by demonstrating the capability to spin pen-like objects. We use reinforcement learning to train a policy and generate a high-fidelity trajectory dataset in simulation. This serves two purposes: 1) pre-training a sensorimotor policy in simulation; 2) conducting open-loop trajectory replay in the real world. We then fine-tune the sensorimotor policy using these real-world trajectories to adapt to the real world. With less than 50 trajectories, our policy learns to rotate more than ten pen-like objects with different physical properties for multiple revolutions. We present a comprehensive analysis of our design choices and share the lessons learned during development. Videos are shown on https://corl-2024-dexpen.github.io/.

Main paper: 
 introduction: 
 Dexterous in-hand manipulation is a foundational skill for various downstream manipulation tasks. For example, one often needs to reorient a tool in hand before using it. Despite decades of active research in this area [1, 2, 3, 4] , in-hand manipulation remains a significant challenge. Manipulating pen-like objects, in particular, is considered one of the most challenging and crucial tasks [5, 6] . This capability is highly practical, as many tools, such as hammers and screwdrivers, have similar shapes. Moreover, spinning pen-like objects requires dynamic balancing and sophisticated finger coordination, making it an ideal testbed for advancing dexterous manipulation systems.Pen spinning has been studied from several perspectives. Classic robotics works demonstrate rotating wooden blocks with open-loop force control [1] . With high-speed cameras and advanced hardware, agile pen spinning can also be achieved [6] . However, these methods rely on accurate object models and cannot generalize to unseen objects. On the other hand, learning-based methods hold the promise of being generalizable with large-scale data. They have indeed achieved significant progress either with imitation learning [7, 8, 9] or sim-to-real [3, 10, 11, 12] . However, they have only demonstrated manipulation of regular spherical or cuboid-shaped objects, and none can extend the capability to pen-like objects. We attribute this to two reasons: For teleoperation-imitation pipeline, current teleoperation systems fail at collecting complex and dynamic demonstrations; for sim-to-real, bridging the gap for dynamic tasks becomes substantially difficult.In this work, we push the boundaries of learning-based in-hand manipulation systems by demonstrating their capability to spin pen-like objects. Similar to previous approaches [10, 11, 13] , we first learn an oracle policy with privileged information using reinforcement learning in simulation. However, when attempting to distill it into a sensorimotor policy, we find the sim-to-real gap too large. While this gap generally exists in previous in-hand manipulation tasks [11, 13] , the extreme difficulty of spinning pen-like objects exposes the gap even further. Fine-tuning the policy with realworld trajectories can be one way to mitigate this gap, but it is challenging to collect demonstrations via teleoperation for this dynamic task. Inspired by recent analysis on open-loop controllers [14, 15] , we instead collect a high-fidelity trajectory dataset in simulation and use it as an open-loop controller on the real robot. The successful trajectories in the real world serve as our high-quality demonstrations. We then bridge the sim-to-real gap by fine-tuning our sensorimotor policy with these real-world trajectories. With simulation pre-training, our sensorimotor policy has the motion prior from diverse data and can adapt to real-world physics with fewer than 50 trajectories.We conduct comprehensive experiments in both simulation and the real world. In simulation, we identify the key factors that enable the oracle policy to learn the challenging pen-spinning task and generate realistic trajectories. We then evaluate different methods of obtaining a deployable policy in the real world. We also conduct ablation experiments showing the importance of pre-training in simulation. We demonstrate that our policy can adapt to real-world physics with fewer than 50 realworld trajectories. To the best of our knowledge, this is the first learning-based system to achieve continuous spinning of pen-like objects in the real world. 
related work: 
 Classic in-hand manipulation. In-hand manipulation has been studied for decades [2, 16] . Classical methods rely on an accurate model and analytically plan a sequence of motions to control the object. For example, Han and Trinkle [17] manipulate objects using sliding, rolling, and finger gaiting motions, while Bai and Liu [18] studies the collaboration of fingers and the palm. Mordatch et al. [19] demonstrates object rotation in simulation by trajectories optimization. Li et al. [20] learns a object-level impedance controller for both grasping and rotating objects. Open-loop manipulation also shows surprising robustness and dexterous behavior [14, 15] . Sieler and Brock [21] uses linearized feedback-control for in-hand manipulation with a soft hand. State-of-the-art systems in this category include full SO(3) reorientation using a compliance-enabled hand [22] and an accurate pose tracker [23] . However, most methods cannot manipulate pen-like objects due to their complex and dynamic nature. Extrinsic dexterity [24] can also be used to achieve dynamic manipulation, but a precise model is necessary. In contrast, our method uses human priors to build a simulator environment but does not rely on an accurate model during deployment.Learning-based dexterous manipulation. Learning-based methods make fewer assumptions and hold the promise of being more generalizable as we acquire more data. Recently, significant progress has been made in this field [3, 4] . The advancement mainly comes from two sources: 1) lowcost and accessible teleoperation systems [7, 8, 25, 26, 27, 28, 29, 30] We first train an oracle policy in simulation using reinforcement learning. This policy provides high-quality trajectory and action datasets. We use this dataset to train a student policy and as an open-loop controller in the real world to collect successful real-world trajectories. Finally, we fine-tune the student policy using this real-world dataset.learning [31, 32] ; and 2) reinforcement learning in simulation [33, 34] combined with sim-toreal [10, 11, 12, 34] . However, both methods have limitations. Current teleoperation cannot support agile and dynamic tasks such as spinning pens, due to the non-negligible communication latency and retargeting errors. On the other hand, sim-to-real approaches demonstrate great generalization and robustness by training policies in randomized environments. They show success in multiple fields such as in-hand manipulation [13, 35, 36, 37, 38, 39, 40, 41] , grasping [42, 43, 44] , long-horizon tasks [45] , and bimanual dexterity [46, 47] . However, the gap between simulation and reality is quite large, and some results are only limited to simulation [48, 49, 50] . Our paper distinguishes itself from all previous work by leveraging the advantages of both fields. We use reinforcement learning in simulation to obtain high-quality demonstrations and use real-world trajectories to bridge the sim-to-real gap.Our work is also related to several recent works on combining real-world and simulation data. Torne et al. [51] and Wang et al. [52] augment real-world human demonstrations by creating a simulated environment, showing this is helpful for policy robustness. Jiang et al. [53] demonstrates that simto-real policies can adapt to real-world complex dynamics with only a few human demonstrations.Our approach also adapts policies trained in simulation to the real world using demonstrations and utilizes simulation data to make the policy more generalizable and robust. However, since our task is more challenging, it is difficult to collect human demonstrations or provide human feedback. Therefore, we need to generate high-fidelity trajectories by learning a policy in simulation.Pen spinning. The specific problem of pen spinning has also been studied extensively due to its challenging nature and practical implications in the real world. Fearing [1] shows an open-loop force control strategy can achieve robust finger gaiting for manipulating a long wooden block. Ishihara et al. [6] and Nakatani and Yamakawa [5] demonstrate high-speed pen spinning using a highspeed robot hand and camera. In the machine learning community, Charlesworth and Montana [54] demonstrate promising results with RL and trajectory optimization. Ma et al. [55] uses a language model for reward design. However, the results are limited to simulation. Bringing simulation results to the real world is a substantially harder task. There are works that involve learning to manipulate long objects using real-world reinforcement learning [56] or augmented with imitation [57] , but it can only do less than half a circle and no finger gaiting. In contrast, we achieve continuous pen spinning using a learning-based approach and commercially available hardware. 
learning to spin pens: 
 An overview of our method is shown in Figure 2 . Our method consists of three steps. First, we train an oracle policy with privileged information to generate realistic trajectories in simulation. With these trajectories, we pre-train a sensorimotor policy in simulation. We then use these trajectories as an open-loop controller to generate demonstrations in the real world, which is used to fine-tune the sensorimotor policy to adapt it to the real-world dynamics.Obtaining high-quality data for pen spinning is itself a challenging task due to the dynamic and complex movements involved. The current teleoperation system is not suitable due to the nonnegligible latency and imperfect retargeting error between the human hand and the robot hand. Alternatively, previous work shows that reinforcement learning can synthesize complex behaviors in simulation [54, 55] . These methods achieve fast and dynamic behavior but may violate real-world physics and hardware constraints. In contrast, we design our approach to generate high-quality trajectories that are realistic enough for use as an open-loop controller in the real world. This is achieved by properly designing the input space, reward function, and initial state distributions.Observations. The observation o t of the oracle policy f is a combination of the following quantities: joint positions q t , previous joint position target a t-1 , binary tactile signals c t , fingertip positions p t , the pen's current pose and angular velocity w t , and a point cloud of the pen at the current state ∈ R 100×3 . To obtain fine-grained tactile responses, we augment the sensor arrangement in [12] to include five binary sensors on each fingertip (see Figure 5 ). The point cloud is obtained by transforming points on the original mesh based on the current ground-truth object pose. We encode the point cloud using PointNet [58] as in [35, 59, 60] . We stack three historical states of joint positions and targets as inputs. We also include physical properties such as mass, center of mass, coefficient of friction, and object size in the input [11] . The dimensions of the inputs are detailed in the appendix.Actions. At each step, the action provided by the policy network f (o t ) is a relative target position.The position command a t = ηf (o t ) + a t-1 , where η is the action scale, is sent to the robot and it will be converted to torque via a low-level PD controller.Reward. The goal of the policy is to continuously rotate the pen around the z-axis. Our reward is defined as a combination of rotation reward and a few energy penalty terms. The reward and penalty terms follow [11, 12] . However, stable gaits do not emerge solely from this. Motivated by [54] , we propose another reward r z , a penalty regarding the height difference between the highest and the lowest points on the pen, encouraging the robot hand to keep the pen horizontal during rotation.In summary, our reward function is (t omitted for simplicity): r = r rot +λ z r z +λ energy r energy , where r rot rewards the pen's rotation velocity and r energy penalizes the object's linear velocity, deviation from initial joint positions, mechanical work, and torque applied (see appendix for details). Initial state design. Our task fundamentally differs from previous work where the object is placed on the palm [12, 13] , a table [10] , or fingertip by gravity [11] , where there is natural support in those cases. Therefore, using randomly sampled poses does not provide meaningful exploration in our case. We find that a proper design of the initial state distribution is critical for policy training. Designing initial states for pen rotation is non-trivial because the initial grasp should be stable enough to facilitate learning subsequent steps of motion. Moreover, exploration can be slow if we repeatedly use the same initial state upon reset. Thus, inspired by human behavior, we manually design multiple patterns of grasping that may occur in the cycle of pen rotation (visualized in Figure 3 ), and then add noise to generate and filter for a set of stable initial states.Policy optimization. We use proximal policy optimization (PPO) [61] to train the oracle policy.Given the state information, we use a Multi-Layer Perceptron (MLP) for both the policy and value networks. We apply domain randomization to perception inputs, physical parameters, object prop- erties, etc. An episode terminates when reset conditions are met or the agent reaches the maximum number of steps T . We prune unnecessary explorations when the pen falls below a height threshold.The oracle policy mentioned above can learn smooth and dynamic behavior during simulation training. However, it cannot be deployed because it requires privileged information as input, which is not accessible in the real world. Previous works typically distill the oracle policy into a sensorimotor policy using DAgger [62] . However, we find this approach does not work well for our pen-spinning task. We experimented with either proprioception [11] or adding visuotactile feedback [13, 35] .While the policy with visuotactile feedback can learn reasonable behavior in simulation, the mismatch between simulation and reality is too large for these two modalities. On the other hand, proprioceptive feedback is the most similar and reliable sensing method between simulation and the real world, but the proprioceptive policy cannot converge even in simulation and always drops the object in the first few steps.For this reason, we propose an alternative approach: we roll out the oracle policy f in simulation, in contrast to previous work using DAgger and rolling out the sensorimotor policy [11, 35] , and collect a dataset of proprioception and actions (s t , a t ). This dataset is used to pre-train a proprioceptive policy in simulation. The goal of this step is to expose the sensorimotor policy to diverse training data. Although training with such data cannot enable direct transfer to the real world due to inaccurate dynamics, it can provide a motion prior, allowing the policy to be efficiently fine-tuned with real-world trajectories.Following [11] , our proprioceptive policy takes 30 steps of joint positions q t-29:t and previous joint targets a t-30:t-1 as input. We use a temporal transformer similar to the one used in [35] to model sequential features and an MLP for the policy network. Such pre-training allows our proprioceptive policy to experience a wider range of circumstances, preventing overfitting to specific trajectories.Due to the large sim-to-real gap of our task, we choose to use real-world trajectories to fine-tune the pre-trained sensorimotor policy to adapt to real-world dynamics. However, obtaining real-world trajectories is challenging. Our key observation is that although the oracle policy cannot be directly distilled and zero-shot transferred to the real world, it does provide motion sequences that are difficult to generate using teleoperation. Inspired by recent work that highlights the effectiveness of open-loop controllers for in-hand manipulation [14, 15] , we use the trajectories generated by the oracle policy as an open-loop controller in the real world.Specifically, after training the oracle policy f , we test it in the simulation environment with different initial poses. We select 15 trajectories from different initial poses that last longer than 800 timesteps. We record these actions and replay them on the real robot with three training objects (Figure 6 ). For each replay, we randomly select one of the objects more than 2π in this trial, we store this trajectory in the dataset. We repeat this process until we collect 15 trajectories per object (45 trajectories in total).Using the learned policy to generate such trajectories has two benefits: First, it naturally provides smoothness driven by our reward definition; Second, compared to alternative approaches such as learning from human videos, it provides trajectory data with actions. We use this dataset to fine-tune our proprioceptive policy π to make it adapt to real-world dynamics. Because the proprioceptive policy has already been pre-trained in diverse simulation environments, it can adapt to the real world with fewer than 50 trajectories. 
experiments: 
 In this section, we compare our approach for pen spinning to several baselines in both simulation and the real world. Specifically, we study 1) the critical design choices in obtaining an oracle policy that can be replayed in the real world; 2) various techniques for sim-to-real deployment. Evaluation metrics. In our simulation experiments, we evaluate the Cumulative Rotation Reward and Duration (seconds) [11, 12, 13 ]. In the real world, we measure the radians of rotation (RR.) over the z-axis and the success rate (Suc.). We define success as the rate at which the policy can rotate target objects at least 180 degrees, which typically corresponds to the policy completing one circle of finger gaiting, where each finger completes a break and re-establishes contact.The goal of the oracle policy is to generate realistic trajectories that can be used both for pre-training the student policy and serving as an open-loop controller in the real world. We compare several critical factors in achieving this, specifically: 1) without a well-designed initial pose distribution; 2) without privileged information; 3) without r z .Q1: How does the initial state distribution help policy training? We study the effect of a welldesigned initial state distribution. The results are shown in Figure 4 We find that although the No Pretrain baseline improves as we increase the number of demonstrations from 15 to 75, it still performs worse than our method, especially on unseen objects. This indicates that training with much more diverse data in simulation is beneficial and can also avoid overfitting to certain objects.samples states around one canonical hand pose, as used in [11, 35] . In contrast, our method defines multiple canonical hand poses inspired by how humans spin pens and achieves better performance compared to using a single canonical pose. We also emphasize that although the curve for single canonical init does increase over time, the finger gaiting cannot emerge, and this policy cannot escape from the local minima. We visualize the behavior in Figure 7 (c) in our appendix and find the finger does not break contact with the object and fails to achieve more than one revolution.Q2: How does privileged information help policy training? We study the importance of privileged information in Figure 4 right. Unlike [11] , the oracle policy cannot be trained only with simple object properties such as object position. We find that without tactile feedback or a point cloud, the policy does not achieve good enough performance. The shape of the pen is important as the policy needs to know when to lift the fingers to spin the pen. Privileged information such as the object's physical properties and finger positions is also critical, without which the policy does not converge.Q3: How does z-reward help policy training? We study the effect of z-reward r z , shown in Figure 7 (b) in our appendix. Although the trajectories look similar to our approach at first glance, the object gets tilted at configurations in the third and sixth sub-figure. This can barely succeed in real-world replay. In contrast, policies trained with the z-reward rotate the pen more stably, keeping the pen approximately horizontal, which facilitates data collection in real-world replay.Although the oracle policy achieves great performance in simulation, it cannot be directly deployed in the real world. To address this issue, we use it as an open-loop controller to collect real-world trajectories. We also pre-train a proprioceptive policy in simulation and fine-tune it using this dataset. We compare our method with several alternatives in the real world. The results are shown in Table 1 .Q4: Is oracle replay a good enough controller? We design our oracle policy so that it achieves decent performance in the real world (Oracle Replay). However, it still performs worse than our method. On Training Objects A/B/C, our method achieves 15%-30% better performance in terms of success rate. On Unseen Objects D/E/F, which are considered out-of-distribution, our method achieves a 10% increase in the radius rotated, despite having a similar success rate. Our method also achieves 15%-30% success rate improvements on objects I/J/K. This result demonstrates that our method generally achieves a longer radius rotated compared to the oracle replay because it is also pre-trained in simulation with more diverse data.Q5: Does distillation work for pen spinning? Previous approaches demonstrate promising results by distilling the oracle policy into the sensorimotor policy [11, 35, 13] using DAgger. However, this approach does not work for our dynamic and contact-rich task (Figure 1 ). First, we try to use segmented depth [35] or two endpoints of the pen, and the visuotactile policy can achieve reasonable performance in simulation. However, the sim-to-real gap is significantly larger compared to previous works. In our real-world deployment, the objects oscillate a lot, making the image distribution far removed from the training one. Secondly, proprioceptive feedback does not have this problem, but using proprioception alone does not achieve good performance in simulation.Q6: How do pre-training and fine-tuning contribute to the final performance? Our approach is first pre-trained in simulation and then fine-tuned using real-world data. We study the contribution of each part in Table 2 . With only pre-training, the policy has limited effectiveness in the real world. It rarely completes finger gaiting on Objects C and D, and the success rate is also low for the remaining objects. This is mainly because the physics gap between simulation and reality becomes more significant in our task. With only behavior cloning, the also does not perform well.For this experiment, we use ACT [63] as the architecture, which is one of the best imitation learning algorithms. It has a 50% lower rotation radius on Training Objects. On out-of-distribution objects, the success rate drops to less than 20%, indicating that its generalization capability is limited.Q7: Can simulation pre-training be replaced by more demonstrations? We also study whether increasing the number of real-world demonstrations can substitute for the advantages gained from pre-training in simulation. The results are shown in Table 3 . We find that although the performance of the No Pretraining baseline can be improved with more demonstrations, it gradually saturates when increasing the number of demonstrations from 45 to 75. In addition, the major improvements come from the training objects (A/B/C), while the performance on unseen objects (D/E/F) is still far worse compared to our methods. This indicates that solely relying on real-world trajectories is likely to overfit to certain objects.In addition to the objects we present in the quantitative study, we also try more different objects for our policy and try to push the limits on objects that are significantly out-of-distribution. Examples are shown in our Figure 1 and our project website. 
conclusion and lessons: 
 In this paper, we present the first learning-based approach for spinning pen-like objects. Through our extensive experiments, we share the lessons we learned as follows:• Simulation training requires extensive design for exploration, such as the proper design of initial distributions to aid exploration and using privileged information to facilitate policy learning. • Sim-to-Real does not directly work for such contact-rich and highly dynamic tasks. Even when isolating touch and vision, the pure physics sim-to-real gap remains significant and cannot be bridged by extensive domain randomization alone. • Simulation is still useful for exploring skills. The dynamic skill of spinning pens with a robotic hand is nearly impossible to achieve with human teleoperation and imitation learning alone. Reinforcement learning in simulation is critical for exploring feasible motion. • Only a few real-world trajectories are needed for fine-tuning. Although a proprioceptive policy learned purely in simulation does not work directly in the real world, it can be fine-tuned to adapt to real-world physics using only a few successful trajectories.Limitations. We have identified several key bottlenecks of using vision and touch during sim-toreal for this dynamic task. However, we are not stating they should not be used. Humans do not seem to need vision to spin a pen, but touch feedback seems important. In future work, we will explore whether using them can help further improve performance. Currently, the system is only capable of rotating along z-axis, it is also a promising direction to extend it to general multi-axis rotation. In addition, our work assumes the object is placed at a stable grasp position following previous work [11, 10] . Incorporating more advanced grasping work [64] or consider chaining different skills [45] together would make our work more general.  
 
 Appendix: In Q4 of our experiment section, the initial hand / object configurations are chosen from the initial configurations of the replay trajectory dataset. This setting brings advantage to the oracle replay baseline. To comprehensively study the performance, we conduct new real-world experiments where the objects are initialized at a randomly chosen stable grasp. We choose 10 random grasps, do 5 trials for each. For each random grasp, we generate the oracle replay trajectory by running the oracle policy in the simulator and select the best trajectory among 1000 simulated environments. In this setting, our method achieves 22.0% (from 54% to 78.0%) better success rate on object D, 36% (from 46 to 82) on object E, and 40% (from 34% to 74%) on object F. The reason that our policy does not drop compared to the number if Table 1 is because it has seen much more diverse data via simulation pre-training.Hardware setup. We use the Allegro Hand for our hardware experiments. The Allegro Hand has four fingers, each with 4 degrees of freedom. Our neural network outputs the joint position target at 20 Hz, which is sent to a low-level PD controller operating at 333 Hz.Simulation setup. We use Isaac Gym [34] for our simulation training. To obtain additional tactile feedback for oracle policy training, we simulate 20 tactile sensors around the fingertips, with 5 on each fingertip. We gather the contact signal from each sensor and binarize the measurement based on a pre-defined threshold [12, 13] . In simulation, the control frequency is 20 Hz and the simulation frequency is 200 Hz. Our reward function is a combination of r rot , r z and r energy . The energy reward consists of r vel , r diff , r ang , r torq , and r work . Here, r vel penalizes the pen's linear velocity, r diff discourages the hand's pose from deviating much from its initial pose, r ang penalizes the pen's angular velocity above a pre-defined threshold to encourage stable rotation, r torq penalizes large torques, and r work penalizes the work of the controller. We follow the same definition of reward in [35] . We combine the above rewards with weights listed in Table 4 .We detail the dimensions of the inputs of our oracle policy in Table 5 . We train our oracle policy with PPO, and the training hyper-parameters are shown in Table 7 . Specifically, we train with 8192 parallel environments. Each environment gathers # steps data to train in each epoch of PPO. The data is split into # minibatches and optimized with PPO loss. γ and λ are used for computing generalized advantage estimate (GAE) returns. We use the Adam optimizer to train PPO and adopt the gradient clip to stabilize training. We train 500 million agent steps in total, which takes less than one day on a single GPU. We train our student policy with Behavior Cloning, and the training hyper-parameters are shown in Table 6 . We collect approximately 50M steps of data in total. The domain randomization parameters are listed in 8.We tune the P and D gain in the low level controller according to the following two metrics: 1) Before we do data collection, we first replaying several finger gait trajectory without objects inhand. We do this for both sim and real and compare the errors between joint positions. We try to 


Decision: The submission analyzes sim2real transfer for manipulating pen like objects.

The submission tackles an ambitious, contact-rich dexterous manipulation problem and includes a thorough literature review. It shows real robot results and provides detailed analysis. It proposes a novel strategy of augmenting dynamics data. 

However, insights on the design of the real data augmentation strategy still need to be improved and some result presentations could be clearer. Overall, the performance, while better than baselines, is still underwhelming and stronger baseline comparisons would be helpful. A key point lies in using open-loop replay on hardware, which might not work for other tasks and could be more broadly explored. Partial simplification arises from the stable initial state being manually specified.

The paper has improved during the rebuttal via detailed communication between reviewers and authors even though not all points have been addressed there has been a shift in reviewer evaluation. Based on the above the paper is recommended for acceptance. Please follow up on the remaining open points for a potential camera ready version.

Reviews: review: title: title: Interesting, if cursory, study of a 'grand challenge' in dexterous manipulation
paper_summary: paper_summary: Simulation-based pre-training and real-world fine-tuning enable sim-to-real transfer for challenging in-hand manipulation.
main_review: main_review: **Summary:** The authors consider the problem of “pen spinning,” a long-challenging task in dexterous manipulation. In particular, the authors present a “systems” result, showing how to combine existing results (with solid design decisions) to enable robust hardware performance.

In particular, the authors train a ‘teacher’ policy with privileged information in simulation, run policies open-loop in real world, and then fine-tune on real-world data. They show that the “oracle” simulation policy can be adapted to real-world settings in <50 trajectories, which I regard to be remarkable data efficiency (and impressive hardware results).

**Strengths:**

- Considers an ambitious, contact-rich dexterous manipulation problem.
- Very thorough literature review that provides an excellent perspective on historical and current approaches to dexterous in-hand manipulation.
- Strong hardware results showing the proposed pretraining + fine-tuning approach is plausible for this problem.
- Defines a good problem for dexterous manipulation (pen-spinning), although still relatively quasistatic compared to the “typical” definition of “pen spinning”.

**Limitations:**

- Performance is better than baselines but still underwhelming qualitatively. Work still needs to be done to get the dynamic, robust motion the authors are imagining in their introduction.
- Baseline comparisons could be stronger, especially w/r/t imitation learning. In particular, strong comparisons would be vs. diffusion policies with imitation of teleoperated demonstrations. I acknowledge this approach has numerous severe challenges (dynamic demo collection is extremely difficult with current telop pipelines for dexterous hands) but think this comparison would strengthen the paper and the "story" motivating an RL approach here overall (especially since the experimental results, as the authors also acknowledge, are relatively underwhelming). 

-----

**Summary after rebuttal phase:**
I largely stand by my original review (aside from some original confusion, now deleted, re: the RL architecture used here). I think the other reviewers are correct to pick at the heavy use of open-loop replay in hardware, and think the paper suffers a lack of clarity around these topics. Overall, however, I view this paper as mostly a *negative result* / temperature check for how well current techniques can perform on a difficult task. I would argue our community needs more of these results (compared to / in addition to minor advancements on less ambitious tasks) and think this clears the bar for CoRL.
questions: questions: - The authors draw a clear contrast between their approach and imitation learning throughout, they do not actually baseline against IL results like diffusion policies. Does the proposed method outperform pure imitation in hardware (which is difficult to collect, but suffers no sim-to-real gap)?
- What are some limitations of the current method? Or, put differently, what would be required to get a robot hand to spin pens dynamically like humans can do?
limitations: limitations: 1
review_summary: review_summary: Weak accept; 'systems' papers are under-appreciated in our community + this is a great study of how exiting techniques work for a 'grand challenge' in manipulation.
title: title: Good study of pen spinning but limited contribution
paper_summary: paper_summary: This paper presents a system study on dexterous hand rotating pens. The overall approach involves oracle simulation training, distillation, open-loop replay in real, and another fine-tuning in real. The different components of the pipeline are ablated in the experiments to showcase their importance.
main_review: main_review: Strength:

1. The paper is well written and most of the technical details are presented clearly in the main text.

2. The figures are nicely done, especially Fig. 1 and Fig. 4. I pick up the idea very quickly from them.

3. I appreciate the thoroughness of the experiment section, carefully ablating the different components of the pipeline.

Limitation:

1. I think the overall contribution is too limited. The use of distillation and real-world fine-tuning is well studied.

2. The main novelty, to me, lies in using open-loop replay on hardware. But meanwhile, I also find it very surprising that this kind of open-loop action trajectories from simulation can work quite well in real, e.g., 78.21% success rate with Object D (Table 1). To me, this means the task is not that difficult, and the sim-to-real gap is not big. Maybe the authors can clarify this a bit.

3. Also, the overall improvement from the oracle baseline is not very impressive in Table 1. I think with multi-stage training the success rates should aim above 80% for the different objects. if there is some fundamental limit, please clarify.
questions: questions: It would be good to mention how the policy is parameterized, e.g., is it a Gaussian distribution, what is the noise scale for the distribution, and whether the noise scale is fixed or learned.
limitations: limitations: 2
review_summary: review_summary: I think the paper does a thorough study on pen spinning using mostly existing methods, but I am concerned with the limited overall contribution.
title: title: Review
paper_summary: paper_summary: This work introduces a method for dexterous pen spinning tasks. Prior approaches struggle due to poor human demonstrations and sim-to-real gaps. Using reinforcement learning, the authors train a policy in simulation and generate high-quality trajectory data. This data is used for pre-training and real-world trajectory replay, requiring fewer than 50 real-world examples to fine-tune the policy. The result is a system that can successfully rotate various pen-like objects with different properties.
main_review: main_review: Strengths:
1. “Spin a pen with a multi-finger robot hand” is one of the most challenging tasks for dexterous manipulation. Although the task is simplified, seeing the system working in the real world and can work on multiple objects is refreshing.
2. This work introduces a novel teacher-student training pipeline, where the demonstrations are replayed on a real-world robot and the successful trajectories are used to fine-tune the policy learned with simulation data. This idea is neat and brings two key benefits: 1) it bridges the sim-to-real gap by leveraging a few real-world, automatically collected data when zero-shot transfer does not work, and 2) it greatly improves sample efficiency.

Questions:
1. Seems like most pen-shaped objects have similar thickness and are evenly weighted along the entire shape. The objects are assumed to be held in the middle at the beginning of the task. The robot motions seem like a specific type of periodic replay. I hope the authors can provide more details on the differences between the training and novel objects, and explain what truly makes generalization difficult here.
2. In the introduction, the authors mention the importance of “dynamic balancing” the pen-like object during spinning. However, there is no object state information input to the policy, which makes me feel the policy does not learn balancing at all. If that is the case, the application of “pen spinning” in this work would be very similar to (1) but only rotate along the z-axis (and without any sensing). The technical method this work uses to help learn dynamic balancing is unclear to me.
3. What would happen if sim2real does not work in the early stages and all the collected trajectories fail? Is the method based on the assumption that a functioning sim2real policy is achievable at the beginning?
4. This work assumes that the action distribution and the low-level PD controller are the same in both simulation and real-world settings, which is not true in most cases. This might also cause most real-world replays to fail, resulting in no successful trajectories being gathered. I hope the authors can introduce more details about aligning action and controller between simulation and real-world settings, and how to guarantee successful trajectories during open-loop replay.
5. The stable initial state is manually specified, which seems to be a limitation for serving as a general approach in other manipulation tasks. Some prior works like (2) and (3) have studied the relation between the way of grasping and following manipulation tasks. It would be nice to add some discussion in the future work section about potential arm+hand motion with both grasping and spinning. This is also a limitation of this work.

(1). General In-Hand Object Rotation with Vision and Touch

(2). Dexterous functional grasping

(3). Sequential dexterity: Chaining dexterous policies for long-horizon manipulation
questions: questions: Same as the limitations mentioned above.
limitations: limitations: 2
review_summary: review_summary: Overall, this work introduces a novel method for sim-to-real dexterous manipulation with good real-world results. However, the assumptions of the controller and task setup need more clarification. Therefore, I start with weak reject and might adjust based on the authors response.
title: title: Interesting results and insights for a long-lasting dexterous manipulation problem
paper_summary: paper_summary: The paper explores sim-to-real designs with an augmentation of real world data to solve in-hand spinning pen-shaped objects, with a success rate of 50-80% on the real-hardware and the capability of generalizing to unseen objects.
main_review: main_review: The paper targets the challenging problem of spinning a pen in-hand. The main contribution lies in an exploration of many design choices in learning the simulated policy and a strategy of blending the real data that are hard to acquire through teleoperation due to the task dynamic. Many insights on the privileged state and distillation design are discussed. The importance of resetting environment to critical frames in successful execution and terminating failing trials are highlighted. The interesting findings on using proprioception and open loop trajectories from high-quality simulated episodes might be inspirational for addressing other tasks. The real-hardware evaluation looks solid , though some observations are not fully addressed. 

Strengths:

- Impressive real robot results on a challenging dexterous manipulation problem.

- Well documented and organised insights on the design consideration, which are mostly reasonable.

- Interesting strategy of augmenting real dynamics data which might inspire other hard-to-demonstrate tasks.

Weakness:

- Limited insights on the design of the real data augmentation strategy.

- Some result presentation can be clearer.

- Minor: the literature review misses the works on rotating objects with trajectory optimisation through contact (a) and model-free compliant control (b).

(a) Mordatch et al., Contact-invariant optimization for hand manipulation, SCA 2012
(b) Li et al., Learning object-level impedance control for robust grasping and dexterous manipulation, ICRA 2014
questions: questions: - Any example to show what typical failure modes the real data help to correct? The realized rotating radians on average are still less than $2\pi$, what are blocking a spinning lasting longer?

- How many successful examples we can typically obtain from executing the 15 open-loop trajectories? Are they just the numbers under oracle reply in Table 1?

- Why the oracle replay appears to work better on out-of-distribution objects D/E/F while much poorer for in-distribution objects? Why the improvement from real data augmentation is more significant for in-distribution objects while marginal for D/E/F?

- How large is the discrepancy between in-distribution and out-of-distribution objects in terms of mass distribution, geometry and surface materials?

-  The augmentation seems to rely on a few seeding sim-to-real trials which might be a bit lucky-based. What if we could not obtain sufficient successful real trials?

- Would more exhaustive domain randomization increase the chance of getting successful real replays or even lead to zero-shot sim2real? What specific physical parameters are randomized here? Why not considering randomizing the actuator parameters or communication delay given the task is time-critical?

- Z-reward design seems vital to pruning out invalid explorations. Will a curriculum learning, e.g. with gradually increased gravity, help on efficient exploration?

- The ultimate hardware control is 20 Hz while both simulated control and engine step operate at 200 Hz. How is the simulated control aligned to the real setup? Is it just receiving proprioception less frequently?

-  Since the simulation is designed to reset at predefined contact modes, will a motion planning algorithm with this predefined constraint switching as a mode-search skeleton be more efficient to generate simulation trajectories?

- The ultimate motion looks conservative: it looks more like carefully supporting the pen with tips and nudging it with tiny motions, instead of with more significant spinning motion with more phalange contacts as humans. Will such a behavior harder to learn given the current simulation and hardware setup?
limitations: limitations: 3
review_summary: review_summary: The paper still has room to improve on its analysis about the method design and experimental results. However, a contribution for such a challenging task on real-hardware and novel objects should already place it above the acceptance threshold. Edit: the authors' rebuttal helps to address some of the limitations. The flaws on lacking systematic and well-grounded conclusions persist. However, I believe the paper has made visible contribution that deserves discussion at the conference.