AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch

Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 19:39:55

Abstract: Human hands are capable of in-hand manipulation in the presence of different hand motions. For a robot hand, harnessing rich tactile information to achieve this level of dexterity still remains a significant challenge. In this paper, we present AnyRotate, a system for gravity-invariant multi-axis in-hand object rotation using dense featured sim-to-real touch. We tackle this problem by training a dense tactile policy in simulation and present a sim-to-real method for rich tactile sensing to achieve zero-shot policy transfer. Our formulation allows the training of a unified policy to rotate unseen objects about arbitrary rotation axes in any hand direction. In our experiments, we highlight the benefit of capturing detailed contact information when handling objects of varying properties. Interestingly, we found rich multi-fingered tactile sensing can detect unstable grasps and provide a reactive behavior that improves the robustness of the policy.

Main paper: 
 introduction: 
 The versatility of manipulating objects of varying shapes and sizes has been a long-standing goal for robot manipulation [1] . However, in-hand manipulation with multi-fingered hands can be hugely challenging due to the high degree of actuation, fine motor control, and large environment uncertainties. While significant advances have been made in recent years, most prominently the work by OpenAI [2, 3] , they have relied primarily on vision-based systems which are not necessarily well suited to this task due to significant self-occlusion. Overcoming these issues often requires multiple cameras and complicated setups that are not representative of natural embodiment.More recently, researchers have begun to explore the object rotation problem with proprioception and touch sensing [4, 5] , treating it as a representative task of general in-hand manipulation. The ability to rotate objects around any chosen axis in any hand orientation displays a useful set of primitives for manipulating objects freely in space, even while the hand is in motion. However, this can be challenging as the object is in an intrinsically unstable configuration without any supporting surfaces, as noted in [6, 7] , and requires high-precision control of secure grasps in the presence of gravity (i.e. gravity invariant): it is harder to hold an object while manipulating it if the palm is not facing upwards. Tactile sensing is expected to play a key role here as it enables the capture of detailed contact information to better control the robot-object interaction. However, rich tactile sensing for in-hand dexterous manipulation has not yet been fully exploited due to the large sim-to-real gap, often leading to a reduction of high-resolution tactile data to low-dimensional representations [8, 9] . One might expect that a more detailed tactile representation could increase in-hand dexterity and enable new tasks.In this paper, we introduce AnyRotate: a robot system for performing multi-axis gravity-invariant inhand object rotation with dense featured sim-to-real touch. Here, we propose to tackle this challenge with sim-to-real RL and rich tactile sensing. We first present our goal-conditioned formulation and dense tactile representation to train an accurate and precise policy for multi-axis object rotation. We then train a tactile perception model to simultaneously predict contact pose and contact force readings from tactile images and capture important features for precise manipulation under noisy conditions. In the real world, we mount tactile sensors onto the fingertips of a four-fingered fullyactuated robot hand to provide rich tactile feedback for performing stable in-hand object rotation.Our principal contributions are, in summary: 1) An RL formulation using auxiliary goals for learning a unified policy to perform in-hand object rotation about any desired axis for any hand orientation relative to gravity. 2) A dense tactile representation, consisting of contact pose and contact force, for learning in-hand manipulation in simulation. We highlight the benefit of these tactile modalities for handling unseen objects with various physical properties, such as mass and shape.3) An approach to achieve zero-shot sim-to-real tactile policy transfer, validated on 10 diverse objects in the real world. Our rich tactile policy demonstrates strong robustness across various hand directions and rotation axes and maintains high performance when deployed on a rotating hand. 
related work: 
 Classical Control. Due to the complexity of the contact physics in dexterous manipulation, work on this topic has traditionally relied on simplified models [10] [11] [12] [13] [14] [15] [16] [17] . With improved hardware and design, these methods have continued to demonstrate an increased level of dexterity [18] [19] [20] [21] [22] [23] [24] [25] [26] . While these methods offer performance guarantees, they are often limited by the underlying assumptions.Dexterous Manipulation. With advances of machine learning, learning-based control has become a popular approach to achieving dexterity [2, 3, 27, 28] . However, most prior works rely on vision as the primary sense for object manipulation [6, 7, 29, 30] , which requires continuous tracking of the object in a highly dynamic scene, and occlusions could lead to poorer performance. Vision also has difficulty capturing local contact information which may be crucial for contact-rich tasks.More recently, researchers have explored the in-hand object rotation task using proprioception and touch sensing [4, 5, 9, 31] . This has so far been limited to rotation about the primary axes or training separate policies for arbitrary rotation axes with an upward facing hand [8, 32] . In-hand manipulation in different hand orientations can be challenging as the hand must perform finger-gaiting while keeping the object stable against gravity. Several works [7, [33] [34] [35] achieved manipulation with a downward-facing hand using either a gravity curriculum or precision grasp manipulation, but the policies were still limited to a single hand orientation. In this work, we make significant advancements to train a unified policy to rotate objects about any chosen rotation axes in any hand direction, and for the first time achieve in-hand manipulation with a continuously moving and rotating hand.Tactile Sensing. Vision-based tactile sensors have become increasingly popular due to their affordability, compactness, and ability to provide precise and detailed spatial information about contact through high-resolution tactile images [36] [37] [38] [39] . However, this fine-grained local contact information has not yet been fully utilized for in-hand dexterous manipulation. Previous studies have reduced the high-resolution tactile images to binary contact [9] or discretized contact location [8] to reduce the sim-to-real gap. In contrast, our system utilizes a dense featured tactile representation consisting of the full contact pose and contact force. We show that this tactile representation can capture important interaction physics that is valuable for dexterous manipulation under unknown disturbances.Sim-to-real Methods: Learning in simulation for tactile robotics has gained appeal as it avoids the practical limitations of large data collection in real-world interactions. This trend has been driven by advancements in high-fidelity tactile simulators [40] [41] [42] [43] and various sim-to-real approaches [44] [45] [46] [47] [48] . Several works have proposed using high-frequency rendering of tactile images for sim-to-real RL [44, 49, 50] . However, this can be computationally expensive and inefficient, limiting these methods to simpler robotic systems. In this work, we extend the sim-to-real framework of Yang et al. [51] by proposing an approach to predict full contact pose and contact force and apply it to a dexterous manipulation task with a robot hand. 
method: 
 We perform in-hand object rotation via stable precision grasping, constituting a process for continuous object rotation without a supporting surface. Gravity invariance is considered by randomly initializing hand orientations between episodes. This is a difficult exploration problem whereby the movement of the fingers and object has a constant requirement of maintaining stability, as any finger misplacement can induce slip and lead to an irreversible state where the object is dropped. To obtain a general policy for multi-axis in-hand object rotation, we formulate the object-rotation problem as object reorientation and adopt a two-stage learning process. First, the teacher is trained with privileged information and reinforcement learning [52] . We use an auxiliary goal formulation and adaptive curriculum to achieve sample-efficient training. The student is then trained via supervised learning to imitate the teacher's policy given only real-world observations. During both stages, we provide the agent with rich tactile feedback. To bridge the sim-to-real gap for rich tactile sensing, we collect contact data to train a tactile perception model, which allows for zero-shot policy transfer to the real world. An overview of the method is shown in Figure 2 and 3 .We formulate the task as a finite horizon goal-conditioned Markov Decision Process (MDP) M = (S, A, R, P, G), defined by a continuous state s ∈ S, a continuous action space a ∈ A, a probabilistic state transition function p(s t+1 |s t , a t ) ∈ P, a goal g ∈ G and a reward function r ∈ R : S × A × G -→ R. At each time step t, a learning agent selects an action a t from the current policy π(a t |s t , g) and receives a reward r. The aim is to obtain a policy π * θ parameterized by θ that maximizes the expected return E τ ∼pπ(τ ), g∼q(g) T t=0 γ t r(s t , a t , g) over an episode τ .Observations. The observation O t contains the current and target joint position q t , qt ∈ R 16 , previous action a t-1 ∈ R 16 , fingertip position f p t ∈ R 12 , fingertip orientation f r t ∈ R 16 , binary contact c t ∈ {0, 1} 4 , contact pose P t ∈ S 8 , contact force magnitude F t ∈ R 4 , and the desired rotation axis k ∈ S 2 . The privileged information provided to the teacher includes object position, object orientation, object angular velocity, gravity force vector, and the current goal orientation. Full details can be found in Table 5 . Action Space. At each time step, the action output from the policy is a t := ∆θ ∈ R 16 , the relative joint positions of the robot hand. To encourage smooth finger motion, we apply an exponential moving average to compute the target joint positions defined as qt = qt-1 + ãt , where ãt = ηa t + (1 -η)a t-1 . We control the hand at 20 Hz and limit the action to ∆θ ∈ [-0.026, 0.026] 16 rad.Simulated Touch. We approximate the sensor as a rigid body and fetch the contact information from its sensing surface; the local contact position (c x , c y , c z ) for computing contact pose, and the net contact force (F x , F y , F z ) for computing contact force magnitude. We apply an exponential moving average on the contact force readings to simulate sensing delay due to elastic deformation. We also saturate and re-scale the contact values to the sensing ranges experienced in reality. Contact force is used to compute binary contact signals using a threshold similar to the real sensor.Auxiliary Goal. When training a unified policy for multi-axis rotation, a formulation using angular velocity can lead to inefficient training and convergence difficulties, as will be shown in Section 5.1. Instead, we formulate the problem as object reorientation to a moving target. Targets are generated by rotating the current object orientation about the desired rotation axis in regular intervals. When a target is reached, a new one is generated about the rotation axis until the episode ends.Reward Design. In the following, we provide an intuitive explanation of the goal-based reward used for learning multi-axis object rotation (with full details in Appendix B):EQUATIONThe object rotation objective is defined by r rotation . We use a keypoint formulation K(||k o i -k g i ||) to define target poses [29] and apply keypoint distance threshold to provide a goal update tolerance d tol . We augment this reward with a sparse bonus reward when a goal is reached and a delta rotation reward to encourage continuous rotation. Next, we use r contact to maximize contact sensing which rewards tip contacts and penalizes contacts with any other parts of the hand. We also include several terms to encourage stable rotations r stable comprising: an object angular velocity penalty; a handpose penalty on the distance between the joint position from a canonical pose; a controller work-done penalty; and a controller torque penalty. Finally, we include an early termination penalty r terminate , if the object falls out of the grasp or the rotation axis deviates too far from the desired axis.Adaptive Curriculum. The precision-grasp object rotation task can be separated into two key phases of learning: first to stably grasp the object in different hand orientations, then to rotate objects stably about the desired rotation axis. Whilst the r contact and r stable reward terms are beneficial for the sim-to-real transfer of the final policy, these terms can hinder the learning process, resulting in local optima where the object will be stably grasped without being rotated. To alleviate this issue, we apply a reward curriculum coefficient λ rew (r contact + r stable ), which increases linearly with the average number of rotations achieved per episode. The training in Section 3.1 uses privileged information, such as object properties and auxiliary goal pose. Similar to previous work [5, 8] , we use policy distillation to train a student that only relies on proprioception and tactile feedback. The student policy has the same actor-critic architecture as the teacher policy a t = π θ (O t , a t-1 , z t ) and returns a Gaussian distribution with diagonal covariancesa t ≡ N (µ θ , Σ θ ). The latent vector z t = ϕ(O t , O t-1 , ..., O t-n) is the predicted low dimensional encoding from a sequence of N proprioceptive and tactile observations. We use a temporal convolutional network (TCN) encoder for the latent vector function ϕ(.).Training. The student encoder is randomly initialized and the policy network is initialized with the weights from the teacher policy. We train both the encoder and policy network via supervised learning, minimizing the mean squared error (MSE) of the latent vectors z t and zt and negative log-likelihood loss (NLL) of the action distributions a t and āt . Without explicit object or goal information, we found the student policy unable to achieve the same level of goal-reaching accuracy as the teacher, which can lead to missing the goal and collecting out-of-distribution data. To alleviate this issue, we increase the goal update tolerance d tol during student training.For sim-to-real transfer, we train a tactile perception model to extract contact features from real tactile images [51] . The dense tactile features consist of contact pose and contact force. We use spherical coordinates defined by the contact pose variables: polar angle R x and azimuthal angle R y .The contact force variable is the magnitude of the 3D contact force ||F ||.Data Collection. We use a 6-DoF UR5 robot arm with the tactile sensor attached to the end effector and a F/T sensor placed on the workspace platform. The tactile sensor is moved on the surface of the flat stimulus at randomly sampled poses. For each interaction, we store tactile images with the corresponding pose and force labels. We then train a CNN model to extract these explicit features of contact from tactile images. More details are given in Appendix H.Deployment. We use the Structured Similarity Index (SSIM) to compute binary contact, which is also used to mask contact pose and force predictions. Given tactile images on each fingertip, we use the tactile perception models to obtain the dense contact features. This is then used as tactile observations for the policy. An overview of the tactile prediction pipeline is shown in Figure 3 . 
system setup: 
 Real-world. We use a 16-DoF Allegro Hand with finger-like front-facing vision-based tactile sensors attached to each of its fingertips. Each sensor can be streamed asynchronously along with the joint positions from the hand. The target joint commands are sent with a control rate of 20 Hz. The hand is attached to the end effector of a UR5 to provide different hand orientations for performing in-hand object rotation, as shown in Figure 1 . Simulation. We use IsaacGym [53] for training the teacher and student policies. Each environment contains a simulated Allegro Hand with tactile sensors attached on each fingertip. Gravity is enabled for both the hand and the object. We perform system identification on simulation parameters in various hand directions to reduce the sim-to-real gap (detailed in Appendix D). We run the simulation at dt = 1/60s and policy control at 20 Hz.(M) (M) (M) (M)Real World Object Set Object Set. We use fundamental geometric shapes in Isaac Gym (capsule and box) for training. In simulation, we test on two out-of-distribution (OOD) object sets (see Figure 4 ): 1) OOD Mass, training objects with heavier mass; 2) OOD shape, selection of unseen objects with different shapes. In the real world, we select 10 objects with different properties (see Table 10 ) to test generalizability of the policy.Evaluation We run each experiment for 600 steps (equating to 30 seconds) and use the following metrics for evaluation: (i) Rotation Count (Rot) -the total number of rotations about the desired axis achieved per episode. In the real world, this is manually counted using reference markers attached to the object (visible as the tape in Figure 4 ).(ii) Time to Terminate (TTT) -time is taken before the object gets stuck, falls out of grasp, or if the rotation axis has deviated away from the target. 
experiments and analysis: 
 First we investigate our auxiliary goal formulation and adaptive curriculum for learning the multiaxis object rotation task (Section 5.1). We then study the importance of rich tactile sensing for learning this dexterous manipulation task and conduct a quantitative analysis on the generalizability of the learned polices (Section 5.2). Finally, using the proposed sim-to-real approach, we deploy the policies in the real world on a range of different object rotation tasks (Section 5.3).We compare our auxiliary goal formulation against angular rotation ("w/o auxiliary goal"), a common formulation for in-hand object rotation [5, 8, 32] . The learning curves are shown in Figure 5 .While the agent can learn in the single-axis setting using an angular rotation objective, it resulted in much lower accuracy with near-zero successive goals reached. In the multi-axis setting, the training was unsuccessful and the learning tends to get stuck where the object is stably grasped with minimal rotation. We suspect this is due to the object being held in an intrinsically unstable configuration whereby small random actions can lead to irrecoverable states, such as dropping the object, leading to the agent taking overly conservative actions. During training, when the angular velocity is low and the rotation axis can be noisy, an angular velocity reward cannot effectively guide the agent out of this local optimum. Conversely, provided with privileged goals and a smoother goal-driven reward, the objective becomes more conducive to learning the multi-axis in-hand object rotation task.The proposed adaptive curriculum also contributes positively to this.Tactile The results for randomized rotation axes in random hand orientations are shown in Table 1 . First, we observe that a policy trained in a fixed hand orientation performed poorly in arbitrary hand orientations, suggesting gravity invariance adds considerable complexity to the task. Table 1 compares our dense touch policy (contact pose and contact force) with policies trained with proprioception, binary touch, and discrete touch (a discretized representation introduced in [8] ).Contrary to the findings in [8] , we find binary touch to be beneficial over proprioception alone. We attribute this to including binary contact information during teacher training, which provides a better base policy. Overall, we found that performance improved with more detailed tactile sensing. The dense touch policy, trained with information regarding contact pose and force, outperformed policies that used simpler, less detailed touch. Moreover, discretizing the contact location led to a drop in performance, suggesting that this type of representation is not as well suited to the morphology of our front-facing sensor.Ablation Studies. The results for each tactile modality showed that contact force can provide useful information regarding the interaction physics when handling objects with different mass properties; contact pose is beneficial when handling unseen shapes; and excluding either feature of dense touch resulted in suboptimal performance.Object rotation performance for various hand orientations and rotation axes are given in Tables 2 and 3 . In both cases, the dense touch policy performed the best, demonstrating a successful transfer of the dense tactile observations. The proprioception and binary touch policies were less effective at maintaining stable rotation, often resulting in loss of contact or getting stuck.Hand orientations. The performance dropped as the hand directions changed from palm up and palm down, followed by base up and base down, to the thumb up and thumb down directions. We attribute this to the larger sim-to-real gap when fingers are positioned horizontally during manipulation. In the latter cases, the gravity loading of the fingers acts against actuation, which weakens the hand in those orientations. However, despite the noisy system, a policy provided with rich tactile information consistently demonstrated more stable performance. Examples are shown in Figure 6 . Table 3 : Real-world results for different rotation axes in the palm-down configuration. We report on average rotation count and time to terminate (TTT) per episode.Rotation Axis. Rotation about z-axis was the easiest to achieve, followed by the xand y-axes. We noticed that binary touch performed less well compared to proprioception when rotating about the z-axis, but performed better for xand y-rotation axes. The latter axes require two fingers to hold the object steady (middle/thumb or index/pinky) while the remaining two fingers provide a stable rotating motion. This requires more sophisticated finger-gaiting, and the policy struggled to perform well without tactile sensing.Behavior. An analysis of the tactile predictions during a perturbed rollout is shown Figure 7 . We apply a grasp offset to the object at n step = 300 to visualize the robustness of the policy. The two key motions for stable object rotation can be seen in the output contact pose and force. Given rich tactile sensing on a multi-fingered hand, the policy can detect unstable grasps under boundary contact and provide reactive finger-gaiting motions that prevent the object from slipping further. This emergent behavior was not seen when using proprioception or binary touch.Gravity Invariance. We also demonstrate that the trained policy can adapt effectively to a rotating hand where the gravity vector is continuously changing in the hand's frame of reference. Sample performance for three hand trajectories are provided in the Appendix K.4 and on our website. This capability to manipulate objects during angular movements of the hand enables 6D reorientation of the object while simultaneously repositioning the grasp location. This gives a new level of dexterity for robot hands that could be beneficial in many tasks, e.g. general pick-and-place. 
conclusion and limitations: 
 In this paper, we demonstrated the capability of a general policy leveraging rich tactile sensing to perform in-hand object rotation about any rotation axis in any hand direction. This marks a significant step toward more general tactile dexterity with fully-actuated multi-fingered robot hands.We found the policies had difficulties with objects that have sharp geometric features, such as corners or edges, and rotating with these features required a more adaptive policy. To improve the performance further, a richer tactile representation and perception model can be used to better capture these precise geometric features, such as the pose of an edge feature, or incorporate visual information about the object's shape. Also, the actuation of the Allegro Hand was significantly weakened under certain hand orientations. Therefore, designing low-cost and more capable hardware is crucial for advancing dexterous manipulation with multi-fingered robotic hands.The goal to manipulate objects effortlessly in free space using a sense of touch mirrors a key aspect of human dexterity and stands as a significant goal in robot manipulation. We hope that our research underscores the importance of tactile sensing and spurs continued efforts towards this goal. 
 
 Appendix: The full list of real-world observations o t and privileged information x t used for the agent is presented in Tables 4 and 5 respectively. The proprioception and tactile dimensions are in multiples of four, representing four fingers. Table 5 : Full list of privileged information x t only available in simulation.The privileged information is used to train the teacher with RL and for obtaining the target latent vector z during student training. Whilst the gravity vector can be inferred using the end effector pose of the robot arm, we did not find any benefit of explicitly including this information in the policy. We suspect that using a history of observation, which includes proprioception and contact forces, can also implicitly infer the gravity direction.We use the following reward function for learning multi-axis in-hand object rotation:EQUATIONwhere,r rotation = λ kp r kp + λ rot r rot + λ goal r goal , r contact = λ rew (λ gc r gc + λ bc r bc ), r stable = λ rew (λ ω r ω + λ pose r pose + λ work r work + λ torque r torque ), r terminate = λ penalty r penaltyThe key terms for defining the object rotation task are r rotation and r penalty . We include contact terms to encourage fingertip dexterity and tactile sensing. We also include various stability terms commonly used in previous work [8, 32] to obtain natural-looking policies and aid the sim-to-real transfer. In the following, we explicitly define each term of the reward function.Keypoint Distance Reward:EQUATIONwhere the keypoint distancekp dist = 1 N N i=1 ||k o i -k g i ||,k o and k g are keypoint positions of the object and goal respectively. We use N = 6 keypoints placed 5 cm from the object origin in each of its principle axes, and the parameters a = 50, b = 2.0.We also formulate an alternative reward function consisting of an angular velocity reward and rotation axis penalty to compare with our auxiliary goal formulation.Angular Velocity Reward:EQUATIONwhere c 2 = 0.5.Rotation Axis Penalty:EQUATIONwhere || ko || is the current object rotation axis.We form the new r rotation reward r rotation = λ av r av + λ rot r rot . We provide an additional object axis penalty λ axis r axis in the r stable term and remove the angular velocity penalty, λ ω = 0. The weights are λ av = 1.5 and λ axis = 1.0. We keep all other terms of the reward function the same.The adaptive reward curriculum is implemented using a linear schedule of the reward curriculum coefficient λ rew (r contact + r stable ) which increases with successive goals are reached per episode,EQUATIONwhere [g min , g max ] determines the ranges where the reward curriculum is active. This changes the learning objective towards more realistic finger-gaiting motions as the contact and stability reward increases. We use [g min , g max ] = [1.0, 2.0].To generate stable grasps, we initiate the object at 13cm above the base of the hand at random orientations and initialize the hand at a canonical grasp pose at the palm-up hand orientation. We then sample relative offset to the joint positions U(-0.3, 0.3) rad. We run the simulation by 120 steps (6 seconds) while sequentially changing the gravity direction from 6 principle axes of the hand (±xyz-axes). We save the object orientation and joint positions (10000 grasp poses per object) if the following conditions are satisfied: -The number of tip contacts is greater than 2.-The number of non-tip contacts is zero -Total fingertip to object distance is less than 0.2 -Object remains stable for the duration of the episode.To reduce the sim-to-real gap of the allegro hand, we perform system identification to match the simulated robot hand with the real hand. We model each of the 16 DoF of the hand with the parameters; stiffness, damping, mass, friction, and armature, resulting in a total of 80 parameters to optimize. We collect corresponding trajectories in simulation and the real world in various hand orientations and use CMA-ES [55] to minimize the mean-squared error of the trajectories to find the best matching simulation parameters. [56] and [57] for a detailed explanation of each hyperparameter.Data Collection. The setup for tactile feature extraction is shown in Figure 8 . We collect data by tapping and shearing the sensor on a flat stimulus fixed onto a force torque sensor and collect six labels for training: contact depth z, contact pose in R x , contact pose in R y , and contact forces F x , F y and F z . In order to capture sufficient contact features needed for the in-hand object rotation task and stay within the contact distribution, we sample the sensor poses with the ranges shown in Table 8 . This provides sensing ranges for contact pose between [-28 • , 28 • ] and contact force of up to 5 N, which are the largest ranges we could reasonably consider for this tactile sensor.Training. The architecture and training parameters of the perception model are shown in Table 9 .For each fingertip sensor, we collect 3000 images (2400 train and 600 test) and train separate models.The prediction error for one of the sensors is shown in Figure 9 . The perception model does not explicitly consider multiple contact points. In practice, we found this to be rare and by assuming a single combined contact, the model was sufficient in producing consistent estimates that did not affect the final performance of the in-hand rotation policy.The tactile sensors provide raw RGB images from the camera module. We use an exposure setting of 312.5 and a resolution of 640 × 480, providing a frame rate of up to 30 FPS. The images are then postprocessed. We convert the raw image to greyscale and resale the dimension to 240 × 135. Binary Contact: We apply a medium blur filter with an aperture linear size of 11, followed by an adaptive threshold with a block size of 55 pixels and a constant offset value of -2. These operations improve the smoothness of the image and filter out noise. The postprocessed image is compared with a reference image using the Structural Similarity Index (SSIM) to compute binary contact (0 or 1). We use an SSIM threshold of 0.6 for contact detection. Contact Pose and Force: We directly use the resized greyscale image for contact force and pose prediction. From the target labels, we use contact pose (R x , R y ) and the contact force components F x , F y , F z (to compute the contact force magnitude ||F ||) to construct the dense tactile representation used during policy training. We use the binary contact signal to mask contact pose and force, thresholding the predictions at ≈ 0.25N . driver board that connects to the computer via a USB cable and can be streamed asynchronously at a frame rate of 30 FPS. We perform post-processing using OpenCV [59] in real-time.Sensor Placement. A common limitation of unidirectional tactile sensors is that they are primarily sensorized over a front-facing area. Contacts with the side of the sensor casing can be slippery and result in unstable behaviors. To alleviate this issue, similar to [31] , we adjusted the sensor direction relative to the fingers to maximize contact with the sensing surface, and placed the tactile fingertips with offsets (thumb, index, middle, ring) = (-45 • , -45 • , 0 • , 45 • ). This allowed the policies to achieve consistent and stable in-hand object rotation performance, providing a basis to validate our learning approach against baselines.Control Pipeline. Each tactile perception model is deployed together with the policy as shown in Figure 11 . We stream tactile and proprioception readings asynchronously at 20 Hz. The joint positions are used by a forward kinematic solver to compute fingertip position and orientation. The relative joint positions obtained from the policy are converted to target joint commands. This is published to the Allegro Hand and converted to torque commands by a PD controller at 300 Hz.Object Properties. Various physical properties of the objects used in the real-world experiment are shown in Table 10 . We include objects of different sizes and shapes not seen during training.We provide additional ablation studies to analyze the design choices for our axillary goal formulation. The effect of goal update tolerance d tol for the student training and the auxiliary goal increment intervals are shown in Table 11 .The performance can be significantly affected by the goal-update tolerance. As the tolerance reduced during student training, the number of average rotations and successive goals reached per episode also reduced. This suggests that the performance of the teacher policy was poorly transferred and the student could not learn the multi-axis object rotation skill effectively. Increasing the goal increment intervals also resulted in fewer rotations achieved. We present the tactile readings of a similar policy rollout in simulation and real-world, in Figure 12 and 13 respectively. We observed a sim-to-real gap in the rotation speed as demonstrated by the higher contact cycles obtained in simulation. However, a matching pattern can be seen from the recorded contact features. By comparing the raw and processed contact readings in simulation, we see the effect of the post-processing functions in Section F. This helped with aligning the simulated tactile readings to that of the real sensors and smoothing out the noisy readings of the contact force. The comparison also demonstrates a successful sim-to-real transfer of the proposed tactile representation.The real-world results for each object for varying rotation axes and hand orientations are shown in Figure 13 and 12 respectively. We observed that larger objects resulted in fewer rotations, likely due to the size and joint limits of the Allegro Hand, making smaller objects easier to maneuver. Objects with sharp corners, such as the cube and gum box, sometimes caused the fingers to get stuck around these points. We believe that this is because navigating around these geometric features requires additional finger extension during rotation, making it challenging for a general tactile policy to handle such shapes effectively. The gum box was the most challenging due to its sharp corners and shifting mass (sloshing gum pieces). These factors led to the least stable rotation and the lowest time to terminate (TTT).Contact Surfaces. While we train the tactile perception models on a flat surface, we found that it can generalize to other uneven contact surfaces, shown by various test object shapes in Figure 4 , demonstrating the robustness of the proposed tactile representation.We test the robustness of the policy by performing in-hand object rotation during different hand movements. In particular, we choose hand trajectories where the gravity vector is continuously changing relative to the orientation of the hand, adding greater complexity to this task. Rollouts for three different hand trajectories are shown in Figure 14 . In particular, for the third hand trajectory (iii), we demonstrate the capability of the robot hand to servo around the surface of the object in different directions while keeping the object almost stationary in free space. This motion also demonstrates the ability to command different target rotation axes during deployment, offering a useful set of primitives for other downstream tasks. 


Decision: The paper makes good advance in the topic of in-hand manipulation with tactile input and achieved good results. The authors should address the clarification questions posted by the reviewers.
---
During the rebuttal phase, the authors improved the paper and addressed most concerns raised by the reviewers. The reviewers all agree to accept the paper now.

Reviews: review: title: title: Interesting Paper and Good Results, Need to Clarify Several Details
paper_summary: paper_summary: This paper presents a pipeline for sim-to-real in-hand object manipulation when the hand orientation is randomized. The authors train a tactile perception model that can map tactile images to contact pose and forces. Using this representation, the authors train a policy in simulation and transfer it to the real-world.
main_review: main_review: Strength:

This paper further pushes the boundaries of in-hand manipulation using sim-to-real by demonstrating how tactile sensing can help the policy maintain stability during changes in the direction of gravity.

This result is particularly interesting as the authors show the usage and analyze the effect of fine-grained touch sensing in this process. This could motivate future researchers in this direction.

The paper is well-written and provides a very comprehensive system design, which can be helpful for the community.

Weakness:

Overall, this paper does a good job of explaining the system design. However, I think this paper could significantly benefit from adding more “insights and discussions” on different choices and experiments. More specifically:
* About the auxiliary goal: one interesting contribution of this paper is the revised reward definition. The current experiment section only describes its better performance compared to the velocity reward. However, from my perspective, the reward is only an integral of the velocity reward and it is not easy to understand why this formulation is better. It would be more helpful to discuss the underlying reason or observations for this design.
* Another interesting observation is the “reactive behavior” mentioned in the abstract. This is discussed in Figure 7. However, I do not see a clear difference between the highlighted blue region and other parts. A more detailed illustration of this would be helpful.

The way this paper collects touch data can only deal with single point contact. If there are multiple points in contact with the finger, or if the contact regions are larger than those used during the data collection time, the force prediction will have problems.
questions: questions: Why is the key point-based auxiliary reward better than the rotation velocity reward?

How should we perceive or understand the reactive behavior from Figure 7? For example, what is the unique behavior for the shaded curve, and are the corresponding videos are in the supplementary material?

How important is the gravity direction as an input? In the student training, the gravity direction can also be inferred from the arm’s pose. Have you considered adding this into student policy?

How different are these tactile sensors? Do you need to train a force prediction model for each sensor? Or how do you calibrate between them. In addition, what about the durability of the sensors? How long can the force prediction model be used?

For the dense touch feature data collection, how large can the contact region be? What happens when there are multiple points of contact?

The reward function has quite a few terms. Could the authors comment on the usefulness of them? Which terms are the most critical?
limitations: limitations: 3
review_summary: review_summary: Overall, I think this is a well-written paper with good results. There are several questions that need to be clarified, but I still recommend acceptance because of the results.
title: title: In-hand object rotation with dense tactile sensing: novel method with comprehensive evaluation
paper_summary: paper_summary: The authors propose a system that leverages dense tactile sensing and reinforcement learning to perform object rotation about arbitrary axes in any hand orientation relative to gravity. They first train a policy using teacher-student policy distillation given rich tactile feedback in simulation, then utilize an observation model trained on contact data to perform zero-shot policy transfer to the real world. They evaluate the system using a 4-fingered 16-DoF tactile robot hand in both simulation and real-world experiments, based on rotations per episode and rotation time until failure.
main_review: main_review: Overall, the paper is structured and readable. The proposed system is novel and relevant to the CoRL audience. The relevant work seems adequate. The proposed system is clearly presented with sufficient detail for reproductions. The problem formulation and policy training method are generalizable to other robotics tasks. The tactile prediction pipeline can also be applied to various scenarios that require tactile feedback.  

Strengths:
- Utilizing contact information provided by dense tactile sensing seems to be a promising approach to dexterous manipulation tasks.
- The zero-shot unified policy transfer demonstrates the robustness of the system in real-world tasks.
- The experiments are thorough, covering various hand orientations, rotation axes, and object properties, with proper baselines selected.
- The ablation experiments indicate that all components of dense tactile sensing are crucial to the performance of the system.  

Weaknesses:
- The results are presented by taking the average of results from all tested objects. It is a little unclear how object properties affect the performance. It is possible that the baselines may have better performance than the proposed method in some cases.

Minor comments:
- In Table 3, sign missing at y-axis Dense Touch TTT.
questions: questions: 1. As the results are presented by taking the average of all tested objects, are there any cases where the baselines outperform the proposed method?
2. The policy is trained using Isaac gym fundamental geometric shapes including boxes, yet the system still has difficulties with box-shaped objects. “Grasping points producing similar tactile information for different states” is a reasonable cause, but is it feasible to replicate this problem in simulation by tuning the simulated touch reading and get a better trained policy?
3. The figures can be improved. As the hand is all black and the fingers look almost the same, it would be helpful to annotate the axes in the figures to help readers understand why rotation axes affect the performance.
limitations: limitations: 2
review_summary: review_summary: The authors present a system that is relevant and novel, they provide detailed information that helps the readers to understand the system and potentially reproduce it. The system is well grounded and thoroughly tested in both simulation and real-world experiments. The authors might further improve this paper by giving some insights on what external factors affect the system performance. Overall, this is a good paper and I suggest acceptance.
title: title: Review
paper_summary: paper_summary: The paper presents a system designed for gravity-invariant multi-axis in-hand object rotation using a tactile robot hand with rich tactile feedback. The system is trained in simulation and employs zero-shot policy transfer to handle real-world tasks without additional retraining.  Key contributions include:  1. An RL framework for learning a unified policy for multi-axis in-hand object rotation. 2. A dense tactile representation that enhances the handling of objects with different properties. 3. Successful zero-shot sim-to-real policy transfer validated on diverse objects.
main_review: main_review: The overall quality and clarity of the paper are good. However, the task and method are similar to previous work, which may limit its contribution. While the addition of tactile information and gravity invariance is noteworthy, these changes may not significantly differentiate it from earlier papers focused on axis rotation. Nonetheless, the system engineering aspect, particularly the use of multi-finger tactile sensing, could be considered a major contribution.

# Strength:

1. The paper demonstrates gravity-invariant multi-axis in-hand rotation, with impressive video evidence of object rotation despite changing end-effector orientations.

2. Achieving sim-to-real transfer for tactile-based in-hand reorientation is a significant engineering accomplishment.

3. The paper deals with challenging tactile simulation and sim-to-real transfer.

4. Using a learning method to extract tactile features from images, rather than relying on raw data, is a notable strength.

# Limitation:
See Questions for Rebuttal.
questions: questions: # Limitation:

Minor Limitation:

1. The method described in the paper is similar to the approach used in "Visual Dexterity: In-Hand Dexterous Manipulation from Depth" for teaching the pipeline. The main difference is that the authors of the current paper incorporate tactile modalities in the training process.
2. For the hardware design, it would be beneficial if the authors explained the rationale behind choosing different orientations for the tactile sensors on various fingers. Additionally, conducting experiments to compare the benefits of having sensors oriented in different directions versus having all sensors aligned in the same direction would provide valuable insight into the impact of sensor orientation on the system’s performance

Major Limitation:

1. While the dense tactile sensor is commendable at the hardware level, its installation in the hand reveals a potential issue. The authors manually adjusted the sensor direction to face the object, thereby increasing the contact area between the objects and the sensor (as mentioned in Appendix J, lines 587-590). Although this design choice is intuitive and aimed at simplifying the task, it limits the system's generalizability. By optimizing the sensor orientation for this specific task, the hand's versatility for other tasks, even similar rotation tasks with different motions, may be compromised. Therefore, this approach may not serve as a general framework for in-hand rotation tasks. I agree that the digit sensor might not have any signal on the sides of cases, which may be the major reason behind this design. However, I still believe the framework could be more generalizable if the authors had placed all the sensors in the same direction and successfully completed the task. 
2. The tactile sensing simulation in the paper raises some questions. The authors treat the sensor as a rigid body to fetch contact information in the simulation. For example, in reality, the contact position will move when the force increases, but this cannot happen in a simulation using a rigid body model. Additionally, it would be beneficial if the authors described how they simulate the delay in contact force response.
3. I assume the authors used Isaac Gym's contact simulation to fetch the contact information without using any external package. It would be great if the authors demonstrated how to obtain contact information in Isaac Gym and visualized both the raw contact data and the processed data. Obtaining accurate contact information in Isaac Gym is challenging, particularly because the contact position can be inaccurate. The study would be much improved if the authors visualized similar contact scenes in both simulation and the real world to compare the tactile representations.
4. In lines 11-13, the authors state, "we found rich multi-fingered tactile sensing can implicitly detect object movement within grasp and provide a reactive behavior that improves the robustness of the policy." This claim needs experimental verification to demonstrate how object movement within the grasp is implicitly detected and how this detection leads to reactive behavior.
5. The task presented is similar to those in previous papers listed in the references, such as "In-Hand Object Rotation via Rapid Motor Adaptation." It would be more impactful if the paper demonstrated enhanced dexterous skills through the use of tactile modalities rather than focusing solely on achieving gravity-invariance. For gravity-invariance, the authors use a relatively simple technique of randomizing initial hand orientations between episodes, which may not be considered a major contribution. It would be beneficial if the authors highlighted how tactile information specifically aids the task, particularly given their significant effort in tactile sim-to-real transfer.
6. The simulation uses a single contact point for tactile information (as visualized in Figure 3), whereas the real-world tactile representation involves a contact area. Bridging the sim-to-real gap between these differing sensing patterns is a key challenge. The paper should address how this discrepancy is managed and how the simulation results translate effectively to the real-world tactile data.
limitations: limitations: 2
review_summary: review_summary: Weak accept. The paper has contribution on the system engineering with both hardware and learning contribution, but lack of major contribution. I recommend weak reject the paper, but may adjust the score if the issues are well addressed.