AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch
Authors: Max Yang
chenghua lu
Alex Church
Yijiong Lin
Christopher J. Ford
Haoran Li
Efi Psomopoulou
David A.W. Barton
Nathan F. Lepora
Venue: CoRL 2024 (2024)
Keywords: Tactile Sensing
In-hand Object Rotation
Reinforcement Learning
TL;DR: We present AnyRotate, a system that achieves multi-axis gravity-invariant in-hand object rotation with dense featured sim-to-real touch.

Abstract: Human hands are capable of in-hand manipulation in the presence of different hand motions. For a robot hand, harnessing rich tactile information to achieve this level of dexterity still remains a significant challenge. In this paper, we present AnyRotate, a system for gravity-invariant multi-axis in-hand object rotation using dense featured sim-to-real touch. We tackle this problem by training a dense tactile policy in simulation and present a sim-to-real method for rich tactile sensing to achieve zero-shot policy transfer. Our formulation allows the training of a unified policy to rotate unseen objects about arbitrary rotation axes in any hand direction. In our experiments, we highlight the benefit of capturing detailed contact information when handling objects of varying properties. Interestingly, we found rich multi-fingered tactile sensing can detect unstable grasps and provide a reactive behavior that improves the robustness of the policy.

Any Rotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch
Max Yang', Chenghua Lu', Alex Church?, Yijiong Lin', Chris Ford Haoran Li',
Efi Psomopoulou', David A.W. Barton"', Nathan F. Lepora?
Uniierssity of Bristol
Cambbiin Robotics
https: https::/maxyang278890 github. cocuubioo/ae /o//ayyotate///////ooooo
Time
Palm up
Palm Down
Thumb Up
Thumb Down
Base Up
Base Down
Figure 1: Setup: A 4-fingered 16-DoF tactile robot hand attached to a URS performing multi-axis in-hand object rotation (top), with experiments in six key hand orientations with respect to gravity: palm up, palm down, thumb up, thumb down, base up and base down (bottom).
Abstract: Human hands are capable of in-hand manipulation in the presence of different hand motions. For robot hand, harnessing rich tactile information to achieve this level of dexterity still remains a significant challenge. In this paper, we present AnyRotate, a system for gravity-invariant multi-axis in-hand object rotation using dense featured sim-to-real touch. We tackle this problem by training dense tactile policy simulation and present sim-to-real method for rich tactile sensing to achieve zero-shot policy transfer. Our formulation allows the training of unified policy to rotate unseen objects about arbitrary rotation axes in any hand direction. In our experiments, we highlight the benefit of capturing detailed contact information when handling objects of varying properties. Interestingly, we found rich multi-fingered tactile sensing can detect unstable grasps and provide a reactive behavior that improves the robustness of the policy.
Keywords: Tactile Sensing, In-hand Object Rotation, Reinforcement Learning
Introduction
The versatility of manipulating objects of varying shapes and sizes has been long- lonn-itanding goal for robot manipulation [1]. However, in-hand manipulation with multi-fingered hands can be hugely
These authors contributed equally. Correspondence to max. wang@brisstor ao..ll...u
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

challenging due to the high degree of actuation, fine motor control, and large environment uncer- tainties. While significant advances have been made in recent years, most prominently the work by OpenAl [2, 3], they have relied primarily on vision-based systems which are not necessarily well suited to this task due to significant self-ocelusion. Overcoming these issues often requires multiple cameras and complicated setups that are not representative of natural embodiment.
More recently, researchers have begun to explore the object rotation problem with proprioception and touch sensing [4, 5], treating it as a representative task of general in-hand manipulation. The ability to rotate objects around any chosen axis in any hand orientation displays a useful set of primitives for manipulating objects freely in space, even while the hand in motion. However, this can be challenging as the object in an intrinsically unstable configuration without any supporting surfaces, as noted in [6, 7], and requires high-precision control of secure grasps in the presence of gravity (i.e. gravity invariant): it is harder hold an object while manipulating it if the palm is not facing upwards. Tactile sensing is expected to play a key role here enables the capture of detailed contact information to better control the robot-object interaction. However, rich tactile sensing for in-hand dexterous manipulation has not yet been fully exploited due to the large sim-to-real gap, often leading to a reduction of high-resolution tactile data to low-dimensional representations [8, 9]. One might expect that more detailed tactile representation could increase in-hand dexterity and enable new tasks.
In this paper, we introduce AnyRotate: a robot system for performing multi-axis gravity- -nnaaittnni in- hand object rotation with dense featured sim-to-real touch. Here, we propose to tackle this challenge with sim-to-real RI. and rich tactile sensing. We first present our goal-conditioned formulation and dense tactile representation to train an accurate and precise policy for multi-axis object rotation. We then train a tactile perception model to simultaneously predict contact pose and contact force readings from tactile images and capture important features for precise manipulation under noisy conditions. In the real world, mount tactile sensors onto the fingertips of four- four-fingered fully- actuated robot hand to provide rich tactile feedback for performing stable in-hand object rotation.
Our principal contributions are, in summary: 1) An RL formulation using auxiliary goals for learning a unified policy to perform in-hand object rotation about any desired axis for any hand orientation relative to gravity.
2) A dense tactile representation, consisting of contact pose and contact force, for learning in-hand manipulation in simulation. We highlight the benefit of these tactile modalities for handling unseen objects with various physical properties, such as mass and shape.
3) An approach to achieve zero-shot sim-to-real tactile policy transfer, validated on 10 diverse ob- jects in the real world. Our rich tactile policy demonstrates strong robustmess across various hand directions and rotation axes and maintains high performance when deployed on a rotating hand.
2 Related Work
Classical Control. Due to the complexity of the contact physics in dexterous manipulation, work on this topic has traditionally relied on simplified models [10-17]. With improved hardware and design, these methods have continued to demonstrate an increased level of dexterity [18-26]. While these methods offer performance guarantees, they are often limited by the underlying assumptions.
Dexterous Manipulation. With advances of machine learning, learning- based control has become a popular approach achieving dexterity [2, 3, 27, 28]. However, most prior works rely on vision as the primary sense for object manipulation [6. 7, 29, 30], which requires continuous tracking of the object in a highly dynamic scene, and occlusions could lead to poorer performance. Vision also has difficulty capturing local contact information which may be crucial for contact-rich tasks.
More recently, researchers have explored the in-hand object rotation task using proprioception and touch sensing [4. 5, 9, 31]. This has so far been limited rotation about the primary axes or training separate policies for arbitrary rotation axes with an upward facing hand [8, 32]. In-hand manipula- tion in different hand orientations can be challenging as the hand must perform finger-gaiting while

---

keeping the object stable against gravity. Several works [7, 33-35] achieved manipulation with downward-facing hand using either a gravity curriculum or precision grasp manipulation, but the policies were still limited to a single hand orientation. In this work, we make significant advance- ments to train unified policy to rotate objects about any chosen rotation axes in any hand direction, and for the first time achieve in-hand manipulation with a continuously moving and rotating hand.
Tactile Sensing. Vision-based tactile sensors have become increasingly popular due to their afford- ability, compactness, and ability to provide precise and detailed spatial information about contact through high-resolution tactile images [36-39]. However, this fine-grained local contact information has not yet been fully utilized for in-hand dexterous manipulation. Previous studies have reduced the high-resolution tactile images to binary contact [9] or discretized contact location [8] to reduce the sim-to-real gap. In contrast, our system utilizes a dense featured tactile representation consisting of the full contact pose and contact force. We show that this tactile representation can capture important interaction physics that is valuable for dexterous manipulation under unknown disturbances.
Sim-to-real Methods: Learning in simulation for tactile robotics has gained appeal as it avoids the practical limitations of large data collection in real-world interactions. This trend has been driven by advancements
high-fidelity tactile simulators 43] and various sim-to-real approaches [44- 48). Several works have proposed using high-frequency rendering of tactile images for sim-to-real RL [44, 49, 50]. However, this can be computationally expensive and inefficient, limiting these methods to simpler robotic systems. In this work, we extend the sim-to-real framework of Yang et al. [51] by proposing an approach to predict full contact pose and contact force and apply it to dexterous manipulation task with a robot hand.
3 Method
We perform in-hand object rotation via stable precision grasping, constituting process for contin- uous object rotation without supporting surface. Gravity invariance is considered by randomly initializing hand orientations between episodes. This a difficult exploration problem whereby the movement of the fingers and object has a constant requirement of maintaining stability, as any fin- ger misplacement can induce slip and lead to an irreversible state where the object is dropped. To obtain a general policy for multi-axis in-hand object rotation, we formulate the object-rotation prob- lem object reorientation and adopt two-stage learning process. First, the teacher is trained with privileged information and reinforcement learning [52]. We use an auxiliary goal formulation and adaptive curriculum to achieve sample-efficient training. The student is then trained via supervised learning to imitate the teacher's policy given only real-world observations. During both stages, we provide the agent with rich tactile feedback. To bridge the sim-to-real gap for rich tactile sensing, we collect contact data train a tactile perception model, which allows for zero-shot policy transfer to the real world. An overview of the method is shown in Figure 2 and 3.
3.1 Multi-axis In-hand Object Rotation
We formulate the task as finite horizon goal-conditioned Markov Decision Process (MDP) M = R, P, G), defined by continuous state E S, continuous action space a E A. probabilistic state transition function p($++1|s;, a.) E P. a goal g G and a reward function r E R S x A x g - RR. At each time step I,, learning agent selects an action az from the current policy x(a st. g) and receives reward r. The aim to obtain a policy z parameterized by f that maximizes the expected return (rrpp(F).
t "rr(se az. g)) over an episode T.
T
Observations. The observation O, contains the current and target joint position 4ts q. 16 previous action dr-1 E R16,, fingertip position J I fingertip orientation Jī R16 binary contact C, E {0, 134, contact pose Pt E 38, contact force magnitude F; E R-, and the desired rotation axis k 8², The privileged information provided to the teacher includes object position, object orientation, object angular velocity, gravity force vector, and the current goal orientation. Full details can be found in Table 5.
3

---

Muttaan Retation
Tracher Trahing
Eroeder
Prilyy
Audlyay Eagpoles
Time
Prstteed (nfomatios
Reakwands Oheercatione [+]
Transtry
Object Prozetko
Proprooarknn
Qui
TCM
Studert
Enooder
Polky
Oranty Deeation
Target ftraaioo Add
CortedPeaae (R, Rg F.)
Ope
Figure 2: Overview of the approach. Left: The object rotation problem formulated as an object reorientation to moving goal. Auxiliary goal keypoints are used to define target poses about the chosen rotation axis. Right: training a policy using teacher-student policy distillation. The teacher is trained using privileged information with RL and the student aims to imitate the teacher's action given real-world observations. Privileged information and real-world observation are shown.
Action Space. At each time step, the action output from the policy is az := 20 E R the relative joint positions of the robot hand. To encourage smooth finger motion, we apply an exponential moving average to compute the target joint positions defined as dr = ft- + ãt, where ã = 1j0 + We control the hand at 20 Hz and limit the action to AO E [-0.0 0.026] 16 rad.
(1 11-
Simulated Touch. We approximate the sensor a rigid body and fetch the contact information from its sensing surface; the local contact position (Cz, Cy. C2) for computing contact pose, and the net contact force (Fz, Fy. F=) for computing contact force magnitude. We apply an exponential moving average on the contact force readings to simulate sensing delay due to elastic deformation. We also saturate and re-seale the contact values to the sensing ranges experienced in reality. Contact force is used to compute binary contact signals using a threshold similar to the real sensor.
Auxiliary Goal. When training unified policy for multi-axis rotation, formulation using angular velocity can lead to inefficient training and convergence difficulties, as will be shown in Section 5. 1. Instead, we formulate the problem as object reorientation to moving target. Targets are generated by rotating the current object orientation about the desired rotation axis in regular intervals. When target reached, a new one is generated about the rotation axis until the episode ends.
Reward Design. In the following, we provide intuitive explanation of the goal-based reward used for learning multi-axis object rotation (with full details in Appendix B):
Trotation + Tcoutact + Tstable + Frerminate-
The object rotation objective defined by Trotation- We use keypoint formulation K[[] - to define target poses [29] and apply keypoint distance threshold to provide goal update tolerance deol- We augment this reward with a sparse bonus reward when goal is reached and a delta rotation reward to encourage continuous rotation. Next, we use Tcontact to maximize contact sensing which rewards tip contacts and penalizes contacts with any other parts of the hand. We also include several terms to encourage stable rotations Tstable comprising: an object angular velocity penalty; a hand- pose penalty on the distance between the joint position from canonical pose; a controller work-done penalty: and a controller torque penalty. Finally, we include an early termination penalty Nterminate> if the object falls out of the grasp or the rotation axis deviates too far from the desired axis.
Adaptive Curriculum. The precision-gratt object rotation task can be separated into two key phases of learning: first to stably grasp the object in different hand orientations, then to rotate objects stably about the desired rotation axis. Whilst the Tcontact and rstable reward terms are beneficial for the sim-to-real transfer of the final policy, these terms can hinder the learning process, resulting in local optima where the object will be stably grasped without being rotated. To alleviate this issue, apply reward curriculum coefficient Nyew wewwwrrcotaan (Fcontact + Fstable), which increases linearly with the average number of rotations achieved per episode.

---

Grasp Exampie
Fosture Extraction
Effediie Representoo
Contact Feature
Data
Indax
Thumb
CNN
Ap. Paly
Figure 3: Tactile prediction pipeline; a) tactile images are preprocessed to grey-scale filtered images, b) models extract explicit contact features, c) visualization of the tactile features: contact pose and contact force are represented by the center and area of the shaded circle respectively.
3.2 Teacher-Student Policy Distillation
The training in Section 3.1 uses privileged information, such as object properties and auxiliary goal pose. Similar to previous work [5, 8]. we use policy distillation to train student that only relies on proprioception and tactile feedback. The student policy has the same actor-critic architecture as the teacher policy a₁ = (O, a1-1, zn) and returns a Gaussian distribution with diagonal covariances a; = (Nu,, Eg). The latent vector 2; ( .... Oz-n) is the predicted low dimensional encoding from a sequence of N proprioceptive and tactile observations. We use temporal convo- lutional network (TCN) encoder for the latent vector function o(-).
Training. The student encoder is randomly initialized and the policy network is initialized with the weights from the teacher policy. We train both the encoder and policy network via supervised learning, minimizing the mean squared error (MSE) of the latent vectors 2 and & and negative log-likelihood loss (NLL) of the action distributions de and dt. Without explicit object or goal information, we found the student policy unable to achieve the same level of goal-reaching accuracy as the teacher, which can lead to missing the goal and collecting out-of -itto---istriuution data. To alleviate this issue, we increase the goal update tolerance deal during student training.
3.3 Sim-to-Real Dense Featured Touch
For sim-to-real transfer, we train tactile perception model to extract contact features from real tactile images [51]. The dense tactile features consist of contact pose and contact force. We use spherical coordinates defined by the contact pose variables: polar angle R₃ and azimuthal angle Rg- The contact force variable is the magnitude of the 3D contact force ||]
Data Collection. We use 6-DoF UR5 robot arm with the tactile sensor attached the end effector and F/T sensor placed on the workspace platform. The tactile sensor is moved on the surface of the flat stimulus randomly sampled poses. For each interaction, we store tactile images with the corresponding pose and force labels. We then train CNN model to extract these explicit features of contact from tactile images. More details are given in Appendix H.
Deployment. We use the Structured Similarity Index (SSIM) to compute binary contact, which is also used to mask contact pose and force predictions. Given tactile images on each fingertip, we the tactile perception models to obtain the dense contact features. This is then used as tactile observations for the policy. An overview of the tactile prediction pipeline is shown in Figure 3.
System Setup
Real-world. We use a 16-DoF Allegro Hand with finger-like tront-facing rision-based tactile sen- sors attached each of fingertips. Each sensor can be streamed asynchronously along with the joint positions from the hand. The target joint commands are sent with control rate of 20 The hand is attached the end effector of UR5 to provide different hand orientations for performing in-hand object rotation, shown in Figure 1.
5

---

Single
Senleack Retwilnn
axb
wns sualltyy goo
Perten may 1.
In kun
2568 2000 2500
300
1090 1040 2000
300
2000
1500 2003
2500
530
1200
1563
Figure 5: Learning curve for different training strategies. Left: Single-axis training fixed about rotation in the z-axis. Right: Multiple-axis training for arbitrary rotation axis. We report on the average rotation count and the number of successive goals reached.
Simulation. We use IsaacGym [53] for training the teacher and student policies. Each environment contains simulated Allegro Hand with tactile sensors attached on each fingertip. Gravity enabled for both the hand and the object. We perform system identification simulation parameters in var- ious hand directions reduce the sim-to-real gap (detailed in Appendix D). We run the simulation at dt- 1/60s and policy control at 20 Hz.
Object Set. We use fundamental geometric shapes in Isaac Gym (capsule and box) for training. In sim- ulation, we test on two outtof-diitrrbutionn (OOD) object sets (see Figure 4): 1) OOD Mass, training objects with heavier mass; 2) OOD shape, selection of unseen objects with different shapes. In the real world, we select 10 objects with different properties (see Table 10) to test generalizability of the policy.
dimulitionn Obppct Set
Ottrd
Evaluation We run each experiment for 600 steps (equating to 30 seconds) and use the following met- rics for evaluation:
Figure 4: Top: Simulation test object set
from [54]. Bottom: Real everyday objects.
(i) Rotation Count (Ror) the total number of rota- tions about the desired axis achieved per episode. In the real world, this is manually counted using reference markers attached to the object (visible as the tape in Figure 4).
(ii) Time to Terminate (TTT) time taken before the object gets stuck, falls out of grasp, or if the rotation axis has deviated away from the target.
5 Experiments and Analysis
First we investigate our auxiliary goal formulation and adaptive curriculum for learning the multi- axis object rotation task (Section 5.1). We then study the importance of rich tactile sensing for learning this dexterous manipulation task and conduct quantitative analysis the generalizability of the learned polices (Section 5.2). Finally, using the proposed sim-to-real approach, we deploy the policies in the real world on a range of different object rotation tasks (Section 5.3).
5.1 Training Performance
We compare our auxiliary goal formulation against angular rotation ("w/o auxiliary goal"), a com- mon formulation for in-hand object rotation [5, 8, 32]. The learning curves are shown in Figure 5. While the agent can learn in the single-axis setting using an angular rotation objective, it resulted in much lower accuracy with near-zero successive goals reached. In the multi-axis setting, the training was unsuccessful and the learning tends to get stuck where the object is stably grasped with minimal rotation. We suspect this is due to the object being held in an intrinsically unstable configuration whereby small random actions can lead to irrecoverable states, such as dropping the object, leading to the agent taking overly conservative actions. During training, when the angular velocity low and the rotation axis can be noisy, an angular velocity reward cannot effectively guide the agent out of this local optimum. Conversely, provided with privileged goals and smoother goal-driven re- ward, the objective becomes more conducive learning the multi-axis in-hand object rotation task. The proposed adaptive curriculum also contributes positively to this.

---

5.2 Simulation Results
The results for randomized rotation axes in random hand orientations are shown in Table 1. First, we observe that a pol- icy trained in a fixed hand orientation per- formed poorly in arbitrary hand orienta- tions, suggesting gravity invariance adds considerable complexity to the task. Ta- ble 1 compares our dense touch policy (contact pose and contact force) with poli- cies trained with proprioception, binary touch, and discrete touch (a discretized representation introduced in [8]).
Tactile Observation
Mass
OOD Shape
Rot
Eplen(s)
Roi
EpLen(s)
Fixed Hand Orn
0.55
11.8,0
0.55ggo
10.
Proprioception
1.34
21.6g0.5
0.82
26.1
Binary Touch
1.900004
20.5go.5
1.57
25.3g0.
Discrete Touch
1.95
22.2g0.4
1.67
26.5go.1
Dense Force (wle Pose)
2.05
22.0g0.5
1.60000000
25.520.4
Dense Pose (wlo Force)
2.05
21.9.0.1
1.73
26.7.0.0
Dense Touch (Ours)
2.1820.05
22.8=0.8
1.77
27.2
Table I: Comparison of different tactile policies on test object sets in simulation. We report average rotation achieved per episode (Rot) and average episode length (EpLen) for arbitrary rotation axis and hand direction.
Contrary to the findings in [8], we find bi- nary touch to be beneficial over proprioception alone. We attribute this to including binary contact information during teacher training, which provides a better base policy. Overall, we found that performance improved with more detailed tactile sensing. The dense touch policy, trained with in- formation regarding contact pose and force, outperformed policies that used simpler, less detailed touch. Moreover, discretizing the contact location led to drop in performance, suggesting that this type of representation is not well suited to the morphology of our front-facing sensor.
The results for each tactile modality showed that contact force can provide useful information regarding the interaction physics when handling objects with different mass properties; contact pose is beneficial when handling unseen shapes; and excluding either feature of dense touch resulted in suboptimal performance.
Ablation Studies.
5.3 Real-world Results
Object rotation performance for various hand orientations and rotation axes are given in Tables 2 and 3. In both cases, the dense touch policy performed the best, demonstrating a successful transfer of the dense tactile observations. The proprioception and binary touch policies were less effective at maintaining stable rotation, often resulting in loss of contact or getting stuck.
Hand orientations. The performance dropped the hand directions changed from palm up and palm down, followed by base up and base down, to the thumb up and thumb down directions. We attribute this to the larger sim-to-real gap when fingers are positioned horizontally during manipula- tion. In the latter cases, the gravity loading of the fingers acts against actuation, which weakens the hand in those orientations. However, despite the noisy system, a policy provided with rich tactile information consistently demonstrated more stable performance. Examples are shown in Figure
Rotation Axis. Rotation about z-axis was the easiest to achieve, followed by the x- and y-axes. We noticed that binary touch performed less well compared to proprioception when ro- tating about the z-axis, but performed better for x- and y-rotation axes. The latter axes require two fingers to hold
Tactile Observation
x-axis
y-axis
z-axis
Rot
TTT(s)
Rot
TTT(s)
Roi
TTT(s
Propriocaption
0.35_ma
10.0
0.17 8.33_sa 1.06 25.3
Binary Touch
0.87 20.5g9.4
0.25 15.9-10a 0.89g0.2 23.8
Dense Touch (Ours)
1.33gn.m22.66..
0.79 27.8 1.33 28, 28.22
Table 3:
Real-world results for different rotation axes in the palm-down configuration. We report average rota- tion count and time to terminate (TTT) per episode.
Tactile Observation
Palm Up
Palm Down
Base Up
Rase Down
Thumb Up
Thumb Down
Rot
TTT(s)
Rot
TTT(
Ret
TTT(9)
Rot
TTTss)
Roi
TTT6)
Roi
TTT6)
Proprioception
1.07 c0
27.6.3.
1.05
25.0ux4
0.841030
26.8144
1.87
22.8
0.78
20,3
0.51asas
9.50
Binary Touch
1. 1.22a
25.5
0.80
23. 23.8
0.88
25.
a.77
23.0g17
0 08
22 2g06
0.4723
13 23
Deuse Touch (Ours)
57 30.0
1.882044
28.2
32
29.8
1.17
29.4
1.082
27.9ga1
0.
29.2530
Table 2: Real-world results of policies trained on different observations for rotating about the z-axis in different hand directions. We report on average rotation count (Rot) and time to terminate (TTT) per episode over all test objects.
7

---

Time
Tima
Assy Sakrtence
Urosable dreup Ricowery
Gredp hol
Corract Pows 3, slgp
Contuct Prem drept
Dontact Porce PM
MUMMMM
mo m
M008 Thuno
mm Am
mmmm UJJ
M Munan
Ww MA
A TRAnAA
Figure 7: Tactile features during rollout. The rotation component of contact seen in Ry, a repeated cycle of the object rolling along the fingertips. A reactive behavior is seen in the blue-shaded region Ry, where after boundary contact detection, the fingers extend reduce contact angle in subsequent cycles achieve more stable grasping. A demonstration included on our website.
Figure 6: Frames of in-hand object rotation for six distinct objects under six hand orien- tations relative to gravity.
the object steady (middle/thumb or index/pinky) while the remaining two fingers provide a stable rotating motion. This requires more sophisticated finger-gaiting, and the policy struggled to perform well without tactile sensing.
Emergent Behavior. An analysis of the tactile predictions during perturbed rollout is shown Figure 7.
We apply a grasp offset to the object at 'lstep = 300 to visualize the robustness of the policy. The two key motions for stable object rotation can be seen in the output contact pose and force. Given rich tactile sensing on a multi-fingered hand, the policy can detect unstable grasps under boundary contact and provide reactive finger-gaiting motions that prevent the object from slipping further. This emergent behavior was not when using proprioception or binary touch.
Gravity Invariance. We also demonstrate that the trained policy can adapt effectively to rotating hand where the gravity vector is continuously changing in the hand's frame of reference. Sample performance for three hand trajectories are provided in the Appendix K.4 and on our website. This capability to manipulate objects during angular movements of the hand enables 6D reorientation of the object while simultameously repositioning the grasp location. This gives a new level of dexterity for robot hands that could be beneficial in many tasks, e.g. general pick-and-place.
6 Conclusion and Limitations
In this paper, demonstrated the capability of general policy leveraging rich tactile sensing to perform in-hand object rotation about any rotation axis in any hand direction. This marks significant step toward more general tactile dexterity with fully-actuated multi-fingers robot hands.
We found the policies had difficulties with objects that have sharp geometric features, such as corners or edges, and rotating with these features required a more adaptive policy. To improve the perfor- mance further, richer tactile representation and perception model can be used to better capture these precise geometric features, such the pose of an edge feature, or incorporate visual informa- tion about the object's shape. Also, the actuation of the Allegro Hand was significantly weakened under certain hand orientations. Therefore, designing low-cost and more capable hardware is crucial for advancing dexterous manipulation with multi-fingered robotic hands.
The goal to manipulate objects effortlessly in free space using sense of touch mirrors key aspect of human dexterity and stands as significant goal in robot manipulation. We hope that our research underscores the importance of tactile sensing and spurs continued efforts towards this goal.

---

Acknowledgments
We thank Andrew Stinchcombe for helping with the 3D-printing of the stimuli and tactile sensors. We thank Haozhi Qi for the valuable discussions. This work was supported by the EPSRC Doctoral Training Partnership (DTP) scholarship.
References
[1] A. M. Okamura, N. Smaby, and M. R. Cutkosky. An overview of dexterous manipulation. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 1, pages 255-262. IEEE, 2000.
[2] Akkaya, M. Andrychowicz. M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint 1rXiv:1910.07113, 2019.
[3] O. M. Andrychowicz, B. Baker, M. Chociej, Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3 2020.
[4] G. Khandate, M. Haas-Heger, and M. Ciocarlie. On the feasibility of learning finger-gaiting in-hand manipulation with intrinsic sensing. In 2022 International Conference on Robotics and Automation (ICRA), pages 2752-2758. IEEE, 2022.
[5] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik. In-hand object rotation via rapid motor adaptation. In Conference on Robot Learning, pages 1722-1732. PMLR, 2023.
[6] T. Chen, J. Xu, and P. Agrawal. A system for general in-hand object re-orientation. In Confer- ence Robot Learning, pages 297-307. PMLR, 2022.
[7] Chen, M. Tippur, S. Wu, V. Kumar, E. Adelson, and Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes. Science Robotics, 8(84))ead:9244, 2023.
[8] H. Qi, B. Yi, S. Suresh, M. Lambeta, Y. Ma, R. Calandra, and J. Malik. General in-hand object rotation with vision and touch. In Conference on Robot Learning, pages 2549-2564. PMLR, 2023.
[9] G. Khandate, S. Shang, E. T. Chang, T. L. Saidi, J. Adams, and M. Ciocarlie. Sampling- based exploration for reinforcement learning of dexterous manipulation. arXiv preprint arXiv:2303.03486 2023.
[10] I.. Han and J. C. Trinkle. Dextrous manipulation by rolling and finger gaiting. In Proceedings. 1998 IEEE International Conference Robotics and Automation (Cat. No. 98CH36146), volume 1, pages 735- IEBE, 1998.
[11] L. Han, Y.-S. Guan, Z. Li, Q. Shi, and C. Trinkle. Dextrous manipulation with rolling contacts. In Proceedings of International Conference on Roboties and Automation, volume 2, pages 992-997. IEEE, 1997.
[12] A. Bicchi and Sorrentino. Dexterous manipulation through rolling. In Proceedings of 1995 IEEE International Conference on Robotics and Automation, volume 1, pages 452-457. IEEE, 1995.
[13] D. Rus. In-hand dexterous manipulation of piecewise-smooth 3-d objects. The International Journal of Robotics Research, 18(4)3355-381 1999.
[14] R. Fearing. Implementing a force strategy for object re-orientation. In Proceedings. 1986 IEEE International Conference Robotics and Automation, volume 3, pages 96-102 IEEE, 1986.

---

[15] S. Leveroni and K. Salisbury. Reorienting objects with a robot hand using grasp gaits. In Robotics Research: The Seventh International Symposium, pages 39-51. Springer, 1996.
[16] R. Platt, A. H. Fagg, and R. A. Grupen. Manipulation gaits: Sequences of grasp control tasks. In IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004, volume 1, pages 801-806. IEEE, 2004.
[17] J.PP. Saut, A. Sahbani, S. El-Khoury, and V. Perdereau. Dexterous manipulation planning using probabilistic roadmaps in continuous grasp subspaces. In 2007 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 2907-2912. IEFE, 2007.
[18] Y. Bai and C. K. Liu. Dexterous manipulation using both palm and fingers. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 1560-1565. IEEE, 2014.
[19] Shi, J. Z. Woodruff, P. B. Umbanhowar, and M. Lynch. Dynamic in-hand sliding manip- ulation. IEEE Transactions on Robotics, (374)7778- 2017.
[20] C. B. Teeple, B. Aktaş, M. C. Yuen, G. R. Kim, R. D. Howe, and R. J. Wood. Controlling palm-object interactions via friction for enhanced in-hand manipulation. IEEE Robotics and Automation Letters, 7(2):2258-2265 2022.
[21] Y. Fan, W. Gao, W. Chen, and M. Tomizuka. Real-time finger gaits planning for dexterous manipulation. IFACC-PpersOnnL.nee 50()):1776-12772, 2017.
[22] B. Sundaralingam and T. Hermans. Geometric in-hand regrasp planning: Alternating opti- mization of finger gaits and in-grasp manipulation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 231 IEEE, 2018.
[23] A. S. Morgan, K. Hang, B. Wen, K. Bekris, and A. M. Dollar. Complex in-hand manipu- lation via compliance-enabled finger gaiting and multi-modal planning. IEEE Robotics and Automation Letters, (122:4821-4822 2022.
[24] Khadivar and A. Billard. Adaptive fingers coordination for robust grasp and in-hand manip- ulation under disturbances and unknown dynamics. IEEE Transactions Robotics, 2023.
[25] X. Gao, K. Yao, F. Khadivar, and A. Billard. Real-time motion planning for in-hand manipu- lation with a multi-fingered hand. arXiv preprint arXiv:2309.06955 2023.
[26] T. Pang, H. T. Suh, L. Yang, and R. Tedrake. Global planning for contact-rich manipulation via local smoothing of quasi-dynamil contact models. IEEE Transactions on Robotics, 2023.
[27] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMLR, 2020.
[28] B. Huang, Y. Chen, T. Wang, Y. Qin, Y. Yang. N. Atanasov, and X. Wang. Dynamic handover: Throw and catch with bimanual hands. arXiv preprint arXiv:2309. 05655,, 2023.
[29] A. Allshire, M. Mittal, V. Lodaya, V. Makoviychuk, D. Makoviichuk, F. Widmaier, M. Wüthrich, Bauer, A. Handa, and Garg. Transferring dexterous manipulation from gpu simulation to a remote real-world trifinger. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11 11802-1111 1189. IEEE, 2022.
[30] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 5977-5984. IEEE, 2023.
[31] S. Suresh, H. Qi, T. Wu, T. Fan, L. Pineda, M. Lambeta, J. Malik, M. Kalakrishnan, R. Ca- landra, M. Kaess, et al. Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation. arXiv preprint 33X44:2312.13469, 2023.
10

---

[32] Z.-H. Yin, B. Huang, Y. Qin, Q. Chen, and X. Wang. Rotating without seeing: Towards in-hand dexterity through touch. arXiv preprint arXiv:2303. 200::2003308880 2023.
[33] L. Sievers, J. Pitz, and B. Bäuml. Learning purely tactile in-hand manipulation with torque- controlled hand. In 2022 International Conference on Robotics and Automation (ICRA), pages 2745-2751. IEEE, 2022.
[34] L. Röstel, J. Pitz, L. Sievers, and B. Bäuml. Estimator-coupled reinforcement learning for robust purely tactile in-hand manipulation. In 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), pages 1-8. IEEE, 2023.
[35] J. Pitz, L. Röstel, L. Sievers, and B. Bäuml. Dextrous tactile in-hand manipulation using modular reinforcement learning architecture. arXiv preprint arXiv:2303. 2023.
[36] W. Yuan, S. Dong, and E. H. Adelson. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force. Sensors, 17(12):2762, Dec. 2017.
[37] B. Ward-Cherrier, N. Pestell, L. Cramphorn, B. Winstone, M. E. Giannaccini, J. Rossiter, and N. F. Lepora. The tactip family: Soft optical tactile sensors with 3d-printed biomimetic morphologies. Soft robotics, 5(2):216-227, 2018.
[38] M. Lambeta, P.-W. Chou, S. Tian, B. Yang. Maloon, V. Most, Stroud, R. Santos, A. Byagowi, G. Kammerer, al. Digit: A novel design for low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3):3838-388 3845, 2020.
[39] N. F. Lepora. Y. Lin, B. Money-Coomes, and J. Lloyd. Digitac: A digit-tactip hybrid tactile sensor for comparing low-cost high-resolution robot touch. IEEE Robotics and Automation Letters, 7(4):9382-99) 9988,, 2022.
[40] P. P. Daniel F. Gomes and S. Luo. Generation of gelsight tactile images for sim2real learning. IEEE Robotics and Automation Letters, 6(2):417741184, Apr. 2021.
[41] S. Wang. M. Lambeta, P.-W. Chou, and R. Calandra. Tacto: fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors. IEEE Robotics and Automation Letters, 7(2):3930-3977, 2022.
[42] Z. Si and W. Yuan. Taxim: An example-based simulation model for gelsight tactile sensors. IEEE Robotics and Automation Letters, pages 2361-2368, 2022.
[43] Z. Chen, S. Zhang, S. Luo, Sun, and Fang. Tacchi: A pluggable and low computational cost elastomer deformation simulator for optical tactile sensors. IEEE Robotics and Automa- fion Letters, 8()):1239-1246, 2023. 1oi:10.1
[44] A. Church, Lloyd, R. Hadsell, and N. Lepora. Tactile Sim-to-Real Policy Transfer via Real- to-Sim Image Translation. In Proceedings of the 5th Conference on Robot Learning, pages 1-9. PMLR, Oct. 2021.
[45] T. Jianu, D. F. Gomes, and S. Luo. Reducing tactile sim2real domain gaps via deep texture generation networks. arXiv preprint arXiv:21 2021.
[46] W. Chen, Y. Xu, Z. Chen, P. Zeng, R. Dang, R. Chen, and Xu. Bidirectional sim-to-real transfer for gelsight tactile sensors with cyclegan. IEEE Robotics and Automation Letters, (3):6187-6194, 2022.
[47] Xu, S. Kim. T. Chen, A. R. Garcia, P. Agrawal, W. Matusik, and S. Sueda. Efficient tactile simulation with differentiability for robotic manipulation. In Proceedings of The 6th Confer- ence on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1488-1498. PMLR, 14-18 Dec 2023. URL https://proceeddngg/ mlr, press/v205/ xu23b. html.
11

---

[48] Q. K. Luu, N. H. Nguyen, et al. Simulation, learning, and application of vision-based tactile sensing at large scale. IEEE Transactions on Robotics, 2023.
[49] Y. Lin, J. Lloyd, A. Church, and N. Lepora. Tactile gym 2.0: Sim-to-real deep reinforcement learning for comparing low-cost high-resolution robot touch. volume 7 of Proceedings of Machine Learning Research, pages 10754-10761. IEEE, August 2022. doi:10. 109/LRA. 2022.3195195. URL htw.h:tt:////tttm /ttte///eeeeeeeeeeeeeeee ieee
[50] Y. Lin, A. Church, M. Yang, H. Li, Lloyd, D. Zhang, and N. Lepora. Bi-touch: Bi- manual tactile manipulation with sim-to-real deep reinforcement learning. IEEE Robotics and Automation Letters, 2023.
[51] M. Yang, Y. Lin, Church, J. Lloyd, D. Zhang, D. A. Barton, and N. F. Lepora. Sim-to-real model-based and model-free deep reinforcement learning for tactile pushing. IEEE Robotics and Automation Letters, 2023.
[52] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1ww.fv:1007.06.... 2017.
[53] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physies simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
[54] S. Brahmbbatt, C. Ham, C. C. Kemp, and J. Hays. Contactdb: Analyzing and predicting grasp contact via thermal imaging. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7709-8719, 2019.
[55] N. Hansen, Y. Akimoto, and P. Baudis. 001:10.528/22cmoo..555566.. Feb. 2019. URL https://doi. htt////ii.or///..///////eeee 2559634.
CMA-ES/pycma
Github.
Zenodo,
[56] D. Makoviichuk and V. Makoviychuk. rl-games: A high-performance framework for rein- forcement learning. https://ggittub..commeeee www.///eya//rrr.ggaaia..... May 2021.
[57] A. Kumar, Z. Fu, D. Pathak, and Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.040044 2021.
[58] N. F. Lepora. Soft Biomimetic Optical Tactile Sensing With the TacTip: A Review. IEEE Sensors Journal, 21(19):21331-2114 Oct. 2021. ISSN 1530-437X, 1558-1748, 2379-9153. doi: 10.11..///JEENN 20211/SEN.222133333 2021.310021442
[59] G. Bradski. The OpenCV Library. Dr. Dobb's 's Journal of Software Tools, 2000.
12

---

A Observations and Privileged Information
The full list of real-world observations o and privileged information xa used for the agent is pre- sented in Tables 4 and respectively. The proprioception and tactile dimensions are in multiples of four, representing four fingers.
Name
Symbol
Dimensions
Proprioception
Name
Symbol Dimensions
Joint Position
16
Object Information
Fingertip Position
f"
12
Position
Po
3
Fingertip Orientation
f
16
Orientation
To
4
Previous Action
Q
Angular Velocity
Wr
3
Target Joint Positions
16
Dimensions
dimn,
2
Tactile
Center of Mass
COM.
3
Binary Contact
Mass
Mo
1
Contact Pose
P
Gravity Vector
9
3
Contact Force Magnitude
Auxiliary Goal Information
Task
Position
Ps
3
Target Rotation Axis
Orientation
r9
4
Table 4: Full list of observations on avail-
Table 5: Full list of privileged information 22 only available simulation.
able in simulation and the real world used for teacher and student policy.
The privileged information is used to train the teacher with RL and for obtaining the target latent vector 2 during student training. Whilst the gravity vector can be inferred using the end effector pose of the robot arm, did find any benefit of explicitly including this information in the policy. We suspect that using history of observation, which includes proprioception and contact forces, also implicitly infer the gravity direction.
B Reward Function
B.1 Base Reward
We use the following reward function for learning multi-axis in-hand object rotation:
Frotation
Fcontact
Tstable
Pterminate:
where,
Frotation Akp ArotTrot Agoal"ggoal
Tcontact Arew (gpoTgo \borba),
Tstable =
Arew (woT + AposeTpose + AworkTwork + )torqueTrorque)) Pterminate =
Apenalty pennaltyTpenatty penalty
The key terms for defining the object rotation task are Trotation and Tpenalty. We include contact terms to encourage fingertip dexterity and tactile sensing. We also include various stability terms commonly used in previous work [8, 32] to obtain natural-looking policies and aid the sim-to-real transfer. In the following, we explicitly define each term of the reward function.
Keypoint Distance Reward:
dkp
Tkp (eax + e0oz)
where the keypoint distance kpaist 1/2 object and goal respectively. We use N = 6 keypoints placed 5 cm from the object origin in each of its principle axes, and the parameters = 50, 2.0.
Σin |k k os and 69 keypoint positions of the
13

---

Rotation Reward:
rot = clip(A@.... (4)
(4)
The rotation reward represents the change in object rotation about the target rotation axis. We clip this reward in the limit Cj 0.025rad.
Goal Bonus Reward:
[ kpdist <
diol
Tgoal
0
otherwise
(5)
where we use keypoint distance tolerance deal to determine when a goal has been reached.
Good Contact Reward:
{1
if up contact 2 2
"ge
otherwise
(
where up contact sum(c). This rewards the agent if the number of tip contacts is greater or equal encourage stable grasping contacts.
Bad Contact Penalty:
{ if Poon too.ttip.commact 2 0
otherwise
where Rgeoauip.contact is defined as the sum of all contacts with the object that is not fingertip. We accumulate all the contacts in the simulation to calculate this.
Angular Velocity Penality:
Tw min([woll Wmax. 0)
where the maximum angular velocity Wmax = 0.6. This term penalises the agent the angular velocity of the object exceeds the maximum.
Pose Penalty:
posee = -111 goll
(9)
where is the joint positions for some canonical grasping pose.
Work Penalty:
Twork -
(10)
Torque Penalty:
Twork
-||T||
(11)
where in the above is the torque applied the joints during actioned step.
Termination Penalty:
--
(k Pdist > dmax) or (Ro > kmax)
T Tterminate
(12)
0
otherwise
Here we define two conditions to signify the termination of an episode. The first condition represents the object falling of grasp, for which we use the maximum keypoint distance of dmax 0.1.
The second condition represents the deviation of the object rotation axis from the target rotation axis (k.)) beyond maximum kmax. We use kinax = 45%
The corresponding weights for each reward term is: Akp = 1.0, Apot = 5.0, Agoal = 10.0, Age = 0.1, Abc 0.2, A = 0.5, Apose = 0.5, Awork 0.1, Atorque 0.05, Apenalty 50.0.
14

---

B.2 Alternative Reward
We also formulate alternative reward function consisting of an angular velocity reward and rota- tion axis penalty to compare with our auxiliary goal formulation.
Angular Velocity Reward:
Pav = clip(w k -22, 02)
(13)
where = 0.5.
Rotation Axis Penalty:
k- Ro
Taxis 1
(14)
||4] |||1||ba1
where ||ko/| is the current object rotation axis.
We form the new Frotation reward Frotation Aav + rotTrot- We provide an additional object axis penalty TaxisTaxis in the Tstable term and remove the angular velocity penalty, A 0. The weights are Àav 1.5 and axis = 1.0. We keep all other terms of the reward function the same.
B.3 Adaptive Reward Curriculum
The adaptive reward curriculum is implemented using linear schedule of the reward curriculum coefficient Xrew (f /oonttct + (ssable) which increases with successive goals are reached per episode,
Jeval
Imin
Arew
(15)
Imax
Imin
where [gmin 9max] determines the ranges where the reward curriculum is active. This changes the learning objective towards more realistic finger-gaiting motions as the contact and stability reward increases. We use [ymin, [man] = [1.0, 2.0].
C Grasp Generation
To generate stable grasps, we initiate the object at 13cm above the base of the hand at random orientations and initialize the hand at canonical grasp pose the palm-up hand orientation. We then sample relative offset the joint positions 4(30.3, 0.3) rad. We run the simulation by 120 steps (6 seconds) while sequentially changing the gravity direction from 6 principle axes of the hand (ixyy--axes). We save the object orientation and joint positions (10000 grasp poses per object) if the following conditions are satisfied:
The number of tip contacts is greater than 2.
The number of non-tip contacts is zero
Total fingertip object distance is less than 0.2
Object remains stable for the duration of the episode.
D System Identification
To reduce the sim-to-real gap of the allegro hand, we perform system identification to match the simulated robot hand with the real hand. We model each of the 16 DoF of the hand with the pa- rameters; stiffness, damping, mass, friction, and armature, resulting in total of 80 parameters to optimize. We collect corresponding trajectories in simulation and the real world in various hand orientations and use CMA-ES [55] to minimize the rrean-suuared error of the trajectories to find the best matching simulation parameters.
15

---

E Domain Randomization
In addition to the initial grasping pose, target rotation axis and hand orientation, we also include additional domain randomization during teacher and student training to improve sim-to-real perfor- mance (shown in Table 6).
Object
Hand
Capsule Radius (m)
[0.025, 0.034]
Controller: Stiffnesss
x 1.1)
Capsule Width (m)
[0.000, 0.012]
PD Controller: Damping
x
Box Width (m)
[0.045, 0.06]
Observation: Joint Noise
0.03
Box Height (m)
[0.045, 0.06]
Observation: Fingertip Position Noise
0.005
Mass (kg)
[0.025, 0.20]
Observation: Fingertip Orientation Noise
0.01
Object: Friction
10.0
Hand: Friction
10.0
Tactile
Center of Mass (m)
[-0.01, 0.01]
Observation: Pose Noise
0.0174
Disturbance: Scale
2.0
Observation: Force Noisc
0.1
Disturbance: Probability
0.25
Disturbance: Decay
0.99
Table 6: Domain randomization parameters.
F
Simulated Tactile Processing
To simulate our soft tactile sensor in rigid body simulator, we process the received contact infor- mation from the simulator to make up the tactile observations. We use contact force information to compute binary contact signals:
c= if F > 0.25 N; otherwise)
(16)
A contact force threshold of 0.25 N was selected to simulate the binary contact detection of the real sensor. For contact force information, we simulate sensing delay caused by elastic deformation of soft tip in the real world by applying an exponential average on the received force readings.
a + (1 - Ft-1
(17)
We use a 0.5. We then apply a saturation limit and re-scaling to align simulated contact force sensing ranges with the ranges experienced in the real world.
= 3pclip(F, Fmin, Fmax)
(18)
We use Br = 0.6, Fmin 0.0N, Fmax 5.0N. We also apply the same saturation and rescaling factor for the contact pose.
Pp. Pmin. Pmax)
(19)
We use Bp = 0.6, Pmin = -0.53 rad, Pmax = 0.53 rad. We use binary contact signals to mask contact pose and contact force observations to minimize noise in the tactile feedback. The same masking technique was applied in the real world.
G Architecture and Policy Training
The network architecture and training hyperparameters are shown in Table The proprioception policy uses an observation input dimension of N 79, the binary touch N 83, and the full touch N 95. We use history of 30 time steps input the temporal convolutional network (TCN) and encode the privileged information into latent vector of size for all the policies.
16

---

Teacher
Student
MLP Imput Dim
18
TCN Input Dim
[30, N]
MLP Hidden Units
[256, 128, 8]
TCN Hidden Units
IN. N
MLP Activation
ReLU
TCN Filters
NJ
Policy Hidden Units
[512, 256, 128]
TCN Kernel
[9. 5, 5]
Policy Activation
ELU
TCN Stride
[2. 1, I]
Learning Rate
5 10
TCN Activation
ReLU
Num Envs
8192
Latent Vector Dim
8
Rollout Steps
8
Policy Hidden Units
[512, 256, 128]
Minibatch Size
32768
Policy Activation
ELU
Num Mini Epochs
5
Learning Rate
3 10
Discount
0.99
Num Envs
8192
G
0.95
Batch Size
8192
Advantage Clip
0.2
Num Mini Epochs
KL Threshold
0.02
Optimizer
Adam 0.25
Gradient Norm
1.0
Goal Update deal
Optimizer
Adam
Goal Update deo)
0.15
Table 7: Policy training parameters. Please refer to ref. [56] and [57] for a detailed explanation of each hyperparameter.
H
Tactile Perception Model
Data Collection. The setup for tactile feature extraction is shown in Figure 8. We collect data by tapping and shearing the sensor on a flat stimulus fixed onto force torque sensor and collect six labels for training: contact depth z, contact pose Rg contact pose Ry. and contact forces Fg. Fg and Fz. order to capture sufficient contact features needed for the in-hand object rotation task and stay within the contact distribution, we sample the sensor poses with the ranges shown in Table 8. This provides sensing ranges for contact pose between [-280, 28 and contact force of up to 5 N, which are the largest ranges we could reasonably consider for this tactile sensor.
Training. The architecture and training parameters of the perception model are shown in Table 9. For each fingertip sensor, we collect 3000 images (2400 train and 600 test) and train separate models. The prediction error for one of the sensors is shown Figure The perception model does not explicitly consider multiple contact points. In practice, found this to be rare and by assuming single combined contact, the model was sufficient in producing consistent estimates that did not affect the final performance of the in-hand rotation policy.
Tactile Image Processing
The tactile sensors provide raw RGB images from the camera module. We use an exposure setting of 312.5 and a resolution 640 x 480, providing frame rate of up to 30 FPS. The images are then postprocessed. We convert the raw image to greyscale and resale the dimension to 240 135.
Binary Contact: We apply a medium blur filter with an aperture linear size of 11, followed by an adaptive threshold with a block size of 55 pixels and constant offset value of -2. These operations improve the smoothness of the image and filter out noise. The postprocessed image is compared with reference image using the Structural Similarity Index (SSIM) to compute binary contact (0 or 1). We use an SSIM threshold of 0.6 for contact detection.
Contact Pose and Force: We directly use the resized greyscale image for contact force and pose prediction. From the target labels, we use contact pose (R.,, R,y) and the contact force components Fz. Fy. F. (to compute the contact force magnitude [[/')) construct the dense tactile representa- tion used during policy training. We use the binary contact signal to mask contact pose and force, thresholding the predictions at 0.25N.
17

---

Pose Component
Sampled range
Depth (mm)
[-1, -4]
Shear Sa (mm)
[-2, 22]
Shear Sy (mm)
[-2, -2]
Rotation R& (deg)
[-28, 28]
Rotation R2 (deg)
[-28, 281
Table 8: Sensor pose sampling ranges used during tactile data collection for training the pose and force prediction models, relative the sensor coordinate frame.
URS Robot Arm
Tactile Sensor
Tactile Perception Model
Conv Input Dim
[240, 135]
Conv Filters
[32, 32, 32, 32]
Sensor
Conv Kernel
111,9,755
Figure 8: Data collection setup for training tactile perception model, including an F/T sensor and URS Robot arm.
Cony Stride
1,
Max Pooling Kermal
[2, 2, 2| [2. 2,
Max Pooling Stride
Output Dim
Batch Normalization
True
Activation
ReLU
Learning Rate
x 10
Batch Size
16
VAC 35X
Num Epochs
100
Optimizer
Adam
Table 9: Tactile perception model training parameters.
Figure Error plots on test data for the per- ception model.
J Real-world Deployment
Tactile Sensor Design. This design of the sensor is based on the DigiTac version [39] of the Tac- Tip [37, 58], a soft optical tactile sensor that provides contact information through marker-tipped pin motion under its sensing surface. Here, have redesigned the DIGIT base be compact with new PCB board, modular camera and lighting system (Figure 10). We also improved the mor- phology of the skin and base connector to provide larger and smoother sensing surface for greater fingertip dexterity. The tactile sensor skin and base are entirely 3D printed with Agilus 30 for skin and vero-series for the markers on the pin-tips and for the casings. Each base contains a camera
Deiver Roard
Tirpe
Trayd keid Oommanck
Prcabosker
Polky
Midole Shell
Camera
Fawced Conraria
Engerri poss
Cossanaa
Hofomatton
Figure 10: CAD models of the fully-actuated (Allegro) robot hand and integrated custom tactile sensors.
Figure 11: pipeline.
Real-world robot hand control
18

---

weal-world Object Set
Dimensions (mm)
Mass (g)
Dimensions (mm)
Mass (g)
Apple
75 x 75 x 70
60
Tin Cylinder
45 x 45 x 63
30
Orange
70 x 72 72
52
Cube
51 51 51
65
Pepper
61 68 65
10
Gum Box
90 80 76
89
Peach
62 x x 55
30
Container
x 80
Lemon
52 x 52 x 65
33
Rubber Toy
80 x 53 x 48
27
Table 10: Dimensions and mass of real-world everyday objects.
driver board that connects to the computer via USB cable and can be streamed asynchronously at a frame rate of 30 FPS. We perform post- post-processing using OpenCV [59] real-time.
Sensor Placement. A common limitation of unidirectional tactile sensors is that they are primarily sensorized over font-faccng area. Contacts with the side of the sensor casing be slippery and result in unstable behaviors. To alleviate this issue, similar to [31], we adjusted the sensor direction relative to the fingers to maximize contact with the sensing surface, and placed the tactile fingertips with offsets (thumb, index, middle, ring) (-45%, -45%, 0% 45%). This allowed the policies to achieve consistent and stable in-hand object rotation performance, providing a basis to validate our learning approach against baselines.
Control Pipeline. Each tactile perception model is deployed together with the policy shown in Figure 11. We stream tactile and proprioception readings asynchronously at 20 Hz. The joint positions are used by a forward kinematic solver to compute fingertip position and orientation. The relative joint positions obtained from the policy are converted to target joint commands. This is published to the Allegro Hand and converted to torque commands by a PD controller at 300 Hz.
Object Properties. Various physical properties of the objects used in the real-world experiment shown in Table 10. We include objects of different sizes and shapes not seen during training.
K
Additional Experiments
K.1
Hyperparamters
We provide additional ablation studies to analyze the design choices for our axillary goal formula- tion. The effect of goal update tolerance diat for the student training and the auxiliary goal increment intervals are shown Table
The performance can be significantly affected by the goal-update tolerance. As the tolerance reduced during student training, the number of average rotations and successive goals reached episode also reduced. This suggests that the performance of the teacher policy was poorly transferred and the student could not learn the multi-axis object rotation skill effectively. Increasing the goal increment intervals also resulted fewer rotations achieved.
Goal Update Tolerance
Rot
TTT(s)
#Success
Goal Increment
Rot
TTT(s)
#Success
deal 0.15
0.75
28.1
3.07
30
1.77
27.2
5.26
deal 0.20
1.36
27.7
4.48
0 40
1.50
26.7
4.36
deol = 0.25
1.77
27.2
5.26
0 50
1.30
27.1
3.86
We compare the performance of policies trained with different design choices in the auxiliary goal formulation. We compare goal update tolerance and goal increment intervals and provide metrics for average successive goals reached, rotation count (Rot), and time to terminate (TTT).
Table 11:
19

---

Timo
wwwmma
Figure 12: Simulated tactile readings during pol- icy rollout. We plot raw and processed contact pose and contact force readings in simulation for rotating ball in the palm-up orientation about the 7-axis.
Figure 13: Real tactile readings during policy rollout. We plot the contact pose and con- tact force predictions from the tactile perception model for rotating a ball in the palm-up orienta- tion about the z-axis.
K.2 Sim-to-Real Tactile Sensing
We present the tactile readings of a similar policy rollout in simulation and real-world, in Figure 12 and 13 respectively. We observed a sim-to-real gap in the rotation speed as demonstrated by the higher contact cycles obtained in simulation. However, matching pattern can be seen from the recorded contact features. By comparing the raw and processed contact readings in simulation, the effect of the post-processing functions in Section F. This helped with aligning the simu- lated tactile readings to that of the real sensors and smoothing out the noisy readings of the contact force. The comparison also demonstrates a successful sim-to-real transfer of the proposed tactile representation.
K.3 Real-world Object Results
The real-world results for each object for varying rotation axes and hand orientations are shown in Figure 13 and 12 respectively. We observed that larger objects resulted in fewer rotations, likely due to the size and joint limits of the Allegro Hand, making smaller objects easier maneuver. Objects with sharp corners, such the cube and gum box, sometimes caused the fingers to get stuck around these points. We believe that this is because navigating around these geometric features requires additional finger extension during rotation, making it challenging for a general tactile policy to handle such shapes effectively. The gum box was the most challenging due to its sharp corners and shifting mass (sloshing gum pieces). These factors led to least stable rotation and the lowest time terminate (TTT).
Contact Surfaces. While we train the tactile perception models flat surface, found that it can generalize to other uneven contact surfaces, shown by various test object shapes in Figure 4, demonstrating the robustness of the proposed tactile representation.
K.4 Rotating Hand
We test the robustness of the policy by performing in-hand object rotation during different hand movements, particular, choose hand trajectories where the gravity vector continuously changing relative the orientation of the hand, adding greater complexity this task. Rollouts for three different hand trajectories are shown in Figure 14. In particular, for the third hand trajec- tory (iii), we demonstrate the capability of the robot hand to servo around the surface of the object different directions while keeping the object almost stationary in free space. This motion also demonstrates the ability to command different target rotation axes during deployment, offering a useful set of primitives for other downstream tasks.
20

---

Tucfile
Apple
Orange
Pepper
TTT())
Rot
TTT()
Rot
ITT(s)
Rot
TTT(s)
TTT(s)
Propricception
25.5_10.1
1.21
25.3
1.51 10.33
28.8
1.54 26.8.a.
1.1
20.3ga.
Binary Touch
1.21
27.7=4.4
1.20
26.2
1.25
243+6.8
1.00
23.2=4.0
0.80g
19.5
Dense Touch (Ours)
1.37 30.0=o.0
1.542o.ss
30.0zo.0
1.50+0 20
30.0
1.8940.25 30.0
1.57
29.5
Tactile Observation
Tin Cylinder
Cube
Gum Box
Container
Rubber
TTT()
TTTO)
TTT68
Propriaception
0.48
17.0-115
0.80
19.0g120
0.57
19.2
a.36 17.7-m
0.49 17.7g68
Binary Touch
16.8
0.5810.00
16.819.1
0.4810.34
13.8
0.59
28.7ia.
0.65gous
21.0g7
Dense Touch (Ours)
28.32a.
0.88
23.0 0.83
24.2
0.59 27.8.aa
1.06 28.3
Table 12: Hand orientation. Real-world results of policies trained on different tactile observations for different objects. We report on average rotation count (Rot) and time to terminate (TTT) per episode averaged over the test hand orientations.
Observation
Apple
Orange
Pepper
TTT()
Rot
TTT()
Ret
TTT(s)
Rot
TTT(s)
TTT(s)
Proprioception
0.5 20
23.3
0.68
21.7
14.7s11
1.00e041
28.7=1.9
0.6840.4
18. 7b9.
Binary Touch
0.78
28.3-24
0,90+
23.0a9.9
0,78+0
25.3e3.s
0,92e042
25.0-4.1
0.88
20.7er.4
Touch
1.03+a. se 30.0=a.e
1.20
30.0+0.0
1.17
30.0+o.0
1.82+0.4
30.0+o.0
1.70+0.39
30.0+0.0
Thctile Observation
Tin Cylinder
Cube
Gum Box
Container
Rubher Toy
TTT()
TTT(s)
TTT()
TTT())
Propriaception
1.28 25 10.0_aa
0.631044
20.31103
0.4710.6
7.33
0.10
6.00_aa a..
0.35
14.
Binary Touch
0.49
19.3=29
0.774020
26.744.7
0.42404
14.7x107
0.21e021
16.7-125
0.47e041
15.7e15.
Dense Touch (Ours)
29.7
1.04
0.65
18.31181
0.42
25.01za
25.7
Table 13: Rotation-axis. Real-world results of policies trained different tactile observations for different objects. We report on average rotation count (Rot) and time to terminate (TTT) per episode averaged over the test rotation axes.
Hand rotation
ii) Hand rotation x-axis
iij) Hand rotation about object pose
Time
Figure 14: Examples of in-hand object rotation for plastic apple on moving hand. Rollouts for three hand trajectories are shown: (i) object rotation about z-axis while the hand rotates about the z-axis from to 2m; (ii) object rotation about z-axis while the hand rotates about the r-axis from π to ग; (iii) object rotation about the y-axis while the hand rotates about the y-axis the opposite direction to keep the object pose stationary.
21


Reviews:
[{"review_id": "LBnD8DID6g", "reviewer": "Reviewer_mNc5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a pipeline for sim-to-real in-hand object manipulation when the hand orientation is randomized. The authors train a tactile perception model that can map tactile images to contact pose and forces. Using this representation, the authors train a policy in simulation and transfer it to the real-world.", "summary_of_recommendation": "Overall, I think this is a well-written paper with good results. There are several questions that need to be clarified, but I still recommend acceptance because of the results.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Why is the key point-based auxiliary reward better than the rotation velocity reward?\n\nHow should we perceive or understand the reactive behavior from Figure 7? For example, what is the unique behavior for the shaded curve, and are the corresponding videos are in the supplementary material?\n\nHow important is the gravity direction as an input? In the student training, the gravity direction can also be inferred from the arm’s pose. Have you considered adding this into student policy?\n\nHow different are these tactile sensors? Do you need to train a force prediction model for each sensor? Or how do you calibrate between them. In addition, what about the durability of the sensors? How long can the force prediction model be used?\n\nFor the dense touch feature data collection, how large can the contact region be? What happens when there are multiple points of contact?\n\nThe reward function has quite a few terms. Could the authors comment on the usefulness of them? Which terms are the most critical?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BxIPjeEAkf", "reviewer": "Reviewer_Si75", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a system that leverages dense tactile sensing and reinforcement learning to perform object rotation about arbitrary axes in any hand orientation relative to gravity. They first train a policy using teacher-student policy distillation given rich tactile feedback in simulation, then utilize an observation model trained on contact data to perform zero-shot policy transfer to the real world. They evaluate the system using a 4-fingered 16-DoF tactile robot hand in both simulation and real-world experiments, based on rotations per episode and rotation time until failure.", "summary_of_recommendation": "The authors present a system that is relevant and novel, they provide detailed information that helps the readers to understand the system and potentially reproduce it. The system is well grounded and thoroughly tested in both simulation and real-world experiments. The authors might further improve this paper by giving some insights on what external factors affect the system performance. Overall, this is a good paper and I suggest acceptance.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. As the results are presented by taking the average of all tested objects, are there any cases where the baselines outperform the proposed method?\n2. The policy is trained using Isaac gym fundamental geometric shapes including boxes, yet the system still has difficulties with box-shaped objects. “Grasping points producing similar tactile information for different states” is a reasonable cause, but is it feasible to replicate this problem in simulation by tuning the simulated touch reading and get a better trained policy?\n3. The figures can be improved. As the hand is all black and the fingers look almost the same, it would be helpful to annotate the axes in the figures to help readers understand why rotation axes affect the performance.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "luo61YahZw", "reviewer": "Reviewer_9rrD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a system designed for gravity-invariant multi-axis in-hand object rotation using a tactile robot hand with rich tactile feedback. The system is trained in simulation and employs zero-shot policy transfer to handle real-world tasks without additional retraining.  Key contributions include:  1. An RL framework for learning a unified policy for multi-axis in-hand object rotation. 2. A dense tactile representation that enhances the handling of objects with different properties. 3. Successful zero-shot sim-to-real policy transfer validated on diverse objects.", "summary_of_recommendation": "Weak accept. The paper has contribution on the system engineering with both hardware and learning contribution, but lack of major contribution. I recommend weak reject the paper, but may adjust the score if the issues are well addressed.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "# Limitation:\n\nMinor Limitation:\n\n1. The method described in the paper is similar to the approach used in \"Visual Dexterity: In-Hand Dexterous Manipulation from Depth\" for teaching the pipeline. The main difference is that the authors of the current paper incorporate tactile modalities in the training process.\n2. For the hardware design, it would be beneficial if the authors explained the rationale behind choosing different orientations for the tactile sensors on various fingers. Additionally, conducting experiments to compare the benefits of having sensors oriented in different directions versus having all sensors aligned in the same direction would provide valuable insight into the impact of sensor orientation on the system’s performance\n\nMajor Limitation:\n\n1. While the dense tactile sensor is commendable at the hardware level, its installation in the hand reveals a potential issue. The authors manually adjusted the sensor direction to face the object, thereby increasing the contact area between the objects and the sensor (as mentioned in Appendix J, lines 587-590). Although this design choice is intuitive and aimed at simplifying the task, it limits the system's generalizability. By optimizing the sensor orientation for this specific task, the hand's versatility for other tasks, even similar rotation tasks with different motions, may be compromised. Therefore, this approach may not serve as a general framework for in-hand rotation tasks. I agree that the digit sensor might not have any signal on the sides of cases, which may be the major reason behind this design. However, I still believe the framework could be more generalizable if the authors had placed all the sensors in the same direction and successfully completed the task. \n2. The tactile sensing simulation in the paper raises some questions. The authors treat the sensor as a rigid body to fetch contact information in the simulation. For example, in reality, the contact position will move when the force increases, but this cannot happen in a simulation using a rigid body model. Additionally, it would be beneficial if the authors described how they simulate the delay in contact force response.\n3. I assume the authors used Isaac Gym's contact simulation to fetch the contact information without using any external package. It would be great if the authors demonstrated how to obtain contact information in Isaac Gym and visualized both the raw contact data and the processed data. Obtaining accurate contact information in Isaac Gym is challenging, particularly because the contact position can be inaccurate. The study would be much improved if the authors visualized similar contact scenes in both simulation and the real world to compare the tactile representations.\n4. In lines 11-13, the authors state, \"we found rich multi-fingered tactile sensing can implicitly detect object movement within grasp and provide a reactive behavior that improves the robustness of the policy.\" This claim needs experimental verification to demonstrate how object movement within the grasp is implicitly detected and how this detection leads to reactive behavior.\n5. The task presented is similar to those in previous papers listed in the references, such as \"In-Hand Object Rotation via Rapid Motor Adaptation.\" It would be more impactful if the paper demonstrated enhanced dexterous skills through the use of tactile modalities rather than focusing solely on achieving gravity-invariance. For gravity-invariance, the authors use a relatively simple technique of randomizing initial hand orientations between episodes, which may not be considered a major contribution. It would be beneficial if the authors highlighted how tactile information specifically aids the task, particularly given their significant effort in tactile sim-to-real transfer.\n6. The simulation uses a single contact point for tactile information (as visualized in Figure 3), whereas the real-world tactile representation involves a contact area. Bridging the sim-to-real gap between these differing sensing patterns is a key challenge. The paper should address how this discrepancy is managed and how the simulation results translate effectively to the real-world tactile data.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper makes good advance in the topic of in-hand manipulation with tactile input and achieved good results. The authors should address the clarification questions posted by the reviewers.
---
During the rebuttal phase, the authors improved the paper and addressed most concerns raised by the reviewers. The reviewers all agree to accept the paper now.
Decision: Accept