General In-hand Object Rotation with Vision and Touch
Authors: Haozhi Qi
Brent Yi
Sudharshan Suresh
Mike Lambeta
Yi Ma
Roberto Calandra
Jitendra Malik
Venue: CoRL 2023 Poster (2023)
Keywords: In-Hand Object Rotation
Tactile Sensing
Reinforcement Learning
Sim2Real
Transformer
Visuotactile Manipulation

Abstract: We introduce Rotateit, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our system is trained in simulation, where it has access to ground-truth object shapes and physical properties. Then we distill it to operate on realistic yet noisy simulated visuotactile and proprioceptive sensory inputs. These multimodal inputs are fused via a visuotactile transformer, enabling online inference of object shapes and physical properties during deployment. We show significant performance improvements over prior methods and highlight the importance of visual and tactile sensing.

General In-Hand Object Rotation with Vision and Touch
Haozhi Qi" 12
Brent Yi'
Sudharshan Suresh 23
Mike Lambeta² 2
Yi Ma'
Roberto Calandra 15 Jitendra Malik
UC Berkeley
M AI CCUU *TU Dresden
5 The Centre for Tactile Internet with nuuman-in-the-LLwe (CeTT)
https://haozhi. io/rotateit/
k
Hardware Setup
Raw Depth
Object Depth
Tactile Image (x4)
Rotation over multiple axes by integrating proprioception, vision, and touch sensing. Rotatelt is trained in simulation and deployed directly the real-world, where it generalizes diverse objects without the need fine-tuning. Please see our website for more videos.
Figure I:
Abstract: We introduce Rotatell, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our system is trained in simulation, where it has access to ground-truth object shapes and physical properties. Then distill it operate realistic yet noisy simulated visuotactile and proprioceptive sensory inputs. These multimodal inputs fused via visuotactile transformer, enabling online inference of object shapes and physi- cal properties during deployment. We show significant performance improvements over prior methods and the importance of visual and tactile sensing.
Keywords: In-Hand Object Rotation, Tactile Sensing, Reinforcement Learning, Rim-to-Real, Transformer, Visuotactile Manipulation
Introduction
Despite recent progress in-hand manipulation for single or few objects [1, 2, 4), generalizable object manipulation remains challenge. In this paper, we present model that integrates visual, tactile, and proprioceptive sensory inputs and achieves fingertip-based in-hand object rotation over multiple different axes. This continuous rotation task is important for achieving large-angle in-hand re-orientation skill and is challenging because requires simultaneously maintaining stable force closure for objects with diverse geometries.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Oracle Policy Training
Proprioception only
E (S(S)
Encodad Extrinsics
Privileged Encoder
Control Policy
Object Mesh
a
Oc-1
d
Physics Parameters
t
c_=1|a,
2.112
Copy
L-= |au
Aul1
(2) Visuotactile Policy Training
Vision, touch, and proprioception
Visuotactile
Control Policy
ko.,
2
O11
Transformer
An overview of our training pipeline. Trainable components are highlighted in green. In oracle policy training, we jointly optimize the privileged encader and control policy using PPO. In the visuotactile policy training. we feed sequence of visuotactile and proprioceptive inputs transformer to infer &r. The visuotactile transformer is trained by minimizing the regression loss between 2; and &
Figure 2:
Our approach draws inspiration from recent advances in training reinforcement learning policies with privileged information [5, 6, 7, 8], more specifically rapid motor adaptation [6. 7]. We first train an oracle policy that conditioned on representation of the privileged information (called extrinsics, denoted 2t, shown in Figure 2), which contains ground-truth physical properties and shapes of the objects. With access to this representation, the oracle policy is able to efficiently and stably manipulate diverse objects over multiple axes in the simulator.
An overview our method, Rotatelt, is shown in Figure 2.
The key challenge for real-world deployment lies in estimating the extrinsics encoding when privileged information is inaccessible. To address this challenge, we use multimodal sensing from vision and touch, just humans do [9, 10, 111. We implement this by designing visuotactile transformer which operates history of multimodal proprioceptive, visual, and tactile inputs infer 2,- Concretely, during training, we rollout the oracle policy in simulation and collect the foreground object depth, contact locations on the fingertips, proprioception, and action history. Then we feed these multimodal streams into a transformer to produce an estimate of the 2,, denoted as 2, The visuotactile transformer trained to minimize the difference between the predicted and estimated encodings of the privileged information. In the real-world, we get the foreground objects using Segment Anything [12, 13], which enables Rotatelt to robust to cluttered backgrounds. We use tactile images from omnidirectional vision-based touch sensor retrieve these contact locations.
We demonstrate Rotatels perform multi-axis object rotation using only its fingertips. In simulation, we quantitatively study the performance of rotating skills over three principal axes in the hand- centric frame and the impact of incorporating vision and touch at various stages (Section 5. and Section 5.2). To further understand what is learned by the policy, we investigate how accurately the latent representation of the policy captures objects' shapes by using it to recover 3D shapes (Section 5.3). Finally, we deploy the learned policies to rotate multiple different objects over multiple axes in the real world (Section 5.4). On the website, we show our policy can rotate objects, including but not limited to, the three canonical axes. Our work highlights the importance of both visual and tactile sensing in manipulation and presenting a step towards general dexterous in-hand manipulation.
Related Work
Classic Control Methods. Classical methods typically minimize a desired cost function using planning methods and a simplified system model [14, 15, 16, 17, 18, 19, 20). State-of-the-art systems in this category include full SO(3) reorientation using compliance-enabled hand [21] and an accurate pose tracker [22]. In contrast, work does not rely
object model: we instead combine multi-sensory inputs with leamning-base based policy that is trained on a large set of objects.

---

Real-Worke Learning. In-hand manipulation skills can be learned directly in the real-world, either by reinforcement learning [23, 24, 25] or imitation learning [26, 27, 28, 29). However, reinforcement learning methods usually suffer from sample efficiency and real-world environment cannot provide enough variation. In contrast, our policy is trained using reinforcement learning via GPU-accelerated simulators, and does not need any human demonstrations.
Sim-to-Reat teal Methods. OpenAT al. [1, 2] first transferred dexterous in-hand manipulation policies to the real-world. Similarly, Sievers al. [30], Pitz et al. [4] uses a torque-controlled hand for cube rotation and reorientation when the hand facing downwards. However, they focus on manipulating one single object. Although generalizable in-hand manipulation of diverse objects can be learned in simulation [31, 32, 33], transferring it to the real world remains a challenge.
Among sim-to-real methods, learning with privileged information [5. 6, 34, 35] shown to be effective for legged locomotion [5, 6] and manipulation [7, 8J. Recently, several works [7, 30, 36, 37] study generalizable in-hand object rotation using tim-to-real and reinforcement learning. In this paper, we use the same robot hand (Allegro [38]) as Qi et al. [7] and similar variety of rotated objects with the significant advance being that the use of visual and tactile information now enables to rotate the object about an arbitrary axis, not just the z-axis. Other previous work [30, 37] shared this limitation of only demonstrating rotation about the z-axis. Compared to [36], our task is more challenging as it does not utilize a supporting surface, which allows constant tactile feedback on fingertips and enables natural finger- -aiting to emerge. While we study continuous rotation (as many revolutions as possible), Chen et al. [8] study the task of object reorientation to arbitrary pose (no limitation to z-axis rotation) and obtain impressive results. There are significant differences to our approach in oracle policy training. hand hardware, and goal specification. In addition to proprioception, they only use vision to sense the object, while our visuotactile policy utilizes both vision and touch. In the experiment section, we show that both these components improve the performance of our system.
Visuotactile Sensing and Learning. Tactile sensors such as GelSight [39], TacTTip [40], DIGIT [41], DTact [42], GelTip [43], ArrayBot [44], and AllSight [45] have been used for numerous applications including grasping [46], playing the piano [47, 48], 3D reconstruction and localization [49, 50, 51,52) 52], and cup unstacking and bottle opening [53]. Previous work explores the usage of vision and touch for manipulation [54, 55, 56] but not for in-hand manipulation. To the best of our knowledge, Rotatelt is the first work that intersects visuotactile sensing and learning to achieve general in-hand object rotation with a dexterous hand.
The transformer architecture [57] was originally proposed for machine translation and later used in computer vision [58]. In robotics, there are growing attempts to incorporate it in imitation learning [59, 60, 61, 62, 63] reinforcement learning [64, 65, 66). Chen et al. [67] also use the transformer multimodal data but their tactile refers to force/torque sensing on robot joints. In contrast, our method uses the transformer for temporal modeling of multimodal proprioceptive, visual, and tactile information.
Transformers in Robotics
General In-hand Object Rotation with Vision and Touch
An overview of our method is shown in Figure 2. Our policy training consists of two stages: First, we train an oracle policy with privileged information. Next, we train visuotactile policy with realistic yet noisy observations. Both of these stages happen in simulation. In this paper, we consider privileged information to be the object physical properties and object shape information. Real-world observations comprise of stream of proprioceptive, visual, and tactile inputs. Our method trains one policy for each rotation axis and we show how to distill them into one general policy in Section A.2.
3.1 Oracle Policy Training
Privileged Information. For the object shape information, we sample Np points from the object's mesh and encode it feature vector 2 zahape with cp dimensions using PointNet [68]. One key difference from previous works [7, 8] is that explicitly encode object shape into the oracle policy, which we find to be critical especially for complex objects that are harder to manipulate.
3

---

a-axis
y-axis
z-axis
Method
RotR
1
TTF1
RotP
4
RotR 1
TIF1
RotP
RotR
1
TTF f
RotP J
Hora
[7]
79.13
0.52
2.55
g.54
0.44
99.83_mmm
0.60
a.39
Oracle
125.23
0.79
0.35 Loaz
118.26_138
0.79
0.30 1ear
140.90 11.26
0.82
0.27 16a1
w/o shape
15.10
0.56:om
0.39 2003
99.92
0.62 too4
0.41 to02
129.38
0.75
0.29 too1
Table 1: We compare the performance improvement over various baselines the rotation task over three different axes, under the same training setting. Compared to [7], we first add object and finger pose (w/o shape entry). This component slightly improves the performance. We then add object shape information into the oracle policy and this significantly improves the performance.
The physics property contains object's mass, center of mass, coefficient of friction, scale, and resti- tution, resulting in a 7-dimensional vector. The pose contains object's position, orientation (as a quaternion), and angular velocity, resulting a 10-dimensional vector. These vectors are concate- nated together and projected to an 8-dim encoding vector 2/ 2,b Our final privileged encoding is concatenated from the shape encoding and physical property encoding ,; = = xx 2ahape).
Observations and Outputs. The oracle policy æ takes the robot's proprioception and the encoded privileged information 2; as input. It outputs the targets of the PD Controller a, R 16 tion pa contains small temporal window of joint positions and actions P: qq-22: a1-3:111 R where q: 16 stands for the joint positions of the robot. Formally, we have a; T(,,, Reward Function. Our reward function is modified from [7] with an additional penalty on undesired angular velocities component:
The observa-
y = Trotr + ArotpTrotp + Apase Tppss + Atinvel Flinvel + Awork Twark + Atorgue Ttorque-
(1)
The object rotation task is defined as Trotr = max(min(w k, Tmax), Tmin) where w is the object's angular velocity and k is the desired rotation axis in the hand-centric axis. Naively applying this reward will result in unstable behaviors when rotating over and y-axis. To alleviate this problem, we add rotation penalty term Trotp" = ||ا k[l2. To make the policy stable, smooth, and energy
efficient [6. 7, 69, 70], we use a few penalty terms: Tpose - q - Qinit 112 is the hand pose deviation penalty, Ttorque - ||= is the torque penalty, Twork - - q is the energy consumption penalty, and flinvll = | 122 is the object linear velocity penalty, where Zinit be the starting robot configuration, T be the commanded torques at each timestep, and is the object's linear velocity.
Policy Optimization. We use PPO [71] to optimize the oracle policy. The weights between the policy and the critic network are shared, with an extra linear projection layer to estimate the value function. During training, each environment is assigned to an object with randomized physical properties and a stable initial grasp. We curate list of hundreds of objects for training as shown in Figure 9.
3.2 Visuotactile Policy Training with Transformers.
We find robust and adaptive finger-gaiting emerges from the oracle policy training. However, it is assumed to know full object physical properties, pose, and shape as the input. To deploy it in the real-world, we need to use real-world observations to infer (representations of) these properties. Qi et al. [7] uses proprioceptive states to estimate such information. In this work, we augment it to include vision and touch and study their important roles in improving manipulation performance.
Touch (Figure 3). To reduce the sim-to-real gap for tactile sensors, choose to use the discretized contact location projected on 2D plane as the proxy of tactile information. In simulation, we directly parse the contact position provided by the simulator, project it onto 2D plane in fingertip frame, and discretize to 8 locations. Specifically, the touch observation o couch is a N, x 9 dimensional array, where No is the number of contact at each timestep. For each contact, contains the discretized contact location (8-dimension) and the index of the finger. During training, since the number of contact points across timesteps are not the same, we use an MLP to each contact information and take average of different contact point features. In the real-world dision-based touch sensors at the fingertips. We track the deformation of the highest intensity pixel
use four omnidirectional

---

Simu
World
Tracked Parra Trcced "onta
Contact Direction Fingerindex
18
Scene
Senser Output
Sconc
Sensor Outout
Output: [0,000,0,0,11 finger index)
t- Contact righ
Happens
Figure Representation for Sim-to-Real Touch Sensing. the simulation, we use discretized contact location provided by the simulator. In real-world, we detect the deformation by tracking colored regions of the sensor outputs, and parse the same information from temporal stream of tactile images.
on each sensor, which serves as proxy for contact position (Figure 3). This 2D keypoint from vision-based touch, similar in spirit to Sodhi al. [72], directly fed into the policy.
Vision (Figure 4). We use object depth the vision representation since 1) it is general representa- tion and does not require human labeling in the real-worke and 2) it is hard to realistically simulate RGB images whereas depth is good abstraction of object shape [35, 73]. In world deployment, instead of using the raw depth from a RGBD camera, we use Segment-Anythinn [12, 13] to segment out the objects to reduce the sim-to-real gap. Formally, given an object depth image oz depth we encode it 3-layer ConvNet to output ddephh An overview of the vision pipeline is shown in Figure 4. We also randomize the camera position and orientation during training, to make the policy robust to minor vicwpoint changes.
Visuotactile Transformer. The goal of our visuotactile policy is accurately infer the learned representation of privileged informa- tion. To tackle these challenges, we use a transformer & architecture to model these mul- timodal sensory stream. We concatenate the encoded depth image f edepth encoded tactile contact points f ftouch, joint positions gt. and action the previous timestep a:- to form the feature vector ft. We feed sequence of features fr {fi-ks.... ft-1, fi} input to the transformer. The transformer outputs 2, as the predicted extrinsic vector.
(a) Simullttad Varke
Real Vaion
Segrrm:
Run Duyth
Object Duath
ReY Dupii
Anything
Figure Representation for Sim-to-Real Vision Sens- ing In simulation. we use the object's foreground depth as the input. In real-world, to reduce the sim-to-real gap. we segment out the object's 'ss depth map using Segment- Anything.
Training. Similar to previous work [5, 6, 7]. we roll out the oracle policy with the predicted extrinsic vectors a, = (P,, 21) where 2, = Φ(fr). Meanwhile we also store the ground-truth extrinsic
vector 2, and construct training set B {(ft') (i) 2 ) Then optimize & minimizing the &2 distance between 2, and 2,, and between a, and d; using Adam [74]. The process is iterated until the loss converges. We apply the same object initialization and dynamics randomization setting.
Evaluation Setup
Hardware Setup. We use an AllegroHand from Wonik Robotics [38] for our experiments. The Allegro hand is dexterous anthropomorphic robot hand with four fingers, with four degrees of freedom per finger. Position commands sent these 16 joints at Hz. The target position commands converted torque using PD Controller at 300 Hz. For depth sensing, use Intel RealSense D435 placed at approximately 36cm from the Allegro base. We use an omnidirectional vision-based touch sensor at the distal end of each finger.
Simulation Setup. We use the IsaacGym [75] simulator. Each environment contains a simulated AllegroHand and a sampled object from our curated object datasets (Figure 9). Each object of different physical properties (the exact parameters are in the supplementary material) and a random initial pose. For depth and viewpoint consistency between the real and simulated cameras, measure the camera-robot extrinsics with an ArUco tag [76] placed the palm of the real-world Allegro. In
5

---

Rotation Reward (X)
Rotation Reward (Y)
Rotation Reward (Z)
150
150
150
130
130
110
90
90
70
50
Proprioception & Vision
Proprioception
Proprioception & Vision & Touch
Proprioception
Oracle Policy
& Touch
Figure 5: importance of vision and touch. We show performance improvement of using vision and touch for sensorimotor policy training. Using vision and touch alone can already significantly improve over the proprioception baseline, especially on rotation reward over z and axis. Combining these two sen sings will further improve the performance.
IsaacGym, we this (E) transformation augmented with random pose noise, and further apply realistic depth noise the resultant images [77].
Object Set. We create a curated dataset for objects used in our experiments from EGAD [78], Google Scanned Objects [79], YCB [80]. and ContactDB [81]. We select objects with width/depth/height (w/d/h) aspect ratio less than 2.0 (see Figure 9 for a visualization).
Evaluation Metric. We use the metrics defined in [7] to evaluate our method both simulation and in the real-world. In addition, we also evaluate undesired rotation penalties in simulation. We find this metric is particularly important for rotation over and y axis.
1. Time-to-Fall (TTF). The average length of the episode before the object falls out of the hand. This value is normalized by the maximum episode length (20s). s).
2. Rotation Reward (RotR). This is the average rotation reward w k of an episode in simulation.
3. Rotation Penalty (RotP). This the average rotation penalty per timestep w x k.
4. Radians Rotated (Rotations). The rotation (in radians) achieved by the policy with respect to the desired axis. This metric is only used in the real world experiments.
Results and Analysis
In this section, we first quantitatively study our method in simulation. In particular, we study the importance of using object shape information for policy training (Section 5.1), as well as the importance of vision and touch the visuotactile policies (Section 5.2). Then, we use a shape prediction task to study the information recovered by estimated extrinsic vectors. We show our visuotactile policy learns object shape representation by predicting the shape of objects using ε, (Section 5.3). We also evaluate our method on a real-world robot (Section 5.4) and finally show how to train single policy to rotate six principle axes.
5.1 Object Shape helps Policy Training
The performance is shown Table 1. We compare Rotatelt with previous work [7] and our method without the usage of point cloud while still using the quaternion. Experiments show that using point-cloud significantly improves the performance all of the metrics and for all rotation axis.
To get more insights, we further plot the relative improvements on varies objects shape for r-- axis rotation, shown in Figure 10 (the "stagel" row). We find that point-cloud gives the largest improvement on objects with non non-uniform w/d/h (width/ddepthhheiggh)) ratios and objects with irregular shapes such as the bunny and light bulb. The improvements on regular objects are smaller but still over 40%. addition, we also evaluate the oracle policy on 15 held-out challenging objects (Figure (b)). We show that not using point cloud results in a 22% decrease in generalization gap while using point-cloud improve it only 8% drop. Point-cloud as input is also used in Qin et al. [82] and Bao al. [83] but they do not explore how use it for in-hand manipulation. Note that our

---

T-axis
g-axis
2-axis
Touch
RotR
1
TTF 1
RotP
RotR t
TIF
RoIP J
RotR f
TTF T
RotP J.
Full
104.29 s002
0.68
0.41
93.05
0.65
a.34
126.73
0.72
0.32
NoTouch
0.46
g.55
67.2
0.48
g.55gmm
108
0.62
0.43
Binary
80.14
0.47=002
0.53
66.29
0.49
0.56=noz
110.24
0.63
0,42+
ContactLoc
102.36
0.41 10.04
92.22
0.64
0.36
122.60
0.73 2o12
0.35 goous
Table 2: The importance using a finer tactile information. We compare Rotatelt which use contact location (ContactLoe) and its variant of using binary contact (Binary) or full contact (position, normal, and scale) information. All methods are without vision information. Binary contact does not provide additional value compared No Touch, since it is already contained in proprioceptive history. We also find using discretized contact locations can match the performance of using full contact in task.
(a) OOD Objects
(b) Oracle Policy Comparison
(c) Sensorimotor Policy Comparison Visuolactile Policy Proprioception Only
With Point Cloud
Without Point Cloud
140
Gracle (OCD)
Gracks
140
with Visten and Touch
Visten Touch (000)
No Point Cloud
120
Point Cloud (DOD)
120
Propriscaption
100
100
80
60
40
8% decrease for OOD Objs
22.6% decrease for OOD Objs
15.4% decrease for OOD objs
decrease for OOD objs
Evaluation. We evaluate policies (a) 15 held-out objects. These objects also more challenging manipulate compare objects reward for four settings. For oracle policies (b), we find that using point cloud will lead 22% performance drop for OOD objects while using point cloud can improve it to only 8%. For sensorimotor policies (c), using proprioception only will lead to 41 % drop while visuotactile information can improve it to 15%.
Figure 6:
training set. We show the episode rotation
design is different from Chen et al. [8], which uses only pose for the oracle policy and uses object shape information only in the student policy. In our setting using object pose not sufficient to achieve good enough performance.
5.2 Visuotactile Transformer
The oracle policy evaluated in Section 5.1 cannot be transferred to the real-world because it needs access to a manipulated object shape and physical properties. We instead learn to infer this repre- sentation during execution from proprioceptive, visual, and tactile history. Figure 5, we show that using either vision or touch alone gives a significant performance improvements compared to proprioceptive inputs. We also find using a combination of vision improve the performance. By integrating visuotactile sensing and temporal transformer, our method match the performance of the oracle policy. In appendix Table 4, we also show transformer has better sequence modeling ability compared to temporal convolutions used in previous work [5, 7].
touch sensing can further
In Figure (c), we show the visuotactile information are critical for OOD generalization. Using proprioception only will lead to a 41 % performance drop while using vision and touch can improve to 15% drop.
Importance Finer Tactile Sensing. In contrast to prior work [37], we find in Table that binary contact does not provide benefits. In contrast, contact locations are vital for improving performance in Rotatelt. We speculate that this discrepancy is because Khandate al. [37] does not proprioceptive and action history.
7

---

Method
Cocoon
Squishy
Baseball
Puzzle
Box
Stego
Hora [7] a.54
0.50-oat 0.26 0.48 0.52 in.s 0.46
Cocoon
Squishy
Baseball
Puzzia
Box
Stego
Rotatelt 12.71 8.29 1.3 6.72. 6.12
5.05 5.01
Figure 8: Rotations rotated (f) over r-axis for Rotatelt and [7] in Real-world Evaluation, We compare Rotatelt and Hora [7] on six different objects. Hora [7] is not able to finish this task and does not learn finger-gaiting to rotate the object. while Rotatelt can.
5.3 Representation Learned in the Latent Space
Next, we study the information that is encoded into 2, and ε.. After we finish training the four policies in Figure 7, freeze the network and then our policy on 20 objects in object dataset (16 for training, 4 for testing). This gives an extrinsic vector dataset for each policy. On each of the datasets, we train one decoder whose input is sub-sequence of extrinsic vectors and output is the voxel grid. After training this decoder, we run it on the held-out testing objects.
In Figure 7, visualize predicted shapes averaged over 100 randomly selected subsequences from roll- outs on novel test objects for four policies: the stage 1 oracle policy with and without shape (mesh) con- ditioning, and the stage 2 policy with and without visuotactile sensory inputs. The results suggest that shape information is preserved and useful for our oracle policy even though the only learning signal is the reward function. Next, results also highlight both the capabilities and limits of proprioception, which we can robustly distinguish between spherical (beige) and cuboidal (green) objects. Shape understanding for more irregular objects like the pear (blue). however, requires additional sensors. This supports the increased benefit of vision and touch for more complex objects that we observe in Section 5.1.
Object medh (ground-muth)
Stage (weo shape)
Stage (wl shapp)
Stage (prap only.
Stage (viustretile)
Figure 7: Inverting encoded extrinsics. We pre- dict 3D shapes on novel objects from learned z and &, Stage 1 results are provided with without shape conditioning, stage 2 results are provided with / without visual and tactile sensors.
5.4 Real-world Evaluations
Finally, we quantitatively compare Rotatelt and Hora [7] in the real-world on rotating different objects over the r-axis. We find that without vision and touch, Hora cannot finish this task. It only learns in-grasp movement with thumb slowly moving to the bottom the object. also not able to maintain stability; the object quickly falls down. In contrast, Rotatelt can successfully manipulate multiple objects with different geometries such as cubes, spheres, or cylinders by ~2m radians within 20 seconds. Note that many real-works objects outside training set such the box, Cocoon, Squishy, and Stego. The real-world physics is also different from the simulated physics. Having successful sim-to-real transfer is a strong evidence of generalization. We show qualitative results on rotation around and beyond the three canonical axes on our website. In the video, we also test a policy trained with the assumption that one of the touch sensors off. The policy performs similarly to the full policy, demonstrating the robustness of the algorithm.
Limitations and Future Work
In this paper, we show the feasibility of training policies that can rotate many objects over multiple axes. We view this capability as an important step towards general-purpon purpose in-hand manipulation. We assume the objects are not too long (c.g. a pencil or a screwdriver) and are within the mechanical limit of the robot hand. Our method not able to utilize real-world experiences during deployment since it is frozen after training. There are also various ways improve the touch processing system since only the dineenennio contact location as the input and do not utilize the full information output by the omnidirectional image-based tactile sensor.
8

---

Acknowledgments
This research was supported as BAIR Open Research Common Project with Meta. In their academic roles at UC Berkeley, Haozhi Qi and Jitendra Malik are supported in part by DARPA Machine Common Sense (MCS), Brent Yi is supported by the NSF Graduate Research Fellowship Program under Grant DGE 2146752, and Haozhi Qi, Brent Yi, and Yi Ma are partially supported by ONR N00014-22-1-2102 and the InnoHK HKCRC grant. Roberto Calandra is funded by the German Research Foundation (DFG, Deutsche Forschungseemeinscaatt)) as part of Germany's Excellence Strategy - EXC 2050/1 - Project ID 390696704- - Cluster of Excellence "Centre for Tactile Internet with Human-in- the-Loop" (CeTT) of Technische Universität Dresden. We thank Shubham Goel, Eric Wallace, and Angjoo Kanazawa, Raunaq Bhirangi for their feedback. We thank Austin Wang and Tingfan Wu for their help on hardware. We thank Xinru Yang for her help on real-world videos.
References
[1] OpenAI, M. Andrychowicz, B. Baker, M. Chociej. R. Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng. and W. Zaremba. Learning dexterous in in-hand manipulation. 1/RR, 2019.
[2] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej. M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and Zhang. Solving rubik's cube with robot hand. arXiv:1910.07113, 2019.
[3] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, Y. Narang, J.-F. Lafleche, D. Fox, and G. State. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In ICRA, 2023.
[4] J. Pitz, L. Röstel, L. Sievers, and B. Bäuml. Dextrous tactile in-hand manipulation using a modular reinforcement learning architecture. In ICRA, 2023.
[5] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 2020.
[6] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. In RSS, 2021.
[7] H. Qi, A. Kumar, Calandra, Y. Ma, and J. Malik. In-hand object rotation via rapid motor adaptation. In CoRL, 2022.
[8] T. Chen, M. Tippur, S. Wu, V. Kumar, E. Adelson, and P. Agrawal. Visual dexterity: In-hand dexterous manipulation from depth. arXiv:22/1.11744 2022.
[9] G. Westling and R. S. Johansson. Factors influencing the force control during precision grip. Experimental Brain Research, 1984.
[10] R. Flanagan, M. C. Bowman, and S. Johansson. Control strategies in object manipulation tasks. Current Opinion in Neurobiology, 2006.
[11] R. S. Johansson and J. R. Flanagan. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nature Reviews Neuroscience, 2009.
[12] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-., Lo, et al. Segment anything. ICCV, 2023.
[13] C. Zhang, D. Han, Qiao, J. U. Kim, S.-H. Bae, S. Lee, and S. Hong. Faster segment anything: Towards lightweight sam for mobile applications. arXiv:2306. 14289, 2023.
[14] L. Han and J. C. Trinkle. Dextrous manipulation by rolling and finger gaiting. In ICRA, 1998.

---

[15] J.-P. Saut, A. Sahbani, S. El- El-Khoury, and V. Perdereau. Dexterous manipulation planning using probabilistic roadmaps in continuous grasp subspaces. In IROS, 2007.
[16] D. Rus. In-hand dexterous manipulation of piecewise-smooth 3-d objects. I/RR, 1999.
[17] Y. Bai and C. K. Liu. Dexterous manipulation using both palm and fingers. In /CRA, 2014.
[18] I. Mordatch, Z. Popović, and E. Todorov. Contact-invariant optimization for hand manipulation. In Eurographics, 2012.
[19] R. Fearing. Implementing a force strategy for object re-orientation. ICRA, 1986.
[20] C. Teeple, B. Aktas, M. C.-S. Yuen, G. Kim, R. D. Howe, and R. Wood. Controlling palm-object interactions via friction for enhanced in-hand manipulation. 2022.
[21] A. Morgan, K. Hang, Wen, K. Bekris, and A. M. Dollar. Complex in-hand manipulation via compliance-enabled finger gaiting and multi-modal planning. RA-L, 2022.
[22] B. Wen, C. Mitash, B. Ren, and K. E. Bekris. Se(3)-tracknet: Data-driven 6d pose tracking by calibrating image residuals in synthetic domains. In IROS, 2020.
[23] H. Van Hoof, T. Hermans, G. Neumann, and Peters. Learning robot in-hand manipulation with tactile features. In Humanoids, 2015.
[24] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In CoRI., 2019.
[25] M. Li, H. Yin, K. Tahara, and A. Billard. Learning object-level impedance control for robust grasping and dexterous manipulation. In ICRA, 2014.
[26] A. Gupta, C. Eppner, Levine, and Abbeel. Learning dexterous manipulation for a soft robotic hand from human demonstrations. In IROS, 2016.
[27] Y. Qin, H. Su, and X. Wang. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation. RA-L, 2022.
[28] S. P. Arunachalam, S. Silwal, B. Evans, and L. Pinto. Dexterous imitation made easy: A learning-based framework for efficient dexterous manipulation. In ICRA, 2023.
[29] Y. Qin, Y.-H. Wu, S. Liu, H. Jiang, R. Yang, Y. Fu, and Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In ECCV, 2022.
[30] L. Sievers. J. Pitz, and B. Bäuml. Learning purely tactile in handnd manipulation with a torque- controlled hand. ICRA, 2022.
[31] G. Khandate, M. Heger, and M. Ciocarlie. On the feasibility of learning finger-gaiting in-hand manipulation with intrinsic sensing. In ICRA, 2022.
[32] T. Chen, J. Xu, and Agrawal. A system for general in-hand object re-orientation. In CoRL, 2021.
[33] W. Huang, I. Mordatch, P. Abbeel, and D. Pathak. Generalization in dexterous manipulation via geometry-aware multi-task learning. arXiv:2111.03000 2021.
[34] D. Chen, B. Zhou, V. Koltun, and Krähenbühh. Learning by cheating. In CoRL, 2020.
[35] A. Loquercio, E. Kaufmann, R. Ranftl, M. Müller, V. Koltun, and D. Scaramuzza. Learning high-speed flight in the wild. Science Robotics, 2021.
[36] Z. Yin, B. Huang Y. Qin, Q Chen, and X. Wang. Rotating without seeing: Towards in-hand dexterity through touch. In RSS, 2023.
10

---

[37] G. Khandate, S. Shang, E. T. Chang, T. L. Saidi, J. Adams, and M. Ciocarlie, Sampling-bast based exploration for reinforcement learning of dexterous manipulation. In RSS, 2023.
[38] WonikRobotics. Allegrohand. https: http://wwwwwnnrrooticcc..co c01/,
[39] W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 2017.
[40] B. Ward-Cherrier, N. Pestell, L. Cramphorn, B. Winstone, M. E. Giannaccini, J. Rossiter, and N. F. Lepora. The tactip family: Soft optical tactile sensors with 3d-printed biomimetic morphologies. Soft robotics, 2018.
[41] M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon, V. R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, et al. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. RA-L, 2020.
[42] C. Lin, Z. Lin, S. Wang, and H. Xu. Dtact: A vision-based tactile sensor that measures high-resolution 3d geometry directly from darkness. In ICRA, 2023.
[43] D. F. Gomes, Z. Lin, and S. Luo. Geltip: A finger-shaped optical tactile sensor for robotic manipulation. In IROS, 2020.
[44] Z. Xue, H. Zhang. Cheng, He, Y. Ju, C. Lin, G. Zhang, and H. Xu. Arraybot: Reinforcement learning for generalizable distributed manipulation through touch. arXiv:2306.16833 168577 2023.
[45] O. Azulay, N. Curtis, R. Sokolovsky, G. Levitski, D. Slomovik, G. Lilling, and A. Sintov. Allsight: A low-cost and high-resolution round tactile sensor with zero-shot learning capability. arXiv:2307.02922 2023.
[46] R. Calandra, A. Owens, D. Jayaraman, W. Yuan, J. Lin, J. Malik, E. H. Adelson, and S. Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. RA-L, 2018.
[47] H. Xu, Y. Luo, S. Wang, T. Darrell, and R. Calandra. Towards learning to play piano with dexterous hands and touch. In IROS, 2022.
[48] K. Zakka, L. Smith, N. Gileadi, T. Howell, X. B. Peng, S. Singh, Y. Tassa, P. Florence, A. Zeng, and P. Abbeel. Robopianist: A benchmark for high sigh-dimensional robot control. In CoRL, 2023.
[49] E. Smith, D. Meger, L. Pineda, R. Calandra, J. Malik, A. Romero Soriano, and M. Drozdzal. Active 3d shape reconstruction from vision and touch. In NeurIPS, 2021.
[50] E. Smith, R. Calandra, A. Romero, G. Gkioxari, D. Meger, J. Malik, and M. Drozdzal. 3d shape reconstruction from vision and touch. In NeurIPS, 2020.
[51] S. Suresh, Z. Si, J. G. Mangelson, W. Yuan, and M. Kaess. Shapemap 3-d: Efficient shape mapping through dense touch and vision. In ICRA, 2022.
[52] S. Suresh, Z. Si, S. Anderson, M. Kaess, and M. Mukadam. Midastouch: Monte-carlo inference over distributions across sliding touch. In CoRL, 2022.
[53] Guzey, B. Evans, Chintala, and L. Pinto. Dexterity from touch: Self-supervised pre-training of tactile representations with robotic play. In CoRL, 2023.
[54] N. Sunil, S. Wang, Y. She, E. Adelson, and A. R. Garcia. Visuotactile affordances for cloth manipulation with local control. In CoRL, 2022.
[55] J. Hansen, F. Hogan, D. Rivkin, D. Meger, M. Jenkin, and G. Dudek. Visuotactile-rl: Learning multimodal manipulation policies with deep reinforcement learning. In ICRA, 2022.
[56] I. Guzey, Y. Dai, B. Evans, S. Chintala, and L. Pinto. See to touch: Learning tactile dexterity through visual incentives. 12N::2309.12300, 2023.

---

[57] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. NeurlPS, 2017.
[58] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. In ICLR, 2021.
[59] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. In CoRL, 2022.
[60] Xiao, Radosavovic, T. Darrell, and Malik. Masked visual pre-training for motor control. arXiv:2203.061733 2022.
[61] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. wrXiv:2212.066 12.06177 2022.
[62] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In RSS, 2023.
[63] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. In CoRL, 2022.
[64] I. Radosavovic, Xiao, B. Zhang, T. Darrell, Malik, and K. Sreenath. Learning humanoid locomotion with transformers. WrXiv:2303.03381, 2023.
[65] R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang. Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers. In ICLR, 2022.
[66] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. In ICML, 2022.
[67] Y. Chen, A. Sipos, M. Van der Merwe, and N. Fazeli. Visuo-tactile transformers for manipulation. In CoRL, 2022.
[68] C. R. Qi, H. Su, K. Mo. and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017.
[69] A. Gupta, Pacchiano, Zhai, S. Kakade, and S. Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. In NeurIPS, 2022.
[70] Y. Zhai, C. Baek, 7. Zhou, J. Jiao, and Y. Ma. Computational benefits of intermediate rewards for goal-reaching policy learning. JAIR, 2022.
[71] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv:1707.0633 2017.
[72] P. Sodhi, M. Kaess, M. Mukadam, and S. Anderson. Learning tactile models for factor graph- based estimation. In ICRA, 2021.
[73] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. In CoRL, 2022.
[74] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[75] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin,
A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simulation for robot learning. In NeurIPS Datasets and Benchmarks, 2021.
[76] J.
R. Muñoz-Salinal and R. Medina- Carnieer. Speeded up detection of squared fiducial markers. Image and vision Computing, 2018.
12

---

[77] S. Choi, Q.-Y. Zhou, and V. Koltun. Robust reconstruction of indoor scenes. In CVPR, 2015.
[78] D. Morrison, P. Corke, and Leitner. Egad! an evolved grasping analysis dataset for diversity and reproducibility in robotic manipulation. RA-I., 2020.
[79] L. Downs, A. Francis, Koenig, B. Kinman, R. Hickman, K. Reymann, B. McHugh, and Vanhoucke. Google scanned objects: high-quality dataset of 3d scanned household items. In ICRA, 2022.
[80] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In ICAR, 2015.
[81] S. Brahmbhatt, C. Ham, C. C. Kemp, and J. Hays. Contactdb: Analyzing and predicting grasp contact via thermal imaging. In CVPR, 2019.
[82] Y. Qin, B. Huang, Z.-H. Yin, H. Su, and Wang. Dexpoint: Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation. In CoRL,, 2022.
[83] C. Bao, H. Xu, Y. Qin, and X. Wang. Dexart: Benchmarking generalizable dexterous manipula- tion with articulated objects. CVPR, 2023.
[84] Q. Li, Y. Zhai, Y. Ma, and S. Levine. Understanding the complexity gains of single-task rl with a curriculum. In ICML, 2023.
[85] D.-A. Clevert, T. Unterthiner, and Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv: arXiv:1511.07289 2015.
13

---

Figure 9: Training objects. We curated diverse combination of objects from EGAD [78]. Google Scanned Objects [79], YCB 180), and ContactDB [81]. We filter out meshes with disconnected components and objects with a widtt/depthhhheihr (w/d/h) ratio larger than 2.0.
Stage +192%
+107%
+89%
+51%
+58%
+46%
+34%
+37%
Stage 2
+118%
+119%
+106%
+89%
+40%
+35%
+20%
+13%
Relative rotation reward improvements before and after shape or visuotactile information. For stage training (oracle policy), we compare our oracle policy and the policy without point- cloud as input. For stage 2 training (visuotactile policy), compare the improvement of Rotatelt and the policy with only proprioceptive input. In both cases, having vision and touch information significantly improve the performance.
Figure 10:
A
Additional Experiments
A.1 Relative Improvements Different Objects
We also find policies without object shape will consider all the objects spherical cuboid objects, which explains the huge improvement objects with large w/d/h ratios Figure 10.
Similar to what we find in the oracle policy training, we observe the visuotactile policy has larger improvements on irregular and non-uniform objects (Figure 10, "stage2" row).
A.2 Multi-axis Training
In previous sections, each oracle pol- icy is trained with a fixed rotation axis k. In this section, we demon- strate it is also feasible to train sin- gle network to perform multi-axis ob- ject rotation. To achieve this, aug- ment the observation space with & and train it with the reward defined in Section 3.1 and the imitation learning objective with the corresponding single-axis oracles.
Method
+
+V
Single-axis 110.19 104.29 93.05
90.20
124.91 126.73
Multi-axis 105 10 85.381971
89.83.
125.32 195.19
Table Episode Rotation Reward comparison between single-axis training and multi-axis training. The distilled multi-axis policy per- forms on par with the single task oracles.
We show the episode rotation reward for both the single-axis oracle policy and the multi-axis policy in Table 3. We empirically find the distilled multi-axis policy performs on par with the single task oracles. We also observe the policy does not converge when training with only reinforcement learning.
A.3 Detailed Comparison of Using Vision and Touch.
Table 4 shows the detailed comparison of using vision, touch, and the transformer architecture. We show each of the component can significantly improve over the baseline and are also complement with each other.
A.4 Randonization of Simulated Vision Sensing.
During training, apply various randomizations to the vision sensing make it robust. We evaluate model under different noise setting in simulation. The results are shown in Table 5.
14

---

Modality
Jt-axis
graxis
2-axis
Method
Vision Touch
RotR
TTF
RotP
+
RotR
TTF
RodP +
RotR
TTF
RotP
Oracle
NIA
NIA
g.79 g,35
118. ma 0.79
0.30
140.90 mas
a. 0.27
66.23
0.41
0.64
54.19
0.38
0.69
19.
0.56
a.47
87.21+120
0.59 0.59
72.51
0.57=003
0.62+am
=02.35
0.68
0.42+00l
Conv
82. 29
0.60
0.57
69. 29.
0.58
a.60
107.73
a.
0.46
98.20
0.70+
0,45
89.82+ =.67
0.47+0.0
113.26=13:5
0.70 0.40+am
79.37 7a72 0.46
67.21
0.48
Looe
0.55 join
108.25
me:
0.62. 0.43
102.36 0.65 hoo 0.41 guos
i2
0.64
0.36 gume
0.35 joo 122.60 mos
0.73
Transformer
99.29 79 0.62
g.43
91.47
a.37
1
0.72.
0.39
118.42 0.75 0.37 =0m 109.31 0.73=oa2 0.31 136.25=11a. a.80 0.29:
Table 4: The importance of vision and touch. We show the performance improvement of using vision, touch, and the transformer architecture. Each of the three components significantly improves the performance of rotating over z/y/z axis.
We add gaussian noise to camera positions and orientations. Cam Pos stands for the value for each different setting (in meters). Cam RPY stands for the extend we randomize the camera rotation (in roll/pitch/yaw values, in radius). The camera field-of-view (fov) is also randomized. The values are set to uniform distribution according to the Cam FOV column. We also simulate segmentation noise (for each pixel, with probability p. the mask is flipped) and segmentation failure (for each timestep. with probability p, the mask is completely 0) to simulate segmentation errors in the real-world.
We find that the model behaves robustly under training randomization and slightly out-of-disttibution noises. However, too large noise will still impact the performance, highlighting the importance of proper camera calibration.
B Implementation Details
Simulation Setup During training, we use 32768 parallel environments to collection samples for training the agent, distributed on 4 GPUs. Each environment contains simulated AllegroHand and sampled objects from our curated object datasets (Figure 9). Each object is of different physical properties and random initial pose. The simulation frequency is 200 Hz and the control frequency is 20 Hz. Each episode lasts for 400 control steps (cquivalent to 20 We reset the episode if the objects fall below 13.5 em with respect to the hand.
Stable Precision Grasp Generation. Our approach assumes the object is grasped at the beginning of the episode. To achieve this, we start from canonical grasping position using fingers. Then we add relative offset the joint positions sampled from (/(-0.2, 0.25) Then forward the simulation by 0.5 We save the grasping pose if all the following conditions are satisfied:
1. The distance between the fingertip and the object should be smaller than 10 cm.
2. At lease two fingers are in contact with the object.
3. The object's 's height should be above 13.5 cm higher than the center of the palm.
In practice, we discretized (each region separated by 0.2) the scales specified in Table and pre-sampled 400 grasping poses for each object and for each scale.
Physical Randomization Parameter. We apply domain randomization during training the oracle policy as well as the visuotactile policy. The parameters are listed in Table 6. Following [1], we apply random disturbance force to the object during training whose scale is 2m where m is the object mass. The force is decayed by 0.9 every 80 ms following [1]. The force is re-sampled at each time-step with probability 0.25.
Reward Hyperparameter. We use Tmax = 0.5, Tmin -0.,, Atorque -0.1, Alivel = 00.. Awork = -2.0, and Ayosp -0.1. We also find that if we apply Arotp -0.1 1 at the start of training, the policy will only learn to stably hold the objects. Therefore set this coefficient to be 0 at the beginning and then linearly decrease it to -0.1 using curriculum learning [84].
15

---

Setting
Cam Pos
Cam RPY
Cam FOV
Seg Noise
Seg Failure RotR 1
Perfect Vision
119.19
Same Noise as training
+0 0.01)
+0(0,, 0.03) = LL(55, 58) 0.2
0.05
118.42
Out-of-distribution Noise
+ (0,0.1155)
+N(0, 0.035) = U(48, 62)
0.25
0.075
115.30
Larger Noise
(0,00, +N((0, 0.04) = U(45, 65) 0.3
0.1
102.80
No Vision
99.29
Table 5: Evaluation performance on noisy vision sensing systems. We evaluate the same visuotactile policy five different settings. We find that the model behaves robustly under training randomization and slightly out-of-distribution noises. However, too large noise still impact the performance, highlighting the importance of proper camera calibration.
Parameter
Range
Parameter
Default Value
Object Scale
[0.46, 0.68]
N₂
100
Mass
[0.01, 0.25] kg
Cp
32
Center of Mass
[-1.00, 1.00] cm
dim 2t
shapey
32
Coefficient of Friction
[0.3, 3.0]
phys dim [2)
External Disturbance
(2, 0.25)
dim (2))
40
PD Controller Stiffness
[2.9, 3.1]
dim J deppty
32
PD Controller Damping [0.09, 0.11]
dim [
(Louch
32
Table 6: Randomization Range of Physics Parameters. Table 7: Default Values for network hyperpa- We sample the physical parameter values from a uniform rameters. distribution.
Network Architecture. The oracle control policy x a multi-layer perceptron (MLP) which takes in the state Pt R6 and the embedding of privileged information 2, R-40, and outputs 16- dimensional action vector a, There are four layers with hidden unit dimension [512, 256, 128, 16]. We use ELU [85] as the activation function. The privileged encoder μ is also three layer MLP with hidden unit dimension [256. 128, 8] and encodes object pose and physical properties to output phys
z E R We use ReLU as the activation function. The PointNet Encoder is three layer MLP with hidden unit dimension [32, 32, 32). The MIP is applied on each of points and then the features are aggregated using max pooling.
The visuotactile transformer takes object depth feature, touch feature, proprioception feature, and action history as input. The object depth image is of size 60 60 and is first be passed to four layer ConvNet and then global average pooling layer to produce the feature of dimension 32. The contact location is 9-dimension vector and is first passed to an MI.P with hidden unit dimension [32, 32, 32]. The contact feature is aggregated using average pooling. For the robot joint position and actions, we first encode them into a 32-dimensional representations for each timestep via a two-layer MLP (with hidden unit dimension [32, 32)). The feature dimension of our transformer is 32 and with depth The self-attention module has parallel head.
Optimization Details. During the oracle policy training, we jointly optimize the control policy æ and the privileged information encoder using PPO [71]. In each PPO iteration, collect samples from 32768 environments with 10 agent steps each (corresponding
0.5 seconds). We train epochs with a batch size 32,768. The learning rate is 5e-3, For the visuotactile policy, we use Adam optimizer [74] to minimize MSE loss. The learning rate is 3c-4.
16


Reviews:
[{"review_id": "v9PnczJurfj", "reviewer": "Reviewer_3t8K", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a policy learning method for in-hand object rotation. The goal is to learn a general and realistic policy that can rotate any object around a fixed axis as fast as possible in simulation and then transfer to real world. The method is based on RMA approach. It first learns an expert policy that acts with input privilege state information, object physical parameters, and object point cloud. Following RMA, the learned expert policy is composed of two parts, a task-relevant encoder, and a control-relevant policy layer. It then distills the expert policy to a policy with real observation as input by learning a new task-relevant encoder to map the history of observations into the corresponding latent space. It extends the previous RMA-based approach [7] by using visual (i.e. depth image) and touch (i.e. contact location) to infer the latent task-related representation. The conducted experiments show the effectiveness of the approach.", "summary_of_recommendation": "While this paper shows a great system for object rotating and demonstrates the results in both simulation and real world, given the listed questions and concerns, I cannot fully assess the technical significance of this paper over the existing works. Therefore I am towards the negative side of the paper for now.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "I have the following questions and suggestions that I hope the authors can clarify and address during the rebuttal:\n\n1. Was the proposed method tested on unseen objects? The input of object mesh might help perform better on trained objects, but it might also result in overfitting to seen objects.\n\n2. Does the trained policy work for an arbitrary axis or each policy is trained just for one single axis? It is unclear to me in the paper.\n\n3. Line 79 mentions that one advantage of the proposed method is that it can manipulate texture-less and symmetrical objects. Is this indeed an advantage of the simpler task of rotating around an axis so that there is no need to determine the success/termination of the task based on the object pose? For instance, can we apply the proposed approach to reorient texture-less objects?\n\n4. The proposed approach is mainly based on RMA approach. It would be more appropriate if the author could give more credit to RMA paper in the method section.\n\n5. For the experiment section 5.3, the authors show that it is able to recover the full geometry from the latent $z$. Did the authors train a decoder separately? It lacks details about how this reconstruction work in the paper. Also it seems that there is no reconstruction loss during training, so what is the secret sauce that encourages the latent space to contain meaningful info to reconstruct the mesh?\n\n6. Line 151: is the 9-dim vector for a single finger or for the full hand?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I summarize the strengths and weaknesses below.\n\n**Strengths**:\n\n1. The paper is clear and the technical part is easy to follow.\n\n2. The experiments effectively demonstrate the advantages of the proposed approach over the compared baselines.\n\n3. The real-world demo in the video shows convincing results on the sim-to-real performance of the approach.\n\n**Weaknesses**:\n\n1. It lacks motivation for the paper and the approach. Given the existing works (especially Chen et. al. [8]), it is unclear to me what aspects this paper tries to solve beyond the existing work's scope and why the proposed technique (e.g. RMA framework, touch information) helps address the goal.\n\n2. Continued from the last question, it is also unclear to me about the advantage of the proposed approach over the existing work (Chen et al. [8])? The single-axis rotation task seems to be a simpler task than the random goal pose re-orientation task solved by [8]. As a suggestion, it would make this paper much more valuable if it can demonstrate some advantages over [8] such as either having higher performance or being more learning efficient on random object random goal pose re-orientation tasks.\n\n3. It seems that the trained policy only works for a fixed axis instead of goal conditioning on a random axis.\n\n4. It is unclear whether the trained policy is able to generalize to unseen objects.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "JWbVdnStg7", "reviewer": "Reviewer_Ms7j", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors tackle object rotation with a teacher-student framework, where PPO is trained with privileged information in the simulator first, and the student is forced to match the encoded extrinsics given observation data. The authors consider certain design choices that are unique compared to previous work: depth is used for training instead of RGB, and dense tactile signals on locations are used. The authors show that these series of choices lead to much improved performance compared to previous works that do vision-only or touch-only.  ", "summary_of_recommendation": "The paper scientifically investigates many of the design choices behind the teacher-student PPO framework within in-hand manipulation, and the results from this investigation can be very informative for other researchers doing in-hand manipulation. The presentation is thorough and the paper is written well, and all the investigations are scientifically done by comparing against appropriate baselines, so I recommend acceptance. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Would a similar strategy work for in-hand translation / commanding general pose commands that combine translation and rotation?\n\n2. Do the author’s proposed method works for object outside the trained dataset? What is the extent to which it generalizes across shapes? \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "<Strengths>\n\n1. Overall, this is a strong empirical systems paper that does many scientific investigations and ablation studies on the effect of various decisions to doing in-hand manipulation within the teacher-student framework. \n\n2. The hardware experiments are extensive and convincing, and the authors do a good job of implementing the system. \n\n<Weaknesses>\n\n1. The parts of the hands that are used to make contact are quite minimal, although this might likely be due to the difficulty of installing tactile sensors among the surface of the hand. \n\n2. Although the technical and scientific contribution of the paper is quite strong, the fundamental methods used in the paper follow the same algorithm, so the algorithmic contribution / novelty is quite low. \n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "Fr22qDOZMB", "reviewer": "Reviewer_ahST", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a system capable of general fingertip-based in-hand object rotation with a dexterous hand. The system uses both visual and tactile sensing as input, which significantly improves the manipulation performance. Meanwhile, the latent representation of the policy captures the 3D shapes of objects and works as primitives for general dexterous manipulation.", "summary_of_recommendation": "The paper presents a system capable of general in-hand object rotation using vision and tactile sensing. The idea is interesting to the community and the presentation is well organized. Hence, I recommend accepting the paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Issues:\n1.\tIt seems the system encodes quite abundant of information into the extrinsics vector including shape from point cloud and physical property. While this makes the system working well in the rotating tasks, it makes the system more complex. I’m wondering if this helps or limit the generalization ability of the system to novel objects. Although the authors show the representation learned of 3 novel objects, these shapes are very common and not very different from the shapes in the training data. How does the system performs on rotating objects with distinct shapes?\n2.\tHow does the accuracy of the vision sensing from the depth camera affect the performance of the system? The depth image in Figure 4(b) looks very rough and has occlusion from the robotic hand.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The quality, originality, and significance of the paper are good. The contributions and technic details are clearly presented in the paper. The literature is well reviewed and organized to understand the contributions of the work. The framework and training pipeline are defined clearly both in diagram and math notation. Results are also reported and compared with relevant existing methods, which help the reader to understand the difference easily. The weaknesses are listed in the issues.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "5k5T7c2VTxT", "reviewer": "Reviewer_Kkn8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper focuses on in-hand object rotation using fingertips, utilizing both vision and tactile information. The main contributions of this paper are as follows:\n\n1. Emphasizing the significance of combining both vision and tactile inputs for effectively performing the in-hand re-orientation task.\n2. Proposing a novel latent manipulation policy that leverages a latent space to implicitly capture the geometrical understanding of objects without requiring direct supervision.\n3. Conducting direct sim2real experiments on multiple different objects and axes to validate the proposed approach's effectiveness.", "summary_of_recommendation": "This work includes Interesting result regarding in-hand re-orientation with vision and touch. I am willing to increase the recommendation if the author update the result with fully working hardware during the rebuttal.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* Is it possible to fix your hardware such that it matches your simulate environment and your methodology descriptions? \n* To improve the manuscript, the author could consider the following:\n1. Enhance clarity by explicitly mentioning the dimensions of each input in section 3.1.\n2. In Figure 4 (c), what does '!' mean?\n3. Figure 5 \"we our oracle policy and our policy\"", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper indeed presents impressive results, particularly regarding multi-axis in-hand reorientation, sim2real outcomes, and the latent space's capability to recover 3D object geometry. Aligning the latent space (z_t) learned from simulation, which has access to all privileged information, with the real-world latent space (\\hat{z}_t) obtained from limited information is a commendable approach. Additionally, the ablations and comparisons with their baseline (Hora) add to the paper's strengths.\n\nHowever, my main concern about this work is their real-world result, where one of the four touch sensors was broken at the time of submission. This is problematic because their result will not match the simulation from training and what they described in the methodology (line 154). I appreciate the author's honesty about this issue, but I think they should have withheld their submission until they fixed their hardware and obtained a non-misleading real-world result.\n\nFurthermore, there are some mistakes in the manuscript and figures. I left details in the following section.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents learning-based control method for in-hand manipulation using depth and tactile information. The method is somewhat common (i.e., teacher-student pipeline) but it is the first application to this problem.

Many reviwers agreed that the method is not new. The technical contribution including the experiments are strong but the weakside is on the novelty of the method.

I accept the fact that the comparisons to Chen et al.[8] is not important now because the paper is not published. The use of tactile information also differentiates the two papers. As I mentioned, the algorithmic contribution is limited in this paper so algorithmic comparisons are not important in my opinion.

The new experiments on held-out objects make this paper stronger. But these have to be added to at least in the supplementary materials.

I belive that the weakness (weak novelty, rotate only in canonical axes) and strength (comprehensive experiments, good integration) are clear. Since all reviwers have positive opinions, I am also willing to accept this paper.
Decision: Accept (Poster)