Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play
Authors: Irmak Guzey
Ben Evans
Soumith Chintala
Lerrel Pinto
Venue: CoRL 2023 Poster (2023)
Keywords: Tactile
Dexterity
Manipulation

Abstract: Teaching dexterity to multi-fingered robots has been a longstanding challenge in robotics. Most prominent work in this area focuses on learning controllers or policies that either operate on visual observations or state estimates derived from vision. However, such methods perform poorly on fine-grained manipulation tasks that require reasoning about contact forces or about objects occluded by the hand itself. In this work, we present T-Dex, a new approach for tactile-based dexterity, that operates in two phases. In the first phase, we collect 2.5 hours of play data, which is used to train self-supervised tactile encoders. This is necessary to bring high-dimensional tactile readings to a lower-dimensional embedding. In the second phase, given a handful of demonstrations for a dexterous task, we learn non-parametric policies that combine the tactile observations with visual ones. Across five challenging dexterous tasks, we show that our tactile-based dexterity models outperform purely vision and torque-based models by an average of 1.7X.
Finally, we provide a detailed analysis on factors critical to T-Dex including the importance of play data, architectures, and representation learning.

Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play
Irmak Guzey
Ben Evans!
Soumith Chintala
Lerrel Pinto
New York University!, Meta?
Abstract: Teaching dexterity multi-fingered robots has been longstanding challenge in robotics. Most prominent work in this area focuses on learning con- trollers or policies that either operate on visual observations or state estimates de- rived from vision. However, such methods perform poorly on fine-grained manip- ulation tasks that require reasoning about contact forces or about objects occluded by the hand itself. In this work, we present T-DEX, a new approach for tactile- based dexterity, that operates in two phases. In the first phase, we collect 2.5 hours of play data, which is used train self-supervised tactile encoders. This is nec- essary to bring high-dimensional tactile readings to lower-dimensional embed- ding, In the second phase, given handful of demonstrations for a dexterous task, we learn non parametretrc policies that combine the tactile observations with vi- sual ones. Across five challenging dexterous tasks, we show that our tactile-based dexterity models outperform purely vision and torque-based models by an aver- age of 1.7X. Finally, we provide a detailed analysis on factors critical T-DEX including the importance play data, architectures, and representation learning.
Keywords: Tactile, Dexterity, Manipulation
1 Introduction
Humans are able solve novel and complex manipulation tasks with small amounts of real-world experience. Much of this ability can be attributed to our hands, which allow for redundant contacts and multi-finger manipulation. Endowing multi-fingered robotic hands such dexterous capabilities has been a long-standing problem, with approaches ranging from physics- based control [i] to sim- ulation to real (sim2real) learning [2, 3]. More recently, the prevalence of improved hand-pose esti- mators has enabled imitation learning approaches to teach dexterity, which in turn improves sample efficiency and reduces the need for precise object and scene modelling [4, 6).
Even with improved algorithms, teaching dexterous skills is still quite inefficient, requiring large amounts of demonstration data and training [5, 2]. While algorithmic improvements in control will inevitably lead to improvements in dexterity over time, an often overlooked source of improvement lies in the sensing modality. Current dexterous robots either use high-dimensional visual data or compact states estimated from them. Both suffer significantly either when the task requires reason- ing about contact forces, or when the fingers occlude the object being manipulated. In contrast to vision, tactile sensing provides rich contact information while being unaffected by occlusions.
So, why is mattle--bsed dexterity hard to achieve? There are three significant challenges. First, tac- tile sensors are difficult to simulate, which limits the applicability sim2real based methods [2, 7] binary contact sensors [8]. Second, for many commonly available tactile sensors, precise calibra- tion of analog readings physical forces is difficult achieve [9]. This limits the applicability of
"orresponddncee rmakguzzy@nyu..edt
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

banile Chservttoon
Toctile
Tactile
Book
Policy
(a) Framework for Tactile-based Dexterity
Visualization T-Dex rollouts three selected robot tasks
Figure T-DEX learns dexterous policies from high dimensional tactile sensors on a multi multi-fingered robot hand. Combined with vision, our tactile representations are crucial to learn fine-grained manipulation tasks.
physics-based control. Third, for multi-fing fungti-i hands, tactile sensors need cover a larger area compared to two-fingered hands. This increases the dimensionality of the tactile observation, which in turn makes learning-basee based approaches inefficient. A common approach alleviate this chal- lenge in vision-based learning is to use pretrained models that encode high-dimensional images to low-dimensional representation. However, such pretrained models do not exist for tactile data.
In this work, we present T-DEX, a new approach to teach tactile-based dexterous skills on multi- fingered robot hands. To overcome issues simulating tactile sensors and calibration, we use an imitation framework that trains directly using raw tactile data obtained by teleoperating the robot. However, directly reasoning about actions from raw tactile data would still require collecting large amounts of demonstrations. address this, we take inspiration from recent works in robot play [10], and pretrain our own tactile representations. This is done by collecting 2.5 hours of aimless data of object manipulation. Tactile data collected through this play is used to train tactile encoders through standard superviedd techniques, mitigating the need for exact force calibration.
Given this pretrained tactile encoder, we it solve tactile-rich dexterous tasks with just handful of demonstrations: demonstrations per task, corresponding to under 10 minutes of demonstration time. To achieve imitation with so few demonstrations, we employ retrieves nearest-neighbor actions from the demonstration set. Such a policy provides significant improvements over fully parameteric policies (Table 1). Importantly, this enables combining tactile encodings with other sensor modalities such as vision without additional training or sensor fusion. This ability combine touch with vision makes T-DEX compatible with tasks that require visual sensing for coarse-grained manipulation and tactile sensing for fine-grained manipulation.
con-parameteric policy that
We evaluate T-DEX across five challenging tasks such as opening a book, bottle cap opening, and precisely unstacking cups (see Figure 1). Through a large-scale experimental study of over 50 hrs of robot evaluation we present the following insights:
1. T-DEX improves upon vision-only and torque-only imitation models with over 170% improve- ment in average success rate (Section 4.2).
Play data significantly improves tactile-based imitation, with an average of 58% improvement over tactile models that do not use play data (Section 4.3).
3. Ablations different tactile representations and architectures show that the design decisions in T-DEX are important for high performance (Section 4.4).
Robot videos and qualitative studies of T-DEX are best viewed on tactile-dexteritty github. ic.
2
Related Work
Our work builds several prior ideas in tactile sensing, representation learning and imitation learn- ing. For brevity, we describe the most relevant below and present more discussion in Appendix A.

---

Tactile Sensing in Dexterity: Multi-fingered robot control has been studied extensively [11, 12, 13]. Initial work focuses on physics-basee modelling of grasping [14, 15] that often used contact force estimates to compute grasp stability. However, contact estimates derived from motor torque only give point estimates and are susceptible to noise due to coupling with the hand's controller. To give robots sense of touch, many tactile sensors have been created for enhancing robotic sens- ing [16, 17, 18]. Prominently, the GelSight sensor has been used for object identification [19], geometry sensing [20]. and pose estimation [21]. However, since GelSight requires a large form factor, it is difficult to cover an entire multifingered hand with it. Instead, 'skin" -like sensors [22] and tactile pads can cover entire hands, yielding high- nigh-dimensional tactile observations for dexterity. In this work, we use the XELA uSkin [23] sensors to cover our Allegro hand.
Learning- based approaches have been employed to leverage high-dimensional readings from tactile sensors for variety of applications such as grasping and manipulation with two-fingered grippers [24, 25, 26], object classification [27] and 3d shape detection [28]. However, these prior works differ from T-DEX in two key ways. First, such tactile learning methods have not been applied to multifingered hands. Second, the tactile representations learned in these works require large amounts of task-centric data for each task. On the other hand, T-DEX uses a large amount of task- agnostic play data, which enables learning tasks with small amounts data per task. Concurrent to our work, binary tactile sensing has shown success for in-hand manipulation [8] through sim2real training. However, its application high-dimensional tactile data is under explored.
Representations Offfine Imitation Imitation Learning (IL) allows for efficient training of skills from data demonstrated by an expert. Given a set of demonstrations, offline imitation methods such as Behavior Cloning (BC) use supervised learning learn a policy that outputs actions similar the expert data [29, 30, 31, 32, 33]. However, such methods often require demonstrations the order of hundreds to thousands trajectories and collecting the same quantity of data for dexterous tasks is difficult due to cognitive and physical demands of teleoperating multi-fingered hands. To learn with fewer demonstrations in high-dimensional action spaces, non-parametric approaches such as nearest neighbors have shown to be more effective than parametric ones [4, 5]. Although efficient, non-parametric learning approaches require good representations of the environment. Several prior work have looked at learning visual features using self- self-supervision [34, 35, 5, 36]. We build on this idea to tactile observations and train tactile features using our human-generated robot play data.
Exploratory and Play Data: Since task-specific data can be expensive to collect, a number of works have examined leveraging off-policy data to improve task performance. Previous work has used play data to learn latent plan representations [37] and to learn a goal-conditioned policy [38]. Recent work in offline RL has noted that including exploratory data improves downstream perfor- mance [39] and that actively straying away from the task improves robustness [40]. These findings are paralleled by studies on motor development in humans. 3-5-month old infants spontaneously explore novel objects [41] and 15-month-old infants produce the same quantity of locomotion in a room with or without toys [42]. Given these motivating factors, we opt to leverage a play dataset of imperfect, tactile-rich interactions in order to improve our representations and task performance.
3 Tactile-Based Dexterity (T-DEX)
T-DFX operates in two phases: pretraining from task-agnostic play data and downstream learning from a few task-specific demonstrations. In the pretraining phase, we begin by collecting a diverse, contact-rich play dataset from a variety of objects by teleoperating the robot (see Appendix B. for the setup). Once collected, we use self-supervised learning (SSL) [43] algorithms on the play data to learn an encoder for tactile observations. In the downstream learning phase, a teleoperator collects demonstrations of solving desired task. Given these demonstrations, non-parametric imitation learning is combined with the pretrained tactile encoder to efficiently learn dexterous policies. See Figure for high-level overview of our framework. Details of individual phases follow.
3

---

Figure Visualization of some of play tasks uses grasping, pinching other manipulation tasks.
3.1 Phase Pre-Training Tactile Representations from Play
The play data is collected from a variety of contact-rich tasks including picking up objects, grasping steering wheel, and in-hand manipulation which are illustrated in Figure 2. We collect total 2.5 hours of play data, including failed examples and random behavior. Because the image and tactile sensors operate at 30Hz and 100Hz, respectively, we sub-sample the data to about 10Hz to reduce the size of the dataset. We only include observations whenever the total changed distance of the fingertips and robot end effector exceed Icm, reducing the dataset from 450k frames to 42k. This filters out the similar states when the robot still, which could potentially bias the SSL phase. All this data, along with camera streams, publicly released on our project website.
Play Data Collection:
Feature Learning: To extract useful representations from the play data own Latent (BYOL) [44], an SSL method which tries to learn informative representations from raw observations. We treat the tactile sensor data as an image with one channel for each axis of force. Fach of the finger's 's 3-axis 4x4 sensors are stacked into a column to produce a 16x4 image for the fingers and a 12x4 image for the thumb. These images are then concatenated to produce a three- channel 16x16 image with constant padding for the shorter thumb. A visualization of tactile images can be seen Figure 3. We pre-train an Alexnet [45] encoder on these tactile images from the play data using BYOL and use that encoder extract tactile features in downstream tasks. More information on BYOL and our use of it on tactile readings can be found in Appendix C.I and D.I.
Bootstrap your
3.2 Phase II: Non-parametric Learning
Demonstration Collection: Six demonstrations are collected for each task by a teleoperator using Holodex [5] framework. Because the nature of our tasks are contact-dependent and the human operator does receive tactile feedback, there relatively high failure rate while collecting demonstrations. Although successful demonstrations correspond at most 10 minutes of robot time, it requires up to 60 minutes of collection time in order to get successful demonstrations which results in 5/6 failure rate in teleoperation. Success of task is determined by visual inspection of robot scene by the teleoperator. This highlights the importance of using tactile feedback for dexterous manipulation along with the necessity of learning from few task-specific demonstrations.
Visual Feature Learning: Many dexterous tasks require coarse-grained information about the location of the object be manipulated. This necessitates incorporating vision feedback as tactile observations not meaningful when the hand is not touching the object. To do this, extract visual features using standard B BYOL augmentations on the images collected from demonstration

---

Phase 1: Self-supervised learning from play
Phase Downstream learning with Nearest-Neighbor retrieval
Play Data Tactile Values
Augmentatioes
Training
Faft
Appleed Action
SSL
Lawo
Jolket
Figure 3: An overview of the T-DEX framework. Left: train factile representations using BYOL large play dataset. Right: we leverage the learned representations using nearest neighbors imitation.
data. The views for each task significantly different, including the play data in the visual representation learning. Similar to prior work [4, 5], we start with ResNet-18 [46] architecture that has been pre-trained on ImageNet [47] classification.
did not observe benefit from
Downstream Imitation Learning: Our action space consists of both the hand pose, specified by 16 absolute joint angles, and the robot end effector position and orientation, specified by a 3- dimensional position and 4-dimensional quaternion. Due to both the high-dimensional action and observation spaces, parametric methods struggle to learn quality policies in the low-data regime. To mitigate this, we use a nearest neighbors-basee imitation learning policy [35] to leverage our demonstrations. For each tuple observations and actions the demonstrations (ol of az), we compute visual and tactile features (vl' v, and store them along-side the corresponding actions. different, we scale both features such that the maximum
Since the scales of the two features distance in the dataset for each feature is 1. At test time given OG, compute (yl vf",, find the datum with the lowest total distance, and execute the action associated with it.
Experiments
We evaluate T-DEX on a range of tasks that are designed to answer the following questions: (a) Does tactile information improve policy performance? (b) How important is play data our repre- sentations? What are important design choices when learning dexterity from tactile information?
4.1 Experimental Overview
Description of Dexterous Tasks: We examine five dexterous contact-rich tasks that require pre- cise multi-finger control. We show the robot rollouts for the selected of the tasks in Figure 1. We describe them in detail in Appendix B.2, showcase the contribution of each taxel to tasks in Ap- pendix C.2 and visualize the rollouts for them in Figure 9. To evaluate various models for dexterity, we first collect six demonstrations for each task in which the object's ' configuration is varying inside a 10x 15em box. Models are then evaluated on new configurations in the convex hull of demonstrated ones. This follows the standard practice of evaluating representations for robotics [48, 49, 50]. Ad- ditional experimental details exist in Appendix B.
We study the impact of tactile information on policies learned through imitation, comparing against a number of baselines. Here, we briefly describe these baselines. Ad- ditional baseline and model details can found in Appendix B.3-D.
Baselines for Dexterity:
1. BC[]]]: We train neural network end-to-end to map from visual and tactile features actions.
2. NN-Image [35]: We perform nearest neighbors with the image features only.
3. NN- NN-Torque [52]: We perform nearest neighbors with the output torques from our PD controller
and visual representations.
4. NN-Tactile: Nearest neighhors with only the tactile features trained on play data.
5

---

Table I: Real-world of the learned policies
BC
NN-Im
NN-Tac
NN-Task
NN-Tor
BET
IBC
GMM
T-DEX
Joystick
0/10
4/T0
6/10
8/10
7/10
0/10
0/10
0/10
8/10
Bottle
0/10
0/10
6/10
3/10
3/10
0/10
0/10
0/10
6/10
Cup
0/10
0/10
0/10
4/10
2/10
0/10
0/10
0/10
8/10
Bowl
0/10
2/10
2/10
3/10
4/10
0/10
0/10
0/10
7/10
Book
0/10
5/10
0/10
6/10
3/10
0/10
0/10
0/10
9/10
Average
0/10
2.2/10
2.8/10
4.8/10
3.8/10
0/10
0/10
0/10
7.6/10
Robot observation
Image-only
Tactile-only
T-Dex
Figure 4: Visualization the camera image, top-two activated tactile sensors, and their nearest neighbors for T-DEX and baselines. We observe that baselines fail to either recognize the contact with the object (image- only) capture the position of the robot (tactile-only).
NN-Task: Instead of training the tactile encoder on the play data, we train it on the 6 task-specific demonstrations. Representation retrieval same as T-DEX.
Behavior Transformer (BeT) [31]: Transformer based behavior cloning baseline that focuses on capturing distributionally multi-modal behaviors from diverse set of demonstration data. Implicit Behavioral Cloning (IBC) [30]: We fit energy based model (EBM) on the joint ob- servation and action space and choose actions that minimize the output of this EBM.
Gaussian Mixture Models [53] on top of BC (BC-GMM) [32]: A probabilistic model that repre- sents the action space combination of multiple Gaussian distributions given observations.
9. T-DEX: This is proposed method with the tactile encoder pre-trained all the play data followed by nearest neighbor retrieval with image and tactile features on task data.
Quantitative results can be found in Table 1, while robot rollouts and qualitative comparisons of the baselines are shown our website tactile- sentile-dexterity gi github.io.
4.2 How important is tactile sensing for dexterity?
Table we report success rates of T-DEX along with baseline methods. We observe that among the nearest neighbor based methods, we find that tactile-only (NN-tactile) struggles Book Opening and Cup Unstacking since the hand fails to local- ize the objects to make first contact. On the other hand, the image-only (NN-Image) struggles on Bot- tle Opening and Cup Unstacking as severe occlu- sions caused by the hand result in poor retrievals, Using torque targets (NN-Torque) instead of tactile feedback proved useful, improving Nm-gmage, but did not match using tactile.
T-DEX
Image only
X
X
Figure S: Visualization of the failure modes of our Image only baseline. Without tactile information, the robot applies either much force does correctly contact objects.

---

We find that combining the coarse- searse-gained localization ability of NN-Image along with the fine- grained manipulation of NN-Tactile results in the strongest results across all tasks. To further analyze why T-DEX performs so well, we visualize the nearest neighbors of states for the image-only and tactile-only methods Figure 4. And show additional failure modes for NN-Image in Figure 5. Our method produces neighbors that seem to capture the state of the world better than other baselines. More details of the robot policy can be found in Appendix B.2 and E.
4.3 Does pre-training on play improve tactile representations?
To understand the importance of pre-training, NN-Task, which pre-trains tactile representations task data. As seen in Table 1, this baseline does quite well on the simpler Joystick Movement task. However, on harder tasks, particularly the Unstacking tasks and Bottle Opening, we find that NN-Task struggles signif- icantly. This can be attributed to poor representational matches when trained limited task data. To mitigate this, we also try training the encoder with a combi- nation of successful and failed demonstrations on the Bowl Unstacking task, getting success rate of 30%, which shows no improvement in task performance.
ofPla
110
Openin
Succeo Rast
Plas
Figure 6: Success rate on unstacking tasks with varying amount of play data. Training only on task data performs moderately well, but is out- performed with just minutes play.
To provide further evidence for the usefulness of tac- tile pretraining, we plot the gains in performance across varying amounts of play data in Figure 6. We see that for easier tasks like Book Opening, even small amounts of play data (20 mins) is suffi- cient to achieve a 90% success rate. However, for harder tasks like Cup Unstacking, we see steady improvements in success rate with larger amounts of play data.
4.4 Analysis of the design decisions in T-DEX
Choice of Policy Class: T-DEX uses non- non-parametric class instead commonly used parametric ones. To study the importance of this choice, we compare across a variety of fully parametric policy architectures including BC [29]. IBC [30], BeT [31], BC-GMM [32] in Table I. We observe that parametrie methods fail to learn high-quality policies and significantly overfits the small amount of data with the high-dimensional action space. This emphasizes the need for retrieval- based policies, which have shown success with few demonstrations [35, 5].
Choice of the encoder architecture: A critical component in T-DEX the architectural details in representing and processing tactile data. In this section, examine various approaches represent tactile features. For simplicity, we study subset of the tasks, Book Opening and Cup Unstacking and show the results in Table 2. Each encoder is trained using BYOL on the play dataset with the same augmentations used in the main method. We compare our main encoder, AlexNet with few different architectures: (a) ResNet: A standard Resnet-18 [46] with weights pre-trained on the ImageNet [47] classification task. (b) 3-layer CNN: A CNN with three layers initialized with random weights. (c) Stacked CNN: Rather than laying out the sensor data of the fingers spatially in the image, consider stacking the sensor output into 45-channel image. (d) Shared CNN: We pass individual pad values to the same network and concatenate the outputs. (e) Raw Tactile: We flatten the raw tactile data into a 720-dimensional vector instead of using an encoder.
Table 2: Success rates various representations tactile data on two of our tasks.
T-DEX
ResNet
3-layer
Stacked
Shared
Raw
Book Opening
9/10
9/10
6/10
5/10
2/10
5/10
Cup Unstacking
8/10
6/10
3/10
1/10
1/10
3/10
7

---

We find that both T-DEX and ResNet perform similarly on Book Opening, although ResNet takes significantly more computation for the same results. On Cup Unstacking we find that ResNet per- forms a little worse than T-DEX, which further informs our architectural choice. While, one may conclude that smaller architectures are better, we see that a simpler 3-layered CNN also performs poorly and does not reach the performance of either of the larger models.
Structure of the tactile input: Apart from the architecture, we find that the structure of inputting tactile data from individual tactile pads is also important. For example, we find that stacking tactile pads channel-wise is substantially worse than T-DEX that stacks the tactile pads spatially. Similarly we find that using a shared encoder for each tactile pad is also poor. This is perhaps because of the noise that exists in high-dimensional raw tactile data, which difficult filter out with the stacked and shared encoders. Hence, one spurious reading in unused tactile pad could yield incorrect neighbor, producing bad action. This hypothesis is further substantiated by the result of the Raw Tactile method, which is roughly on par with the Stacked method.
Alternative representations: We additionally run three experiments with different tactile repre- sentations on the Bowl Unstacking task to amalyze our choice of representation. We run PCA on the Raw Tactile features on the play dataset and use the top 100 components as features, achieving success rate of 40%. When PCA fails, it is not able to capture fine- grained tactile information that is necessary to solve the task. Next, we sum the activations of each 4x4 tactile sensor in each dimen- sion to create a 45-dimensional feature, which does not succeed on any task. Finally, we shuffle the order of the pads in the tactile image, which achieves 20% success, which is much lower than using the structured layout (Section 3.1), showing that the layout of tactile data crucial for performance.
4.5 Generalization to Unseen Objects
To examine the generalization ability of T-DEX,
the Bowl Unstack- ing and Cup Unstacking tasks with unseen crockery. T-DEX policies for each of these tasks were trained using only one set of objects, seen in Fig- ure (b). Without any modifications, the policy is then run on new objects, with the objects placed in 10 different configurations. As seen in Figure 7, T-DEX is able to generalize to these new objects in varying degree. For objects of similar shapes and size as the training object, T-DEX does quite well. However, it begins to fail when the objects change significantly. For example, the inner bowl in the right-most column for Bowl Unstacking too small for the hand to piek up.
Cup
6/10
5/10
5/10
4/10
Bowl
7/10
3/10
3/10
0/10
Figure 7: We show success rates of T-DEX objects not seen during demonstration collection.
5 Limitations and Conclusion
In this work, we have presented approach for tactile-based dexterity (T-DEX) that combines tactile pretraining play data along with efficient downstream learning a small amount of task- specific data. Our results indicate that T-DEX can significantly improve over prior approaches that use images, torque and tactile data. However, we recognize two key limitations. Although T-DEX succeeds on several uuttof-distribution examples, the success rate is lower than the training setting. The second, is that our approach currently limited to offline imitation, which limits the ability of our policies learn from failures. Both limitations could be addressed by integrating online learning and better factile-vision fusion algorithms. While these aspects out of scope to this work, hope that the ideas introduced in T-DEX spur future work in these directions.
8

---

Acknowledgments
We thank Vaibhav Mathur, Jeff Cui, Ilija Radosavovic, Wenzhen Yuan and Chris Paxton for valuable feedback and discussions. This work was supported by grants from Honda, Meta, Amazon, and ONR awards N00014-21-1-7588 and N00014-22-1-2775.
References
[1] V. Kumar, E. Todorov, and S. Levine. Optimal control with learned local models: Application to dexterous manipulation. In 2016 IEEE International Conference on Robotics and Automa- lion (ICRA), pages 378-383, 2016. doi:10.1 109/ICRA.2016..48815555
[2] OpenAI, M. Andrychowicz. B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder,
L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation, 2019.
[3] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving rubik's cube with robot hand. arXiv, 2019.
[4] P. Arunachalam, S. Silwal, B. Evans, and Pinto. Dexterous imitation made easy: A learning-base framework for efficient dexterous manipulation. arXiv preprint arXiv:2203. 13251, 2022.
[5] S. P. Arunachalam, I. Güizey, S. Chintala, and L. Pinto. Holo-dex: immersive mixed reality, 2022. URL https: //arxiv..vvv//////////a///aaaav owgg/bs///// 06463.
Teaching dexterity with
[6] A. Handa, K. Van Wyk, W. Yang, J. Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and
D. Fox. Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164-91700 2020. 10110.11////RA4494455 20209999995522299999929
[7] Y. Wang. W. Huang, Fang, Sun, and C. Li. Elastic tactile simulation towards tactile-visual perception. In Proceedings of the 29th ACM International Conference Multimedia, pages 2690-2698, 2021.
[8] Z.-H. Yin, B. Huang, Y. Qin, Q. Chen, and X. Wang. Rotating without seeing: Towards in-hand dexterity through touch. arXiv preprint arXiv:2303. 10880, 2023.
[9] H. Lee, H. Park, G. Serhat, H. Sun, and K. J. Kuchenbecker. Calibrating a soft ert-based tactile sensor with multiphysics model and sim-to-reat transfer learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1632-1638. IEEE, 2020.
[10] Young. Pari, P. Abbeel, and L. Pinto. Playful interactions for representation learning. arXiv preprint a1X1v:2107.09046, 2021.
[11] M. T. Ciocarlie, C. Goldfeder, and P. K. Allen. Dexterous grasping via eigengrasps A low- dimensional approach to a high-complexity problem. In Dexterous Grasping via Eigengrasps Low-dimensional Approach to a High-complexity Problem, 2007.
[12] V. Kumar, Y. Tassa, T. Erez, and E. Todorov. Real-time behaviour synthesis for dynamic hand- manipulation. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 6808-6815. IEEE, 2014.
[13] Shigemi. ASIMO and Humanoid Robot Research at Honda, pages 1-36. Springer Nether- lands, Dordrecht, 2018. ISBN 978 978-94-007-7194499 URL httph::t/////...... ptttss://doii /ww.//o.iia org.
9-2.

---

[14] A. M. Okamura, N. Smaby, and M. R. Cutkosky. An overview of dexterous manipulation. In Proceedings 2000 ICRA. Millennium Conference. IEFE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 1, pages 255-262. IEEE, 2000.
[15] L. U. Odhner, L. P. Jentoft, M. R. Claffee, N. Corson, Y. Tenzer, R. R. Ma, M. Buehler, R. Ko- hout, R. D. Howe, and A. M. Dollar. compliant, underactuated hand for robust manipulation. The International Journal of Robotics Research, 33(5):736-752, 2014.
[16] Wang, She, Romero, and E. Adelson. Gelsight wedge: Measuring high-resolution 3d contact geometry with compact robot finger, 2021. URL https:://rxxvv.orge 2106. 08851.
[17] R. M. Bhirangi, T. L.. Hellebrekers, C. Majidi, and A. Gupta. Reskin: versatile, replaceable, lasting tactile skins. CoRR, abs/2111.0007/. 2021. URL https://azxiv.org//
2111.00071.
[18] A. Alspach, K. Hashimoto, N. Kuppuswamy, and Tedrake. Soft-bubble: A highly com- pliant dense geometry tactile sensor for robot manipulation. In 2019 2nd IEEE International Conference Soft Robotics (RoboSoft), pages 597-604. IEEE, 2019.
[19] R. Patel, R. Ouyang, B. Romero, and E. Adelson. Digger finger: Gelsight tactile sensor for object identification inside granular media. In Experimental Robotics: The 17th International Symposium, pages 105-115. Springer, 2021.
[20] S. Dong, W. Yuan, and E. H. Adelson. Improved gelsight tactile sensor for measuring geom- etry and slip. In 2017 TEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 137-144. IEEE, 2017.
[21] T. Kelestemur, R. Platt, and T. Padir. Tactile pose estimation and policy learning for unknown object manipulation. arXiv preprint arXiv:2203. 10685, 2022.
[22] R. Dahiya, N. Yogeswaran, F. Liu, L. Manjakkal, E. Burdet, V. Hayward, and H. Jörntell. Large-area soft e-skin: The challenges beyond sensor designs. Proceedings of the IEEE, 107 (10):2016-2033, 2019, doi: 10.1 109///0O...00199.... 292491366.
[23] T. P. Tomo, M. Regoli, A. Schmitz, L. Natale, H. Kristanto, S. Somlor, L. Jamone, G. Metta, and S. Sugano. A new silicone structure for uskin a soft, distributed, digital 3-axis skin sensor and its integration on the humanoid robot icub. IEEE Robotics and Automation Letters, 3(3): 2584-2591, 2018. 10t.00.119/LRA..20
2812915.
[24] A. Murali, Y. Li, D. Gandhi, and A. Gupta. Learning to grasp without seeing. In J. Xiao, T. Kröger, and O. Khatib, editors, Proceedings of the 2018 International Symposium on Ex- perimental Robotics, pages 375-386, Cham, 2020. Springer International Publishing. ISBN 978-3-030-33950-0.
[25] R. Calandra, A. Owens, D. Jayaraman, Lin, W. Yuan, Malik, E. H. Adelson, and Levine More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters, 3(4):3300-3307, 2018.
[26] Y. She, Wang, S. Dong, N. Sunil, Rodriguez, and Adelson. Cable manipulation with tactile-reactive gripper, 2019, URL https: /ttp/aaaivv//.////////..///..a.. org/ abb/1910/1111././///// 02860.
[27] M. Zambelli, Y. Aytar, Visin, Zhou, and R. Hadsell. Learning rich touch representations through cross-modal self-supervision. In Conference on Robot Learning, pages 1415-1425. PMLR, 2021.
[28] S. Wang, J. Wu, X. Sun, W. Yuan, W. T. Freeman, B. Tenenbaum, and E. H. Adelson. 3d shape perception from monocular vision, touch, and shape priors. In 2018 JEEE/RSJ Interna- tional Conference Intelligent Robots and Systems (IROS), pages 1606-1613, IEEE, 2018.
10

---

[29] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In NeurlPS, pages 305-313, 1989.
[30] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mor- datch, and J. Tompson. Implicit behavioral cloning, In Conference on Robot Learning, pages 158-166 158-168. PMLR, 2022.
[31] N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L.. Pinto. Behavior transformers: Cloning $k$ modes with one stone. In Advances in Neural Information Processing Systems, 2022. URL https: http:////peeeeieeeeeeeeeeeeeeeee ew net/ / forum? x--
[32] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. arXiv preprint arXiv:2210. 1339, 2022.
[33] A. Mandlekar, D. Xu, J. Wong. S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese,
Y. Zhu, and R. Martin-Martin. What matters learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:21085 03298, 2021.
[34] Sermanet, C. Lynch, Y. Chebotar, Hsu, E. Jang. S. Schaal, Levine, and G. Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pages 1134-1141. IEEE, 2018.
[35] J. Pari, N. M. Shafiullah, S. P. Arunachalam, and L. Pinto. The surprising effectiveness of representation learning for visual imitation, 2021.
[36] A. Zhan, R. Zhao, L. Pinto, P. Abbeel, and M. Laskin. A framework for efficient robotic manipulation. In Deep RL Workshop NeurIPS 2021, 2020.
[37] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play, 2019.
[38] Z. J. Cui, Y. Wang, N. Muhammad, L. Pinto, al. From play to policy: Conditional behavior generation from uncurated robot data, arXiv preprint arXiv:2210.10047. 2022.
[39] D. Yarats, D. Brandfonbrener, H. Liu, M. Laskin, P. Abbecl, A. Lazaric, and L. Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201.13422 2022.
[40] D. Brandfonbrener, S. Tu, A. Singh, S. Welker, C. Boodoo, N. Matni, and J. Varley, Visual backtracking teleoperation: A data collection protocol for offline image-based reinforcement learning, 2022. URL https: / /ttp///avvvv//.////////.e//..avv org/ abs 2210. 02343.
[41] P. Rochat. Object manipulation and exploration in 2-to 5-month-ole infants. Developmental Psychology, (5)6) 1989.
[42] J. Hoch, S. M. O'Grady, and K. E. Adolph. It's the journey, not the destination: Locomotor exploration in infants. Developmental Science, 22(2), Oct. 2018.
12740.
URL https::t///........ / doi. wwwoi.org/1000 /r1g//0.1111111 w1w.1/des..11e1e desc.
[43] L. Ericsson, H. Gouk, C. C. Loy, and T. M. Hospedales. Self-supervised representation leamn ing: Introduction, advances, and challenges. IEFE Signal Processing Magazine, 39(3)-22-62, 2022.
[44] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS, 2020.
[45] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.

---

[46] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[47] Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierar- chical image database. In 2009 IEEE conference on compuler vision and patlern recognition, pages 248-255. Icce, 2009.
[48] G. Zhou, V. Dean, M. K. Srirama, A. Rajeswaran, J. Pari, K. B. Hatch, A. Jain, T. Yu, P. Abbeel, L. Pinto, C. Finn, and A. Gupta. Train offline, test online: A real robot learn- ing benchmark. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022. URL https: http//ppppeeeeeeeeeeeeeeeeeeeeee .we.ee.eeeeeeeeeeeeeeeeeeeeeeeee worormiid=egr/nneeee
[49] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. arXiv preprint arXiv:2233//2601, 2022.
[50] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training, 2022. URL https: //axxvv./// wttp:/vv.rg//b///////1 09.09.
[51] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In NIPS, 1989.
[52] L. Sievers, J. Pitz, and Bäuml. Learning purely tactile in-hand manipulation with torque- controlled hand. In 2022 International Conference Robotics and Automation (ICRA), pages 2745-2751, 2022. doi: WWW//ICCRA46633.222222999
[53] D. Reynolds. Gaussian Mixture Models, pages 659-663. Springer US, Boston, MA, 2009. ISBN 978-0-387-73003-5. org/10 1007/9787887 3-7 1130003-5_190
196. URL https: //doi.
[54] I. Mordatch, Z. Popović, and Todorov. Contact-invariant optimization for hand manipula- tion. In Proceedings of the ACM SICGRAPPH/Eurographies symposium computer anima- tion, pages 137-144, 2012.
[55] K. Lowrey, A. Rajeswaran, S. Kakade, E. Todorov, and I. Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. arXiv preprint arXiv: 1.01848, 2018.
[56] A. Nagabandi, K. Konoglie, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. arXiv, 2019.
[57] W. Huang, I. Mordatch, P. Abbeel, and D. Pathak. Generalization in dexterous manipulation via geometry- multi-task learning, arXiv preprint arXiv:21 2021.
[58] T. Chen, J. Xu, and P. Agrawal. A system for general in- hand object re-orientation. Conference on Robot Learning, 2021.
[59] H. 7hu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In ICRA, 2019.
[60] K. Lowrey, S. Kolev, J. Dao, A. Rajeswaran, and E. Todorov. Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system. In 2018 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR), pages 35-42. IEEE, 2018
[61] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, Y. Narang, J.-F. Lafleche, D. Fox, and G. State. Dextreme: Transfer of agile in m-hand manipulation from simulation to reality. arXiv, 2022.
12

---

[62] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. In RSS, 2018.
[63] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 512-519. IEEE, 2016.
[64] D. Ha and Schmidhuber. World models. arXiv preprint 1803. 2018.
[65] L. Pinto, D. Gandhi, Y. Han, Y.-.. Park, and A. Gupta. The curious robot: representations via physical interactions. In ECCV, 2016.
Learning visual
[66] P. R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In Conference on Robot Learning, pages 373-385. PMLR, 2018.
[67] B. Chen, A. Sax, G. Lewis, I. Armeni, S. Savarese, A. Zamir, J. Malik, and L. Pinto. Ro- bust policies via mid-level visual representations: An experimental study in manipulation and navigation. arXiv preprint arXiv:2011.06692 2020.
[68] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709. 2020.
[69] M. Caron, I. Misra, Mairal, Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances neural information processing systems, 33:99 12-9924, 2020.
[70] A. Bardes, J. Ponce, and Y. LeCun. Vicreg: Yariane-iivariance-svvaraece regularization for self-supervised learning. arXiv preprint arXiv:2105.0495 049906 2021.
[71] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Reinforcement learning with prototypical rep- resentations. In International Conference on Machine Learning, pages 11920-1 17200-1933 1131.. PMLR, 2021.
[72] M. Laskin, A. Srinivas, and P. Abbeel. Curl: reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
Contrastive unsupervised representations for
[73] D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, and Kashino. BYOL for Audio: Exploring pre-trained general-purpose audio representations. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:137-151, 2023. ISSN 2329-9304. doi:10. 109/ 3221007. URL httptttpp/.....ee/etetttptpttttt http://tt.......o..oo.ooo//// 11..0..1191/AA. .0...222222222...2..22222222.. 3221007.
[74] M. Afham, I. Dissanayake, Dissanayake, A. Dharmasiri, K. Thilakarathna, and R. Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9902-9912,2077 2022.
13

---

Appendix
A Related Work
A.1 Learning in Dexterous Manipulation
There are several methodologies for using learning in dexterity. Model-based reinforcement learn- ing (RL) methods have been shown to work in both simulation [54, 55] and the real world [56, I]. Model-free RL has been used to train policies both in simulation [57, 58] and directly on hard- ware [59]. Simulation real transfer has also shown success [60, 2, 61, 52], though it often requires extensive randomization, significantly increasing training time. The of expert demonstrations reduce the amount of real-world interactions needed to learn dexterous policy [62. 59, 4]. The works mentioned above either use visual observations or estimates of object state, which suffer during heavy occlusion of the object.
A.2 Representation Learning for Robotics
Learning concise representations from nigh-dimensional observations is an active area research in robotics. A wide variety approaches using auto-encoder. [63, 50, 64]. physical interaction [65], dense descriptors [66], and mid-level features [67] have been studied.
In computer vision, self-supervised learning (SSL) is often used pre-train visual features from unlabeled data, improving downstream task performance. Contrastive methods learn features by moving features of similar observations closer to one another and features of dissimilar observations farther from one another [68, 69). These methods require sampling negative pairs of datapoints, which adds an additional layer of complexity. Non-contrastive methods typically try to learn fea- tures by making augmented versions of the same observation close [44, 70] and do not require sampling negative examples. Self-supervision has been adopted for visual RL [71, 72, 49, 50] and robotics [34, 35, 5, 36] to improve sample efficiency and asymptotic performance. SSL methods have also been applied to other sensory inputs like audio [73] and depth [74]. We build on this idea of self-supervision and extend it to tactile observations. However, unlike visual data, for which large pretrained models or Internet data exists, neither are available for tactile data. This necessitates the creation of large tactile datasets, which we generate through robot play.
B Experiment Details
B.1 System Details and Robot Setup
Our robotic system, visualized in Fig- ure 8, consists of robotic arm and hand. The arm is 6-dof Kinova Jaco and the hand is a 16-dof Allegro hand with four fingers. The arm can be teleoperated through
Kinova Robot Arm
Allegro Hand
Realsense Cameras
built-in Ki- nova joystick, while the the hand can be teleoperated using the the Holo- Dex framework [5]. Here, teleop- erator uses virtual reality headset to both visualize robot images and con- trol the hand in real time. The headset returns a pose estimate for each fin- ger of the hand which is re-targeted the Allegro Hand. Inverse Kine- matics is then used to translate target Cartesian positions in space to joint
Oculus Headset
Kinova Joystick
Figure 8: Hardware setting T-DEX. We use an Oculus Headset to teleoperate the Allegro hand and the built Kinova joystick to control the arm. Visual observations are streamed through two different Realsense cameras and tactile observations saved with XELA touch sensors on the Allegro hand.
14

---

Joystick
Mov
Bettle Opening
Cup Unitakingg
Bowl Unstacking
Book Opening
Figure 9: Visualization of all rohot rollouts from T-DEX policies. Note the severe visual occlusions when the robot makes contact with the object.
angles, which are fed into the low-level hand controller. achieve robust position control, we use low-level PD joint position controller with gravity compensation allow the robot maintain hand pose at different orientations in space. Our action space is Cartesian position and orientation of the arm (3D position and 4D quaternion for orientation) and the 16-dimensional joint state of the hand for total of 23 dimensions.
The Allegro hand fitted with XELA uSkin tactile sensors, on each finger and on the thumb. Each sensor has 4x4 resolution output of tri-axial force reading (forces in translational y, 2) information, which amounts to 720-dimensional tactile reading. The force readings uncalibrated, susceptible to hysterisis, and can change when strong magnets metals in the vicinity. Due to this, we opt against explicit calibration of the 720 sensor units. To supplement the tactile sensors, we also use two RGB cameras with 640x480 resolution to capture visual information in the scene, though our policies only uses information from one to execute. Our choice of camera for tasks depends on which one captures the most visual information about the objects to ensure fairness when comparing to baselines and enable better joint vision and tactile control.
B.2 Task Details
Here we explain the details of our tasks and visualize the robot rollouts for each of them in Figure 9.
1. Joystick Movement:
Starting over arcade gamepad, the hand is tasked with moving down and pulling a joystick backwards. This task is difficult because the hand occludes the gamepad when manipulating it. We collect demonstrations of the joystick in two different positions and evaluate on different positions and orientations not seen during training. A trial is successful if the joystick has been pulled within 60 seconds.
2. Bottle Opening:
This task requires the hand to open the lid of a bottle. We collect three demonstrations with the bottle orientation requiring the use of the thumb, and three other requiring the use of the middle finger. The task is considered successful if the lid is open within 120 seconds.
3. Cup Unstacking: Given two cups stacked inside one another, the tasks is to remove the smaller cup from the inside of the larger one. addition to occlusion, this task requires making contact both the inner and outer cups before lifting the inner cup with the index finger. It is considered success if the smaller cup is raised outside the larger cup without dropping it or knocking the cup off the table within 240 seconds.
15

---

4. Bowl Unstacking:
This task is similar to the previous, but with bowls instead of cups. Since the bowls are larger, multiple fingers are required to lift and stabilize them. A run is successful if it has lifted the bowl within 100 seconds.
5. Book Opening: This task requires opening a book with three fingers. After making contact with the cover, the hand must pull up with an arm movement, remaining contact until is fully open. The task is considered success if the book is open within 300 seconds.
B.3 Baseline Details
Here we explain the details of all our baselines. Unless explicitly explained all the image features are received by an image encoder trained with BYOL on the task-specific demonstrations and all the tactile features are received by tactile encoder pre-trained on all the play data. Success rates of each of the baselines are shown in
1. Behavior Cloning (BC) [51]: We train neural network end-to-end to map from visual and tactile features to actions.
2. Nearest Neighbors with Image only (NN-Image) [35]: We perform nearest neighbors with the image features only. During evaluation, to ensure fairnsss we use viewpoints that can convey maximal information about the scene.
3. Nearest Neighbors with Torque and Image (NN-Torque) [52]: We perform nearest neigh- bors with the output torques from our PD controller and visual observations. The torque targets be used proxy for force, providing some tactile information.
4. Nearest Neighbors with Tactile only (NN-Tactile): Nearest neighbors with the tactile fea- tures trained on play data. Unlike T-DEX we do not use vision data for this baseline.
5. Nearest Neighbors with Tactile Trained on Task Data (NN-Task): Instead of training the tactile encoder on the play data, we train it on the 6 task-specific demonstrations.
6. Behavior Transformer (BET) [31]: We train a transformer predict action modes and off- sets given image and tactile features. This method is known to capture the multi-mont modality of the environment given unlabeled multi-task data.
7. Implicil Behavioral Cloning (IBC) [30]: We train a model that outputs the energy of the representation space given image and tactile features and optimize for actions that minimize the energy landscape. This method is proven to capture the stochastic nature of the envi- ronments and recover better in out-of distribution modes compared to the explicit behavior cloning (BC) approach.
8. Gaussian Mixture Models (53) on top of BC (BC-GMM) We fit a Gaussian Mixture Model on the action space and sample actions from a weighted sum of Gaussian component den- sities from given image and tactile features.
9. Nearest Neighbors with Tactile Trained on Play Data (T-DEX): This is our main method with the tactile encoder pre-trained on all the play data followed by nearest neighbor re- trieval task data.
the joystick, bottle, and book the table. This mimics having another manipulator keep the object in place while the hand manipulates the object. The bowls and cups not secured, making the problem of unstacking much more difficult.
For simplicity,
To ensure fair evaluation, we start each method with the object in the same configuration for each index trial. This corresponds to 10 different starting positions, each which used at the start of each baseline run.
C Tactile Data Details
In this section give more details regarding the mapping of the tactile readings to our robotic setup and the importance of each of the sensor readings to tasks.
16

---

C.1 Mapping of the tactile images
Figure 10 showcase where each tactile pad is on the Allegro hand and how they are mapped to tactile readings and images.
Tactilc Readings
15
Tactile Image
13
Figure 10: We show the location of each tactile pad and how they are mapped to the corresponding tactile readings and images. Locations could be observed by following the numbered pads and visualizations.
C.2 Contribution of each tactile reading to tasks
In order to explore the contribution of each tactile reading to our tasks we created figure to show the heatmaps for the variances of each taxel through robot trajectories for different tasks. We observe that the lower segments the fingers have very high variance due to the lower segments touching the palm of the hand in most of the tasks. The variance in the other segments show the importance of each pad as follows:
Bowl Unstacking: The 2nd and 3rd segment tactile pads of the thumb.
Bottle Cap Opening: Tips of middle, index and the thumb.
Cup Unstacking: Tip, 2nd and 3rd segment of the thumb.
Joystick Movement: Tips of index, middle and ring finger.
Book Opening: Tips index, middle and ring finger and the 3rd segment of the thumb.
Heatmaps for the variances can be seen in Figure 11.
D
Model Details
Here we provide additional details about our method and baselines for easier reproduction.
For all image-based models, we normalize the inputs based on the mean and standard deviation of the data seen during training. For the tactile-based models, we normalize the inputs to be within the range [0, 1].
D.I BYOL Details
Bootstrap your Latent: BYOL has both primary encoder fe, and a target encoder fe, which is an exponential moving average of the primary. Two augmented views of the same observation and o' are fed into each to produce representations y and y', which are passed through projectors ge
17

---

Hcatmap of Usage of Each Tactile Pad
Bowl Unstacking
Bottle Cap Opening
Cup Unstacking
Joystick Movement
Book Opening
2200
Figure Heatmaps variances each taxels through our robot trajectories.
and 95 to produce and z', which are higher dimensional. The primary encoder and projector are then tasked with predicting the output of the target projector. After training, we use extract features from observations.
Using Tactile Readings with BYOL: After getting the tactile images as mentioned in 3.1, we scale the tactile image up to 224x224 to work with standard image encoders. For the majority of our experiments, we use the AlexNet [45] architecture, also starting with pre-trained weights. Unlike SSL techniques in vision [44], we only apply the Gaussian blur and small random resized crop augmentations, since other augmentations such color jitter and grayscale would violate the assumption that augmentations do not change the tactile signal significantly. Importantly, unlike vision, since all of the tactile data is collected in the frame of the hand, the sensor readings are invariant to hand pose and can be easily reused between tasks.
The complete list of BYOL hyperparameters has been provided in Table We take the model with
the lowest training loss out of all the epochs.
D.2 Nearest Neighbors Details
We give equal weight to visual and tactile distances for all of the tasks except bottle cap. where tactile and image features were given weights of and 1, respectively. We do this because the quality of the neighbors on image data was poor and emphasizing the tactile data slightly vastly improves performance.
While executing NN imitation, keep buffer of recently executed neighbors that call the reject buffer. Given new observation, we pick the first nearest neighbor not in the reject buffer. This prevents the policy from getting stuck in loops if a chain of neighbors and actions are cyclical. We set the reject buffer size to 10 for every task except Joystick Pulling, which set to 3. The buffer, combined with the 2cm spatial subsampling are critical for the success NN policies.
D.3 BC Details
We train BC end-to-end using standard MSE loss the actions with the same learning rate as BYOL and a batch size of 64.
18

---

Parameter
Value
Optimizer
Adam
Learning rate
1e-3 3
Weight decay
le-3
Max epochs
1000
Batch size (Tactile)
1024
Batch size (Image)
64
Aug. (Tactile)
Gaussian Blur (3x3) (1.0, 2.0) p= 0.5
Random Resize Crop (0.9, 1.0) 0.5
Aug. (Image)
Color Jitter (0.8, 0.8, 0.8, 0.2) p 0.2
Gaussian Blur (3x3) (1.0, 2.0) p = 0.2
Random Grayscale = 0.2
Table 3: BYOL Hyperparameters.
Explaieed Variazee Number c/Cmppnen
390
200
4.03
390
602
Figure 12: Explained variance ratio for PCA the play tactile data. Most variance is captured in the first 100 components.
D.4 NN-Torque Details
Our hand does not have torque sensors, but is actuated by torque targets from low-level PD position controller: We use the torque targets proxy for torque information since the desired torque will be higher when the finger is in contact with an object, but trying to move further inside, and lower when it is not in contact.
D.5
PCA Details
We run PCA on the tactile play data and take the top 100 components for use as features. The captured variance about 95% and the entire explained variance ratio can be seen Figure By visualizing the reconstructions (Figure 13), we can see that retains a majority of the tactile information.
D.6 Shuffled Pad Details
For this experiment, we permute the position of the 15 4x4 tactile sensors using the same permutation for both pretraining and deployment. This ensures that we'ne ee'r inputting the same data from each sensor to each location in the tactile image, but does not leverage the spatial locations of the pads
19

---

Trath
Figure 13: Tactile data and the PCA reconstruction of two using 100 components for two tactile readings. Most of the information is preserved, but we can see minor differences in magnitude and offset.
Bottle Opening
Miinag
Figure 14: Additional rollouts for the Bottle Opening task.
the hand. If spatial layout had no effect, we would expect no difference in the performance hetween this and T-DEX.
E Additional Rollouts
We visualize extra rollouts for each task in Figures 14-18.
Tactile Image Visualization
We visualize tactile images for each task in Figures 19-23.
20

---

Book Opening
Figure 15: Additional rollouts for the Book Opening task.
Bowl Unstacking
Figure 16: Additional rollouts for the Bowl Unstacking task.
21

---

Cup Unstacking
Figure 17: Additional rollouts for the Cup Unstacking task.
Joystick Pulling
Figure 18: Additional rollouts for the Joystick Pulling task.
22

---

Figure 19: Tactile Image for the Bottle Opening task.
Figure 20: Tactile Image for the Book Opening task.
23

---

Figure 21: Tactile Image the Bowl Unstacking task.
Figure 22: Tactile Image for the Cup Unstacking task.
24

---

Figure 23: Tactile Image for the Joystick Pulling task.
25


Reviews:
[{"review_id": "yJ41xj-wxr", "reviewer": "Reviewer_Tv7A", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a method to learn dexterous manipulation from visuo-tactile data combining techniques of play and offline non-parametric imitation learning. To do this, the authors present a two-phase approach. In stage 1, large amounts of unrestrained play data is collected using teleoperation on a set of objects. Then an encoder is trained on the tactile sensory images using BYOL in a self-supervised fashion. In stage 2, a small number of demonstrations are collected for each task and embedded using the pretrained encoder in stage 1. During test time, the input observations are embedded and actions are outputted based on a nearest-neighbor search on the demonstration embeddings. The results indicate that the play data for tactile representation learning plays a crucial role when combined with visual data for learning dexterous manipulation.", "summary_of_recommendation": "The paper demonstrated how tactile play data can be leveraged for pretraining encoders for learning tactile representations using SSL to be later used for offline IL. The evaluation is thorough and the results demonstrate the efficacy of visuo-tactile learning in high-dimensional action spaces.\n\n### Post Rebuttal\nThe authors have updated the paper according to the provided suggestions and added necessary details about the tactile hardware and observation space. I retain my recommendation of strong accept.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. The section describing the tactile sensor data in Section 3.1 under “Feature Learning” is unclear. Perhaps, it would be helpful to refer to the original tactile data in Fig. 11 in this section. I recommend instead adding a separate figure to explain this. This can include a full view photo of the hand with all sensors along with the 16x16 tactile map with each pad numbered by the corresponding finger and axis. This will make things very clear without having to explain too much in words. I suggest adding such a figure in the supplementary.\n\n2. What is the contribution of different tactile pads to different tasks? Did you conduct such a study? It will be interesting to see which fingers and which parts of the hand are the most impactful for performing different manipulation skills.\n\n3. What are the limitations of non-parametric approaches compared to parametric ones? Are certain tasks more difficult in the non-parametric case, such as more complex long-horizon tasks, or ones including repetitive motions?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Strengths:\n1.\tThe proposed method to pretrained tactile representations on play data performs very well in combination with visual representations. The efficacy of the tactile data is demonstrated with a rigorous comparison with non-tactile image-only observations which doesn’t perform as well for the given tasks. Although inspired by prior works using BYOL, the results in the context of using tactile data and the associated analysis are new and useful.\n2.\tThe evaluation conducted is comprehensive and covers a wide range of baselines spanning parametric as well as non-parametric approaches. For the tasks considered, the parametric approaches overfit when provided with limited demonstrations. In this context, the proposed NN based approach adopted by T-Dex serves very useful in generalizing with sparse training data. Amongst the non-parametric methods, image-only and tactile-only are shown to perform worse than T-Dex as expected. \n3.\tThe analysis of design choices in section 4.4 is an added bonus and the authors have done a great job testing out each component of their proposed method including architecture and tactile input structuring. \n4.\tThe paper is very well written and is a joy to read. Each section is precise and thorough in its formulation.\n\n\n### Weaknesses:\n1.\tEvaluating on more tasks would have added greater insight and impact to the results. Given that the play data was not dependent on successful interactions and took only 2.5 hours for 5 tasks, it seems to be relatively easy to scale to more diverse tasks.\n2.\tSection describing tactile pad structure is unclear. Refer to rebuttal question.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "y2wL3AmOQH", "reviewer": "Reviewer_x3zE", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents the T-Dex framework trained with robot play data to learn non-parametric policies that combine the tactile observations with visual ones. The robot play data provides the necessary demonstration that involves high-dimensional tactile data for training skills for manipulation, which the authors achieved through the T-Dex framework in a series of tasks. By comparing with a series of existing methods, the author show that the proposed method is superior in performance in five selected tasks of object manipulation. The authors explored three research questions in experiments and generalized the proposed method to unseen objects. Limitations of this work is also explained briefly and the video demonstration is helpful to understand this work. ", "summary_of_recommendation": "Overall, I do not recommend this paper for acceptance due to the lack of innovation with repetition of known understandings, limitations in the method and data that is not clear on sharability and reproducibility, and the close resemblance to another submission to CoRL 2023.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "1. Please further elaborate on the data collected for training and the robot play part, whether it is through human teleoperation or autonomous/preprogrammed exploration of the robot by itself. Human preference could have an impact on the quality of data collected. What are the nature of the objects or tasks? Are they standard available for other researchers? What about the tactile data collected?\n\n2. Please improve the literature review section on the current literature of tactile sensing supporting vision data for robot learning. Please improve on the current literature on high-dimensional tactile data collection and dataset. Please improve on the existing work of robot manipulation explore the scene with or without human demonstration for collection of large dataset. \n\n3. For the results reported in Table 1, I'm surprised to see that methods such as BC, BET, IBC, and GMM failed in all cases. How are they implemented? This result is not very consistent with the existing literature and I'm not sure the way the authors implemented these methods are suitable for the proposed robot setup. Please justify how to compare these method using the authors robot platform in a fair and reasonable way for analysis. \n\n4. The tactile data presented in Fig. 4 is not very informative for understanding. What's the meaning of the red and green boxes? How to understand the relationship of these images in what sequence? The tactile data seems to be all active in large regions without further differentiation. Please improve this figure for a clearer understanding. \n\n5. The results reported in Fig. 5 and 7 shares the similar problem as they are not particular helpful in understanding the contribution of this experiment or this work. The results in Table 2 shares the same problem as in Table 1. Please explain and elaborate on the results and the suitability of the networks used. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "While the paper is technically complete in demonstrating the idea, the main issue is probably the innovation behind, which is lacking. It is widely accepted that adding tactile data to visual information helps and improves grasping performances, which is proved again in this work. The involvement of high-dimensional tactile data in this case is no different at all. The robot play data is also well-adopted method in the current literature, although in this particular case it seems it is not exactly the robot playing by itself, but teleoperated by a human operator to play with the robot system during playing. It is unclear what the T-Dex fully represents, the dataset itself, the algorithms developed, or both. Further breakdown statistics of the tasks or objects performed during robot play helps the readers to better understand the nature of this dataset itself. The learning part is also pretty standard and well-explored using Nearest-Neighbor retrieval. \n\nThe three research questions need further improvement. Question 1 is already explored in the currently literature extensively, and proved again in this work. Question 2 is essentially a known conclusion that pre-training improves learning performances, whether it is robot play data or not. And for Question 3, it is interesting to see that a much simpler Alex-net outperforms the other more complexed algorithms, especially after pre-training. While the authors did a lot of experiments with variations on the task and objects, it would be great to focus on a few representative ones with more tests than just 10 trails for a more robust conclusion, given that the authors have already conducted a lot of experiments. \n\nAnd finally, the most concerning part is the close resemblance of this paper with another submission, No. 161 See-to-touch paper for CoRL. These two papers used exactly the same setup and share almost the same paper structure (different placement of related works section). Although they are positioned differently, I don't think this should be encouraged, making the work more like a production from paper mill or automatic generation instead of genuine research. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "unLHxC2PWP", "reviewer": "Reviewer_rRsP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a self-supervised pre-training learning method for tactile encoders using play data.\nThis paper uses Holo-Dex framework as a teleoperation framework, Allgro hand as a robot hand, Kinova robot arm, Realsense cameras, and XELA uSkin as a tactile sensor.\nIn the first phase, this method collects play data and learns tactile representation.\nIn the second phase, the method collects expert task demonstrations with vision and learns tasks using the nearest neighbor method to select a hand and finger pose. \n\nThe main contribution of the paper is to present a method for self-supervised pre-training of tactile presentation with play data for learning dextrous manipulation.", "summary_of_recommendation": "The paper is well-written and has clarity, originality, and significance.\nHowever, the downstream tasks do not seem challenging dexterous manipulation requiring multiple fingers, and it is not unclear whether the proposed method can be used for more complex manipulation.\nTherefore, the reviewer recommends weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Listing the contributions of the paper in the introduction may help the readers to understand these contributions easily.\n- The experiment section should show the five downstream tasks again and introduce the teleoperation system. \n- If the proposed method can be used for the non-nearest neighbor method, explaining how to do that helps the reader to understand the significance of the paper.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strenghs\n- The paper is written clearly.\n- Imitation learning dextrous manipulation, unsupervised learning tactile representation, leveraging play data, and pre-training for learning manipulation are significant issues that the paper tackles.\n- The description of the method and experiments sound technical.\n- The originality of the paper is to address the pre-training unsupervised tactile representation with play data for learning downstream dexterous manipulation.\n\nWeaknesses\n- The downstream tasks, such as opening a book, bottle cap opening, and unstacking cups, do not seem challenging. A simple parallel gripper can execute these tasks. The tasks do not express the dexterity of the multi-finger robot hand. \n- Their nearest neighbor method does not have scalability. Their method may be used for more complex manipulation task which requires continuous control.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "dK7gLBSz5V", "reviewer": "Reviewer_27e9", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents a method for learning self-supervised tactile encoders from unlabeled play data that are useful in downstream dexterous manipulation tasks.", "summary_of_recommendation": "This work shows evidence that tactile representations learned from large amounts of task-agnostic play data are useful in downstream dexterous imitation learning tasks, and generalize to new visually distinct object instances. This feels like an important finding to communicate to the community, as it derisks a strategy for learning from more scalable forms of supervision than those currently considered for tactile dexterous manipulation.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "N/A", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n* Well motivated technique for pretraining tactile representations from unstructured play data. The techniques ability to learn from either successful or unsuccessful demonstrations seems particularly well aligned with the realities of teleoperation for dexterous manipulation studied in this work, where the failure rate of teleoperators is 5/ 6.\n* Clear description of the method and helpful and practical descriptions of how the dataset was constructed and filtered.\nStrong evidence under extensive real world evaluations (50 hours) of the efficacy of the proposed technique. \n* Good ablations quantifying design decisions and highlighting the importance of pre-training on play, and how separate tasks scale differently with more play.\n* (Pending the actual release) The 2.5 hours of high rate vision and tactile sensing play data could be useful to the community as a source of pretraining or for studying multimodal representation learning.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for pretraining tactile representations using unstructured play data. When combined with visual observations, this method demonstrates improved performance in fine-grained manipulation tasks through imitation learning, using just a handful of demonstrations.

The overall sentiment from the reviewers regarding this paper is predominantly positive. They recognize the significance of the tasks undertaken, the clarity of the method's description, the breadth of real-world evaluations, the importance of the findings in tactile dexterous manipulation, and the anticipated usefulness of the dataset to the community.

However, some reviewers raised concerns about the novelty of the proposed approach, the actual dexterity of the tested tasks, and the scalability of the nearest neighbor method. While the community broadly acknowledges that tactile information enhances complex contact-rich manipulation tasks and play data can facilitate the learning process, this paper primarily offers a systems contribution. Given the capabilities showcased in the evaluations, the detailed system design, and the key findings, this work can be of great value to the community. As a result, I am inclined towards acceptance.

The authors have promised a number of changes to the final version of the paper. Please make sure to include those in your revision.

Additionally, Reviewer x3zE expressed concerns regarding a perceived similarity between this paper and another submission, No. 161 See-to-touch, also under review at CoRL. As the Area Chair for Paper 49, I do not have access to Paper 161 for a detailed comparison. And the discussion of the resemblance might violate anonymity requirements. The Program Chair recommends handling this concern through a different channel, so the final decision on this issue rests with the PC.
Decision: Accept (Poster)