Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks
Authors: Jialiang Zhao
Yuxiang Ma
Lirui Wang
Edward Adelson
Venue: CoRL 2024 (2024)
Keywords: Tactile Sensing
Representation Learning
Heterogeneous Learning
Robot Manipulation
Robot Learning
TL;DR: A framework that learns a tactile representation that transfers between diverse sensors and tasks, and an aggregated tactile dataset that is the largest and most diverse to date.

Abstract: This paper presents T3: Transferable Tactile Transformers, a framework for tactile representation learning that scales across multi-sensors and multi-tasks.T3 is designed to overcome the contemporary issue that camera-based tactile sensing is extremely heterogeneous, i.e. sensors are built into different form factors, and existing datasets were collected for disparate tasks. T3 captures the shared latent information across different sensor-task pairings by constructing a shared trunk transformer with sensor-specific encoders and task-specific decoders. The pre-training of T3utilizes a novel Foundation Tactile (FoTa) dataset, which is aggregated from several open-sourced datasets and it contains over 3 million data points gathered from 13 sensors and 11 tasks. FoTa is the largest and most diverse dataset in tactile sensing to date and it is made publicly available in a unified format. Across various sensors and tasks, experiments show that T3 pre-trained with FoTa achieved zero-shot transferability in certain sensor-task pairings, can be further fine-tuned with small amounts of domain-specific data, and its performance scales with bigger network sizes. T3 is also effective as a tactile encoder for long horizon contact-rich manipulation. Results from sub-millimeter multi-pin electronics insertion tasks show that T3 achieved a task success rate 25% higher than that of policies trained with tactile encoders trained from scratch, or 53% higher than without tactile sensing. Data, code, and model checkpoints are open-sourced at https://t3.alanz.info.

Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks
Jialiang Zhao!
Yuxiang Ma²
Lirui Wang" MIT CSAII.
Edward H. Adelson'
(alanzhao, adelson) @delson]@csail @csail mit.edu [yxma20, liruiii @mit.ed edu
Abstract: This paper presents T3: Transferable Tactile Transformers, framework for tactile representation learning that scales across multi-senson and multi-tasks. T3 is designed to overcome the contemporary issue that camera-based tactile sensing is extremely heterogeneous, i.e. sensors are built into different form factors, and existing datasets were collected for disparate tasks. T3 captures the shared latent information across different sensor-tar task pairings by constructing a shared trunk transformer with sensor- speecif encoders and task-specific decoders. The pre-training of T3 utilizes a novel Foundation Tactile (FoTa) dataset, which is aggregated from several open-sourced datasets and it contains over 3 million data points gathered from 13 sensors and 11 tasks. FoTa the largest and most diverse dataset in tactile sensing date and it is made publicly available in unified format. Across various sensors and tasks, experiments show that T3 pre-trained with FoTa achieved zero-shot transferability in certain sensor-task pairings, can be further fine-tuned with small amounts of domain-specific data, and its performance scales with bigger network sizes. T3 is also effective as tactile encoder for long horizon contact-rich manipulation. Results from sub- mibbiiilietter multi-pin electronics insertion tasks show that T3 achieved task success rate 25% higher than that of policies trained with tactile encoders trained from scratch, or 53% higher than without tactile sensing. Data, code, and model checkpoints are open- sourced at https://3..alaz..img
1 Introduction
The tactile sensing modality has gained increasing popularity within the robotics community, by providing important fine-grained contact information for dexterous and contact-rich manipulation tasks, such as robot grasping [1, 2] and fabric manipulation [3, 4]. Camera- tactile sensing [5], a sensing method that operates by embedding camera beneath soft elastomer to capture the fine-gre -reigrd interactions with the environment, is among the most popular methods of tactile sensing for its higher resolution and lower cost. However, camera-bases tactile sensors are extremely heterogeneous, and there has not been converged sensor design widely adopted by the robotics community. Different tactile sensors can differ significantly in shapes, types and numbers of cameras, placement and colors of illumination, etc. Such inherent heterogeneity hinders roboticists from building a general-purpose tactile encoder that is transferable across different sensors and downstream tasks.
Existing learning architectures and datasets focus on one specific sensor-task pairing, and when comes to a newly emerged sensor or task, data recollection and training an encoder from scratch are often required. This issue harms learning efficiency in more significant way in longer horizon tactile manipulation tasks, where the training of the tactile encoder is guided by a more sparse reward. Intuitively, although different tactile sensors produce vastly different tactile images and various tasks extract different information from tactile input, there should be shareable latent information due to the inherent similarities in tactile sensing across sensors and tasks. Therefore, it is both technically viable and practically desirable to design an architecture capable of extracting such shared latent representations and transferring them across different sensor-task pairings.
8th Conference on Robot Learning (CoRI. 2024), Munich, Germany.

---

Spark Frendrt
Deaeder
Sentor
Sharod Trunk
Take Nadel hois Ensddta
Snew Spert Escaderi
Pravérmin & Trunsfrr
Sensor
Cliess Objest Objecr
Decdder
3: Chordiattos
Twx Sparie
Sensoer
30mm
Toleszed
Tedket Paue Regremkkn
Figure 1: Architecture illustration of Transferable Tactile Transformers (T3). T3 learns shared representation across heterogeneous tactile sensors and downstream tasks with a shared trunk between sensor-specific encoders and task-specific decoders. The encoders and the shared trunk constructed with transformer blocks. The decoder architectures are chosen according to the types of the tasks: transformer for generative tasks like reconstruction for masked auto encoding, MLP for classification tasks, and CNN + MLP for pose estimation.
Learning from heterogeneous tactile inputs shares many similarities with multi-modal representation learning. Recent works in multi-modal foundation models have demonstrated the potential capability bridge the gap between different modalities and learning a common latent representation, such as in the perception domain [6, 7, 8] as well as the robotic manipulation domain [9, 10]. However, learning such foundation models often requires temporally aligned multi-modal inputs distance function that describes the resemblance between inputs from different modalities. In heterogeneons tactile learning. the hardware-leve disparity often makes ill-defined to align images collected from different embodiments.
In this work, we tackle the problem of heterogeneous tactile learning using unaligned tactile data, with the goal of learning a scalable representation that can be shared across different sensors and tasks. We first aggregate existing open-sourced tactile datasets and assemble the Foundation Tactile (FoTa) dataset, which the largest and most diverse tactile dataset so far to the best of the authors' knowledge. We then propose Transferable Tactile Transformers (T3) to learn shared representation across multi-sensors and multi-tasks. T3 is constructed with sensor-specific encoders, shared trunk, and task-specific decoders. Our experiments show that T3 pre-trained with FoTa achieves reasonable zero-shot transferability, it can be further fine-tuned with small domain-specific datasets, and its performance scales with bigger network sizes. We also demonstrate that T3 can be used as a tactile encoder for hoger-horizonn manipulation policies with 3 behavior cloning- bbsee sub-millimeter level electronics insertion tasks. In summary, this paper proposes:
Foundation Tactile (FoTa): dataset containing 3, 083, 452 tactile images collected with 13 sensors for 11 tasks in a unified format. is aggregated with several of the largest open-sourced tactile datasets and additional data collected in-house by the authors.
Transferable Tactile Transformers (T3): a neural network framework that learns shared repre- sentation across all sensors and tasks in FoTa, Pre-trained weights are provided, and we demonstrate that T3 can be easily fine-tuned to new task or new sensor with few fine-tuning data.
2 Related Works
Camera-based tactile sensors Camera-based tactile sensors are often constructed with soft elastomer as the contact medium, reflective layer, light source to provide illumination, and one or more than one camera to capture the deformation of the elastomer, which therefore provides detailed view of the contact surface. Specific design choices camera-based tactile sensors are highly diverse, such as monochrome [11] v.s. colored [12, 13, 14] in illumination, with v.s. without markers [15, 16, 17], and different form factors such as flat [12, 13], dome-shaped [18, 14], finger- shaped [19, 20), compliant [21, 22], multi-linked [23] even palm-shaped [24]. Such diversity

---

and disparity have enabled researchers to use those sensors across spectrum of applications such as in defect inspection [25] and robotic tool-using [9]. Nevertheless, they also add a substantial barrier in leveraging machine learning to encode the information coming from those sensors. A unique sensor-task pairing often requires data re-collection and neural network re-training.
Datasets for tactile manipulation Several large-scale tactile sensing datasets have been released in recent years tackling visual-tactile cross-modality reasoning [26, 27, 28], object or material classification [28, 29, 4], longer horizon tasks such as robotic grasping stability analysis [2], or shape reconstruction [30]. Those datasets also cover interactions with a wide variety of objects, including household objects and toys [2, 26, 27, 28, 30J, fabrics and clothes [4], natural objects such as wood and gravel [29]. Quantity-wiss several of those datasets contain well over a million images[26, 30], making the total number of available tactile images on par with ImageNet- 1K [31].
While quantity is often not an issue when using existing open-sourced tactile datasets, one significant drawback is that all the aforementioned datasets lack diversity in terms of sensors and tasks. The vast majority of tactile datasets were collected entirely with only one specifie tactile sensor. With this constraint, even if sufficiently well verf-prrforning tactile encoder can be trained from one dataset, such a pre-trained encoder will not be able to transfer to another sensor. In order to learn general- purposs representation for tactile sensing like [32, 33] does in the natural image domain, a large and diverse tactile dataset with a wide variety of sensor-task pairings is a prerequisite.
Representation learning with heterogencous data Learning from heterogeneous data domains, such as taking multi-sensory feedback as inputs or predicting outputs for different tasks, has been an active area of research, especially in the robot learning domain. Many existing works combine tactile, vision, and occasionally other modalitics such as robot states and sound, by passing each modality through one modality-speciite encoder, then concatenating them and passing them through a decoder network to predict the final output [34, 15, 35, 9, 36]. Apart from learning-drived neural networks, it is also common use analytical tools, such as factor graphs convex optimization, to combine predictions from multiple domains [37].
Another branch of multi-modal robot learning actively reasons the relationship among heterogeneous domains and cross-links them to learn a shareable latent representation. Extensive research has explored using contrastive learning to join temporally aligned modalities [6. 10, 38). To connect vision and touch, Chen et al. designed representation, while Li et al. proposed cross- modsl Generative Adversarial Network and were able to predict tactile images from visual images and vice versa [26]. Bachmann et al. applied masked auto-encoding (MAE) aligned RGB images, depth maps, and semantic segmentation and learned a shared representation across three domains [7]. However, although cross-modality representation learning shares lot of commonalities with heterogeneous tactile representation learning, one significant difference that heterogeneous tactile representation learning lacks aligned data collected with different sensors for different tasks. It is often ill-defined to infer the resemblance of two tactile images collected from different hardware, which is a prerequisite for many self- supervised learning techniques. It is therefore desirable to devise new architecture that is able to learn from unaligned tactile data and produce a representation that is transferable across different tactile sensors and different tasks.
multi-modal self-attention mechanism [39] to learn shared
3 The FoTa Dataset
We provide a large tactile dataset by aggregating existing public datasets as well as adding new data collected in-house. The FoTa dataset is provided in unified, I/O efficientt WebDataset [40] format. It is by far the largest and most diverse dataset for tactile perception, with 3, 083, 452 data points collected on 13 tactile sensors, and high-quality labels are provided for 1l tasks. Mixture of constituents of FoTa is illustrated in Fig. 2.a, and example tactile images from each sensor are shown in Fig. 2.b. Details and statisties about public datasets aggregated in FoTa well as the pre-prec pre-processing method are provided in Appendix A.2.
Besides aggregating publicly available tactile datasets, we add new data collected with several recently emerged tactile sensors including GelSight Finray [5], GelSight Svelte [19], GelSight Wedge [13), Dense Tact 2.0 [14], and GelSight 360 [18]. Those new data were also used for final performance
3

---

Datasets
Tactile Sensors
Task Labels
Fo Data
Outgget 200
Onigge 114
GelSight var.1
GelSight17 var.1
GelSight17 var.2
Ge15ight17 var.3
DIGIT
GelSight Mini
GelSigh var.4
GelSight Wedge
GelSight 360
GelSight Finray
GelSight Svelte
DenseTact2.0
GelSight var.2
FoTa dataset visualizations. (a) We show the mixture and distribution of constituent datasets, sensors, and tasks of the FoTa dataset. Note that not all tasks utilized in the training of T3. (b) We visualize one tactile image from each constituent sensor in FoTa. Note that in the training of T3, similar sensors share encoders. For example, GelSight17 var. (1-4) share the encoder, and GelSight var. (1 [1-2 share the same encoder.
Figure 2:
benchmarking well transferability testing for the proposed learning framework. Two platforms were built to collect these additional data.
A 7-DoF robotic platform collects tactile data by attaching 2 tactile sensors to parallel jaw gripper of 7-DoF Franka Emika Panda robot arm and commanding the robot arm to explore object that is fixed at known location. Robot pose in (3) is recorded for each step and then transformed into the sensor's coordinate frame. Object IDs as well as names are also recorded.
3-DoF gantry platform is built with a desktop 3-axis CNC mill. Six 3D printed probes with different textures are attached
force/torque sensor, which is then attached the Z axis of the CNC. During data collection, a sensor is fixed at known location, and the CNC machine probes the entire sensing surface of the sensor. Translational poses, probing forces, and probe IDs are recorded for each tactile image frame.
The systems introduced in detail in Appendix A.4.
4 Heterogeneous Tactile Learning with T3
We propose Transferable Tactile Transformers (T3) as the backbone network to learn from diverse tactile sensors and produce outputs for diverse downstream tasks.
Network architecture A training dataset contains inputs from N, different sensors and N, dis- tinguished tasks. T3 is constructed with N, encoders {Enc1, ..., EneN, ). one for each individual tactile sensor; N₂ decoders ( [Dec1...., ..., Decv, }, each responsible for one downstream task; and one shared trunk, denoted as Trunk. During both training and inference, we ensure that one batch of data always comes from the same sensor- censor-task pairing. For given pair of data point (X;, Yj) that is collected from sensor i for task j. the corresponding encoder Ene; and decoder Dec, are attached to the trunk. The constituent network components of T3 can take different architectures. In practice, we use ViT [41] in the trunk and encoders for its proven learning capacity and scalability in the ViT for generative tasks (reconstruction for MAE), natural image domain. For decoders,

---

ResNet+MLLP for pose estimation tasks, and MLP for classification tasks. More details about network configurations are discussed in Appendix A.5.
Learning objectives For each task j [1, NN.. we define loss function L,- Different tasks require different numbers of tactile images as inputs. For example, tasks like object classification and material classification only require one tactile image to perform, while the pose estimation task requires two tactile images, e.g. ,x?, because the prediction goal is the relative pose between the two instances. We pass each individual tactile image through the encoder and trunk separately, and we then concatenate them together before passing them through the decoder. The loss between data pair (X,, Y2) or ([X), x;), Yj) calculated as
loss( X,, Y2) = (Y,, Dec; (Trunk(Enn loss loss( [X] x21, Yy) = L,(Y, Decy(Trunnii (Trunk (x,,(())) @ Prunk(Enn;(({))
where @ denotes concatenation.
The training process of T3 is split into two pre-training phases and one optional fine-tuning phase. Pre-training is divided into two phases to improve data utilization and to attain both local ene-graieed understanding and global semantic understanding. Phase focuses on more pixel-level understanding using self-sup eef-supervised learning, and it is able to utilize all data from the FoTa dataset. Phase 2 focuses on semantic understanding and it requires suitable task labels that only a subset of the FoTa dataset possesses.
Pre-training I: Self-supervised learning with Masked Auto Encoding (MAE) MAE was first proposed as self-supervised pre-traini pre-training technique for natural images [42] and it has gained increased popularity in other domains such as audio processing [8], point-cloud processing [43], and multi-modal representation learning [7]. operates by first randomly masking out a certain portion of the original inputs, then it constructs decoder to reconstruct the missing portion, Neural networks that were pre-trained with MAE have been shown to be able to outperform ones that were trained without MAE in various domains. This pre-training stage is able to utilize all data in FoTa, including unlabeled data (e.g. intermediate tactile images collected during the settling period a robot grasp in [2], data with semantic labels for which loss functions are hard to define (e.g. natural language description of a scene [27]).
In this pre- pre-training stage, one decoder Deco that is designed to reconstruct partially masked images shared by all domains. Deco constructed with 8 Tranoreransformer blocks [41]. The loss Lo is 1.2 pixel-wise loss normalized within each patch, following [42].
Pre-training II: Supervised learning with labels distilled from public datasets In this stage, T3 trained under supervision from selected task labels in the FoTa dataset. For each individual task j E [1, a decoder Dec, and a loss function Lj is defined. We define 10 tasks for this training stage.
Classification tasks include object classification for [2, 29], material classification for [30], and fabric smoothness, fuzziness, textile type classification for [4). Decoders for each of them are defined as MLP neural networks, and each Lj is cross-entropy loss function.
Regression tasks include SE(3) relative pose estimation bewteen two overlapped tactile images for [13, 44, 45] (with ResNet+MLP decoders), and variance of Laplacian (a metric describing the amount of information contained in tactile images, more details in Ap- pendix A.2) estimation for [26] (with MI.P decoders). Each loss L, is mean squared error loss function.
Supervised learning with task-specific data In this phase, T3 is further fine-tumed with data collected for the specific downstream task with the specific sensor that the user aims to use. This stage can be optional if the sensor-task pairing already exists in the pre-training dataset. Architecture-wise T3 configured the same way as it is in Pre-training but with only one target task and sensor.
Fine-tuning:
5

---

(a) Task performance v.s.
Task performance v.s.
model size, training stages, and amount of finetuning data
masking ratio at stage
1,00
Classiiiation Accuracy (top
Classfiiatioo Accuracy (top-1) higher better
0.09
0.6
4.65
0.85
0.4
0.80
1950
from SCI aich
0.75
fnneuune from protrain
40%
90%-05N
Stage
(c) Transferability classification task
(d) Transferability pose estimation task
0.40
Clessification Accuracy (top-1)
0.35
Pose Extimation RMSE (mm)
10
14
12
highar la botter
lower is better
0.25
Deraset Avg. Zero-shot Transfered Pinetuned
0.2
0.15
Random 0.10
Zero
Finetuned
0.0
GelSight
Denselaact
0.00
GelSight Svelte
Densefbact 2.0
Figure 3: Experiments on Task Performance and Transferability of T3. (a) Eval performance with 4 network sizes (tiny, small, medium, large), training schemes (train from scratch, fine-tune from pre-train 1, fine-tune from pre-train 2), and 2 amounts of fine-tuning data (half data and full data) (b) Eval performance with different masking ratios during Pre-training (c) Transferability test on classification task. (d) Transferability test pose estimation task.
Experiments and Discussions
In this section we discuss the evaluation performance of T3 with numerous ablations and benchmarks. For both pre-training and II, we use all data from the FoTa dataset except data collected on the gantry system in-house, including the object classification task and the 3D pose estimation task. Those data are therefore not included for pre-training the foundation model and are set aside for evaluation and fine-tuning only. We aim to answer the following questions in this section:
How to train efficiently and how much does pre-training improve performance? Sec. 5.1.
Is pre-trained T3 zero-shot transferable new tasks and sensors? Sec. 5.2.
Does FoTa provide meaningful improvements longer-horizon robotic tasks? Sec. 5.3.
5.1
The worthiness of pre-training
The fine-tuning and evaluation task for this section an object classification task (6 categories) for sensors (GelSight Wedge, GelSight Mini) with around 3,300 data points for each sensor. Averaged top-1 classification accuracy from the two sensors reported.
The effect of pre-training and the scaling behavior on different network sizes We conducted the classification experiment with all combinations of the following: (a) 4 network variations: finy-12M, 455, medium- 174M, and large-308M; from peetrraningg I, and fine-tune from pre-training I & II; (c) 2 amounts of training data: half data (<s 1,600 training data) and full data (R33300 training data). All experiments were tested on same validation dataset, which was separated from the training set. The evaluation accuracy is shown Fig. 3.a. We observed that: (a) Pre-training significantly improved the evaluation performance with all network configurations with an median improvement of 24%. (b) The evaluation performance
(b) 3 training schemes: train from scratch, fine-tune

---

improved with larger networks, where large's classification accuracy was 19% higher than that of tiny. However, the performance difference between medium and large was insignificant. (c) When fine-tuned with half of the fine-tuning data, pre-trained networks performed better than trained from scratch networks, and larger pre-trained models performed better than smaller pre-trained models. In fact, for medium and large, the performances between fine-tuned with half data and fine-tuned with full data were very close, signaling those models generalized better thus they require less data to fine-tune to novel task.
More qualitatively analysis, including visualizations of the encoder attention maps and the trunk attention maps, can be found in Appendix. A.1.
Masking ratios in Pre-training Stage I We tested 8 different masking ratios in the MAE recon- struction pre-training stage, and we performed the same Pre-training II stage as well as the fine-tuning stage the 8 tests. The evaluation results are reported in Fig. 3.b. We observe a small performance variance, ranging from 85% (mr = 30%) to 89% (mr = 80%), and a masking ratio of 80% yielded the highest performance.
5.2 Zero-shot transferability of pre-trained T3 to new sensors or tasks
We tested to what extent a pre-trained T3 can be transferred to a new sensor or downstream task with and without fine-tuning. We picked two novel sensors (GelSight Svelte, DensetTact2.00 and two seen tasks (object classification, pose estimation between two overlapped tactile images). The two seen tasks were pre-trained with data from GelSight Wedge, GelSight Finray, and GelSight Mini. During this experiment, we use the encoder pre-trained for GelSight Wedge as the encoder for GelSight Svelte, and GelSight Mini's encoder in place of Dense encoder. Results on the 4 combinations are shown in Fig. 3.c and Fig. 3.d. Zero-shot transfer yielded only minor improvements over random guesses in the object classification task; however, it demonstrated significant improvement over dataset averages in the pose estimation task. We further fine-tuned the networks with a small amount of 2,000 data points. The classification accuracy immediately bounced up to 17% with fine-tuning, and the pose estimation root mean squared error (RMSE) was reduced by 5.5mm for DenseTact2.0 O. On the GelSight Svelte sensor, the pose estimation errors before and after fine-tuning were nearly identical and close to optimal. These results indicate that zero-transferability can be achieved in certain sensor-task pairings, and that better outcomes can be attained with minimal fine-tuning.
5.3
T3 in long-horizon manipulation tasks
Tactile sensing can be especially helpful in contact-rich manipulation tasks. Numerous human studies have shown that the sense of touch is important for humans to perform delicate manipulation tasks such as surgery [46, 47). However, most state-of-the-an tactile manipulation research opted to train a tactile encoder from scratch, with fresh data collected with the specific sensor-task pairing, due to the lack of a pre-trained tactile encoder. In the case of long-horizon multi-modal manipulation tasks, training from scratch is often inefficient, due to the sparseness of trajectory rewards and the number of components that need to be trained (e.g. modality encoders and policy networks) especially in many reinforcement learning and behavior cloning applications such as [48, 9]
To investigate whether a pre-trained T3 helps to bridge this gap, we designed robotic precision insertion task with behavior cloning. The goal of this task is to insert 3 electronics parts: a 3-pin toggle switch, a 12-pin double-stack USB port, and 17-pin VGA connector, onto PCB with corresponding mounting holes to each component. This task requires high precision, where the clearance between the holes on the PCB and the pins on the parts is only 0.4mm. Achieving this precision requires active exploration with tactile feedback. In real-world applications, relying vision alone is often insufficient due to heavy occlusion.
Our experiment setup consists of two GelSight Wedge tactile sensors mounted on parallel jaw gripper, which is attached to the end-effector of a 7-DoF Franka Emika Panda robot, PCB board fixed at a known location on the workbench, the 3 electronics parts, and a RGB camera on the side of the workbench. An illustration of the setup is shown in Fig. 4 and the parts are shown in Fig. 4.b. Data is collected by controlling the robot in "guide mode", i.e. under zero stiffness control, and a human operator manually grabs and moves the end-effector until the part fully inserted. At both data collection and inference time, the sockets are fixed, and we add randomizations to
7

---

Trugle the kil (lige))
VIDA Connacer 117pp00
0.7
25
USB
0.8
Switch
20
Task Surcies Rate
Teg suggorr better
0.3
22
0.3
0.0
Bartle
Trained from acrath
T) Teetlte Erecider
Buctle
Rectile Baiinoll from Seritcs
T3 Bactle Erccader
(b)
(C)
Figure 4: Real world sub-millimtter robotic insertion tasks. (a) Hardware setup. (b) The 3 evaluation electronics parts and their respective mounting places on a PCB. (c) Insertion success rate and average steps taken for each task. Policies trained with the T3 tactile encoder achieved the highest success rate and lowest averaged episode length.
the part-in-hand pose and gripper starting pose before each insertion, both generated from 3D zero-mean, Imm standard deviation normal distribution.
We train and evaluate 3 policies: a baseline policy without tactile input, policy with tactile input encoded by neural network trained from scratch, and policy with tactile input encoded by T3. Besides tactile imputs, all 3 policies have access to the same robot state modality encoded by MLP, and external vision modality encoded by pre-trained ResNet18. All three policies take observations of the current step inputs and predict 3-DoF action which the robot executes at the next time step. At inference time, the robot executes predicted 3-DoF actions in about 2Hz for up to steps. episode is deemed successful if the robot successfully inserts the component within 30 steps. The success rate and the average steps taken of successful episodes across 15 episodes are reported in Fig, 4.c.
The results show that (1) the tactile modality is vital for this electronics insertion task, where the vision-only policy failed all tests to insert the two more challenging parts; (2) using pre-trained T3 as the tactile encoder for this policy helped further improve the overall performance, where the task success rate is higher across all three parts; (3) T3 also helped to reduce the number of tactile exploration steps needed to insert a part.
Limitations and Future Works
One limitation of the current training pipeline is that the FoTa dataset unbalanced. Data collected with the 2 most popular sensors constitutes over 50% of the entire dataset. Therefore, the trained policy could also be biased towards those more popular sensors. Another limitation is that the current architecture of T3 focuses on per-image encoding and T3 is trained with explicit labels during pre-training II and fine-tuning. Representation learning on tactile image sequences with sparse or implicit labels can be a future direction of this work. Furthermore, T3 is currently limited to learning representations for camera-base tactile sensors. Extending the FoTa dataset and the T3 framwork to non-camera- based sensors is another future line of work.
7 Conclusions
This paper presents T3, large sumera-based tactile sensing framework that transfers across mul- tiple sensors and tasks; and FoTa, a tactile dataset that is by far the largest in quantity and most diverse in sensors and downstream tasks. Experiments show that T3 pre-trained with FoTa improves performance for various tasks, including longer-horizon manipulation tasks such as sub-millimeter electronics insertion, and that T3 transfers new sensor-task pairing with little fine-tuning.
8

---

Acknowledgments
The authors thank Kaiming He for the many insights and discussions around network architecture and training. This paper partially funded by Toyota Research Institute and Amazon Science Hub. The authors also thank Won Kyung Do from the Stanford ARMLab for providing a DenseTact2.15 for experiments conducted in the paper.
References
[1] R. Calandra, A. Owens, D. Jayaraman, Lin, W. Yuan, Malik, E. H. Adelson, and S. Levine, "More than feeling: Learning to grasp and regrasp using vision and touch," IEEE Robolics and Automation Letters, vol. 3, no. 4, pp. 3300-3307, 2018.
[2] R. Calandra, A. Owens, M. Upadhyaya, W. Yuan, J. Lin, E. H. Adelson, and S. Levine, "The feeling of success: Does touch sensing help predict grasp outcomes?" arXiv preprint arXiv: 1700055210 2017.
[3] N. Sunil, S. Wang, Y. She, E. Adelson, and A. R. Garcia, "Visuotactile affordances for cloth manipulation with local control," in Conference on Robot Learning. PMLR. 2023, pp. 1606.
[4] W. Yuan, Y. Mo, S. Wang, and E. Adelson, "Active clothing material perception using tactile sensing and deep learning," 2018.
[5] S. Q. Liu and E. H. Adelson, "Gelsight fin ray: Incorporating tactile sensing into a soft compliant robotic gripper," in 2022 IEEE 5th International Conference on Soft Robotics (RoboSoft). IEEE, 2022, pp. 925-931.
[6] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra, "Imagebind: One embedding space to bind them all," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15 180-15 190.
[7] R. Bachmann, D. Mizrahi, A. Atanov, and A. Zamir, "Multimae: Multi-modal multi-task masked autoencoders." 2022.
[8] P.-Y. Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer, "Masked autoencoders that listen," in NeurlPS, 2022.
[9] L. Wang. J. Zhao, Du, E. H. Adelson, and R. Tedrake, "Poco: Policy composition from and for heterogeneous robot learning," " arXiv preprini 4rXiv:2402.02511, 2024.
[10] R. Shah, R. Martín-Marthn and Y. Zhu, "Mutex: Learning unified policies from multimodal task specifications," " arXiv preprint arXiv:2309.1433 2023.
[11] Z. Si, G. Zhang, Q. Ben, B. Romero, Z. Xian, C. Liu, and C. Gan, "Difftactile: A physics- based differentiable tactile simulator for contact-rich robotic manipulation," arXiv preprint arXiv:2403.08716 2024.
[12] Dong, W. Yuan, and E. H. Adelson, "Improved gelsight tactile sensor for measuring geometry and slip," in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, Sept. 2017. [Online]. Available: http://dx..oi.org////
[13] S. Wang, Y. She, B. Romero, and E. Adelson, "Gelsight wedge: Measuring high-resolution 3d contact geometry with compact robot finger," in 2021 IEEE International Conference Robotics and Automation (ICRA). IEEE, 2021, pp. 6468-6475.
[14] W. K. Do and M. Kennedy, "Densetact: Optical tactile sensor for dense shape reconstruction." in 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022, pp. 1188-61944
[15] J. Li, S. Dong, and E. Adelson, "Slip detection with combined tactile and visual information," 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018, pp- 7772-7777.

---

[16] W. Yuan, R. Li, M. A. Srinivasan, and E. H. Adelson, "Measurement of shear and slip with gelsight tactile sensor," in 2015 IEEE International Conference on Robotics and Automation (ICRA). IEFE, 2015, pp. 304-311.
"
[17] A. Yamaguchi and C. G. Atkeson, "Combining finger vision and optical tactile sensing: Re- ducing and handling errors while cutting vegetables," in 2016 IEFE-RAS 16th International Conference on Humanoid Robots (Humanoids). IEEE, 2016, pp. 1045-1051.
[18] M. H. Tippur and E. H. Adelson, "Gelsight360: An omnidirectional samera-based tactile sensor for dexterous robotic manipulation," 2023 IEEE International Conference on Soft Robotics (RoboSoft). IEEE, 2023, pp. 1-8.
[19] J. Zhao and E. H. Adelson, "Gelsight svelte: human finger-shaped single-camera tactile robot finger with large sensing coverage and proprioceptive sensing," in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, pp. 8979-8984.
[20] H. Sun, K. J. Kuchenbecker, and G. Martius, "A soft thumb-sized iision-basct sensor with accurate all-round force perception," " Nature Machine Intelligence, vol. 4, no. 2, 2022.
[21] S. Q. Liu, Y. Ma, and E. H. Adelson, "Gelsight baby fin ray: A compact, compliant, flexible finger with high-resolution tactile sensing." " in 2023 IEEE International Conference on Soft Roboties (RoboSoft). IEEE, 2023, pp. 1-8.
[22] Y. Ma, A. Agarwal, S. Q. Liu, W. Yuan, and E. H. Adelson, "Scalable simulation simulation-guided compliant tactile finger design," in 2024 IEEE 7th International Conference Soft Robotics (RoboSoft). IEEE, 2024, pp. 1068-1074.
[23] Y. Ma, J. Zhao, and E. Adelson, "Gellink: A compact multi-phalant finger with vision- based tactile sensing and proprioception." arXiv preprint arXiv:2403. 2024.
[24] F. Zhao, B. Huang, M. Li, M. Li, Z. Fu, Z. Lei, and M. Li, "A novel tactile palm for robotic object manipulation," 2023.
[25] J. Jiang, G. Cao, D. F. Gomes, and S. Luo, "Vision-guided active tactile perception for crack de- tection and reconstruction," in 2021 29th Mediterranean Conference on Control and Automation (MED). IEEE, 2021, pp. 930-936.
[26] Y. Li, J.-Y. Zhu, R. Tedrake, and A. Torralba, "Connecting touch and vision via cross-modal prediction," " in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[27] L. Fu, G. Datta, H. Huang, W. C.-H. Panitch, J. Drake, Ortiz, M. Mukadam, M. Lambeta, R. Calandra, and K. Goldberg, "A touch, vision, and language dataset for multimodal alignment," arXiv preprint arXiv:2402.132. 13232, 2024.
[28] S. Suresh, Z. Si, J. G. Mangelson, W. Yuan, and M. Kaess, "Efficient shape mapping through dense touch and vision," arXiv preprint arXiv:2109.0988, 2021.
[29] F. Yang, C. Ma, J. Zhang, J. Zhu, W. Yuan, and A. Owens. "Touch and go: Learning from human-collected vision and touch." arXiv preprint arXiv:2211.1244 2022.
[30] R. Gao, Z. Si, Y.-Y. Chang, S. Clarke, J. Bohg, L. Fei-Fei, W. Yuan, and J. Wu. "Objectfolder 2.0: A multisensory object dataset for sim2real transfer," in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2022, pp. 10688-
[31] Deng, W. Dong, R. Socher, L.-J. Li, Li, and Fei-Fei, "Imagenet: large-scale hierar- chical image database." in 2009 IEEE conference computer vision and pattern recognition. Ieee, 2009, pp. 255-25..
[32] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp- 770-778.
10

---

[33] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, et al., "Learning transferable visual models from natural language supervi- sion," in International conference on machine learning. PMLR, 2021, pp. 8748 -763.
[34] H. Li, Y. Zhang, J. Zhu, S. Wang, M. A. Lee, H. Xu, E. Adelson, L. Fei-Fei, R. Gao, and J. Wu, "See, hear, and feel: Smart sensory fusion for robotic manipulation," arXiv preprint 2022.
arXiv:2212.0388
[35] H. Qi, B. Yi, S. Suresh, M. Lambeta, Y. Ma, R. Calandra, and Malik, "General in-hand object rotation with vision and touch," in Conference on Robot Learning. PMLR, 2023, pp. 2549-2564.
[36] L. Wang. Chen, Zhao, and K. He, "Scaling proprioceptive-visual learning with heteroge- neous pre-trained transformers," in Arxiv, 2024.
[37] J. Zhao, M. Bauza, and E. H. Adelson, "Fingerslam: Closed-lood unknown object localization and reconstruction from visuo-tactile feedback," " in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 8033-8039.
[38] V. Dave, F. Lygerakis, and E. Rueckert, "Multimodal visual-tactile representation learning through self-super contrastive pre-training," arXiv preprint arXiv:2401.12024, 2024.
[39] Y. Chen, M. Van der Merwe, A. Sipos, and N. Fazeli, "Visuo-tactile transformers for manipula- tion," in 6th Annual Conference on Robot Learning, 2022.
[40] W. developer team, "Webdataset," https///gttubb.commwwedaaseeeeet 2020, [Online].
[41] A. Dosovitskiy, L.. Beyer, Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010. 11999, 2020.
[42] K. He, X. Chen, Xie, Li, P. Dollár, and R. Girshick, "Masked autoencoders are scalable vision learners," in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2022, pp. 16 100
[43] Y. Pang, W. Wang, F. E. Tay, W. Liu, Y. Tian, and L. Yuan, "Masked autoencoders for point cloud self-supervised learning," in European conference on computer vision. Springer, 2022, pp. 604-621.
[44] M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon, V. R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, er al., "Digit: A novel design for low-cost compact nigh-resolution tactile sensor with application to in-hand manipulation," IEEE Robotics and Automation Letters, vol. 5, no. 3, pp. 3838-38888 3845, 2020.
[45] G. Inc., "GelSight Mini,"
2022, [Online].
[46] M. Eltaib and J. Hewit, "Tactile sensing technology for minimal access surgery a review," Mechatronics, vol. 13, no. 10, pp. 1163 1177, 2003.
[47] Schostek, C.-N. Ho, D. Kalanovic, and M. O. Schurr, "Artificial tactile sensing in minimally invasive surgery-a new technical approach," Minimally invasive therapy & allied technologies, vol. 15, no. 5, pp. 296-304, 2006.
[48] S. Dong, D. Jha, D. Romeres, S. Kim, D. Nikovski, and Rodriguez, "Tactile-rl for insertion: Generalization to objects of unknown geometry," 2021.
11

---

A
Appendix
A.1
Visualization of attention maps
Original
Encoder Mask
Trunk
Mask
Figure 5: Visualization of self-attention weights of the encoders and the trunk pre-trained T3. Top: the original tactile images. Middle: encoder attention weights applied on the original images as masks. Bottom: trunk attention weights applied on the original images as masks
We calculate the attention weights attn; = softmar( scale at layer each self-attention layers for all encoders and the trunk. We then obtain a joint attention map, calculated the product of the attention weights of all layers within each encoder or trunk, map = II. at/m;- This joint attention map is then normalized and applied the original tactile images masks. A few tactile images as well as them applied with encoder masks and trunk masks are visualized in Fig. 5. Qualitatively, we observe that the encoder self-attentions highlight the region of the contact object, whereas the trunk self-an ooff-attentions are more spread out and they cover other features like the markers on a sensor. This result seems show that the encoder is more focused on the sensor agnostic regions, i.c. the "hot" regions of the contact, while the trunk also attends to sensor specific areas.
Q-K
A.2 Public datasets aggregated in the FoTa dataset
The FoTa dataset does not modify the images labels from the constituent datasets. Pre-processing was performed on each them to encode the tactile images into .ppg format pictures and to extract all labels into jsson format dictionaries. In one of the constituent public datasets, VisGel [26], more than half of the included tactile images are "flat" images, i.c. tactile images captured when there is no contact. We sub- sampled VisGel reduce the amount of flat images. With an image frame and true flat frame Zy. the variance of Laplacian of the different image a, is calculated with Eqn. 2. We then remove all images whose
oz. We empirically chose o, 4.24 to remove most flat images.
0: Var VVarllaalacian(I: I,)), E (1...N}
Note that do not aim to eliminate flat images entirely as flat images also contain important information such as the sensor geometry and the illumination design. Statistics of the aggregated datasets in FoTa are listed in Tab. 2.
A.3 Data format
Each constituent dataset of FoTa split into one train dataset and one val dataset, and each of them is packaged in pre-sharded WebDataset [40] format. Each shard of the FoTa dataset contains up to 10, 000 data points packed in one .ar archive, where each data point composed of one jpg tactile image and one json file for various task labels. Utility scripts are also provided to add new data the FoTa dataset to modify the pre- processes procedures on the constituent public datasets.
A.4 Data collection platforms
A.4.1 7-DoF robotic platform
We built a robotic data collection platform with 7-DoF Franka Emika robotic arm with a custom- made parallel jaw gripper attached at the end-efficttr. Two tactile sensors are attached at the tips
12

---

Table 1: Network Sizes
Params
D Denc
Hene
Lene
Liru
incl. all encoders and decoders
T3 T3-tiny
192
3
3
9
12M
T3-small
384
9
45M
T3-medium
768
12
3
9
174M
T3-large
1024
8
3
9
308M
of the parallel jaw gripper, and the transformation between the camera frame and the end-effector frame is accurately measured. During tactile data collection, test object, which is assumed to be rigid enough to be clamped firmly, is fixed on the workbench of the robot a known location. The robot explores the object with random movements on all 6 dimensions (translation and rotation), and tactile images from both sensors as well as the robot pose are collected each step.
A.4.2 3-DoF gantry platform
Another gantry-based 3-DoF data collection platform was built from a desktop CNC milling machine. The spindle was replaced with 3D-printed probe, which is attached to the z-axis of the CNC with 6-axis force torque sensor (MMS101, Mitsumi Electric Company). The probe can be easily swapped out, and we designed in total probes with different textures. During data collection, tactile sensor is attached firmly to the CNC bed, and the CNC probes the tactile sensor by applying vertical force until the 7-axis force reaches threshold (uniformly sampled from 0.3 2.1N for most sensors, except GelSight Mini and GelSight Svelte for which we use 1.5 5N due to higher gel stiffness). Tactile images, probing positions in the sensor's coordinate frame (3D translational poses), and the force values are recorded.
A.5 Network configurations
In principle, all components of T3 can adopt different architectures, provided they maintain coherence. This section lists the specific architecture and
choices used in this work.
Encoders We use Trasoorrr (ViT) [41] blocks as the encoders for tactile images. image is first resized to 256 x 256 then center-cropped to 224 x 224, before split into 196 image blocks each measures 16 x pixels. Augmentation is performed during training. Specifically, color jittering is performed for all tasks. Random cropping, horizontal and vertical flipping are performed for all tasks except pose estimation tasks, which rely on the spatial information to make predictions. We use MIP ratio of 4.0, an embedding dimension of Denes Hene heads, and Tene layers across encoders for all sensors.
Trunk We use ViT blocks the trunk. We use MLP ratio of 4.0, an embedding dimension of DEn = Denes Heru Hene heads, and Lira layers.
Decoders
The decoder architecture varies based on the tasks. The reconstruction decoder for also VIT, with a MLP ratio of 4.0, an embedding
masked auto encoding during pre-training dimension of 512, 16 heads, and 8 layers. No pooling is performed after the trunk for this decoder.
The decoders for a; regression (Eqn. 2) and classification tasks are MLP decoders. We used the same size of 3 hidden layers [256, 128, 64] for all these MLP decoders. The output dimension varies for the classification tasks depending on the number of categories. Only the classification token is kept from the output of the trunk before it is passed through these decoders.
The decoders for both 3D and 6D pose estimation tasks constructed with 2 ResNet [32] blocks followed by average pooling then a MIP with hidden layers [256, 64). Before passing through the token outputs from the trunk into the ResNet blocks, we first remove the classification token, then reshape the output from B, T, C] to B.C.VTT-L. I, I]. Intuitively, transform the embedding space back to 2D map, on which convolution performed by the ResNet blocks.
Choices of hyper parameters are listed in Tab. 1.
13

---

Objects Table 2: Dataset Statistics
Source VisGel[26]-
Labels
Sensor
Marker
Size
Houschold ob- jects and toys. Flat images from the original dataset were filtered.
Variance of the Laplacian of diff.
Gel-
Yes
726,740
small*
Sight' 17 [12]
images
TVL [27]
Household jects
ob- semantic description
object
DIGIT [44]
No
82,463
Touch-and- Go [29]
Natural objects
Object IDs
Gel-
Yes
262,082
Sight'
[12]
Calandra') 17 [2]
Household
ob-
Object IDs, grasp
Gel-
Yes
24,1 118
jects
outcome
Sight'
[12]
Yuan' [4]
Clothes
11 properties such as smoothness, softness
Gel-
Yes
494,655
Sight' 17 [12]
TCB-Sight [28]
Household ob- jects from the YCB object set
Object IDs, 6D
GelSight variant No. and simulation
No
480 real 1,800 sim
poses
ObjectFolder- real [30]
Household jects
ob- Object, material IDs
GelSight variant No.
No
1,417,600
2
Tippur'23 23 [18]
Probing sphere
3D poses
GelSight
No
13,341
360
GelSight Wedge [13]
No
10,000
Object IDs 6D poses
Tool objects
GelSight Mini [45]
Ycs&No
24,000
DIGIT [44]
No
4,000
New data collected in-house
GelSight
No
3,516
Wedge
[13]
GelSight
No
6,432
Finray
Probes with engraved letters
Object IDs
[21]
3D poses
2 variants
GelSight
No
5,335
Svelte
[19]
Dense-
Yes
3,531
Tact2.0
[14]
GelSight
Yes
3,359
Mini [45]
Vislll-small is down- sowplsa from the original VisGel and most flat images removed.
14


Reviews:
[{"review_id": "8tPMXjsl3z", "reviewer": "Reviewer_1X3T", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Firstly, this paper proposes a large scale camera-based tactile sensing dataset with > 3M data points. Secondly, a transformer model is trained on this dataset and experiments are conducted to verify that the model is useful as a pre-trained starting point for various unseen tasks.", "summary_of_recommendation": "I am recommending to accept this paper because of its strengths and the usefulness of the contribution to the research community. The paper can be improved by addressing the weakness mentioned above.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Referring to the weakness mentioned above, I suggest the authors to do unseen task experiments, like the ones in Section 5.2, and compare the performance of a model trained from scratch on the unseen task to the performance of the finetuned model. This will shine a light on the usefulness of the proposed foundation model in a practically relevant setting.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rJeoXjq1q2", "reviewer": "Reviewer_aci4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposed a framework T3 for tactile representation learning that scales across multi-sensors and multi-tasks. The framework captures the shared latent information across different sensor-task pairings by constructing a shared trunk transformer with sensor-specific encoders and task-specific decoders. This paper also presents a novel Foundation Tactile dataset over 3 million data points. T3 pre-trained with FoTa achieved a certain degree of zero-shot transferability and good small-data-finetuning transferability. The creation and open-sourcing of the FoTa dataset represent a substantial contribution to the field, providing a valuable resource for future research.", "summary_of_recommendation": "Overall, this work is a valuable addition to the literature and holds promise for future advancements in tactile sensing. However, further investigations are required to conclude convincing results.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. As shown in Figure 1, the architecture of T3 includes sensor-specific encoders and task-specific decoders to work with any sensor-task combination. This means we need to create a new encoder every time we include a new sensor. To what extent can we call it a new sensor? Is it possible that those similar sensors share the same encoder? It seems to the reviewer that this architecture limits the generality of the framework to novel sensors. Did the authors investigate the benefits of having an encoder for each sensor?\n2. Some results are over-stated. For example, line 212 says, “We observed that pre-training significantly improved the evaluation performance with all network configurations with an average improvement of 31%”. However, as shown in Fig. 3a, finetuning from pre-train 1 seems only slightly better than training from scratch for medium model size. The significant improvement with a large model size may simply be due to the mismatch between the model size and the amount of training data. (The data size is not mentioned in Section 5/1).\n3. The comparison in 5.1 may not be fair. It is a common understanding that transfer learning is more effective than training from scratch. The experiment setup does not convince whether the pretraining has successfully learned the tactile representation. What if the authors use pre-trained weights from other datasets? \n4. The authors should investigate the learned tactile representation more instead of simply showing the different numbers in performance. Are there any more intuitive or fundamental understandings? Is there any pattern in the shared trunk?\n5. The zero-shot transferability is not towards totally new sensors or tasks but their combinations. As shown in Table 1 in the Appendix, the sensor Gelsight Svelte has only two tasks: classification and 3D poses. Does that mean the pre-trained model has never seen any data on Gelsight Svelte?\n6. The dataset is said to include 11 tasks. However, as shown in Fig. 2, most task labels are actually the same type of task, i.e., object classification. Unlike tasks such as force estimation or slip detection, the object classification task is much more affected by the tested objects than the sensors. So, a natural question is: Are the tasks included in this paper the right choices for learning tactile representation?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "amcoOCccPy", "reviewer": "Reviewer_V6xF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents a modular approach to tactile representation learning that is compatible with different visuo-tactile sensors and valuable for multi-task learning. The approach achieves this by curating a large dataset of visuo-tactile sensor recordings across various tasks. Different encoders are used for each sensor and different decoder heads for each task, allowing a common central backbone to be learned. This backbone enables various encoder/decoder combinations to be plugged in, depending on the sensor/task combination. The model is trained using unsupervised learning loss in conjunction with task-specific supervision loss. Results suggest that using the pretrained central backbone accelerates learning with new sensors or on new tasks.", "summary_of_recommendation": "Overall, the paper is well-written, scoped, and targeted towards improving tactile representation learning for few-shot transfer to new visuo-tactile sensors. Additional analysis on the sample efficiency and qualitative insight will further strengthen the paper.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- The results show that interesting transfer learning occurs when tested on a new sensor-task combination, even though the training data contains the sensor on a different task and the task with a different sensor. I am curious about the performance for a sensor that has not been seen at all in the dataset.\n\n- How wide is the task initialization range (socket position, gripper starting pose, plug in-hand position)? It seems that there is no such randomization. If the setting is fixed, an explanation about the success rate capping at 70% would be helpful. Would training for a longer period or increasing the demonstration data improve the performance? What is the performance bottleneck?\n\n- **Qualitatively**, how does training from scratch compare to T3? Does increasing the dataset size improve performance? A comparative analysis of the dataset size would be valuable to support the paper’s argument that T3 improves sample efficiency.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses learning tactile representation across a variety of vision-based tactile sensors and tasks.

The reviewers agree in the relevance and importance of this task and highlight several strengths including the dataset, the novelty in the training paradigm, and the overall clarity of the presentation.

Reviews, however, highlight a number of weaknesses with the strength and presentation of the results. Most importantly the reviewers note that there is only a marginal increase in performance using pre-training and there's a lack of broader insights / qualitative analysis provided in the results.

The authors addressed the reviewers concerns sufficiently well in the rebuttal phase, that I can now recommend acceptance.
Decision: Accept