Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks

Authors: jialiang  zhao
yuxiang  ma
lirui  wang
edward h adelson
mit  csail
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-03 03:50:23

Abstract: This paper presents T3: Transferable Tactile Transformers, a framework for tactile representation learning that scales across multi-sensors and multi-tasks.T3 is designed to overcome the contemporary issue that camera-based tactile sensing is extremely heterogeneous, i.e. sensors are built into different form factors, and existing datasets were collected for disparate tasks. T3 captures the shared latent information across different sensor-task pairings by constructing a shared trunk transformer with sensor-specific encoders and task-specific decoders. The pre-training of T3utilizes a novel Foundation Tactile (FoTa) dataset, which is aggregated from several open-sourced datasets and it contains over 3 million data points gathered from 13 sensors and 11 tasks. FoTa is the largest and most diverse dataset in tactile sensing to date and it is made publicly available in a unified format. Across various sensors and tasks, experiments show that T3 pre-trained with FoTa achieved zero-shot transferability in certain sensor-task pairings, can be further fine-tuned with small amounts of domain-specific data, and its performance scales with bigger network sizes. T3 is also effective as a tactile encoder for long horizon contact-rich manipulation. Results from sub-millimeter multi-pin electronics insertion tasks show that T3 achieved a task success rate 25% higher than that of policies trained with tactile encoders trained from scratch, or 53% higher than without tactile sensing. Data, code, and model checkpoints are open-sourced at https://t3.alanz.info.

Main paper: 
 introduction: 
 The tactile sensing modality has gained increasing popularity within the robotics community, by providing important fine-grained contact information for dexterous and contact-rich manipulation tasks, such as robot grasping [1, 2] and fabric manipulation [3, 4] . Camera-based tactile sensing [5] , a sensing method that operates by embedding a camera beneath a soft elastomer to capture the fine-grained interactions with the environment, is among the most popular methods of tactile sensing for its higher resolution and lower cost. However, camera-based tactile sensors are extremely heterogeneous, and there has not been a converged sensor design widely adopted by the robotics community. Different tactile sensors can differ significantly in shapes, types and numbers of cameras, placement and colors of illumination, etc. Such inherent heterogeneity hinders roboticists from building a general-purpose tactile encoder that is transferable across different sensors and downstream tasks.Existing learning architectures and datasets focus on one specific sensor-task pairing, and when it comes to a newly emerged sensor or task, data recollection and training an encoder from scratch are often required. This issue harms learning efficiency in a more significant way in longer horizon tactile manipulation tasks, where the training of the tactile encoder is guided by a more sparse reward. Intuitively, although different tactile sensors produce vastly different tactile images and various tasks extract different information from a tactile input, there should be shareable latent information due to the inherent similarities in tactile sensing across sensors and tasks. Therefore, it is both technically viable and practically desirable to design an architecture capable of extracting such shared latent representations and transferring them across different sensor-task pairings. Learning from heterogeneous tactile inputs shares many similarities with multi-modal representation learning. Recent works in multi-modal foundation models have demonstrated the potential capability to bridge the gap between different modalities and learning a common latent representation, such as in the perception domain [6, 7, 8] as well as the robotic manipulation domain [9, 10] . However, learning such foundation models often requires temporally aligned multi-modal inputs or a distance function that describes the resemblance between inputs from different modalities. In heterogeneous tactile learning, the hardware-level disparity often makes it ill-defined to align images collected from different embodiments.In this work, we tackle the problem of heterogeneous tactile learning using unaligned tactile data, with the goal of learning a scalable representation that can be shared across different sensors and tasks. We first aggregate existing open-sourced tactile datasets and assemble the Foundation Tactile (FoTa) dataset, which is the largest and most diverse tactile dataset so far to the best of the authors' knowledge. We then propose Transferable Tactile Transformers (T3) to learn a shared representation across multi-sensors and multi-tasks. T3 is constructed with sensor-specific encoders, a shared trunk, and task-specific decoders. Our experiments show that T3 pre-trained with FoTa achieves reasonable zero-shot transferability, it can be further fine-tuned with small domain-specific datasets, and its performance scales with bigger network sizes. We also demonstrate that T3 can be used as a tactile encoder for longer-horizon manipulation policies with 3 behavior cloning-based sub-millimeter level electronics insertion tasks. In summary, this paper proposes: Foundation Tactile (FoTa): a dataset containing 3, 083, 452 tactile images collected with 13 sensors for 11 tasks in a unified format. It is aggregated with several of the largest open-sourced tactile datasets and additional data collected in-house by the authors.Transferable Tactile Transformers (T3): a neural network framework that learns a shared representation across all sensors and tasks in FoTa. Pre-trained weights are provided, and we demonstrate that T3 can be easily fine-tuned to a new task or a new sensor with few fine-tuning data. 
related works: 
 Camera-based tactile sensors Camera-based tactile sensors are often constructed with a soft elastomer as the contact medium, a reflective layer, a light source to provide illumination, and one or more than one camera to capture the deformation of the elastomer, which therefore provides a detailed view of the contact surface. Specific design choices in camera-based tactile sensors are highly diverse, such as monochrome [11] v.s. colored [12, 13, 14] in illumination, with v.s. without markers [15, 16, 17] , and different form factors such as flat [12, 13] , dome-shaped [18, 14] , fingershaped [19, 20] , compliant [21, 22] , multi-linked [23] or even palm-shaped [24] . Such diversity and disparity have enabled researchers to use those sensors across a spectrum of applications such as in defect inspection [25] and robotic tool-using [9] . Nevertheless, they also add a substantial barrier in leveraging machine learning to encode the information coming from those sensors. A unique sensor-task pairing often requires data re-collection and neural network re-training.Datasets for tactile manipulation Several large-scale tactile sensing datasets have been released in recent years tackling visual-tactile cross-modality reasoning [26, 27, 28] , object or material classification [28, 29, 4] , longer horizon tasks such as robotic grasping stability analysis [2] , or shape reconstruction [30] . Those datasets also cover interactions with a wide variety of objects, including household objects and toys [2, 26, 27, 28, 30] , fabrics and clothes [4] , natural objects such as wood and gravel [29] . Quantity-wise, several of those datasets contain well over a million images [26, 30] , making the total number of available tactile images on par with ImageNet-1K [31] .While quantity is often not an issue when using existing open-sourced tactile datasets, one significant drawback is that all the aforementioned datasets lack diversity in terms of sensors and tasks. The vast majority of tactile datasets were collected entirely with only one specific tactile sensor. With this constraint, even if a sufficiently well-performing tactile encoder can be trained from one dataset, such a pre-trained encoder will not be able to transfer to another sensor. In order to learn a general-purpose representation for tactile sensing like [32, 33] does in the natural image domain, a large and diverse tactile dataset with a wide variety of sensor-task pairings is a prerequisite.Representation learning with heterogeneous data Learning from heterogeneous data domains, such as taking multi-sensory feedback as inputs or predicting outputs for different tasks, has been an active area of research, especially in the robot learning domain. Many existing works combine tactile, vision, and occasionally other modalities such as robot states and sound, by passing each modality through one modality-specific encoder, then concatenating them and passing them through a decoder network to predict the final output [34, 15, 35, 9, 36] . Apart from learning-driven neural networks, it is also common to use analytical tools, such as factor graphs or convex optimization, to combine predictions from multiple domains [37] .Another branch of multi-modal robot learning actively reasons the relationship among heterogeneous domains and cross-links them to learn a shareable latent representation. Extensive research has explored using contrastive learning to join temporally aligned modalities [6, 10, 38] . To connect vision and touch, Chen et al. designed a multi-modal self-attention mechanism [39] to learn a shared representation, while Li et al. proposed a cross-modal Generative Adversarial Network and were able to predict tactile images from visual images and vice versa [26] . Bachmann et al. applied masked auto-encoding (MAE) on aligned RGB images, depth maps, and semantic segmentation and learned a shared representation across three domains [7] . However, although cross-modality representation learning shares a lot of commonalities with heterogeneous tactile representation learning, one significant difference is that heterogeneous tactile representation learning lacks aligned data collected with different sensors for different tasks. It is often ill-defined to infer the resemblance of two tactile images collected from different hardware, which is a prerequisite for many selfsupervised learning techniques. It is therefore desirable to devise a new architecture that is able to learn from unaligned tactile data and produce a representation that is transferable across different tactile sensors and different tasks. 
the fota dataset: 
 We provide a large tactile dataset by aggregating existing public datasets as well as adding new data collected in-house. The FoTa dataset is provided in a unified, I/O efficient WebDataset [40] format. It is by far the largest and most diverse dataset for tactile perception, with 3, 083, 452 data points collected on 13 tactile sensors, and high-quality labels are provided for 11 tasks. Mixture of constituents of FoTa is illustrated in Fig. 2 .a, and example tactile images from each sensor are shown in Fig. 2 .b. Details and statistics about public datasets aggregated in FoTa as well as the pre-processing method are provided in Appendix A.2.Besides aggregating publicly available tactile datasets, we add new data collected with several recently emerged tactile sensors including GelSight Finray [5] , GelSight Svelte [19] , GelSight Wedge [13] , DenseTact 2.0 [14] , and GelSight 360 [18] . Those new data were also used for final performance benchmarking as well as transferability testing for the proposed learning framework. Two platforms were built to collect these additional data.A 7-DoF robotic platform collects tactile data by attaching 2 tactile sensors to a parallel jaw gripper of a 7-DoF Franka Emika Panda robot arm and commanding the robot arm to explore an object that is fixed at a known location. Robot pose in SE(3) is recorded for each step and then transformed into the sensor's coordinate frame. Object IDs as well as names are also recorded.A 3-DoF gantry platform is built with a desktop 3-axis CNC mill. Six 3D printed probes with different textures are attached to a force/torque sensor, which is then attached to the Z axis of the CNC. During data collection, a sensor is fixed at a known location, and the CNC machine probes the entire sensing surface of the sensor. Translational poses, probing forces, and probe IDs are recorded for each tactile image frame.The two systems are introduced in more detail in Appendix A.4. 
heterogeneous tactile learning with t3: 
 We propose Transferable Tactile Transformers (T3) as the backbone network to learn from diverse tactile sensors and produce outputs for diverse downstream tasks.Network architecture A training dataset contains inputs from N s different sensors and N t distinguished tasks. T3 is constructed with N s encoders {Enc 1 , ..., Enc Ns }, one for each individual tactile sensor; N t decoders {Dec 1 , ..., Dec Nt }, each responsible for one downstream task; and one shared trunk, denoted as T runk. During both training and inference, we ensure that one batch of data always comes from the same sensor-task pairing. For a given pair of data point (X i , Y j ) that is collected from sensor i for task j, the corresponding encoder Enc i and decoder Dec j are attached to the trunk. The constituent network components of T3 can take different architectures. In practice, we use ViT [41] in the trunk and encoders for its proven learning capacity and scalability in the natural image domain. For decoders, we use ViT for generative tasks (reconstruction for MAE), configurations are discussed in Appendix A.5.Learning objectives For each task j ∈ [1, N t ], we define a loss function L j . Different tasks require different numbers of tactile images as inputs. For example, tasks like object classification and material classification only require one tactile image to perform, while the pose estimation task requires two tactile images, e.g. X 1 , X 2 , because the prediction goal is the relative pose between the two instances. We pass each individual tactile image through the encoder and trunk separately, and we then concatenate them together before passing them through the decoder. The loss between a data pairEQUATIONwhere ⊕ denotes concatenation.The training process of T3 is split into two pre-training phases and one optional fine-tuning phase. Pre-training is divided into two phases to improve data utilization and to attain both local fine-grained understanding and global semantic understanding. Phase 1 focuses on more pixel-level understanding using self-supervised learning, and it is able to utilize all data from the FoTa dataset. Phase 2 focuses on semantic understanding and it requires suitable task labels that only a subset of the FoTa dataset possesses.Pre-training I: Self-supervised learning with Masked Auto Encoding (MAE) MAE was first proposed as a self-supervised pre-training technique for natural images [42] and it has gained increased popularity in other domains such as audio processing [8] , point-cloud processing [43] , and multi-modal representation learning [7] . It operates by first randomly masking out a certain portion of the original inputs, then it constructs a decoder to reconstruct the missing portion. Neural networks that were pre-trained with MAE have been shown to be able to outperform ones that were trained without MAE in various domains. This pre-training stage is able to utilize all data in FoTa, including unlabeled data (e.g. intermediate tactile images collected during the settling period of a robot grasp in [2] ), or data with semantic labels for which loss functions are hard to define (e.g. natural language description of a scene in [27] ).In this pre-training stage, one decoder Dec 0 that is designed to reconstruct partially masked images is shared by all domains. Dec 0 is constructed with 8 VisionTransformer blocks [41] . The loss L 0 is L2 pixel-wise loss normalized within each patch, following [42] .Pre-training II: Supervised learning with labels distilled from public datasets In this stage, T3 is trained under supervision from selected task labels in the FoTa dataset. For each individual task j ∈ [1, N t ], a decoder Dec j and a loss function L j is defined. We define 10 tasks for this training stage.• Classification tasks include object classification for [2, 29] , material classification for [30] , and fabric smoothness, fuzziness, textile type classification for [4] . Decoders for each of them are defined as MLP neural networks, and each L j is a cross-entropy loss function.• Regression tasks include SE(3) relative pose estimation bewteen two overlapped tactile images for [13, 44, 45] (with ResNet+MLP decoders), and variance of Laplacian (a metric describing the amount of information contained in tactile images, more details in Appendix A.2) estimation for [26] (with MLP decoders). Each loss L j is a mean squared error loss function.Fine-tuning: Supervised learning with task-specific data In this phase, T3 is further fine-tuned with data collected for the specific downstream task with the specific sensor that the user aims to use. This stage can be optional if the sensor-task pairing already exists in the pre-training dataset. Architecture-wise, T3 is configured the same way as it is in Pre-training II but with only one target task and sensor.  
experiments and discussions: 
 In this section we discuss the evaluation performance of T3 with numerous ablations and benchmarks. For both pre-training I and II, we use all data from the FoTa dataset except data collected on the gantry system in-house, including the object classification task and the 3D pose estimation task. Those data are therefore not included for pre-training the foundation model and are set aside for evaluation and fine-tuning only. We aim to answer the following questions in this section:• How to train efficiently and how much does pre-training improve performance? Sec. 5.1.• Is pre-trained T3 zero-shot transferable to new tasks and / or sensors? Sec. 5.2.• Does FoTa provide meaningful improvements in longer-horizon robotic tasks? Sec. 5.3.The fine-tuning and evaluation task for this section is an object classification task (6 categories) for 2 sensors (GelSight Wedge, GelSight Mini) with around 3,300 data points for each sensor. Averaged top-1 classification accuracy from the two sensors is reported. improved with larger networks, where large's classification accuracy was 19% higher than that of tiny. However, the performance difference between medium and large was insignificant. (c) When fine-tuned with half of the fine-tuning data, pre-trained networks performed better than trained from scratch networks, and larger pre-trained models performed better than smaller pre-trained models. In fact, for medium and large, the performances between fine-tuned with half data and fine-tuned with full data were very close, signaling those models generalized better thus they require less data to fine-tune to a novel task.More qualitatively analysis, including visualizations of the encoder attention maps and the trunk attention maps, can be found in Appendix. A.1.Masking ratios in Pre-training Stage I We tested 8 different masking ratios in the MAE reconstruction pre-training stage, and we performed the same Pre-training II stage as well as the fine-tuning stage on the 8 tests. The evaluation results are reported in Fig. 3 .b. We observe a small performance variance, ranging from 85% (mr = 30%) to 89% (mr = 80%), and a masking ratio of 80% yielded the highest performance.We tested to what extent a pre-trained T3 can be transferred to a new sensor or downstream task with and without fine-tuning. We picked two novel sensors (GelSight Svelte, DensetTact2.0) and two seen tasks (object classification, pose estimation between two overlapped tactile images). The two seen tasks were pre-trained with data from GelSight Wedge, GelSight Finray, and GelSight Mini. During this experiment, we use the encoder pre-trained for GelSight Wedge as the encoder for GelSight Svelte, and GelSight Mini's encoder in place of DenseTact2.0's encoder. Results on the 4 combinations are shown in Fig. 3 .c and Fig. 3 .d. Zero-shot transfer yielded only minor improvements over random guesses in the object classification task; however, it demonstrated significant improvement over dataset averages in the pose estimation task. We further fine-tuned the networks with a small amount of 2,000 data points. The classification accuracy immediately bounced up to 17% with fine-tuning, and the pose estimation root mean squared error (RMSE) was reduced by 5.5mm for DenseTact2.0. On the GelSight Svelte sensor, the pose estimation errors before and after fine-tuning were nearly identical and close to optimal. These results indicate that zero-transferability can be achieved in certain sensor-task pairings, and that better outcomes can be attained with minimal fine-tuning.Tactile sensing can be especially helpful in contact-rich manipulation tasks. Numerous human studies have shown that the sense of touch is important for humans to perform delicate manipulation tasks such as surgery [46, 47] . However, most state-of-the-art tactile manipulation research opted to train a tactile encoder from scratch, with fresh data collected with the specific sensor-task pairing, due to the lack of a pre-trained tactile encoder. In the case of long-horizon multi-modal manipulation tasks, training from scratch is often inefficient, due to the sparseness of trajectory rewards and the number of components that need to be trained (e.g. modality encoders and policy networks) especially in many reinforcement learning and behavior cloning applications such as [48, 9] To investigate whether a pre-trained T3 helps to bridge this gap, we designed a robotic precision insertion task with behavior cloning. The goal of this task is to insert 3 electronics parts: a 3-pin toggle switch, a 12-pin double-stack USB port, and a 17-pin VGA connector, onto a PCB with corresponding mounting holes to each component. This task requires high precision, where the clearance between the holes on the PCB and the pins on the parts is only 0.4mm. Achieving this precision requires active exploration with tactile feedback. In real-world applications, relying on vision alone is often insufficient due to heavy occlusion.Our experiment setup consists of two GelSight Wedge tactile sensors mounted on a parallel jaw gripper, which is attached to the end-effector of a 7-DoF Franka Emika Panda robot, a PCB board fixed at a known location on the workbench, the 3 electronics parts, and a RGB camera on the side of the workbench. An illustration of the setup is shown in Fig. 4 a and the parts are shown in Fig. 4 .b. Data is collected by controlling the robot in "guide mode", i.e. under zero stiffness control, and a human operator manually grabs and moves the end-effector until the part is fully inserted. At both data collection and inference time, the sockets are fixed, and we add randomizations to The results show that (1) the tactile modality is vital for this electronics insertion task, where the vision-only policy failed all tests to insert the two more challenging parts; (2) using a pre-trained T3 as the tactile encoder for this policy helped further improve the overall performance, where the task success rate is higher across all three parts; (3) T3 also helped to reduce the number of tactile exploration steps needed to insert a part. 
limitations and future works: 
 One limitation of the current training pipeline is that the FoTa dataset is unbalanced. Data collected with the 2 most popular sensors constitutes over 50% of the entire dataset. Therefore, the trained policy could also be biased towards those more popular sensors. Another limitation is that the current architecture of T3 focuses on per-image encoding and T3 is trained with explicit labels during pre-training II and fine-tuning. Representation learning on tactile image sequences with sparse or implicit labels can be a future direction of this work. Furthermore, T3 is currently limited to learning representations for camera-based tactile sensors. Extending the FoTa dataset and the T3 framwork to non-camera-based sensors is another future line of work. 
conclusions: 
 This paper presents T3, a large camera-based tactile sensing framework that transfers across multiple sensors and tasks; and FoTa, a tactile dataset that is by far the largest in quantity and most diverse in sensors and downstream tasks. Experiments show that T3 pre-trained with FoTa improves performance for various tasks, including longer-horizon manipulation tasks such as sub-millimeter electronics insertion, and that T3 transfers to a new sensor-task pairing with little fine-tuning. of the parallel jaw gripper, and the transformation between the camera frame and the end-effector frame is accurately measured. During tactile data collection, a test object, which is assumed to be rigid enough to be clamped firmly, is fixed on the workbench of the robot at a known location. The robot explores the object with random movements on all 6 dimensions (translation and rotation), and tactile images from both sensors as well as the robot pose are collected at each step. 
 
 Appendix: Another gantry-based 3-DoF data collection platform was built from a desktop CNC milling machine.The spindle was replaced with a 3D-printed probe, which is attached to the z-axis of the CNC with a 6-axis force torque sensor (MMS101, Mitsumi Electric Company). The probe can be easily swapped out, and we designed in total 6 probes with different textures. During data collection, a tactile sensor is attached firmly to the CNC bed, and the CNC probes the tactile sensor by applying vertical force until the z-axis force reaches a threshold (uniformly sampled from 0.3 -2.1N for most sensors, except GelSight Mini and GelSight Svelte for which we use 1.5 -5N due to higher gel stiffness). Tactile images, probing positions in the sensor's coordinate frame (3D translational poses), and the force values are recorded.In principle, all components of T3 can adopt different architectures, provided they maintain coherence. This section lists the specific architecture and hyper-parameter choices used in this work.Encoders We use VisionTransformer (ViT) [41] blocks as the encoders for tactile images. An image is first resized to 256 × 256 then center-cropped to 224 × 224, before split into 196 image blocks each measures 16 × 16 pixels. Augmentation is performed during training. Specifically, color jittering is performed for all tasks. Random cropping, horizontal and vertical flipping are performed for all tasks except pose estimation tasks, which rely on the spatial information to make predictions. We use a MLP ratio of 4.0, an embedding dimension of D enc , H enc heads, and L enc layers across encoders for all sensors.Trunk We use ViT blocks as the trunk. We use a MLP ratio of 4.0, an embedding dimension of D tru = D enc , H tru = H enc heads, and L tru layers.Decoders The decoder architecture varies based on the tasks. The reconstruction decoder for masked auto encoding during pre-training I is also a ViT, with a MLP ratio of 4.0, an embedding dimension of 512, 16 heads, and 8 layers. No pooling is performed after the trunk for this decoder.The decoders for σ i regression (Eqn. 2) and all classification tasks are MLP decoders. We used the same size of 3 hidden layers [256, 128, 64] for all these MLP decoders. The output dimension varies for the classification tasks depending on the number of categories. Only the classification token is kept from the output of the trunk before it is passed through these decoders.The decoders for both 3D and 6D pose estimation tasks are constructed with 2 ResNet [32] blocks followed by average pooling then a MLP with hidden layers  


Decision: This paper addresses learning tactile representation across a variety of vision-based tactile sensors and tasks.

The reviewers agree in the relevance and importance of this task and highlight several strengths including the dataset, the novelty in the training paradigm, and the overall clarity of the presentation.

Reviews, however, highlight a number of weaknesses with the strength and presentation of the results. Most importantly the reviewers note that there is only a marginal increase in performance using pre-training and there's a lack of broader insights / qualitative analysis provided in the results.

The authors addressed the reviewers concerns sufficiently well in the rebuttal phase, that I can now recommend acceptance.

Reviews: review: title: title: good contribution to the camera-based tactile sensing research community
paper_summary: paper_summary: Firstly, this paper proposes a large scale camera-based tactile sensing dataset with > 3M data points. Secondly, a transformer model is trained on this dataset and experiments are conducted to verify that the model is useful as a pre-trained starting point for various unseen tasks.
main_review: main_review: ### Strengths
- Collecting many existing tactile sensing dataset, adding new self-collected data, and making them available in a single dataset is a good contribution to the tactile sensing research community.
- The foundation model training uses a novel two-step pre-training method to improve data efficiency, and the efficacy of both the steps is experimentally verified.
- Section 5.3 shows that this pretrained model is indeed useful as a starting point for unseen manipulation tasks that require a tactile sensing component.
- The paper is well written and easy to understand.

### Weaknesses
- A pre-trained foundation model is most useful when training from scratch for a task is difficult, for example because of a scarcity of labelled training data for that task. This paper evaluates that the proposed model is indeed useful in this kind of scenario (Fig. 3a) but the evidence is weak. The difference in accuracy between a model trained from scratch vs. finetuned not very large. Additionally, this experiment tests only one task (object classification). It is not clear from the text (please clarify) but the foundation model is also trained on this task.
- some possibly typing errors:
  - L165 processes -> possesses
  - L216 base -> medium
questions: questions: Referring to the weakness mentioned above, I suggest the authors to do unseen task experiments, like the ones in Section 5.2, and compare the performance of a model trained from scratch on the unseen task to the performance of the finetuned model. This will shine a light on the usefulness of the proposed foundation model in a practically relevant setting.
limitations: limitations: 3
review_summary: review_summary: I am recommending to accept this paper because of its strengths and the usefulness of the contribution to the research community. The paper can be improved by addressing the weakness mentioned above.
title: title: Tactile representation learning with great potential
paper_summary: paper_summary: The paper proposed a framework T3 for tactile representation learning that scales across multi-sensors and multi-tasks. The framework captures the shared latent information across different sensor-task pairings by constructing a shared trunk transformer with sensor-specific encoders and task-specific decoders. This paper also presents a novel Foundation Tactile dataset over 3 million data points. T3 pre-trained with FoTa achieved a certain degree of zero-shot transferability and good small-data-finetuning transferability. The creation and open-sourcing of the FoTa dataset represent a substantial contribution to the field, providing a valuable resource for future research.
main_review: main_review: The quality of the paper is good. The work is written in clear language, and technical details are described and depicted clearly in the main text and figures. The reviewer appreciates the authors’ efforts in pulling up such a comprehensive dataset for tactile sensing. The main strength of the paper is that it gives a good review of the existing vision-based tactile sensing in terms of training dataset and various tasks. The paper shows the potential of learning a shared representation of tactile sensing across diverse sensors and tasks. 

Nonetheless, the reviewer has some concerns regarding the originality and significance of the paper to a broader audience in the relevant field. The main weaknesses of the paper are:
1. While T3 achieves zero-shot transferability and performs well with minimal fine-tuning across new sensors and tasks, the diversity of tactile sensors is largely limited to GelSight or its variations. These sensors share common sensing principles. The authors are suggested to discuss this in the limitations and future works.
2. Some of the conclusions are not convincing with quantitative solid evidence. Refer to the list of issues below.
3. The results is only in terms of numbers in performance. More investigations and understanding are expected.
questions: questions: 1. As shown in Figure 1, the architecture of T3 includes sensor-specific encoders and task-specific decoders to work with any sensor-task combination. This means we need to create a new encoder every time we include a new sensor. To what extent can we call it a new sensor? Is it possible that those similar sensors share the same encoder? It seems to the reviewer that this architecture limits the generality of the framework to novel sensors. Did the authors investigate the benefits of having an encoder for each sensor?
2. Some results are over-stated. For example, line 212 says, “We observed that pre-training significantly improved the evaluation performance with all network configurations with an average improvement of 31%”. However, as shown in Fig. 3a, finetuning from pre-train 1 seems only slightly better than training from scratch for medium model size. The significant improvement with a large model size may simply be due to the mismatch between the model size and the amount of training data. (The data size is not mentioned in Section 5/1).
3. The comparison in 5.1 may not be fair. It is a common understanding that transfer learning is more effective than training from scratch. The experiment setup does not convince whether the pretraining has successfully learned the tactile representation. What if the authors use pre-trained weights from other datasets? 
4. The authors should investigate the learned tactile representation more instead of simply showing the different numbers in performance. Are there any more intuitive or fundamental understandings? Is there any pattern in the shared trunk?
5. The zero-shot transferability is not towards totally new sensors or tasks but their combinations. As shown in Table 1 in the Appendix, the sensor Gelsight Svelte has only two tasks: classification and 3D poses. Does that mean the pre-trained model has never seen any data on Gelsight Svelte?
6. The dataset is said to include 11 tasks. However, as shown in Fig. 2, most task labels are actually the same type of task, i.e., object classification. Unlike tasks such as force estimation or slip detection, the object classification task is much more affected by the tested objects than the sensors. So, a natural question is: Are the tasks included in this paper the right choices for learning tactile representation?
limitations: limitations: 2
review_summary: review_summary: Overall, this work is a valuable addition to the literature and holds promise for future advancements in tactile sensing. However, further investigations are required to conclude convincing results.
title: title: Transformer-based Visuo-tactile representation learning for fast task learning.
paper_summary: paper_summary: This work presents a modular approach to tactile representation learning that is compatible with different visuo-tactile sensors and valuable for multi-task learning. The approach achieves this by curating a large dataset of visuo-tactile sensor recordings across various tasks. Different encoders are used for each sensor and different decoder heads for each task, allowing a common central backbone to be learned. This backbone enables various encoder/decoder combinations to be plugged in, depending on the sensor/task combination. The model is trained using unsupervised learning loss in conjunction with task-specific supervision loss. Results suggest that using the pretrained central backbone accelerates learning with new sensors or on new tasks.
main_review: main_review: The paper is clear and easy to follow, addressing the challenge of fast transfer learning of tactile-based policies to new visuo-tactile form factors.

The work used a number of interesting ideas for improved transfer learning across diverse visuo-tactile sensors, including:
- training a common trunk into which features from new sensors can be integrated. 
- leveraging both unsupervised learning and supervised learning for training the trunk, encoders and decoders.

However, more qualitative insights are needed to fully understand the results. The paper states that "results show that (1) the tactile modality is vital for this electronics insertion task, where the vision-only policy failed." While the raw success rates may indicate this, it is unclear why solving the task is otherwise impossible. Previous works, such as [1], have solved insertion in a fixed socket setting, so it is a little puzzling that the vision-only policy would completely fail in this setting without even a small success rate. A qualitative analysis of the failure cases would be helpful here.




- [1] Offline Meta-Reinforcement Learning for Industrial Insertion. Zhao et al
questions: questions: - The results show that interesting transfer learning occurs when tested on a new sensor-task combination, even though the training data contains the sensor on a different task and the task with a different sensor. I am curious about the performance for a sensor that has not been seen at all in the dataset.

- How wide is the task initialization range (socket position, gripper starting pose, plug in-hand position)? It seems that there is no such randomization. If the setting is fixed, an explanation about the success rate capping at 70% would be helpful. Would training for a longer period or increasing the demonstration data improve the performance? What is the performance bottleneck?

- **Qualitatively**, how does training from scratch compare to T3? Does increasing the dataset size improve performance? A comparative analysis of the dataset size would be valuable to support the paper’s argument that T3 improves sample efficiency.
limitations: limitations: 3
review_summary: review_summary: Overall, the paper is well-written, scoped, and targeted towards improving tactile representation learning for few-shot transfer to new visuo-tactile sensors. Additional analysis on the sample efficiency and qualitative insight will further strengthen the paper.