Sensor-Invariant Tactile Representation
Authors: Harsh Gupta
Yuchen Mo
Shengmiao Jin
Wenzhen Yuan
Venue: ICLR 2025 Poster (2025)
Keywords: Tactile sensing
representation learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a representation to perform zero-shot transfer across vision-based tactile sensors

Abstract: High-resolution tactile sensors have become critical for embodied perception and robotic manipulation. 
However, a key challenge in the field is the lack of transferability between sensors due to design and manufacturing variations, which result in significant differences in tactile signals. 
This limitation hinders the ability to transfer models or knowledge learned from one sensor to another. 
To address this, we introduce a novel method for extracting Sensor-Invariant Tactile Representations (SITR), enabling zero-shot transfer across optical tactile sensors. 
Our approach utilizes a transformer-based architecture trained on a diverse dataset of simulated sensor designs, allowing it to generalize to new sensors in the real world with minimal calibration. 
Experimental results demonstrate the method’s effectiveness across various tactile sensing applications, facilitating data and model transferability for future advancements in the field.

Published as a conference paper at ICLR 2025
SENSOR-INVARIANT TACTILE REPRESENTATION
Harsh Gupta" Yuchen Mo Shengmiao Jin
Wenzhen Yuan
University of Illinois Urbana- Champaagn {hgupt3, yuchenm7 / silliiiis.ee.eeeee
ABSTRACT
High-resolution tactile sensors have become critical for embodied perception and robotic manipulation. However, key challenge the field is the lack of transfer- ability between sensors due to design and manufacturing variations, which result in significant differences tactile signals. This limitation hinders the ability to another. To address this, Tactile Representations
transfer models or knowledge learned from we introduce a novel method to extract (SITR), enabling zero-shot transfer across optical tactile sensors. Our approach utilizes a transformer-based architecture trained on a diverse dataset of simulated sensor designs, allowing generalizability to minimal calibration. Experimental results demonstrate our method's effectiveness across various tactile sensing applications, facilitating data and model transfer- ability for future advancements in the field.
in the real world with
Real object
GelSight Mini
GelSight Mini v1
GelSight Mini v2
DIGIT
GelSight Hex
GelSight Wedge
Figure 1: Vision-based tactile sensors vary in both optical design and physical properties. Even with the same contact object, screw, the tactile images produced by each differ significantly. These variations highlight the challenge of transferring models from one sensor to another.
INTRODUCTION
Tactile sensing crucial modality for intelligent systems to perceive the physical world. Among the various tactile technologies, the GelSight sensor (Yuan et al. 2017a) and its variants (Wang et al. 2021 GelSight 2024 Zhho Adelson 2023) have recently emerged as one the most influential factile technologies, offering rich and detailed information contact surfaces. GelSight captures fine contact geometries through an optical system that transforms tactile data into visual images. This enables robots precisely detect object shapes, recognize materials, and perform fine-grained manipulations with a high degree of accuracy (Yuan et al. 2018 Dong et al. 2019 Hogan et al.
2020
Ota et al. 2023 Yang et al. 2023 Shirai et al. 2023).
Despite their advantages, GelSight-like sensors, and iision-based tactile sensing in a more general sense, still face key challenge: sensor variance. Differences in the optical design manufacturing process can result in significant discrepancies in sensor output. Consequently. machine learning models trained data from sensor often fail to generalize to other sensors. This challenge is further compounded by the high cost and effort of collecting tactile datasets, creating a major barrier sensor transferability in tactile perception.
cqual contribution

---

Published as a conference paper at ICLR 2025
In this paper, we address the challenge of data transferability between GelSight sensors by tackling sensor variance arising from optical design and manufacturing differences. The key issue lies in en- abling generalization to new sensors the domain gap between individual sensors is substantial and unpredictable. Previous methods, such as Yuan et al. (2018); Calandra et al. (2018), attempted to improve generalization by using multiple GelSight sensors to gather diverse tactile datasets, but this approach offered limited gains. More recently, T3 (Zhao et al. (024) sought to improve transferabil- ity by pre-training a transformer model across multiple sensors and tasks. However, their reliance on category- spprcfcccppiicccccc encoders limited the ability of their model to generalize to unseen sensors.
In contrast, we propose that achieving sensor transferability requires learning effective sensor- invariant representations by ensuring the model is trained on sufficiently diverse sensor variations. We introduce novel framework for generating sensor-invariant feature representations from high- resolution tactile readings, enabling zero-shot transfer to unseen sensors across multiple downstream tasks. Our framework incorporates three core innovations:
1. We utilize a small set of easy-to-acee to--uquiuur calibration images to characterize individual sen- sors. We then transformer model as the encoder to effectively combine the calibration images with the tactile reading.
2. We employ supervised contrastive learning (SCL) (Khosla et al. 2020) to emphasize the geometric aspects of tactile data, encouraging the clustering of similar contact geometries across multiple sensors. This training is further supervised by measuring geometric accu- racy.
3. We develop large-scale synthetic dataset using a physics- based simulator that models sen- sor optical systems, capturing variations in both sensor characteristies and contact geome- tries. This dataset, consisting of IM examples across 100 sensor configurations, provides the diversity necessary for robust model training using precise ground truth of the contact geometries.
Our motivation comes from the belief that contact geometry one of the most critical features for most tactile-driven tasks, including shape recognition, texture classification, and contact localiza- tion. By focusing on geometric accuracy and using calibration remove sensor-specific variations, we ensure the development of robust, sensor-invariant representations. Leveraging physics-based simulations allows us to efficiently generate diverse tactile datasets, reducing the time and cost of real-world data collection.
We evaluate the generalizability our method across various downstream tasks using multiple real- world GelSight sensors. Our results demonstrate that models trained one sensor can be seamlessly transferred to others in a zero-shot manner, significantly outperforming existing approaches. This framework paves the way for easier transferability of machine learning models and datasets between different sensors, thereby enhancing the future development of the tactile-sensing community.
2 RELATED WORKS
In the realm of vision-based tactile images, the application of computer vision models and algo- rithms has become common practice due the visual nature of the data these sensors capture (Dong ctt al. 1 Li et al. 2019 Calandra et al. 2018). Researchers have adapted mature representation learning methods from the vision community tactile images. One popular approach contrastive leariing. Both tactile and disual-tactile representations have been explored for specific tasks Yuann
al. 2017b; Yang et al. [0222] Tian et al.
2020 Kerr al., 2022, Guzey et al. 2023 Grill al.
2020
Zambelli et al.] 2021). Another technical approach is based on auto-encoding representa-
tion. Cao et al. (0023) and Xu et al. (2024) leveraged Masked Auto-Encoder (MAE) to learn tactile
representations.
However, many works that directly apply existing representation learning methods to the tactile modality ignore the significant domain gap seen between sensors. Representations trained on one sensor may work well on the exact same sensor or the same type, but the domain gap between different sensors makes models based on such representation fail generalize to other sensors. To address this, Zhao et al. (2024) trained individual encoder-decoder pairs for different sensor-task combinations, focusing learning the shared features and improving fine-tuned performance on
2

---

Published as a conference paper at ICLR 2025
new sensor-task combinations. Yang al. 2024 sought to address this by proposing a general- purpose multimodal representation for vision- based tactile sensors. By integrating multiple tactile datasets into large language model (LLM) framework and encoding sensor types tokens, they try to inform the L.M explicitly of the domain gap among different types of sensors. [Higuera et al. (2024) introduced family of self-supervised models for vision-hased tactile sensing that learns general-purpose representations by leveraging masking and self-distillation pretraining across multiple sensors. They aim improve tactile perception and downstream task performance under limited labeled data budget. However, these methods often depend on large datasets and treat sensor types fixed categories, failing account for variations within the same sensor type and lacking the flexibility to generalize zero-shot to unseen sensors.
supervision, supervised con- trastive learning, and sensor-specific calibration images. The calibration images capture sensor- specific domain features, such as optical properties unique to each sensor, which help the encoder adapt these characteristics. By accounting for subtle variations both within the same sensor type and across different types, our method enhances zero-shot generalization across tactile tasks and demonstrates strong transferability to new sensors.
Our framework introduces novel combination
3 SENNOR-INVARIANT REPRESENTATION LEARNING
Class Token
Patch Stacking
Seroor ID Contact ID 836
S
Negetives
Patch Embedding
Semsor ID Contact 070
Transformer
Samaor 8225 Contact ID 436
Tactile Image
Map
Normal Map Decoder
C normal
Contact
Predicted Normal Map
Truch Normal Map
Bickgrondd
Sensor-Invrrint Representation
Our invarrann representation learning framework. Each tactile image x is paired with set of calibration images c. We patchify and linearly project and to tokens. Additionally, e patches are region-wise stacked before projection. We concatenate the input tokens with a class token and pass it through transformer encoder. The class token is trained with SCL, while patch tokens are supervised by normal map reconstruction loss. We highlight grey the concatenation Sensor- Sensor-Invariant Tactile Representation (SITR) for
Figure 2:
of the output class token and patch tokens downstream tasks.
In this section, introduce our framework for training Sensor-Invariant Tactile Representation (SITR). We explain how calibration images capture sensor-specific information and use normal maps to preserve contact features. We introduce our implementation of SCL to align tactile features across sensor domains. We provide details on the role of calibration Section 3.1 followed by the network architecture and training process in Section 3.2
3.1 CALIBRATION IMAGES FOR TACTILE SENSORS
GelSight-like sensors map RGB values at each pixel to the local surface gradient, enabling the recon- struction of the contact surface. However, these sensors exhibit variations in physical properties that

---

Published as a conference paper at ICLR 2025
introduce sensor-specific artifacts in tactile images. A widely adopted calibration technique involves pressing a ball of known radius onto the sensor pad at various points. The tactile images captured during this process. combined with the known geometry of the ball, establish the correspondenee between RGB changes and the local surface gradient at different locations. This method generates sensor-specific look-up table. While traditional techniques assume pixel-invariant projection for simplicity, neural networks can further learn precise and pixel-dependent projections.
In the pre-training stage of SITR we adopt these steps inform the model of sensor characteristics. We include a cube in our calibration to inform SITR about how the gel deforms around edges and
corners. Thus, we press two objects 4mm diameter ball and cube corne- at nine locations each, roughly arranged in a 3 x 3 grid pattern across the sensor surface as seen in Fig. 3 These calibration images guide the encoder to identify and factor out -pnoor- features.
Formally, given tactile image E R#xWxC we select K calibration images G. Hxxxx where (H, W) is the resolution of the original im- age and is the number of channels. To efficiently encode multiple calibration images we reshape Cr into the form C E # in effect stacking the patches. We then linearly project æ; and e; into a sequence of N flattened 2D patches Tp E N and Cp E NxPC similar to standard ViT, where N = HW/P2, The resulting token sequences from Tps Cp. and a class token 2 are concatenated as input to the transformer encoder.
Figure 3: Calibration images used in SITR, obtained by pressing two objects a 4mm ball and a cube corner at nine different lo- cations each in a x 3 grid.
3.2 NETWORK ARCHITECTURE
Input: We use the tactile image and a set of calibration images for the sensor as inputs for the net- work. We subtract the sensor background from all the input images to get the pixel-wine color change as described in Section 3.1 Following the process described in Vision Transformer (ViT) (Doso-)
vitskiy et al. 2 and Section 3.T we linearly project the input and calibration images to tokens. Note that calibration images need only be tokenized once per sensor.
We modify VIT to process both image and calibration tokens. Adapted from ViT, we add positional encoding to them based on their 2D coordinates and then pass them into the encoder. We apply two supervision signals to train this encoder. One is the pixel-wise normal map reconstruction loss for the output patch tokens, and additional contrastive loss for the class token.
Encoder:
Normal map reconstruction:
During the SITR pre-training phase, we apply a lightweight decoder reconstruct the contact surface as normal map from the encoder output. Normal maps record the orientation of each 3D point on the contact surface. This feature is invariant to the variance across different sensors, contains rich geometry information for downstream tasks, and is viable for many GelSight-like dision-based tactile sensors. Therefore, we apply a pixel-wise MSE loss Lnomal between predicted normal map ň and ground truth normal map n.
Supervised contrastive learning: SCL is an extension of contrastive learning that leverages label information to learn more effective representations. Traditional contrastive learning aims to pull together similar samples and push apart dissimilar ones in the embedding space, typically relying on data augmentations to create positive pairs. SCL enhances this approach by utilizing class labels to define similarity, allowing for more semantically meaningful contrasts.
We employ SCI. create sensor-invariant representations from our labeled simulated tactile dataset. We label positive pairs from tactile images with the same contact geometry across multiple sensors, while negative pairs are labeled from images of different contact geometries or locations. batched implementation, we include captured by two different sensors. This approach allows us to learn discriminative features for downstream tasks while being robust to variations in sensor characteristics.
our views for each sample: tactile images of the same contact
Formally, given a batch of N samples, let class token 2; E Rd represent the encoded feature vector for sample where is the dimension of the embedding space. Let y denote its corresponding
4

---

Published as a conference paper at ICLR 2025
contact label. Let A() denote the of all samples in the batch except for sample itself. For each anchor sample i, we define the set of positive samples as P(i) = E A(i) Up = Yc. with [P())) being its cardinality. The supervised contrastive loss for batch of samples is then formulated
LSCL =
Σ RET
Σ
log
exp
P())
LeEA(s)
exp
PEP())
where 7 is temperature parameter that scales the similarity values to control the concentration of the distribution.
In summary, the total loss for SITR is defined L = Anormal Lnormal + ASCL LSCL where Anormaal and ÁSCL are loss weighting hyperparameters. Refer to Section A.1 for more implementation de- tails.
DATASETS
We collect three datasets for model training and evaluation. The first dataset contains purely syn- thetic data and used to train the encoder for SITR. The other two datasets are collected across real sensors
specific tactile applications: object classification and contact localization. These two datasets are used to evaluate the zero-shot transferability of SITR for downstream tasks.
4.1 SIMULATED TACTILE DATASET
We construct a large-scale synthetic dataset that spans wide range tactile sensor configurations, providing tactile signals of contact geometries along with their corresponding normal maps. The sensor's configuration is defined by its optical design, such as the location and optical properties of the lights, cameras, and reflective surfaces. These attributes quantify the major variances seen in real tactile sensors. The core idea is to train SITR with large distribution simulated sensors so SITR generalize to, and be aligned across, real-world sensors. This dataset is designed to be sensor-aligned, where each contact geometry is sampled across all sensor configurations for SCL.
We Physics-based Rendering (PBR) (Pharr et al.] (023)) to simulate GelSight sen- sors (Agarwal et al. 202I) and implement the algorithm in Blender. PBR simulates the cam- era images by tracing the path of light rays trav- eling in the scene and how they interact with optical components. Therefore, the technology models the physical behavior of the optical sys- tem and can simulate GelSight sensor's 's read- ing with parameterized optical settings. We de- sign the simulator platform to customize the sensors modulating the locations and char- acteristics of each optical component. Fig. 4 illustrates example setup where three light sources surround a deformable surface. A cam- positioned above captures the change of color on the surface caused by object contact.
Camera
Light Source
Gel pad
Figure Demonstration of our physics-based rendering (PBR) model to simulate GelSight sen- sors. We parameterize the sensor's optical design in the environment.
Sensor variation: To mimic the variance across real-world tactile sensors, we identify key param- eters that highlight the differences between real-world tactile sensors. Specifically, we look at the differences among GelSight Mini (GelSight, 2024), GelSight Hex (Yuan al., (017a). GelSight Wedge (Wang et al. 2021) GelSlim 3.0 (Taylor et al.] 2022), GelSight Finray (Liu & Adelson, 2022), and DIGIT (Lambeta et al., 2020). This includes light properties (shape, orientation, angle, color), gel properties (stiffness, specularity), and camera properties (FOV, sensing area). In total, we generate 100 unique simulated sensor configurations. More details on the sensor configurations and examples of rendered images for the same contact object can be found in Section A.2 For each sensor, we also collect a set of calibration images as described in Section 3.I We introduce random variability in the calibration positions make the training robust the real-world setting.
5

---

Published as a conference paper at ICLR 2025
Object diversity: To enable SITR generalize across diverse contact geometries, utilize 50 high-resolution 3D meshes of common household objects. These meshes include tools, kitchenware, toys, and clothing items, which are often used in robotics research. During simulation, the objects are randomly scaled, rotated, and placed at varying locations on the gel pad. For each contact geometry, render tactile images using all sensor configurations and pair them with ground-truth surface normal maps. We generate total of 1OK contact configurations through this process.
With 10K unique contact configurations across 100 different sensor configurations, we pre-train SITR encoder solely using our IM synthetic dataset.
4.2 REAL-WORLD TACTILE DATASET
We collect real-world datasets for training and evaluating downstream tasks across different base- lines. Compared to the synthetic dataset we used for SITR encoder pre-training phase, we keep SITR encoder frozen and train only the corresponding task-specific decoder head for downstream tasks. We use seven different sensors for our datasets: four GelSight Minis (GelSight) 2024) with varying sensor bodies and in-house gel pad modifications, GelSight Hex (Yuan et al. 2017a). Gel- Sight Wedge [Wang et al. 2021), and DIGIT (Lambeta et al., 2020).
For the classification task, we select 16 objects and press them against the sensor in various poses and depths, recording 1K tactile images for each object. We repeat this process for all 16 objects across the 7 sensors, resulting in a dataset with 112K tactile images, with 16K samples per sensor. Section A.3 shows that tactile signals vary even when using the same object across different sensor configurations.
For the pose estimation task, modify Ender-3 Pro 3D printer by replacing its extruder with 3D- printed indenters and mount the tactile sensors onto the print bed. This setup provides the accurate ground truth pose of each contact, including metric J, y. and values. During the data collection process, we press indentors at various locations and depths on the sensor surface. We collected IK samples per indentor for 6 different indentors across 4 sensors. This results in a dataset of 24K tactile images with precise pose labels, with 6K samples per sensor. More details can be found in Section A.4
EXPERIMENTS
In this section, show several experiments to evaluate the zero-shot transferability of our model to different real sensors. We evaluate model performance on three downstream tasks: shape recon- struction, object classification, and contact localization.
5.1 EXPERIMENT SETTING
We conduct experiments with multiple real tactile sensors that can be divided into two groups:
Intra-sensor set: GelSight Mini of different gel pads. These sensors have the same optical design, i.e., placement of camera and light sources, but differ in brightness and color of tactile signals due manufacturing differences and choice of coating materials.
Inter-sensor set: GelSight Mini 1, GelSight Wedge, GelSight Hex, and DIGIT. These sen- sors are designed with very different optical structures and, therefore, generate tactile sig- nals that are significantly different from each other.
For each downstream task, we freeze the SITR encoder and only train the downstream task task-specific decoder on single sensor. We evaluate this model using the rest of the sensors in the set. Formally, let S = (S1, S2, Sn.} be the set of sensors. Let Ai represent the performance (e.g., classi- fication accuracy or pose estimation error) when trained on S, and evaluated on Sy. The transfer performance across all sensors in the set is computed as
Transfer Performance
A
nnn
1)
J

---

Published as a conference paper at ICLR 2025
We also compute the score when training and testing on the same sensor i acting upper bound of the performance: No Transfer Performance = -
Baseline: We compare SITR with ViIs that are either trained from scratch fine-tuned from ImageNet weights to show the effectiveness of our method. As there no previous work that also compare against T3 (Zhao et al., 2024) and UniT (Xu et al. 2024). T3 focuses on improving few-shot fine-tuning results across different sensors and has the potential for zero-shot transfer. UniT learns dense representations for various downstream tasks and shows preliminary results on transferring among GelSight Mini sensors. We evaluate their available models for our experiments to compare the transferability of these representations. Additionally, we ablate the method used in T3 (MAE) and UniT (VQGAN)
directly focuses transferable tactile representations,
when trained on our synthetic dataset to test the effectiveness of our architecture in Section A.6 We
describe model configurations and decoders for each task in Section A.1
5.2 ZERO-SHOT TRANSFER FOR SHAPE RECONSTRUCTION
Simulation
Simulation
Galsight Mini
DIGIT
GelSight
GelSight Wedge
Figure 5: Reconstruction examples for various sensors. The top row shows input tactile images, the middle row presents 3D reconstructions, and the bottom row shows the contact objects. Simulated sensors (Simulation and 2) are in the training set, while real sensors (GelSight Mini, DIGIT, Hex, Wedge) are not.
We qualitatively evaluate how SITR preserves geometry and texture information by reconstructing the contact height map. As shown in Fig. 5 we reconstruct normal maps for objects in our real-world classification dataset and integrate them to generate their corresponding height maps. These 3D re- constructions capture fine-grained geometry and texture details of the contact surface. Though, these reconstructions are naturally constrained by the resolution and sensitivity limitations of the sensors. Despite these limitations, the preservation of dense surface features demonstrates the robustness of SITR in accurately modeling the contact geometry across varying sensor inputs.
5.3 OBJECT CLASSIFICATION
We compare SITR with baselines using our real-world classification dataset from Section 4.2 and report top-1 accuracy. We freeze our SITR encoder and train the downstream classifier using cross- entropy loss. For T3, iments. Since T3 does not provide encoder weights for GelSight Hex or DIGIT, we report inter- sensor results only for the GelSight Wedge and Mini. Note that T3's encoders were trained on marked sensors, so the results in our unmarked evaluations may not reflect their full potential. UniT demonstrates transferability only within GelSight Minis, so we exclude from inter-sensor exper- iments. We train a UniT encoder our unmarked real-world dataset and evaluate its intra-sensor transfer performance.
their released GelSight Mini encoder weights for intra-sensor exper-
As shown in Table SITR outperforms all baselines by large margin regarding classification accuracy when transterred across sensors. Note that most models perform well under the no- no-transfer setting, but fail to generalize when tested understand tactile features learned in the same domain, but SITR can capture meaningful features that are robust to changes in the sensor domain. We also find that the VIT pre-trained on ImageNet
different sensor. This indicates that baselines

---

Published as a conference paper at ICLR 2025
Method
Intra-sensor set f
Inter-sensor set
Wedge-Mini 1
No transfer
ViT-Base Scratch
36.90 + 22.19
24.02 + 1488
52.56 4.95
96.76 = 1.41
ViT-Base Pre-trained
73.22 22.42
48.10 22.82
76.28 +: 17.06
99.01 =: 1.14
Vir-Large Pre-trained
78.38 1: 17.79
54.34 + 23.04
79.04 1: 16.44
99.44 1 0.43
T3-Medium
38.66 20.63
17.02
93.77 = 2.87
UniT
46.39 23.30
92.53 = 4.19
SITR (Ours)
90.23 8.16
81.94 + 12.92
90.80 + 2.85
99.72 0.22
Table I: Results of object classification accuracy on 16 classes for model transfer and no-transfer performance. We report the mean and standard deviation of transfer accuracy percent among the sensor sets specified. Random guess classification accuracy corresponds to 6.67%.
performs better than that trained from scratch, which indicates the effectiveness of pre-training on the image domain.
Moving to feature-level analysis. Fig. 6 presents the t-SNE visualization of the SITR features for the contacts in our real-world classification dataset. The visualization illustrates that the of contrastive learning significantly improves feature clustering, bringing together samples of the same object across different sensors. This indicates that SITR successfully aligns the tactile signals from different sensors, highlighting its capacity to eliminate sensor-variant features.
No SCL
T =
Screw (Side)
Tey Avocado
Lego
Power
Microphome
GelSight
GelSiggt
GelSight Wedge
Figure 6: t-SNE visualization the feature space. We qualitatively show that our contrastive loss term helps cluster those similar contacts from different sensors together.
However, the results also reveal some challenges. The features from the DIGIT sensor somewhat more difficult to cluster with those from other sensors. This is better demonstrated in our detailed transfer results in Sec. A.S.II where see relatively worse classification transferability from the DIGIT sensor. We attribute this to DIGIT's distinct optical design, which differs from the GelSight designs our simulation dataset. We believe the result will be improved the future we extend our synthetic dataset to cover optical designs similar to the DIGIT sensor.
and
5.4 POSE ESTIMATION
In this task, try estimate the 3-DoF (x, y, 22) position change of the object in contact using an initial and final tactile image. We separately feed 2 tactile images of the same object into the frozen SITR encoder, concatenate their features, and train a decoder to learn the pose change with mean square error (MSE) loss. For baseline models, we use similar pipelines detailed in Section A. We evaluate this task on the inter-sensor set to see how each model handles differences scale across sensors. Each sensor in this set has different physical design, meaning they capture tactile signals at varying scales. Variations in object size may create significant challenges for zero-shot transfer tasks like pose estimation.
As shown in Table 2 SITR demonstrates strong performance on the pose estimation when tested on different sensor, reducing the RMSE by about 50% compared to baselines. Remarkably, all models have similar RMSE errors for the no-transfer setting. This may suggest sub-millimeter inaccuracies present in our data collection process. Nonetheless, the no-transfer setting serves the upper bound
8

---

Published as a conference paper at ICLR 2025
Method
Inter-sensor set
Wedge-Mini
No transfer
ViT-Base Scratch
1.63 0.20
1.69 = 0.13
0.56 0.02
ViT-Base Pre-trained
1.58
1.65 4 0.13
0.49 0.01
Vir-Large Pre-trained
1.49 0.25
1.45 +: 0.01
0.50 1: 0.02
T3-Medium
1.7 0.07
0.51 0.02
SITR (Ours)
0.80 1: 0.21
0.62 1 0.11
0.51 0.01
Table 2: Results of pose estimation with 6 objects. We report the mean and standard deviation of transfer pose estimation root mean square error (RMSE) in mm among the sensor sets specified. Random guess pose estimation RMSE corresponds to 2.52mm.
for transfer setting. We also find that compared VIT trained from scratch, the VIT pre-trained on ImageNet only marginally improves this task. This indicates that features learned from natural images may not transfer adequately to the tactile domain for accurate regression tasks like pose estimation.
ABLATIONS
6.1 NUMBER AND TYPE CALIBRATION IMAGES
Effect of number of calibration images Classification, Intra-senser f
(i)
iii
Classification, Inter-sensor f
(iii) Pose cstimation, Inter-sensor
$1.94
Acoansy
$7.5
96.54
Aceuncy
60
76,46
77.08
79.73
RMSSE [mm)
01:7
30.76
70
69
89
13"
155
16"
Number of Calibration Imsgges
Number Calibration Imagus
Number of Calibration Imagges
Ablation study on the number and type of calibration images used in SITR. showing their effect on Classification accuracy for inter-sensor transfer, (ii) Classification accuracy for intra- sensor transfer, and (iii) Pose estimation error for inter-sensor transfer.
Figure 7:
We conduct an ablation study to investigate the impact of the number and type of calibration images on the performance of SITR. the standard SITR setup, we press two objects-a ball and cube corner-at at nine locations roughly arranged in 3x3 grid pattern across the sensor surface. To ex- plore variations, we retrained SITR using different subsets of these calibration images and evaluated performance across all downstream tasks.
We test on five calibration configurations: No calibration images (0); Ball pressed at 4 corners (4); Ball pressed in 3x3 grid (9); Ball and cube pressed at 4 corners (8*); Ball and cube pressed in a 3x3 grid, which the standard setup (18*). Fig. 7 illustrates how different numbers and types of calibration images impact SITR's performance. We observe that increasing the number of calibration images increases performance across all tasks. However, the performance gains diminish images of the same object are added (as in the progression from cases (0) to (4) to (9)). Introducing second calibration object with distinct geometry. such as the cube (cases (4) to (8+)), results larger performance boost compared to simply adding more images the same object (cases (4) to (9)). The effect of calibration images is particularly notable in the inter-sensor setting, where see upwards of 20% increase in classification accuracy from case (0) (18*). We choose case (18*) for SITR since increasing the number of calibration images does not incur additional inference costs, as calibration tokens are computed only once per sensor.
6.2 CONTRASTIVE LOSS AND TEMPERATURE
We conduct an ablation study to assess the effect SCL and varying contrastive temperatures on SITR's 's performance. Specifically, compared models with and without the SCL. term and tested

---

Published as a conference paper at ICLR 2025
Effect of contrastive learning temperature
Classification, Intra sensor
(ii) Classification, Inter-sensor 1
(iit) Pose estimation, Inter-sensor
$2.18
Accarnty
60
Accuracy [N)
80
74.12
75.49
RMSE [mm]
0.82
70
66
SCL
0.10
SCL.0.25
0.07
SC1.
0.07
0.00
Temperature
Temperature
Temperaure
Figure 8: Ablation study examining the impact of SCL and varying contrastive temperature T on SITR's s performance. Subplots (i) and (ii) show classification accuracy in inter-sensor and intra- sensor settings, respectively, while (iii) shows the effect pose estimation RMSE.
five contrastive temperatures: 0.25, 0.10, 0.07, 0.03, and 0.01. No SCL corresponds to using only the normal map reconstruction loss during pre-training. Results Fig. show that contrastive temperature of 0.07 achieves the best classification performance in the intra-sensor setting, while 0.03 performs best for the inter-sensor setting. Lower or higher temperatures lead to reduced per- formance in both cases. For the pose estimation task, the addition of SCL has negligible impact on the RMSE. These results suggest that contrastive learning helps align features across sensors in classification tasks. However, in the pose estimation task, the model's performance is more depen- dent on the fine-grained geometry information from the contact surface. For SITR, we choose a temperature of 0.07 for its strong performance in the classification task.
7 DISCUSSION
Our qualitative and quantitative results indicate that SITR can generalize across sensors while pre- serving key geometric and texture features from tactile interactions. Our model has been largely trained and evaluated on optical tactile sensors with flat gel pads within GelSight family. De- spite this, SITR can be adapted to broader range of sensors. Our PBR environment can be easily expanded to accommodate new parameters to explore distinct optical properties in flat tactile sen- For more complex optical sensors like GelSight Svelte (Zhao & Adelson 2023) DIGIT 360 (Lambeta et al. 2024). adaptation remains feasible using an appropriate PBR model and con- tact surface mapping.
One future direction of our framework to generalize to traditional array-based tactile sensors. The challenge lies in bridging the signal modalities of low-resolution normal force distribution to the high-resolution contact geometry from GelSight sensors. One possible approach is to downsample dision-based tactile sensors' depth maps to approximate low-resolution tactile signals while estab- lishing a meaningful invariant relationship between depth and force. While this approach provides a step towards a unified tactile modality, its effectiveness in maintaining transferability requires further validation and exploration.
Another direction of future work is incorporating marker-based tactile information to SITR. Many variations of GelSight are equipped with markers-disting patterns embedded within the gel sur- face-that provide force and torque information. Currently, these markers are reconstructed using simple computer vision techniques to generate marker motion field. We believe that unifying marker motion fields between sensors may possible with adaptations to calibration in SITR. This extension would broaden the applicability of our model to wider range of tactile sensing tasks.
8 CONCLUSION
In this paper, we introduced SITR, tactile representation that transfers across various vision-based tactile sensors in zero-shot manner. We build large-scale, sensor-aligned datasets using synthetic and real-world data, and propose a method to train SITR to capture dense, sensor-invariant fea- tures. Our experimental results demonstrate that SITR outperforms baseline models and other lated tactile representations in different downstream tasks, showcasing robust transferability and effectiveness. SITR represents step towards unified approach tactile sensing, where models can generalize seamlessly across different sensor types, facilitating advancements in robotic and tactile research.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like thank Amin Mirzace and Ruohan Zhang for their help with hardware design and prototyping. The authors thank Ruihan Gao for her assistance with revising the paper. We thank Arpit Agarwal for his insightful discussions on tactile simulation. This work was supported in part by the NSF award #2024882, the Indira Gunda Saladi Engineering Research Prize, and the Illinois Office of Undergraduate Research.
REFERENCES
Arpit Agarwal, Timothy Man, and Wenzhen Yuan. Simulation of vision-based tactile sensors using physics based rendering. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 1-7, 2021. doi: 10.1 11119//CRA45506.200 200.:4444500.2219956611111
Roberto Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, Edward Adelson, and Sergey Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters, 3(4):3300-3307, 2018.
Guanqun Cao, Jiaqi Jiang, Danushka Bollegala, and Shan Luo. Learn from incomplete tactile data: Tactile representation learning with masked autoencoders, In 2023 JEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pp. 10800-10805. IEEE, 2023.
Siyuan Dong, Daolin Ma, Elliott Donlon, and Alberto Rodriguez. Maintaining grasps within slip- ping bounds by monitoring incipient slip. 2019 International Conference on Robotics and Au- tomation (ICRA), May 2019. doi: 10. 10.1109/icra.20) 2011/999.....88993333....
Siyuan Dong, Devesh K Jha, Diego Romeres, Sangwoon Kim, Daniel Nikovski, and Alberto Ro- driguez. Tactile-rl for insertion: Generalization objects of unknown geometry. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 6437-6443, IEEE, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x words: Transformers for image recognition at scale, 2021. URL httpt:tpppp................p arriv.iv... http:////////oooooooaaaaa/////00
20 1 1929
Inc GelSight. Gelsight mini, 2024. http:::/wwww.gels.t
com/ htmmmgglsighttii// [Ac-
cessed: 2024-10-01].
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21253 2020.
Irmak Guzey, Ben Evans, Soumith Chintala, and Lerrel Pinto. Dexterity from touch: Self-supervised pre-training of tactile representations with robotic play. arXiv preprint arXiv:2303. 12076, 2023.
Carolina Higuera, Akash Sharma, Chaithanya Krishna Bodduluri, Taosha Fan, Patrick Lancaster, Mrinal Kalakrishnan, Michael Kaess, Byron Boots, Mike Lambeta. Tingfan Wu, and Mustafa Mukadam. Sparsh: Self-supervised touch representations for vision-based tactile sensing, 2024.
URL http////www.oo.///////////////// / 24090
Francois Hogan, Jose Ballester, Siyuan Dong, and Alberto Rodriguez. Tactile dexterity: Manip- ulation primitives with tactile feedback. In 2020 IEEF international conference robotics and automation (ICRA), pp. 8863-8869. IEEE, 2020.
Justin Kerr, Huang Huang, Albert Wilcox, Ryan Hoque, Jeffrey Ichnowski, Roberto Calandra, and Ken Goldberg. Self-supervised visuo-tactile pretraining locate and follow garment features. arXiv preprint arXiv:2209. 2022.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Samna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18666 8661-1877, 2020.

---

Published as a conference paper at ICLR 2025
Mike Lambeta, Po-Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, et al. Digit: A novel design for low-cost compact high-resolution tactile sensor with application to in-hand manipulation. JEEE Robotics and Automation Letters, 3(3):3838-384 2020.
Mike Lambeta, Tingfan Wu, Ali Sengul, Victoria Rose Most, Nolan Black. Kevin Sawyer, Romeo Mercado, Haozhi Qi, Alexander Sohn, Byron Taylor, Norb Tydingco, Gregg Kammerer, Dave Stroud, Jake Khatha, Kurt Jenkins, Kyle Most, Neal Stein, Ricardo Chavira, Thomas Craven- Bartle, Eric Sanchez, Yitian Ding, Jitendra Malik, and Roberto Calandra. Digitizing touch with an artificial multimodal fingertip, 2024. URL
attaaa///all. /411.411111.00 02479
Yunzhu Li, Jun-Yan Zhu, Russ Tedrake, and Antonio Torralba. Connecting touch and vision via cross-modal prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10609-10618, 2019.
Sandra Q Liu and Edward Adelson. Gelsight fin ray: Incorporating tactile sensing into a soft com- pliant robotic gripper. In 2022 IEEE 5th International Conference Soft Robotics (RoboSoft), pp. 925-931. IEEE, 2022.
Kei Ota, Siddarth Jain, Mengchao Zhang, and Devesh K Jha. Tactile pose feedback for closed-loop manipulation tasks. In Robotics: Science and Systems workshop. 2023.
Matt Pharr, Wenzel Jakob, and Greg Humphreys. Physically based rendering: From theory to implementation. MIT Press, 2023.
Yuki Shirai, Devesh K Jha, Arvind U Raghunathan, and Dennis Hong. Tactile tool manipulation. arXiv preprint ar arXiv:2301.066 006669, 2023.
Ian H Taylor, Siyuan Dong, and Alberto Rodriguez. Gelslim 3.0: High-resolttion measurement of shape, force and slip in a compact tactile-sensing finger. In 2022 International Conference on Robotics and Automation (ICRA), pp. 10781-10787. IEEE, 2022.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020.
Shaoxiong Wang, Yu She, Branden Romero, and Edward Adelson. Gelsight wedge: Measuring high-resolution 3d contact geometry with a compact robot finger. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6468-6475. IEEE, 2021.
Zhengtong Xu, Raghava Uppuluri, Xinwei Zhang, Cacl Fitch, Philip Glen Crandall, Wan Shou, Dongyi Wang, and Yu She. Unit: Unified tactile representation for robot learning. arXiv preprint arXiv:2408.0644 2024.
Fengyu Yang. Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, and Andrew Owens. Touch and go: Learning from human-collected vision and touch. arXiv preprint arXiv:2211.12499 2022.
Fengyu Yang. Chao Feng, Ziyang Chen, Hyoungseob Park. Daniel Wang, Yiming Dou. Ziyao Zeng, Xien Chen, Rit Gangopadhyay, Andrew Owens, et al. Binding touch to everything: Learning unified multimodal tactile representations. In Proceedings of the IEEE/VVF Conference on Com- puter Vision and Pattern Recognition, pp.
2024.
Wenyan Yang, Alexandre Angleraud, Roel S Pieters, Joni Pajarinen, and Joni- Kristian Kämäräinen. Seq2seq imitation learning for tactile feedback-based manipulation. arXiv preprint 2:X.::2303.02646,, 2023.
Wenzhen Yuan. Siyuan Dong, and Edward Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017a.
Wenzhen Yuan, Shaoxiong Wang. Siyuan Dong, and Edward Adelson. Connecting look and feel: Associating the visual and tactile properties physical materials. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5580-5588, 2017b.
12

---

Published as a conference paper at ICLR 2025
Wenzhen Yuan, Yuchen Mo, Shaoxiong Wang, and Edward H. Adelson. Active clothing material perception using tactile sensing and deep learning. 2018 IEEE International Conference on Robotics and Automation (ICRA), pp.
2018.
doi: 10.1109/ICR.2011 8.8461164,
Martina Zambelli, Yusuf Aytar, Francesco Visin, Yuxiang Zhou, and Raia Hadsell. Learning rich touch representations through cross-modal self-supervision. In Conference on Robol Learning, pp. 1415-1425. PMLR, 2021.
Jialiang Zhao and Edward H Adelson. Gelsight svelte: A human finger-shaped single-camera tactile robot finger with large sensing coverage and proprioceptive sensing. In 2023 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pp. 8979-8984, IEEE, 2023.
Jialiang Zhao, Yuxiang Ma, Lirui Wang, and Edward Adelson. Transferable tactile transformers for representation learning across diverse sensors and tasks. arXiv preprint arXiv:2406. /3640, 2024.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 IMPLEMENTATION DETAILS
This section outlines the detailed implementation steps, including pre- pre-processing, architecture, train- ing settings, and decoder choices for all models.
A.1.1 PRE-PROCESSING
For SITR, apply the following pre pre-proeessing steps across real and simulated sensors:
1. All input images are resized 224 224. For the GelSight Wedge sensor, an affine transformation is applied to correct distortions in the tactile images.
2. Batched data augmentations are applied during training to both the tactile input and cali- bration images, including color jitter and Gaussian blur.
3. Background subtraction is performed on each image to isolate the tactile signal. All im- ages are then normalized based on the mean and standard deviation calculated from the simulated dataset.
A.1.2 ARCHITECTURE
Encoders: Table shows the number of parameters used in each encoder.
Model
Number of Parameters
ViT-Base
86M
ViT-Large
307M
T3-Medium
173M
UniT
25M
SITR (Ours)
96M
Table 3: Comparison of model parameters.
Our SITR model is derived from the ViT-Base architecture. The key modification is in the patch em- bedding, where we tokenize the tactile input and calibration images separately and add a positional embedding before passing them through the transformer.
SITR Training Decoders: During the pre-training phase for SITR, use decoders:
We apply simple linear projection the output tactile image tokens from SITR. We reshape and unpatchify the output to create a feature image map. We supervise with MSE loss Anormal against the ground truth normal map.
Normal Map Reconstruction Decoder:
Class Token Decoder: The class token passed through linear projection dimensional embedding. We then supervise this embedding with SCL loss ASCL.
128-
Loss Terms The total loss during training is weighted sum of these two loss terms: L Anormal Lnormal + ASCL LSCL We set both loss weighting hyperparameters Anormal and ÀSCL to 1.
Downstream Task Decoders: We try several decoders for downstream tasks for each baseline and task and report the best-performing ones here.
1. Classification Decoders We use Cross Entropy Loss for this task.
SITR: We unpatchify the output tokens 2; to a feature map and pass it through a ResNet- 18 network. The resulting feature vector is concatenated with the class token 2;. We then apply a 3-layer MLP decoder with dimensions [256, 128, 16]. The SITR encoder is frozen during this process.
14

---

Published as a conference paper at ICLR 2025
VIT: For all ViT encoders, we linearly project the class token to output of 16 dimensions. We also find that unfreezing the ViT pre-trained weights during training improves performance.
T3: We unpatchify the oulput tokens feature map and pass it through a ResNet-18 network with an output dimension of 16. The T3 encoder is frozen for this process.
UniT: We directly apply their proposed pooling and MLP decoder blocks to output dimension of 16. We find that unfreezing the UniT encoder provides better results.
2. Pose Estimation Decoders
use MSE loss for this task.
SITR:
We pass 2 tactile images 21 and x2 into the network separately. We unpatchify the output tokens from œ and 22 and concatenate their feature maps. We pass the concatenated feature maps into modified ResNet-18 with 6-channel input. We then linearly project the resulting feature vector to an output dimension of 3. The SITR encoder is frozen during this process.
VIT: For all VIT encoders, we pass 2 tactile images 21 and 22 into a modified ViT network allowing 6 channel imput. We then linearly project the resulting class token to an output dimension of We unfreeze the ViTs when training.
T3: We follow the same procedure described in SITR's s pose estimation decoder. tactile images 21 and 22 are passed into the network separately. We unpatchify the output tokens from di and and concatenate their feature maps. We pass this feature into a modified ResNet- 18 and linearly project the resulting feature vector an output dimension of 3. We keep the T3 encoder frozen during this training process.
15

---

Published as a conference paper at ICLR 2025
A.2
SIMULATED DATASET
Lower bound
Upper bound
Lower bound vis.
Upper bound vis.
Lower bound env.
Upper bound env.
Parameter
Light
shape
point
area
Light orienta- tion
sides
corners
Light angle
5
30
Light
color rand
rand
Gel
low
high
stiffness
Gel specular- low ity
high
Camera FOV
40
90"
Sensing area
4cm
16cm²
Table 4: Visualization of parameter bounds in the simulated dataset.
16

---

Published as a conference paper at ICLR 2025
construct large-scale simulated dataset that includes wide range of tactile sensor configurations. Figure Soaure@illun illustrates a sample of tactile images from different simu- lated sensor configurations and contact geometry within the dataset. The samples can be retrieved from our dataset with the sensor IDs and contact IDs provided.
As discussed in Section 4.1
#922
#1295
#1347
#1008
#1950
#3022
#3315
#3748
#4238
#5000
25004
#7248
2622
18420
#9424
Figure 9: Samples from the simulation dataset.
17

---

Published as a conference paper at ICLR 2025
A.3 CLASSIFICATION DATASET SAMPLES
Figure T0 and 11 show the real-world classification dataset that we used to generate the result dis- cussed in Section 5.3 Each row corresponds to different object class, and each column represents a different sensor.
Objects
Gelsight
GelSight
GelSight
GelSight
GelSight
GelSight Wedge
Figure 10: Samples from the classification dataset. (Part 1)
18

---

Published as a conference paper at ICLR 2025
Objects
GelSight Mini 1
GelSight Mini 2
GetSight Mini
GelSight Mini
DIGIT
GetSight Hex
GelSight Wedge
KHP
Figure 11: Samples from the classification dataset. (Part 2)
19

---

Published as a conference paper at ICLR 2025
A.4
POSE ESTIMATION DATASET SAMPLES
Figure 12
shows the world classification dataset that we used to generate the result discussed in
Section 5.4
Each row corresponds to different object class, and each column represents different
sensor.
Objects
GelSight Mini
DIGIT
GelSight Hex
Gelsight Wedge
UC
a
Figure 12: Samples from pose estimation dataset.
Figure 13 shows the modified Ender-3 3D printer. We mount indentors and collect the pose estima- tion dataset for multiple sensors.
Indentor
GelSight Mini
DIGIT
GelSight Hex
GelSight Wedge
Figure 13: Modified Ender-3 Pro 3D printer
20

---

Published as a conference paper at ICLR 2025
A.5 TRANSFERABILITY DETAILS
In this section, we present the full results of the sensor transfer downstream experiments. Details of experiments be found in Section 5.3 and 5.4 Figure 14 and [5 show the classification results and Figure 16 shows pose estimation results.
A.5.1 CLASSIFICATION
100
100
900
GB
Hestad on 84r007
DIGIT
Go 1885
Textted os
-20
-20
GR 18t
54.09
Mirl
GS Minia
G6
GS
GS
(a) ViT-Base Scratch Intra-sensor
ViT-Base Scratched Inter-sensor
09.78
80.88
78.50
84.54
54
44.44
75.22
88:34
-00
50
Taaked 24TTCE
44.56
98.83
33.70
47.78
-00
19.89
19.88
45.16
15.19
-00
26.38
73.75
98 DG
00.00
50.00
50.04
50:47
37.75
GO M444
00.75
71.78
$0.54
500 277
22-22
21.06
44:07
98,81
125 Mex
C5 Minz
CS Minid
CS Mini
DX31
C25 Hic
GS Wedge
Trained th deneo?
Tra red on semsor
(c) VIT- Base Pre-trained Intra sensor
(d) ViT-Bass Pre-trained Inter-senson
09:09
75.15
04.00
79.78
90.00
65 M512112
Teattd sarsor
$1.25
98.78
66.71
Scoteey 20
59.35
Thothed
60.32
-42
-23
-20
CC 1ee
100.00
semsor
(c) ViT-Large Pre-trained Intra-sensor
(f) ViT-I Large Pre-trained Inter-sensor
Figure 14: Transferability on classification tasks. (Part 1)
21

---

Published as a conference paper at ICLR 2025
94.78
10.13
20.01
40.77
90
03 0012112
Lesttd on Betroo
38-41
$0.81
21.91
23,41
00
20
Go 100
28:44
18.72
01.00
811 25
GC 18
$4.38
22.88
09 04
04.55 53
68 Mirl
G5 Minia
G6
Wedlpe
Tral od on sonsor
Trannd on senaac
(a) T3 Intra-sensor
(b) T3 Inter-sensor
02 10:
42.91
Teuted on conter
Accuocy
-20
(c) UniT Intra-sensor
100
M
00.70
75:06
-00
-00
Tostad on MATGO
08.28
90.50
20.01
-00
50
1000 105
GO 3
Tratted
91.25
91.31
00.88
98.88
90.88
00:88
24.08
05
GO 1984
56.78
96.25
90.44
100.00
66.99
88:34
99:38
69 Ni61
69 Mir2
65 Mi1
66 490
69 Miri
DISIT
68 Hox
65 Wesge
Trained seroor
Trannd
SITR Intra-sensor
(c) STTR Inter-sensor
Figure 15: Transferability on classification tasks. (Part 2)
22

---

Published as a conference paper at ICLR 2025
A.5.2
POSE ESTIMATION
G5
0.08
1.47
145
1.98
0.51
1.52
1.51
Tooked an SerTar
0003
1.70
0.00
1:90
1:44
1.05
0.40
1.81
1.25
RMBE DONE
GC M66
Teekked
1.77
1.70
2:57
1-42
1:45
1.87
0.50
1.06
GD Pilgee
1.82
1.30
1:73
2:53
1.78
1:35
1.44
0.49
09 Mir
0/ Mox
US Woddge
C5 Mi
DX31
US Hox
GS Wedge
Trained 0h seneo!"
Traned on sersace
(a) ViT-Basc Scratch Inter- sensor
(b) ViT-Base Pro-trained Inter-sensor
GG MN
1:45
1:54
-20
0.50
1:77
Testad on sutteo
1:51
0.52
1:35
1.02
pmng
RUSE lmn
GS liee
Thitee
-10
1,65
0.58
GO Vengge
.05
(c) ViT-Large Pre-trained Inter-sensor
(d) T3 Inter-sensor
25
00
20
Torked on GonEo
DIGTT
1:09
RMBE(
11.
08 Man
0.70
0.56
-18
0.69
1.01
0.87
0.50
005
C35
DK21
Traned
SITR Inter-sensor
Figure 16: Transferability on pose estimation tasks.
23

---

Published as a conference paper at ICLR 2025
A.6 ADDITIONAL ABLATIONS
This section presents ablation experiments to evaluate the impact of loss terms, alternative supervi- sion signals, and dataset size SITR's performance.
A.6.1 CONTRIBUTION OF LOSS TERMS
We conduct ablation study to evaluate the contributions of the normal map loss and SCL loss to SITR's performance. As shown in Table either loss term independently serves as an effective supervision signal. However, their combination yields the strongest results. This evaluation is ducted on the dataset visualized in Figure further highlighting how these two loss terms synergize improve representation learning.
Method
Classification (%)
Normal loss only
84.21 = 14.01
SCL loss only
78.86 =
Normal + SCL losses
91.43 +: 9.88
Table 5: Ablation study showing the impact of different loss terms on classification accuracy trans- ferability.
A.6.2 CHOICE OF SUPERVISION SIGNAL
There are alternative supervisions our normal map, such using MAE or VQGAN to reconstruct tactile images, as employed T3 and UniT. To evaluate the effectiveness SITR, we adapt these supervisions to train representations using ferability as described in Section [.3 and Section 5.4 SITR consistently outperforms MAE and VQGAN, highlighting the benefits of SITR's architecture and training pipeline.
simulated dataset. We evaluate the models' trans-
Classification (%)
Pose estimation (mm) Inter-sensor set +
Method
Intra-sensor set Inter-sensor set
MAE
45.81 21.44
26.46 19.54
1.13 0.19
VQGAN
59.41 = 19.50
31.02 = 22.01
1.18 + 0.14
SITR (Ours)
90.23 + 8.16
81.94 + 12.92
0.80 + 0.21
Table 6: Comparison of MAE, VQGAN, and SITR performance on intra-sensor and inter-sensor classification tasks (%) and inter-sensor pose estimation (mm)
A.6.3 EFFECT OF SIMULATION DATASET SIZE
We evaluate how the size of the simulation dataset and the variety of sensor configurations impact classification transfer performance on inter-set classification. Table | shows that increasing the number of samples per sensor and the number of sensor variations lead to increases in performance. This demonstrates the benefit diverse and large-scale training dataset.
Sensor
Samples per sensor
Variations
IK
5K
10K
10
45.82 = 21.12
57.00
21.55
61.44 22.81
50
55.86 = 25.04
68.55
11.96
76.78 13.91
100
62.85 = 16.45
73.71 14.27
81.94 4:
Table 7: Transfer classification accuracy (%) on the inter-set dataset across different sensor varia- tions and samples per sensor.
24


Reviews:
[{"review_id": "yxGQSsJpFo", "reviewer": "Reviewer_3T4J", "summary": "This paper aims to learn a tactile representation that can be used across different tactile sensors, including the same sensor design but with different instances, or different sensor designs. To achieve this, it collects a large dataset in simulation and uses contrastive learning to learn the representation. It divides a tactile image into patches and encodes it using a transformer. Meanwhile, it also passes simple calibration images, which contain simple objects pressing on different corners, to this transformer. The feature is further refined by learning to predict the normal map. The learned representation can be fine-tuned on downstream tasks and significantly outperforms baselines and other state-of-the-art methods.", "questions": "What is the performance of UniT or T3 when trained on the simulation dataset gathered in this paper?\n\nWhat is the effect of using the normal map prediction loss?\n\nIs the statement “increasing the number of calibration images does not incur additional inference costs, as calibration tokens are computed only once per sensor” correct? Although it does not introduce additional computational cost for tokenization, the self-attention operator cost is O(n^2), where n is the sequence length. If you increase the number of calibration images, that cost will grow quadratically.\n\nHow is the “No SCL” baseline trained? Is it trained only with the normal prediction loss?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of using simple calibration images is interesting and inspiring. I believe this design should become a standard technique in future tactile representation learning.\n\nThe performance of the learned representation is quite good. On the three downstream tasks they evaluate, it significantly outperforms baselines and other methods in this area.\n\nIt also presents comprehensive ablation experiments on the role of calibration images and contrastive learning losses.\n\nThe paper provides nice visualizations of the learned feature space.", "weaknesses": "I believe there should be another important ablation experiment: training UniT/T3 on the simulated images collected in this paper. This is because there are two major differences between previous state-of-the-art methods (UniT/T3) and the proposed method. The first difference is the specific architectural design, such as contrastive learning, the transformer, and the use of calibration images. The second is the use of simulation data versus real-world data. This paper already presents experiments on architectural design but does not show whether the good performance mainly comes from a large simulation dataset. Specifically, I think the authors could train a representation with the loss proposed in UniT/T3 and compare its performance with the proposed method. This would provide more insight into where the performance improvement comes from.\n\nIt seems the authors do not compare the performance gain achieved by predicting the normal map.\n\nAnother weakness of this work is that it only evaluates sensors with flat gel pads, which have very similar optical designs. In contrast, other work, such as T3, also includes curved sensors. This point should be clearly discussed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lUMhfFreC4", "reviewer": "Reviewer_rhaT", "summary": "To address the problem of current vision-based tactile sensors' lack of transferability, this paper proposes a transformer-based method called SITR. SITR is trained on simulated tactile data, and is generalizable to real-world sensors though calibration. Experiments show the zero-shot performance on three downstream tasks: shape reconstruction, object classification and contact localization.", "questions": "1. Referring to my first concern in Weaknesses section, it would be interesting to test the model's transferability to real-world datasets with more complicated settings. For example, Touch and Go [1] proposes three material understanding tasks that evaluate the model's understanding of material categories/ smoothness/ hardness given GelSight images. It would be interesting to see if the features learnt from pure simulation data can be transferred to these tasks.\n\n2. Referring to my second concern, I'm curious about the computation overhead caused by using 18 calibration images. Also I wonder if a more careful design can be used in encoding the calibration images to reduce redundant information. As is shown in Fig. 3, only one out of the nine grids contains useful information. One simple potential improvement would be cropping the useful regions out from the 9 calibration images and concatenating them into a single image.\n\nReferences\n[1] Yang, Fengyu, Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, and Andrew Owens. \"Touch and go: Learning from human-collected vision and touch.\" arXiv preprint arXiv:2211.12498 (2022).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Sensor variance, which is the problem this paper tries to solve, is very important in the field of vision-based tactile sensing. Recently, different methods [1, 2, 3] have been proposed to address this problem, and this work proposes a model that is complementary to the prior works.\n\n2. The framework proposed by this paper does not require any sensor-specific design in the model architecture, which seems to be its major contribution. Instead of designing different encoders or tokens for specific sensors, SITR only relies on calibration images to distinguish between different sensors. This can both reduce the model size and enable transferability to entirely novel sensors (as long as calibration images can be obtained).\n\n3. The zero-shot results on downstream tasks show that the model transfers well across various sensor settings, which is a significant improvement upon the prior works.\n\nReferences\n[1] Zhao, Jialiang, Yuxiang Ma, Lirui Wang, and Edward H. Adelson. \"Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks.\" arXiv preprint arXiv:2406.13640 (2024).\n[2] Rodriguez, Samanta, Yiming Dou, Miquel Oller, Andrew Owens, and Nima Fazeli. \"Touch2touch: Cross-modal tactile generation for object manipulation.\" arXiv preprint arXiv:2409.08269 (2024).\n[3] Higuera, Carolina, Akash Sharma, Chaithanya Krishna Bodduluri, Taosha Fan, Patrick Lancaster, Mrinal Kalakrishnan, Michael Kaess et al. \"Sparsh: Self-supervised touch representations for vision-based tactile sensing.\" In 8th Annual Conference on Robot Learning.", "weaknesses": "1. The dataset the model is trained on all comes from simulation. The tactile simulator uses 3D meshes of the object to synthesize RGB tactile images, which assumes that all objects are rigid and without micro-geometry on the object surface (e.g., wood grain). Although this might be sufficient for geometry-based tasks such as shape reconstruction and object classification, the features extracted from this model may lack the ability to classify the fine-grained details (e.g., smoothness, hardness) in real-world setting. See the Questions section for more suggestions.\n\n2. Despite being a sensor-agnostic model architecture, the calibration images are necessary for the SITR model. It's also shown in the ablation study that the model performance will drastically drop when calibration images are removed, and more calibration images usually lead to better performance. This leads to two potential concerns: (i) the calibration of new sensors requires specific objects (ball and cude) that might not be easily available to some of the users, which limits the application of the proposed model; (ii) the calibration images can lead to significant computation overhead, and most of the regions in the calibration images are redundant, i.e., not contacted by the object (shown in Fig. 3). See the Questions section for more suggestions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i7jY8XZNLS", "reviewer": "Reviewer_x9aX", "summary": "This study introduces a novel approach for extracting Sensor-Invariant Tactile Representations (SITR) aimed at enhancing the transferability of models across different vision-based tactile sensors. The authors employ a transformer-based architecture combined with supervised contrastive learning to create a robust tactile representation that allows for zero-shot transfer across various optical tactile sensors, specifically within the GelSight series. Experimental results validate the approach’s effectiveness across multiple tactile sensing applications, indicating potential advancements in model generalization and transferability for similar sensor designs.", "questions": "1. Calibration Object Choice: Could you elaborate on the theoretical and practical reasoning for selecting a ball and cube corner as calibration objects? How do these specific shapes contribute to achieving sensor invariance, and would the method remain effective with other shapes?\n2. Generalizability to Non-Vision-Based Sensors: Has the SITR approach been tested or considered for use with non-vision-based tactile sensors (e.g., capacitive or resistive)? If not, what challenges or limitations do you foresee in generalizing this approach to other sensor types?\n3. Temperature Parameter in Contrastive Learning: Could you provide insights into how the temperature parameter was tuned in contrastive learning? This parameter plays a crucial role in the learning process, and understanding its tuning would aid in assessing the approach’s sensitivity and reproducibility. Additionally, given that GelSight sensors tend to generate considerable heat during use, with noticeable temperature changes within a minute, how might such temperature variations impact the performance or stability of your model?\n4. Feasibility for Real-Time Applications: Given the computational intensity of transformer-based architectures, have you assessed the feasibility of this approach in real-time applications? Would it be possible to optimize the model for deployment on resource-constrained platforms?\n5. Given that GelSight sensors are tested independently in your experiments rather than being integrated on a robotic platform (e.g., mounted on both sides of a gripper), could you discuss any potential modifications or limitations of your method for practical deployment in space-constrained environments, such as on robotic end-effectors?\nThank you for your thorough and thoughtful work in this area. The study is innovative and contributes valuable insights to the tactile sensing field. We look forward to your clarifications and insights, which I believe will further enrich the impact of this research and its relevance to the ICLR community！", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Originality and Innovation: The integration of a transformer-based architecture with supervised contrastive learning to achieve sensor-invariant tactile representation is a novel approach within the tactile sensing field. The method is original and demonstrates the potential for addressing the long-standing issue of sensor-specific dependency in tactile applications.\n2. Technical Rigor and Methodological Soundness: The paper presents a technically robust framework, comprising a two-stage process of normal map reconstruction and contrastive learning. The experiments cover a range of scenarios, lending empirical support to the effectiveness of the proposed method. The methodology is well-designed to achieve sensor invariance, and the use of transformers in tactile representation learning is a valuable contribution.\n3. Potential for Real-World Impact: Achieving zero-shot transfer with minimal calibration across sensors is valuable in robotics and other domains that involve tactile sensing. By minimizing the need for sensor-specific calibration, the proposed method has potential implications for resource efficiency and ease of deployment in tactile applications.", "weaknesses": "1. Limited Generalizability: The approach is specifically tailored to GelSight sensors, limiting its generalizability to other types of tactile sensors. GelSight sensors are vision-based, and the proposed method may not be easily transferable to non-vision-based tactile sensors, such as capacitive or resistive types. This restriction diminishes the broader applicability of the findings, potentially limiting its impact within the ICLR community.\n2. Practical Constraints of GelSight Sensors: GelSight sensors, due to their relatively large size and offline nature, present challenges for real-time or embedded applications, especially in constrained environments such as robotic end-effectors. The physical limitations of GelSight sensors may limit the practical deployment of the method in real-world applications, where compactness and real-time processing are often required.\n3. Insufficient Justification for Calibration Object Selection: The study uses a ball and cube corner for calibration, yet lacks a theoretical explanation for why these shapes are specifically suited for achieving sensor invariance. A more detailed discussion of the rationale and potential generalization of these calibration objects would improve the clarity of the methodology and support its robustness across varied sensor designs.\n4. Parameter Tuning Transparency: The paper does not provide adequate information on the parameter tuning process, particularly for the temperature parameter in the contrastive learning phase. This parameter significantly influences the performance of contrastive learning, and a more transparent tuning discussion would enhance reproducibility and provide insight into the method’s robustness.\n5. Computational Complexity and Real-World Feasibility: Transformer models, while powerful, are computationally intensive and may not be optimal for real-time applications. The authors do not discuss computational demands or potential limitations in deploying the model on resource-constrained platforms. An assessment of computational requirements would provide a more comprehensive view of the method’s practicality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "geMQXjEvP0", "reviewer": "Reviewer_cH4B", "summary": "This work proposes a sensor-invariant tactile representation method that learns cross-sensor representations, enabling zero-shot generalization to new tactile sensors. To achieve this, the authors collect a large set of simulated tactile images paired with human-annotated images, where sensor contacts with the same objects in same pose are matched for positive pairs. Using a small amount of tactile data for downstream tasks, they fine-tune a task-specific head while keeping the pre-trained encoder frozen. They also demonstrate zero-shot generalization across unseen tactile sensors. The key contributions include: (i) an open-source large-scale dataset, (ii) zero-shot generalization to novel sensors, enabled by the model design, and (iii) a well-written, accessible paper.", "questions": "See weakness section. If my concerns are decently addressed, I am happy to increase the rating.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The strengths of this paper are:\n\nOriginality: The problem is novel and relevant to practical applications of tactile-based learning.\n\nQuality: he proposed method is well-founded, built upon established principles in the field, and presents a coherent theoretical framework. The experimental results, while only partially demonstrating the method's effectiveness, provide a solid initial validation of the approach. These findings indicate the potential for further development and optimization, suggesting that with additional refinement, the method could yield even more impactful results. However, a more comprehensive evaluation, including a broader range of experiments, would strengthen the evidence for the method’s effectiveness and applicability.\n\nClarify: The writing is clear and easy to follow.\n\nSignificance: Achieving zero-shot generalization to new tactile sensors is a valuable contribution to the field, addressing a critical challenge in tactile-based learning and manipulation. This capability has the potential to greatly expand the applicability of tactile sensing technologies across diverse robotic systems and tasks. By demonstrating that a model can effectively generalize to previously unseen sensors, the authors open up new avenues for research and development, ultimately advancing the state of the art in robotics and sensory integration. However, the tasks considered in this work is a bit limited.", "weaknesses": "There are several concerns:\n\nMajor:\n\n1). The authors highlight the high cost of data collection, but recent studies (e.g., [1], [2]) have developed low-cost tools for gathering data across various manipulation tasks. I would like to see the authors' perspective on whether cross-sensor representation learning (SITR) remains necessary if data collection for each new sensor becomes straightforward. In other words, if sensor-specific data for any downstream task can be obtained easily, might single-policy approaches be more effective than a cross-sensor approach for certain tasks? Including a comparison in experiments could clarify this point, such as assessing whether the proposed method enhances sample efficiency in downstream task policy learning. This would help determine if easy access to sensor-specific data could potentially weaken the contribution of the cross-sensor approach in this work.\n\n[1]. Chi, Cheng, et al. \"Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots.\" arXiv preprint arXiv:2402.10329 (2024).\n\n[2]. Yu, Kelin, et al. \"MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation.\" 8th Annual Conference on Robot Learning.\n\n2). For the SCL loss, collecting labeled data in simulation requires additional effort, as human experts must ensure correspondence in data collection (i.e., matching contact poses on the same object). Could advanced unsupervised methods, such as the approach in [3], be used instead to avoid direct label generation? My concern is that while such controlled data collection is feasible in simulation—where we can enforce contact poses and objects—it would be challenging in the real world, as replicating data collection across different tactile sensors is cumbersome.\n\n[3]. Xu, Mengda, et al. \"Xskill: Cross embodiment skill discovery.\" Conference on Robot Learning. PMLR, 2023.\n\n3). The zero-shot experiments focus on estimation results, but how does the method perform in manipulation tasks that require control inference? Such tasks could be challenging, as the pre-training phase is limited to static simulated tactile images, lacking the temporal reasoning essential for dynamic manipulation tasks. Additionally, the authors generate 100 simulated sensor configurations, which may introduce bias from the rendered images during simulation pre-training. This setup could potentially increase the sim-to-real gap, particularly for complex manipulation tasks.\n\n4). The dataset is extensive, with 1 million samples. It would be helpful to know if such a large dataset is necessary for achieving the reported results. An ablation study on dataset size could provide valuable insights into how dataset scale impacts performance.\n\n5). The baseline evaluation may be somewhat unfair. It appears the authors used existing models for T3 and UniT, but these models may not have been trained on a dataset as large as the one used for the proposed method. If so, this could lead to a performance drop for the baseline models. Ensuring comparable training conditions would provide a more accurate assessment of model performance.\n\n6). The calibration process partially reflects sensor characteristics. However, even without calibration images, the model’s performance remains significantly higher than T3, as shown in Table 1 and Figure 8. This raises concerns that the large-scale dataset and loss design may contribute more to performance than the calibration process or the model architecture itself. To better understand the actual impact of calibration and the proposed model architecture, it would be essential to train T3 models using the same dataset and loss.\n\n7). In Figure 9, the performance without SCL loss is not substantially lower, even for classification tasks, though Figure 7’s t-SNE results show a visibly worse latent space without SCL. Could the authors clarify this discrepancy? Given that the SCL loss is central to this work, it’s surprising that in pose estimation tasks, performance without SCL is almost identical. This raises the question of whether SCL is as critical as suggested, since the reconstruction loss alone—commonly used in other works—seems sufficient to achieve cross-sensor learning if the model can reliably reproduce contact images. If so, this would suggest that the simulation data collection pipeline and use of contact IDs may not be as effective as proposed.\n\nMinor:\n\n8). Figure 2 is incorrect; it’s not the object ID that should be close in the latent space, but rather the contact ID, which encompasses both the object ID and a similar contact pose.\n\n9). Is the learned decoder for reconstruction still utilized in downstream tasks? Since there is a fine-tuned task-specific decoder for each task, I assume the answer is no. It seems that the decoder used during pre-training is solely for the reconstruction loss to facilitate encoder training. Please clarify this in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)