AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors
Authors: Ruoxuan Feng
Jiangyu Hu
Wenke Xia
TianciGao
Ao Shen
Yuhao Sun
Bin Fang
Di Hu
Venue: ICLR 2025 Poster (2025)
Keywords: Tactile Representation Learning
Visuo-tactile Sensors
Cross-sensor Transferring
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Visuo-tactile sensors aim to emulate human tactile perception, enabling robots to precisely understand and manipulate objects. Over time, numerous meticulously designed visuo-tactile sensors have been integrated into robotic systems, aiding in completing various tasks. However, the distinct data characteristics of these low-standardized visuo-tactile sensors hinder the establishment of a powerful tactile perception system. We consider that the key to addressing this issue lies in learning unified multi-sensor representations, thereby integrating the sensors and promoting  tactile knowledge transfer between them. To achieve unified representation of this nature, we introduce TacQuad, an aligned multi-modal multi-sensor tactile dataset from four different visuo-tactile sensors, which enables the explicit integration of various sensors. Recognizing that humans perceive the physical environment by acquiring diverse tactile information such as texture and pressure changes, we further propose to learn unified multi-sensor representations from both static and dynamic perspectives. By integrating tactile images and videos, we present AnyTouch, a unified static-dynamic multi-sensor representation learning framework with a multi-level structure, aimed at both enhancing comprehensive perceptual abilities and enabling effective cross-sensor transfer. This multi-level architecture captures pixel-level details from tactile data via masked modeling and enhances perception and transferability by learning semantic-level sensor-agnostic features through multi-modal alignment and cross-sensor matching. We provide a comprehensive analysis of multi-sensor transferability, and validate our method on various offline datasets and in the real-world pouring task. Experimental results show that our method outperforms existing methods, exhibits outstanding static and dynamic perception capabilities across various sensors. The code, TacQuad dataset and AnyTouch model are fully available at gewu-lab.github.io/AnyTouch/.

Published as a conference paper at ICLR 2025
ANYTOUCH: LEARNING UNIFIED STATIC-DYNAMIC REPRESENTATION ACROSS MULTIPLE VISUO-TACTILE SENSORS
Ruoxuan Feng*
Jiangyu Hu
Wenke Xia
Tianci Gaot
Ao Shen
Yuhao Sun
Bin Fang
Di Hu
Renmin University of China
Wuhan University of Science and Technology BBijjig University of Posts and Telecommunications
ABSTRACT
Visuo-tactile sensors aim to emulate human tactile perception, enabling robots to precisely understand and manipulate objects. Over time, numerous meticulously designed visuo-tactile sensors have been integrated into robotic systems, aiding in completing various tasks. However, the distinet data characteristics of these low- www-stadardiized visuo-tactile sensors hinder the establishment of a powerful tac- tile perception system. We consider that the key addressing this issue lies learning unified multi-sensor representations, thereby integrating the sensors and promoting tactile knowledge transfer between them. To achieve unified represen- tation of this nature, introduce TacQuad. aligned multi-men modal mulli-sensor tactile dataset from four different visuo-tactile sensors, which enables the explicit integration of various sensors. Recognizing that humans perceive the physical en- vironment by acquiring diverse tactile information such as texture and pressure changes, further propose to learn unified multi-sensor representations from both static and dynamic perspectives, By integrating tactile images and videos, we present AnyTouchl a unified static-dynamic multi-sensor representation learning framework with multi-level structure, aimed at both enhancing comprehensive perceptual abilities and enabling effective cross-sens sensor transfer. This multi-leve architecture captures pixel-level details from tactile data via masked modeling and enhances perception and transferability by learning semantic-level sensor- aagoosc features through multi-modal alignment and cross-sensor matching. We provide comprehensive analysis of multi-sensor transferability, and validate our method on various offline datasets and in the real- wolldwrlld pouring task. Experimental results show that our method outperforms existing methods, exhibits outstanding static and dynamic perception capabilities across various sensors. The code, TacQuad dataset and AnyTouch model are fully available at gggee-llab, catpuullab.gtthu..... Any Touch/
INTRODUCTION
Tactile perception is an important sense through which humans perceive the physical world. For many years, researchers have been working to endow robots with human-like tactile perception abilities through diverse tactile sensors (Liu et al.) 2022 Maiolino et al. 2013 Yuan et al.) 2017). Among them, with high resolution comparable human touch, various types of visuo-tactile sen- sors have garnered widespread attention (Yuan et al., 2017]] Donlon et al. 2018 Lambeta et al.] 2020). Many studies have attempted to robots equipped with visuo-tactile sensors perform manipulation tasks such as grasping (Xu et al. 2024) and inserting (Li et al. 2014).
However, due the low standardization of visuo-tactile sensors, different sensors may exhibit dis- crepancies in perceiving the same tactile information. This variability poses challenges to building precise robotic tactile systems, as sensor-specific data collection (Yang et al.) 20222] Gao et al. 2023) and model training limit the data scale and diversity for the model of single sensor and lead to suboptimal perception capabilities. To address this issue, some initial efforts have explored us-
"Equal corresponding authors.

---

Published as a conference paper at ICLR 2025
ing multi-sensor data collaboratively to enhance cross-sensor knowledge transferability YYngg al., 2024 Zhao et al. 2024). Nevertheless, the lack of aligned multi-sensor data has hindered these at- tempts from effectively integrating disparate sensors and constructing unified representation space. Considering this data issue, Rodriguez et al. (024) collected a dual-sensor paired dataset to enable cross-sensor generation. However, their focus specific manipulation tasks limited the variety of sensors and collected objects. Moreover, they overlooked the potential benefits of paired multi- modal data for enhancing sensor transferability and achieving comprehensive tactile perception.
To enhance support for multi-sensor integration, we collect TacQuad, an aligned multi-modal multi- sensor tactile dataset containing 72,606 contact frames, using four different visuo-tactile sensors. We select these representative sensors from publicly available sensors, self-made sensors, and force field sensors to ensure diversity. To balance the trade-off between the cost of data collection and the accuracy of pairing, we collect fine-grained spatio-temporal aligned data on calibration platform, while larger-scale coarse-grand grase--raineed spatial aligned data is acquired through the handheld collection, as shown in Figure] I Additionally, we capture the objects being touched using camera and annotate tactile attribute descriptions for each collection, forming a comprehensive touch-vision-language dataset. This dataset utilizes paired multi-modal data bridge to mitigate the impact of sensor variability on the understanding of tactile semantic features, and enables the explicit integration of sensors into unified multi-sensor space for effective knowledge transfer between sensors.
Building on this solid foundation, we further revisit the challenge of learning unified multi-sensor representations: How
obtain unified multi-sensor representations adaptable to a wide array of tasks? We recognize that the human tactile perception is a combination of static and dynamic processes, as humans derive comprehensive tactile perception from multiple types of information such as texture, sliding, and pressure changes. Drawing on this insight, we propose learning unified representations from both static and dynamic perspectives to accommodate a range of tasks.
To obtain multi-sensor representations of this nature, we introduce AnyTouuhh a unified static- dynamic multi-sensor tactile representation learning framework. This framework integrates the input forms of tactile images and videos, collectively utilizing them to reinforce the model's abilities to perceive both static properties and dynamic changes. Moreover, we design a multi-level architecture to comprehensively strengthen the model's capabilities for capturing pixel-level tactile details and semantic-level sensor-agnostic features. Specifically, we utilize masked modeling (He et al. 2022 Tong al. 2022) maximize the use of multi-sensor data for learning fine- ane-grained, pixel-level details. Subsequently, we conduct modtl aligning and a novel cross-sensor matching task to understand semantic-level tactile properties of objects across different sensors and extract sensor- agnostic features. We aim for the multi-sensor representations to share a common space and cluster by the tactile information of the object they represent, thereby reducing the gap between sensors. To further promote knowledge transfer across multiple sensors, we propose randomly replacing the sensor-specific tokens (Yang et al. 2024) with universal sensor tokens during training. This strategy ensures the model maintains its ability to process and perceive tactile data across various sensors, while also providing knowledge from all seen sensors for generalization to unseen sensors.
We conduct both quantitative and qualitative experiments to analyze the transferability of multi- sensor data and assess the impact of our framework on the multi-sensor representation space. Build- ing on this, we comprehensively evaluate the static and dynamic tactile perception capabilities of AnyTouch across various tactile datasets and through a real-world experiment: fine-grai teeeggraied pouring. The experimental results demonstrate the static and dynamic perception abilities and cross-sensor transferability of AnyTouch We hope the approach of learning unified multi-sensor representa- tions from both static and dynamic perspectives will establish standardized learning paradigm for visuo-tactile perception and further inspire research in multi- sensor representation learning.
2 RELATED WORK
Multi-Source Learning. Learning from multi-source data with greater data scale and diversity is expected to enhance the model's performance and generalization ability, but faces challenges inte- grating the data representation spaces. Researchers have found that multi-source models often strug- gle to capture a unified representation suffering from the discrepancies between data sources (Glorot et al. 2011 1 Zhao et al. 2019). Contrastive learning (Wang al. 2022b Zhao et al. 2023) is pro- posed to learn language- language-agnostic agnostic representations for soutie language training. while cycle consistency loss (Zhu al. 2017 Kim et al. 2022) aligns the target domain distributions in multi-
2

---

Published as a conference paper at ICLR 2025
TacQuad: Aligned Multi-Modal Multi-Senssor
Tactile Dataset
Four Visuo-tactile Sensors
Data Collection
DICIT (Palie)
(Palltc)
Dell-snnde
(Farec Fiddi
Flinegraiind Spatio-tmperral Aligned
Touches
17,624 Frames
The batilla waser louchas the top center of glas par, which las hard, smooth, and tramparent surface with saries of tuxture.
Coarse-grained Spatial Aligned
Uader (uabae)) Objerts Taurhe Pramess
scaser fooches top center reugh orange. contact point reletively firm roughness slightly clastic
Figure 1: TacQuad: an aligned multi-modal multi-sensor tactile dataset from four visuo-tactile sensors. We select GelSight Mini (Inc.) and DIGIT (Lambeta et al. (020)) from publicly available sensors, DuraGel (Zhang et al., 2024) from self-made sensors, and Tac3D (Zhang et al., 2022) from force field sensors for diversity. There is noticeable gap between the data from these sensors. We use the four sensors to touch the same position on the same object to obtain aligned data. To maximize aligned data collection, we use two methods to gather subsets with different alignment accuracy. We collect fine-grained spatio-temporal aligned data on a calibration platform, while larger-scale maase-grained spatial aligned data acquired through handheld collection.
source image generation. Similarly, to integrate multi-source tactile data from different sensors, techniques such as multi-sensor joint training (Zhao et al. 2024 Higuera et al. 2024| Gupta al. multi-modal alignment (Yang et al., 2024). and cross-sensor generation (Rodriguez et al, 2024 have emerged. However, these methods overlook the benefits of jointly utilizing multi-modal data and aligned multi-sensor data to bridge the sensor gap. In this work, we collect an aligned multi-modal multi-sensor dataset and propose learning unified multi-sensor representations.
2025).
Visuo-tactile Perception. Visuo-tactile sensors have garnered widespread attention due to their high resolution Yuaan et al. 2017 Donlon et al.) 2018 Lambeta al.) 2020 Zhang et al., 2024 2025). Nowadays, many works utilize visuo-tactile sensors to capture contact deformations. enabling the completion of dexterous manipulation such as dense packing (Li et al., 2022), grasping (Xu et al.) 2024.. and insertion (Li et al. 2014). Visuo-tactile sensors can also collaborate with other sensors to assist the robot in performing more complex manipulation tasks. For instance, |Feng et al.) (2024) dynamically integrates visuo-tactile signals with visual and audio data to collaboratively accomplish fine-grained manipulation tasks, such as pouring and peg insertion with keyway. In addition to these tasks requiring dynamic tactile perception, visuo-tactile sensors are also used in static tasks such as material classification (Yang et al.) (2022) and shape reconstruction (Gao et al.] (022b). However, due to the low standardization of visual-tactile sensors, these methods fail to leverage larger and more diverse data from other sensors and lack sensor transferability. In this work, we propose learning a unified multi-sensor representation from both static and dynamic perspectives.
Representation Learning, Representation learning has achieved remarkable success in improving model generalization in various fields. Techniques like BERT (Devlin) 2018) and masked autoen- coder (MAE) (He et al. 2022) have enhanced the model's performance across various downstream applications. With the rise of multi-modal learning, representation learning has expanded its impact across fields. Vision-languee
pre-training (Radford et al. (021) has seen tremendous success, and more modalities, including audio (Guzhov et al.) (2022), touch YYng et al.] 2024), and 3D point clouds (Xue et al.) 2023), are being integrated. Among them, tactile information from visuo-tactile sensors can be expressed as images, allowing vision-related techniques to make strides in touch. Applying MAE (Cao et al. 2023 Higuera et al. 2024) or multi-modal aligning (Yang et al. 2024 Cheng et al. 2024) has enhanced tactile model capabilities. However, these efforts have not ex- plored how to obtain unified visuo-tace cactole representation suitable for various tasks. Our research addresses this challenge from both static and dynamic perspectives, enhancing cross senssr transfer- ability across various tasks through semantic-level multi-modal aligning and cross-sensor matching.
3

---

Published as a conference paper at ICLR 2025
3 ALIGNED MULTI-MODAL MULTI-SENSOR TACTILE DATASET
The low standardization of visuo-tactile sensors and the gap between multi-sensor data have resulted in insufficient data for individual sensors and the poor cross-sensor transferability of tactile models. Roddiguez et al. (2024) has made an initial attempt to address it by collecting a dataset with 32,256 pairs of tactile images from two sensors with a limited variety of objects for specific manipula- tion tasks. It does not consider tactile properties such as material and hardness, and overlooks the potential to enhance sensor transfer capabilities through multi-modal information.
In this work, we present a more comprehensive solution to this problem by providing mensi- aligned data with text and images, explicitly enabling the model to learn semantic-level tactile at- tributes and sensor-agnostic features to form a unified multi-sensor representation space through data-driven approaches. We collect TacQuad, an aligned multi-modal multi-sensor tactile dataset with a greater variety of objects, larger data volume, and more types of sensors. To ensure sensor diversity, we select GelSight Mini (Inc.) and DIGIT (Lambeta et al. 2020 from publicly available sensors, DuraGel (Zhang et al. (024) from self-made sensors, and D Zhang et al.) 2022) from force field sensors for data collection. The first three sensors are used to collect tactile images, while the Tac3D is used to capture deformation force fields. However, considering collecting fin-graiied aligned tactile data is very costly, to collect data on a larger scale while ensuring as much data pairing as possible, we use both coarse and fine methods to collect aligned data:
Fine-grained spatio-temporal aligned data. We fix the four sensors side by side in a rectangular container and connect them to the movable end of calibration platform. We use a program to control the four sensors to press the same position on the same object in sequence, sharing the same speed and depth. Consequently, we obtain a set of continuous spatio-temporally aligned contact frames. Due to the high requirement of precision, the process is consumin,, thus we try our best to collect 30 sets of aligned data across 25 objects. This portion of the data contains a total of 17,524 contact frames, which can be used for fine-grained tasks such as cross-sensor generation. See more details for the fine-grained data collection in Sec. Af5 the Appendix.
Coarse-grained spatial aligned data. We collect data in a handheld manner by sequentially press- ing the same location on the same object with four sensors. While pressing, we introduce some twisting motions to the handheld sensors to better simulate the authentic dynamic touch experience of humans. This method allows us obtain a larger amount of aligned data in a short time. Using this approach, we collect 151 sets of aligned data from 99 objects, including both indoor and outdoor scenes. This portion of the data contains a total of 55,082 contact frames.
Each tactile frame in the dataset has a paired visual image and tactile attribute descriptions that generated using GPT-4o and manually corrected. We aim to bridge the gap between sensors and achieve more comprehensive tactile perception by aligning with the multi-modal data. As a result, we obtain an aligned multi-sensor multi-modal tactile dataset, as shown in Figure
4 METHOD
In this section, we introduce Any LnyTo, a unified multi- sensor tactile representation learning frame- work from the perspectives of both static and dynamic perception, as shown in Figure 2 Concretely, integrate the input format of tactile images and videos (Sec. 4.1 and foeus learning both fine- grained pixel-level details for refined tasks (Sec. (4.2) and semantic-level sensor-agnostic features for understanding properties (Sec. 4.3) and building unified space (Sec. 4.4) by a multi-level structure. We also propose universal sensor tokens for better knowledge transfer.
4.1 UNIFIED INPUT FORMAT FOR STATIC AND DYNAMIC TACTILE PERCEPTION
In daily life. human tactile perception includes both static and dynamic processes. A brief touch allows quick recognition of properties like material and texture, while tasks such as unlocking a lock require continuous dynamic perception. These two types of perception complement each other, enabling us to comprehensively understand the physical surroundings and engage in a variety of interactions. This inspires to learn unified multi-sensor representation from the perspective of combining static and dynamic perception, using tactile images and videos respectively.
Given static tactile image I E R x Xx33 and dynamic tactile video clip V FF*xxxX consider tactile images as single-frame frame static videos to unify tactile images and videos. Concretely, replicate 7 along the time axis for F times, and unified 4-D tensor Xy E RR Fxxxx
4

---

Published as a conference paper at ICLR 2025
Stage 1: Levell
Universal Sensor Taken
Masked Medeling: Food -Lve- Detail
Frandday
Toach
Track
P
Traske
Textlle
Textle Videe
Romatioation
Near Trems
Pecistion
(Liar iacrad)
Stage 2: Semantic-Level
Baplac: (Pribniilty PP)
Multi-Modi Medal Aligning: Semontic-Leal Properties
Vidien Encader
Touch
Senvor Takee
Seaser
(Skrod]
Rnooder
losaiii aiiiig wilh hard
Tasile liage
Taiiil Vileo
Conrastie Leurain
Legends
Matching:
Sencor Agasticc Festures
on !
Teuch
Task
Unterall Seave Tikes
Escader
Braunic
the wite pastic dqject?
Tastie laugg Tortle Videe
Semglnn
Masket Taken D Clan Side
Figure 2: Overview AnyTouch. Our framework integrates static tactile images and dynamic tac- tile videas, aiming to learn unified multi-sensor representation suitable for various tasks. Through multi-level architecture, we employ masked modeling to learn pixel-Ievel tactile details, and use multi-modal aligning and cross-sensor matching to understand semantic-level sensor-agnostic tac- tile properties. We also use universal sensor tokens to integrate and transfer sensor information.
to represent both I and V Girdhar al. 2022 2023), where F is the number of frames and H, W denote the shape of images. We then process XT E ER Fx x3 into spatio-temporal tokens E Nxd through shared patch projection layer, where N is the length of tokens and d represents the feature dimension. By unifying the processing of images and videos in this manner, our approach integrates tactile images and video input, enhancing the model's ability comprehend both static and dynamic information, and endows the model with the potential to accomplish various tasks.
4.2 MASKED MODELING: LEARNING PIXEL-LEVEL DETAILS
Visuo-tactile images are fine-grained data with pixel-level details of subtle tactile deformations and continuous changes during dynamic processes, especially for refined perception tasks. To enhance the fine-gre mne-grained perception capabilities the tactile representation model, we employ the masked autoencoder technique (He et al. 2022: Tong et al. 2022). compelling the model to capture pixel- level details across multiple sensors. Concretely, we randomly mask the tokens of both tactile images and videos with a masking ratio p. and build decoder obtain the reconstructed static images f and dynamic videos V. The corresponding loss function L and C are mean squared error (MSE) loss between the original masked tokens and reconstructed ones in the pixel space:
F ΣΣ 3 PER
Σ pEtt
(p(p) - I(p),, cP
-y)))
(1)
1aM
FlSM
where p the token index, Ωa is the of masked tokens and Vy is the f-th frame in the video V. We use masked modeling to learn fine Aie-ggained tactile deformation features at the pixel level, as well as the temporal dynamics tactile changes.
To further enhance the model's understanding of continuous deformation changes, we introduce an additional task of predicting the next frame Vr+1 while reconstructing the dynamic video V. The loss function CD Cp is MSE loss between the original frame VF+1 and the predicted frame V
Cpred
1 N
Σ -V++())-V++((>)3
(2)
4.3 MULTI-MODAL ALIGNING: UNDERSTANDING SEMANTIC-LEVED PROPERTIES
After obtaining tactile representations with fine-grained perceptual details via masked modeling. we aim to further understand semantic-level tactile properties and use paired multi-modal data as a bridge to narrow the gap between sensors. Therefore, we propose using multi multi-modal aligning, which binds data from various sensors with paired modalities for more comprehensive semantic- level perception and reduce perceptual differences between sensors. However, differences in data collection scenarios across various datasets (e.g., simulation vs. reality) make simple vision-tactile
5

---

Published as a conference paper at ICLR 2025
Touch-Vision
Touch-Vision-Laame Laingu- Contrastive Learning
Contrastive Learning
UniTonch
Ours
Visioa
INagle
(Mutii
(Multi-attzs))
OMalli
oegin?
Text
Text
Figure 3: Comparison with existing multi-modal aligning methods. Our method not only uses multi-modal data to bridge the gap between sensors, but also explicitly clusters representations of the same position on the same object from different sensors together.
alignment less effective in bridging sensor gaps. Therefore, we seleet the text modality, which consistently describes tactile attributes across datasets, an anchor to align touch, vision, and text modalities. Since tri-modal tactile datasets are rare, with most containing only vision-touch pairs, we explore two strategies: automatically expanding the amount of text modality pairings and designing aligning methods that are compatible with missing modalities. We first select representative datasets for each sensor and then use GPT-4o generate expand the text modality within these datasets. Through this method, we create new text pairs for 1.4 million samples across the four datasets.
touch-vision language contrastive learning method to leverage the paired data between touch and other modalities for aligmment. We maximize the use paired data by selecting the largest subset for each modality combination within the batch for multi-modal aligning. Considering a pair of uni-modal represen- tations (IT, TV, .r).) derived from uni-modal encoders, where øT E Rdd is the touch representation, IV E Rd U Ø is the vision representation and Tt E E U Ø is the text representation. We then perform multi-modal alignment (Radford et al. 2021) within the batch as (taking vision-language aligning with missing modalities example):
Based these extensive tactile datasets,
develop
moddalty-miising-avve
exp(xvs ILS/T)
CV-L = 1y 7
log
Ejen.nnn exp(av, ZLL./T)²
iEftyn
where B is the batchsize, Ωl, Un are sets of indices for the samples containing vision and text, and J the scalar temperature. The computation of similar -r-v and CT L is shown in Sec. A.. of the Appendix. This approach maximizes the use of paired data with missing modalities by aligning the sample intersections between modalities. The computation of Lv-T, LL-TT and LL-V similar but in the opposite direction. We then obtain the joint aligning loss as:
Lalign
CTV
2 (LT-V + Lv
CTL
(LT + LL-T) + QVL (Lv + fL-V),
(4)
2
2
where aTV, ATL and OVL are hyper-parameter to control the alignment strength.
4.4 CROSS-SENSOR MATCHING: EXTRACTING SENSOR-AGNOSTIC FEATURES
To fully utilize multi-sensod aligned data and build unified space by clustering multi-sensor tactile representations of the same object, introduce novel cross-sensor matching task. In this task, the model needs to determine whether two tactile images or videos are collected from the same position on the same object. We aim to cluster representations of the same tactile information from different sensors while performing multi-modal aligning, thereby enhancing the learning of sensor-agnostic features and forming a unified multi-sensoo representation space, as shown Figure 3
We treat data collected from the same object and position by two different sensors as positive pair, and data from different objects or positions as negative pair. The model is trained to distinguish between positive and negative pairs. For each image and video sample Xy in our TacQuad, we randomly select one sample from the same object the same location captured by another sensor as the positive sample X and choose another sample from any dataset of any other object or location as negative sample X-. We element-wisely multiply the touch representation JT with at and 27. and then input each result into an MLP to compute the matching scores m and y.
m M LL 2F),
-MPP(TT IT))
(5)
where IT, Di and wn are the representations of XT. X and X-. The loss function Lmaach is Binary Cross Entropy Loss similar to Lin et al. (2020):
Ematthh -(y* log(m*) + (1 - 4² log((1 )) - (u log(m ) + (1 - y log(1 mmm) -m))). (6)

---

Published as a conference paper at ICLR 2025
This task requires the model to distinguish features with the same semantics from different sensors, thus explicitly clustering representations with the same object information form unified multi- sensor representation space. As shown in Figure E] Any AnyTouch, incorporating this task, differs from existing multi-modal aligning efforts. The construction of this unified multi-sensor representation space explicitly reduce the gap between sensors and aid in generalizing unseen sensors.
As both this task and multi-modal aligning focus on semantic-level features, we combine them as the second stage, with masked modeling the first stage. This multi-level training approach allows us to develop unified multi-sensor representations adaptable to tasks of varying granularities.
4.5 UNIVERSAL SENSOR TOKEN
In addition to building a multi-senson representation space, we aim to extract and store information related to each sensor to aid the understanding of input data. More importantly, we want to integrate and effectively transfer this information when generalizing to new sensors. Using sensor-specific tokens a method for extracting sensor-specific information, but this approach cannot fully transfer information from all seen sensors when generalizing to sensors (Yang et al. 2024).
Therefore, we propose using universal sensor tokens to integrate and store information related to various sensors, thereby maximizing the utilization of multi-sensor data when generalizing to new sensors. During training, we randomly replace the sensor-specific tokens with the universal sensor tokens, expecting them to aid in understanding input data from various sensors. Specifically, introduce a set of learnable sensor tokens {&x K U Su. where K is the number of sensor types, 8k E IR 4xd are the sensor-specific tokens for the k-th sensor, 8y E 4xd are universal sensor tokens and T. is the number of sensor tokens for each sensor. When inputting the tactile token sequence from the k-th sensor into the encoder Penc to obtain its representation z7, we randomly select one from 8& and sy to concatenate with z, as follows:
i Su + 1) 8k. i im B(wu).
TT Peno( (2,
where Pu is the probability of using universal sensor tokens Su. During inference, we consistently use universal sensor tokens for data from new sensors.
4.6 TRAINING PARADIGM
Our framework has multi-level structure, with the training of two stages conducted sequentially. In the first stage, we simultaneously perform the reconstruction of static tactile images and dynamic tactile videos, well as the unique next frame prediction task for tactile videos. The loss for the first stage Lstagel is as follows:
Estagel
L. + CP + -pred
(8)
In the second stage, we continue to use both tactile images and videos, and simultancously perform multi-modal aligning and cross-sensor matching tasks. Hence, the loss function for the second stage is the sum of the losses from these two tasks:
Cstage2 = Culign + XLmatchs
where À is hyper paper-par controlling the weight of cross-sensor matching task.
EXPERIMENTS
In this section, we explore the answers to the following questions through quantitative and qualitative experiments: (Q1) How much benefit does the data of each sensor provide when it is integrated? (Q2) What does the unified multi-sensor representation space constructed by Any look like? (Q3) Is the unified multi-sensor representation more advantageous in various static and dynamic perception tasks? We analyze Q1 in Section expolore Q2 in Section 5.3 and answer Q3 through comparisons with existing methods in Sections 5.4 5.5 and 5.6
5.1 DATASET AND BASELINES
We use different tactile datasets for training, including: Touch and Go (TAG) Yang et al. 2022
VisGel (Li et al. 2019), Cloth (Yuan et al. 2018), ObjectFolder Real (OF Real) (Gao al. 2023
7

---

Published as a conference paper at ICLR 2025
Table 1:
The impact of adding data from multiple sensors on the seen dataset (TAG), the unseen dataset from seen sensors (Feel), and the unseen dataset from unseen sensors (OF 1.0 and OF 2.0).
Data Volume
TAG Material
Feel Grasp
OF 1.0
OF 2.0
Tactile Training Data
Material
Material
No Tactile Pre-trainin -r-iiiiiiig (CLIP)
52.96
72.37
41.00
73.16
TAG, VisGel, Cloth
996k
83.55 (130.59)
79.12 (*6.75) 46.12 (15.12)
75.10 (119
TAG, VisGel, Cloth, OF Real
2161k
79.67 (43.88)
79.28 (70.16) 47.55 (11.43)
75.53 (10.43)
TAG, VisGel, Cloth, OF Real,
2388k
79.61 (10.06) 79.10 (J0.18) 48.00 (10.45) 75.57 (?0.04)
TVL, SSVTP, YCB-Slide
TAG, VisGel, Cloth, OF Real,
2427k
79.70 (10.09) 79.40 (10.30) 48.75 (10.75) 75.66 (70.09)
TVL, SSVTP, YCB-Slide. Octopi
AnyTouch (MAE Only)
Any Touch (MAE Align)
Any Touch (MAE Align Match)
CLIP
GetSight Miel
DIGIT
DaraGGr
Figure 4: The impact of components in Any AnyTouch on the multi-sensor representation space. We t-SNE visualize the representations on the unused fine-grained subset of TacQuad, starting with CLIP and sequentially incorporating the modules. Each color represents a single touch, and samples from three sensors that touch the same position are connected by dashed lines.
TVL (Fu et al. 2024), YCB-Slide (Suresh et al. 2023) and SSVTP (Kerr et al. 2022) Octopi Yu al. 2024) and the coarse- grainedd subset of our TacQuad. We leverage the continual frames avail- able in these datasets for dynamic perception. We also select four datasets: TAG, Feel (Calandra jet al. 2017). ObjectFolder 1.0 (Gao et al., (022a), ObjectFolder 2.0 (Gao et al. 2022b) as the downstream datasets. We compare Any Touch with several single-sensor models: VIT-LENS-2 (Lei jet al. 2024), TLV-Link (Cheng et al. 2024), and Omnibind (Lyu et al., 2024). We also compare our method with the multi-sensor model UniTouch (Yang et al. (024). We use the largest subset of all data that meets the requirements of UniTouch and TLV-Link to train them, remarked as Uni- Touch² and TLV-Link For the real- world dynamic perception task, compare our method with multi-sensor model T3 (Zhao et al.) 2024), which is trained on 3M data, more than Any AnyTou.h. The detailed dataset and baseline introduction are provided in Appendix A.I A.3 and A.4
5.2 SENSOR TRANSFERABILITY ANALYSIS (Q1)
Since we have integrated data from multiple sensors into unified representations, we aim to inves- tigate the contributes of the knowledge transferred from each sensor's data to downstream tasks. Therefore, we incorporate data from GelSight, GelSlim (Donlon et al. 2018). DIGIT and GelSight Mini into the training of AnyTouch obtain four different models, and compare them across four downstream tasks. As shown in Table [ training with only GelSight data significantly improves performance on downstream datasets for all sensors, compared the CLIP model which has not encountered any tactile data. This indicates that tactile representation pre-training is crucial and transferable across new sensors. After sequentially integrating data from GelSlim, DIGIT, and Gel- Sight Mini into the training, we observe performance improvements across the three unseen datasets, with greater enhancements for unseen sensors than seen sensors. This suggests that the knowledge from the data of GelSlim, DIGIT, and GelSight Mini can transfer to the GelSight and other sensors.
However, we also observe two interesting phenomena: (1) the material classification task of the seen dataset TAG, the model trained solely on GelSight data performs the best, while incorporating data from more sensors leads to a performance drop. This is because TAG is included in the pre- training data and integrating more data reduces the proportion of TAG data in pre- pre-training. This aligns with the CLIP paper's ' finding that greater overlap between the downstream task dataset and the pre-training dataset may improve performance. (2) Although the integrated data volume from
8

---

Published as a conference paper at ICLR 2025
Evaluation of static perception capabilities on the seen sensor (GelSight). *Note that Feel is seen for the corresponding UniTouch and AnyTouch models. &Note that original TLV-Link uses frames after grasping which are much easier Feel, whereas other models use frames during grasping. Any Touch achieves a result of 99.0 using frames after grasping.
Table 2:
Touch and Go Roughness
Feel Grasp
Method
Tactile Training Data
Material
Hardness
CLIP
52.96
84.09
88.34
72.37
VIT-LENS-2
TAG Touch10Ok TAG
63.0
85.1
92.0
TLV-Link
67.2
84.7
91.3
94.5%
OmniBind
67.45
UniTouch
TAG, Feel*, YCB, OF 2.0
61.3
82.3
TLV-Link
TAG, TVL, SSVTP, OF Real, TacQuad
74.12
85.94
94.18
76.97
AnyTouch
TAG, Feel", YCB, OF 2.0 TAG. VisGel, Cloth, TVL, SSVTP. YCB-Slide, OF Real, Octopi, TacQuad
82.74
86.01
94.24
87.17
AnyTouch
80.82
86.74
94.68
80.53
DIGIT is larger, the benefits are less compared to incorporating data from GelSight Mini. This may suggest that the images from DIGIT differ more from the images of GelSight and other sensors than the images from GelSight Mini do due to the hardware difference.
5.3 MULTI-SENSOR REPRESENTATION SPACE (Q2)
To verify whether AnyTouch clusters the representations with the same tactile information from different sensors together as expected, we use t-SNE (Van der Maaten Hinton 2008) to visualize the tactile representations. We extract one aligned contact frame from each sensor for the 30 touches in the unused fine-grained subset of TacQuad. We input these samples into the CLIP model and the Any AnyTouch model which gradually incorporates masked modeling, multi-modal aligning, and cross- sensor matching, and visualize their representations in Figure Due to the lack of exposure to tactile images, CLIP struggles distinguish the same tactile information from different sensors, instead clustering samples by sensor. After introducing masked modeling, the representations become more centralized within each sensor, as this method focuses on pixel-level tactile features, which sensor-dependen However, this is not ideal for cross-sensor generalization, as we want multi-sensor tactile representations cluster based the object's 's tactile information they represent, minimizing sensor gaps. After incorporating multi-modal aligning, the multi-sensol tactile representations begin blend and cluster by the objects they represent. This indicates that cross-sensor generalization is beginning to emerge, but there is still distinct tendency for sensor-specific clustering. With our cross-sensor matching task, the representations from different sensors fully mix shared multi- sensor space, clearly clustering by the object's tactile information. This indicates that our model possesses the ability to extract sensor-agnostic features, enabling generalization to unseen sensors.
5.4 STATIC PERCEPTION ON SEEN SENSORS (Q3)
To validate the benefit of unified multi-sensor representations in transferring knowledge from multi- ple sensor data
sensors, we compared it to baselines on the seen dataset TAG and the unseen dataset Feel from the GelSight sensor. Since UniTouch uses different training data, we also train an AnyTouch model with the same data UniTouch ensure a fair comparison. As shown in Table the TLV-Link* trained with multi-sensor data outperforms all single-sensor models and the original TLV-Link in all three tasks of TAG. The AnyTouch trained with the same data as UniTouch, outper- forms UniTouch in all four tasks. With the integration of dynamic perception and more multi-sensor data, TnyTouch trained on all data achieved the best results in hardness and roughness classifica- tion in TAG and comparable results to UniToouch in Feel, despite UniTTouch having seen this data. These demonstrate the static perception capabilities of our framework on seen sensors. Notably, the original TLV-Link uses frames after grasping, while other models use frames during grasping. Any noychuh achieves a result of 99.0 using frames after grasping. is worth mentioning that the Any- Touch trained with less data outperforms the trained with all data in TAG material classification, similar to Table [ while exposure to more multi-sensor data enhances performance in hardness and roughness classification. This is because these binary hardness and roughness classification tasks

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation of static perception capabilities on unseen sensors (TACTO and Taxim) using linear probing. *Note that OF 2.0 is seen for the corresponding UniTouch and AnyTouch models.
ObjectFolder 1.0
ObjectFolder 2.0 Material
Method
Tactile Training Data
Material
CLIP
41.00
73.16
UniTouch
TAG, Feel, YCB-Slide, OF 2.0
41.3
85.4
TAG, VisGel, TVL SSVTP. OF Real, TacQuad
UniToucht
47.25
75.29
AnyTouch
OF 2.0 TAG, VisGel, Cloth, TVL,, SSVTP, SCide, OF Real, Octopi, TacQuad
TAG, Feel,
46.50
85.87
AnyTouch
49.62
76.02
Table 4: Evaluation on the real-world pouring task using GelSight Mini.
Gdbiigtt Miai
Dynamic Perception
Mean Error (g) J
Method
Fine-tune
Freeze
Cyllnder 180g beads
CLIP x
5.22
49.1
T3
x
2,33
9.74
AnyTouch
x
2.45
9.60
AnyTouch
1.56
8.22
Figure 5: Setup of real-world pouring task.
much simpler, and the tactile text descriptions in other datasets also include these two binary attributes, which have less impact on the data distribution.
5.5 STATIC PERCEPTION ON UNSEEN SENSORS (Q3)
To verify the generalization of our method on unseen sensors, compare it with the multi-sensor models UniTouch and UniToucht on two datasets from unseen sensors, OF 1.0 and OF 2.0. As shown in Table 3 the AnyTouch trained on the same data as Unitouch outperforms it on both datasets, demonstrating the static perception capability of our method across different sensors. Both UniTouch and AnyTouch, perform better on the unseen OF dataset, confirming that integrating multi-sensor data aids generalization to unseen sensors. In addition, the AnyTouch trained on the full dataset achieves the highest performance on the unseen OF 1.0, demonstrating that learning sensor-agnostic semantic-level tactile features and constructing unified multi-sensor representation space is an effective approach for cross-sensor transfer.
5.6 REAL WORLD DYNAMIC PERCEPTION (Q3)
To test the dynamic perception capability of our method in real-world object manipulation tasks, conduct experiments on a real-world task: fine-grained pouring, as shown Figure [] The detailed task setup is located at A.8 We compare AnyToocc with recent multi-sensor model T3 and use static-only Any Touch model trained solely on tactile images as a baseline. Neither model includes modules specifically designed for dynamic perception. We conduct 10 real-world
runs and record the error between the poured mass and the target mass for each test. We then average the error across the runs to get the "mean error" and use as the metric, shown in Table 4 When the tactile encoder is frozen and only the policy network is fine-tuned, CLIP, which has not seen tactile images, struggles with the task, highlighting the challenges fine- ane-grrined dynamic perception. In contrast, the static-only Any AnyThouch performed comparably to T3 which was trained on more data. After integrating dynamic perception capabilities, AnyTouch achieved the best performance. This demonstrates the importance of learning unified multi-sensor representations from both static and dynamic perspectives for completing various tasks including real-world tasks.
6 CONCLUSION
In this paper, we collect TacQuad, an aligned multi-modal multi-sensor tactile dataset that enables the explicit integration of various sensors. Based on this, present AnyTouch, unified multi- sensor tactile representation learning framework from the perspectives of both static and dynamic perception. We also explore the multi-sensor representation space and sensor transferability.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This project is sponsored by CCr-Zhipu.Al Large Model Innovation Fund and also supported by the Zhipu Company. We want thank Xinjie Tang for providing design and drafting support.
REFERENCES
Roberto Calandra, Andrew Owens, Manu Upadhyaya, Wenzhen Yuan, Justin Lin, Edward H Adel- son, and Sergey Levine. The feeling of success: Does touch sensing help predict grasp outcomes? arXiv preprint arXiv: 1710.05512, 2017.
Guanqun Cao, Jiaqi Jiang, Danushka Bollegala, and Shan Luo. Learn from incomplete tactile data: Tactile representation learning with masked autoencoders. In 2023 IEEE/RSI International Con- ference on Intelligent Robots and Systems (IROS), pp. 10800-10805. IEEE, 2023.
Ning Cheng, Changhao Guan, Jing Gao, Weihao Wang, You Li, Fandong Meng, Jie Zhou, Bin Fang, Jinan Xu, and Wenjuan Han. Touch100k: large-scale touch mouch-language-iin vision dataset for touch-centric multimodal representation. arXiv preprint amXiv:2406.03813 2024.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive iaguage-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1.Xivv/81000440 2018.
Elliott Donlon, Siyuan Dong, Melody Liu, Jianhua Li, Edward Adelson, and Alberto Rodriguez. Gelslim: A high-resolution, compact, robust, and calibrated tactile-sensing finger. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1927-1934.
IEFE, 2018
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Ruoxuan Feng, Di Hu, Wenke Ma, and Xuelong Li. Play to the score: Stage-guided dynamic multi- sensory fusion for robotic manipulation. arXiv preprint wrXh::2408.01366, 2024.
Letian Fu, Gaurav Datta, Huang Huang, William Chung-Ho Panitch, Jaimyn Drake, Joseph Ortiz, Mustafa Mukadam, Mike Lambeta. Roberto Calandra, and Ken Goldberg. A touch, vision, and language dataset for multimodal alignment. In Forty-first International Conference on Machine Learning, 2024. http:::t/pss:eeeeeteetee
net. forum? : d==tFOOH9eHO)
Ruohan Gao, Yen- Yu Chang, Shivani Mall, Li Fei-Fei, and Jiajun Wu. Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations. In Conference on Robot Learning, pp. 466-476. PMLR, 2022a.
Ruohan Gao, Zilin Si, Yen- Yu Chang, Samuel Clarke, Jeannette Bohg, Li Fei-Fei, Wenzhen Yuan, and Jiajun Wu. Objectfolder 2.0: A multisensory object dataset for sim2real transfer. In Proceed- ings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 10598-10608, 2022b.
Ruohan Gao, Yiming Dou, Hao Li, Tanmay Agarwal, Jeannette Bohg, Yunzhu Li, Li Fei-Fei, and Jiajun Wu. The objectfolder benchmark: Multisensory learning with neural and real objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp- 17276-17286 2023.
Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens Van Der Maaten, Armand Joulin, and Ishan Misra. Omnivore: single model for many visual modalities. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16102-16112 2022.

---

Published as a conference paper at ICLR 2025
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceedings of the 28th international conference machine learning (ICMI.-II), pp. 513-520, 2011.
Harsh Gupta, Yuchen Mo, Shengmiao Jin, and Wenzhen Yuan. Sensor invariaat tactile representa- tion, 2025. URL htttt::t/axxiiie org/ att//aaa//////////////// 19638
Andrey Guzhov, Federico Raue, Jöm Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-20200 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pp. 976-980. IEEE, 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022.
Carolina Higuera, Akash Sharma, Chaithanya Krishna Bodduluri, Taosha Fan, Patrick Lancaster, Mrinal Kalakrishman, Michael Kaess, Byron Boots, Mike Lambeta, Tingfan Wu, et al. Sparsh: Self Supersupe touch representations for vision-based tactile sensing. In 8th Annual Conference on Robot Learning, 2024.
GelSight. Inc. GelSight Mini. (2022).
URL htttp::/www.geliigthe htt.://wwwwwegsgggiiiiiieee
Justin Kerr, Huang Huang, Albert Wilcox, Ryan Hoque, Jeffrey Ichnowski, Roberto Calandra, and Ken Goldberg. Self-supervised visuo-tactile pretraining to locate and follow garment features. arXiv preprint arXiv: /3022, 2022.
Kunhee Kim, Sanghun Park, Eunyeong Jeon, Tachun Kim, and Daijin Kim. A style-aware discrimi- nator for controllable image translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18239-18248, 2022.
Mike Lambeta, Fei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, et al. Digit: A novel design for low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3):3838-3845 2020.
Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu. Vitgan: Training gans with vision transformers. In International Conference on Learning Representations, 2022.
Weixian Lei, Yixiao Ge, Kun Yi, Jianfeng Zhang, Difei Gao, Dylan Sun, Yuying Ge, Ying Shan, and Mike Zheng Shou. Vit-lens: Towards omni-modal representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26647-26657 2024.
Hao Li, Yizhi Zhang, Junzhe Zhu, Shaoxiong Wang, Michelle A Lee, Huazhe Xu, Edward Adelson, Li Fei-Fei, Ruohan Gao, and Jiajun Wu. See, hear, and feel: Smart sensory fusion for robotic manipulation. arXiv preprini 2rXiv:22/2.03888 2022.
Rui Li. Robert Platt, Wenzhen Yuan, Andreas Ten Pas, Nathan Roscup, Mandayam A Srinivasan, and Edward Adelson. Localization and manipulation of small parts using gelsight tactile sensing. In 2014 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3988-3993. IEEE, 2014.
Yunzhu Li, Jun-Ya Zhu, Russ Tedrake, and Antonio Torralba. Connecting touch and vision via cross-modal prediction. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10609-10618, 2019.
Junyang Lin, An Yang, Yichang Zhang. Jie Liu, Jingren Zhou, and Hongxia Yang. Interbert: Vision- and-language interaction for multi-modal pretraining. arXiv preprint arXiv:2003. 1ato:2003.11999 13198, 2020.
12

---

Published as a conference paper at ICLR 2025
Fengyuan Liu, Sweety Deswal, Adamos Christou, Yulia Sandamirskaya, Mohsen Kaboli, and Ravinder Dahiya. Neuro-inspired electronic skin for robots. Science robotics, 7(67))eabl7344, 2022.
I Loshchilov. Decoupled weight decay regularization arXiv preprint arXiv:1711.0510. 2017.
Yuanhuiyi Lyu, Xu Zheng, Dahun Kim, and Lin Wang. Omnibind: Teach to build unequal-seale modality interaction for anni-bind of all. arXiv preprint arXiv:2405.16100, 2024.
Perla Maiolino, Marco Maggiali, Giorgio Cannata, Giorgio Metta, and Lorenzo Natale. A flexible and robust large scale capacitive tactile system for robots. IEEE Sensors Journal, 13(10)33100 3917, 2013.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
Samanta Rodriguez, Yiming Dou, Miquel Oller, Andrew Owens, and Nima Fazeli. Touch2touch: Cross-modal tactile generation for object manipulation. arXiv preprint arXiv:2409.08266 2024.
Zilin Si and Wenzhen Yuan. Taxim: An example-based simulation model for gelsight tactile sensors. IEEE Robotics and Automation Letters, 7(2):2361-2368, 2022.
Sudharshan Suresh, Zilin Si, Stuart Anderson, Michael Kaess, and Mustafa Mukadam. Midastouch: Monte-carlo inference over distributions across sliding touch. Conference on Robot Learning, pp. 319-331. PMLR, 2023.
Zhan Tong, Yibing Song, Jue Wang. and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training, Advances in neural information process- ing systems, 35:10078- 10093, 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Shaoxiong Wang, Mike Lambeta, Po-Wei Chou, and Roberto Calandra. Tacto: A fast, flexible, and open-source simulator for rego-resolution vision-based tactile sensors. IEEE Robotics and Automation Letters, 7(2))3930-32) -9330,0-93-2200 2022a.
Yau-Shian Wang, Ashley Wu, and Graham Neubig. English contrastive learning can learn universal cross-lingual sentence embeddings. arXiv preprint (iXXix22/11061122 2022b.
Zhengtong Xu, Raghava Uppuluri, Xinwei Zhang, Cael Fitch, Philip Glen Crandall, Wan Shou, Dongyi Wang, and Yu She. Unit: Unified tactile representation for robot learning. arXiv preprint a4Xiv:2408. 06481, 2024.
Le Xue, Mingfei Gao, Chen Xing, Roberto Martin-Martin, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1179-1189, 2023.
Fengyu Yang, Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, and Andrew Owens. Touch and go: Learning from human-collected vision and touch. arXiv preprint urXiv:2211.1244 2022.
Fengyu Yang, Chao Feng, Ziyang Chen, Hyoungseob Park, Daniel Wang. Yiming Dou, Ziyao Zeng, Xien Chen, Rit Gangopadhyay, Andrew Owens, et al. Binding touch to everything: Learning unified multimodal tactile representations. In Proceedings of the IEEE/CVF Conference on Com- puler Vision and Pattern Recognition, pp. 26340-26633 26353, 2024.
Samson Yu, Kelvin Lin, Anxing Xiao, Jiafei Duan, and Harold Soh. Octopi: Object property rea- soning with large tactile-language models. arXiv preprint arXiv:2405.0279.. 022944 2024.
13

---

Published as a conference paper at ICLR 2025
Wenzhen Yuan, Siyuan Dong, and Edward H Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017.
Wenzhen Yuan, Yuchen Mo, Shaoxiong Wang, and Edward H Adelson. Active clothing material perception using tactile sensing and deep learning. In 2018 IEEE International Conference on Robotics and Aulomation (ICRA), pp. 4842-4849. IEEE, 2018.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuner audio-visual language model for video understanding. arXiv preprint arXiv:23066. 02855, 2023.
Lunwei Zhang, Yue Wang, and Yao Jiang. Tac3d: A novel vision-based tactile sensor for measuring forces distribution and estimating friction coefficient distribution. arXiv preprint arXiv:2202.06211, 2022.
Shixin Zhang. Yiyong Yang, Fuchun Sun, Lei Bao, Jianhua Shan, Yuan Gao, and Bin Fang. A compact visuo-tactile robotic skin for micron-level tactile perception. IEEE Sensors Journal, 2024.
Shixin Zhang, Yiyong Yang. Yuhao Sun, Nailong Liu, Fuchun Sun, and Bin Fang. Artificial skin based on visuo-tactile sensing for 3d shape reconstruction: Material, method, and evaluation. Advanced Functional Materials, 35(1):241 1686, 2025.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. International conference on machine learning, pp- 7523-7532. PMLR, 2019.
Jialiang Zhao, Yuxiang Ma, Lirui Wang, and Edward Adelson. Transferable tactile transformers for representation learning across diverse sensors and tasks. arXiv preprint arXiv:2406.1300 2024.
Kaiyan Zhao, Qiyu Wu, Xin-Qiang Cai, and Yoshimasa Tsuruoka. Leveraging multi-lingual positive instances in contrastive learning to improve sentence embedding. arXiv preprint arXiv:2309.08929, 2023.
Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2223222 2017.
14

---

Published as a conference paper at ICLR 2025
Table 5: Training dataset statistics. The text modality in Touch and Go and ObjectFolder Real is generated by GPT-4o. *Note that Cloth and YCB-Slide contain vision modality, but we intentionally do not use demonstrate our method's compatibility with modality absence. We only count the number of contact frames used for training in each dataset.
Dataset
Vision
Text
Video
Sensor
Size
Touch and Go Yang et al 2022
GelSight
250k
VisGel [Li
2019)
GelSight
587k
Cloth Yuan et al
2018
GelSight
587k
TVL Fu ci al. 2024)
V
DIGIT
39k
SSVTP
Kerr
2022
X
DIGIT
4.5k
YCB-Slide (Suresh el al.
22223
DIGIT
183k
ObjectFolder
Real
Gao
2023
GelSlim
1165k
Octopi el al. 2024
GelSight Mini
39k
GelSight, DIGIT, DuraGel
TacQuad
55k
GelSight Mini
A APPENDIX
A.1 TRAINING DATASET STATISTICS
In this section, we provide a detailed presentation the sensor type, modality pairing and data scale of the datasets used during the training phase. We
total of 9 datasets from
different
sensors for training, including: Touch and Go (TAG) Yang et al. 2022). VisGel (Li et al.
2019) and
Cloth Yuan et al. 2018) from GelSight Yuan et al. [217); ObjectFolder Real (OF Real) (Gao al.
2023 from GelShim (Donlon et al. 2018); TVL (Fu et al. 2024), YCB-Slide (Suresh et al. 2023
and SSVTP (Kerr et 2022) from DIGIT (Lambeta et al. 2020 Octopi (Yu al. 2024 from
GelSight Mini (Inc.): ) and the coarse- coarse-grained subset of our Tae QuaQ from DIGIT, GelSight Mini and DuraGel (Zhang et al. 2024). We filter the contact frames with tactile deformations by calculating the difference between each tactile image and the corresponding background frame in these datasets. Eventually, we extract a total of 2,481,703 1,770 tactile contact frames from these datasets for model training. We also leverage the continual frames available in these datasets to train the model's dynamic perception capabilities. The detailed training dataset statistics are shown in Table [ We generate text descriptions for Touch and Go and ObjectFolder Real using GPT-4o. We also expand the text descriptions in TVL, SSVTP and Octopi. We remove the text modality of the training samples included in the test set of downstream tasks for fairness. It is worth saying that Cloth and contain vision modality originally, but we intentionally do not use it to demonstrate our method's compatibility with missing modalities.
A.2 MULTI-MODAL ALIGNING LOSS
When training AnyTouch, we maximize the use of paired data by selecting the largest subset for each modality combination within the batch for multi-modal aligning. Considering a pair of uni- modal representations (er, derived from uni-modal encoders, where ær E IRd is the touch representation, IV E R the vision representation and æy E Rdd U the text representation. We then perform multi-modal alignment (Radford et al.) 2021 within the batch as:
1
exp(xT,5 zV./T)
CT =
log
[22vl
jeny exp(xt, #V3/T)"
explxTy FL,/TT)
LT
log
(10)
12
Σ Cjefz exp(xt, ILS/T)²
vEl
explaT; #1,1/T
LV = 1aa n 1u
Σ log Ejea,n,, expluTs x1a/T)²
ig nar.
where the batchsize, 12,52 are sets indices for the samples containing vision and text, and is the scalar temperature.
15

---

Published as a conference paper at ICLR 2025
A.3 DOWNSTREAM DATASETS
In this section, we provide a more detailed introduction to the downstream datasets. Specifically, we compare the static perception capabilities of AnyTouch and the baselines on four downstream datasets: TAG, Feel (Calandra et al. 2017), ObjectFolder 1.0 (OF 1.0) (Gao et al. 2022a) and OjectFolder 2.0 (OF 2.0) (Gao et al. 2022b). TAG includes three tactile properties understanding tasks: material, hardness, and roughness classification. Feel is a robotic dataset from GelSight containing grasp success prediction task. We follow the data split in Yang al., 2024 Cheng
iet al. 2024) for Feel. ObjectFolder 1.0 and OjectFolder 2.0 are two simulated object datasets using
TACTO (Wang al. 2022a) and Taxim (Si & Yuan 2022). We use them as unseen datasets from
unseen sensors, and follow the data split in Yang et al. 2024
A.4 BASELINES
In the static perception task, we compared AnyTouch with several recent single-sensor and multi- sensor baselines. VIT-LENS-2 (Lei et al. 2024). TLV-Link (Cheng et al. 2024), and Omnibind (Lyu et al. 2024 are three single-sensor models included, all of which conduct multi-modal alignment us- ing data from GelSight. As for multi-sensor models, we compare our method with UniTouch (Yang eet al., 2024). which currently demonstrates the SOTA cross-sensor performance. We also intend to train Uni Touch and TLVV-Link using all available multi-sensor data for comparison. However, since Tnuhoouch requires touch-vision paired data and TLVVLink can only be trained three-modal paired data, we extract the largest subset of data that meets the requirements to train them, remarked as UniTouch* and TLV-Link
In the real-world dynamic perception task, we compare with a recent multi-sensor model T3 Zhao al. 2024). This model has been validated have strong capabilities in completing manipulation tasks. It is important to note that the amount of training data used by this model is approximately 3M, which more than our AnyTouch. Additionally, since this model utilizes labels from various downstream tasks during pre-training, comparing its static perception capabilities with other models would be unfair. Therefore, we only use it in the dynamic perception task.
A.S DETAILS FOR GIAIIGR DATA COLLECTION
In this section, we provide a more detailed introduction of the data collection process for the fine- grained spatio-temporal aligned data. The calibration platform we built consists of three main parts: a platform, a movable end effector, and a 3D-printed container that holds the sensor. The four sensors are fixed side by side the container. The movable end on our calibration platform can be programmed to move at a specified speed to designated position within the coordinate system defined by the base. Therefore, as long as we pre-measure the relative positions of the centers of the four sensor surfaces within the container and compensate for the relative positions during each set of data collection, we can ensure that all four sensors make contact with the object from the same initial position and at the same speed, thereby achieving both temporal and spatial alignment.
A.6 IMPLEMENTATION DETAILS
We base our encoders on OpenCLIP-Large (Cherti et al. 2023 For the tactile decoder, we use a Vision Transformer (ViT) Dosoviiskiy et 2020) with 8 layers and a dimension of 512. We use the AdamW (Loshchilov 2017) optimizer with a learning rate of 2e-4. After a warm-up period of 1 epoch, we implement linear learning rate decay. For each tactile video clip, we use T = 3 frames. We train the first stage for 20 epochs and the second stage for 12 epochs on 4 NVIDIA A800 GPUs We alternate between training with tactile images and video clips throughout the entire training process. We use a mask ratio p = 0.75. During the alignment, we use the text modality as the anchor, freezing the text encoder while performing LoRA fine-tuning on the vision encoder. We set the alignment strength arv = aTT = 1.0 and Qy = 0.2, and set the weight of sensor matching Ā= = 0.1. Following (Yang al. 2024), sensor. In both stages, we set the probability of using universal sensor tokens p., to increase linearly from 0 to 0.75.
use L 5 sensor tokens for each type of
16

---

Published as a conference paper at ICLR 2025
Focus on the object being touched by the handheld tactile sensor. Please provide a description of the tactile properties (exact location of the touch, materials at the contact point, texture, roughness level, hardness level, and more than these properties). using scrics of adjectives. Pleasc do not use ambiguous adjectives like 'moderate' 'medium' Ensuring your response does not exceed 60 words, counting punctuation separate words.
The sensor touches a rough, splintered wooden bench with peeling paint, Textured surface, visibly cracked, and fragmented. Hardness lovel high, unevenness significant, and the material at the contact point is weathered, aged wood coated in deteriorating paint chips.
Output:
Figure 6: Prompt and raw output for TAG and TacQuad.
A.7 GPT-40 ANNOTATION
In this work, generate paired text descriptions of tactile properties for Touch and Go, Object- Folder Real and our TacQuad. We input paired visual images and predefined text prompts into GPT-4o to obtain text descriptions. We borrow the prompt from (Cheng al.] 2024) and make appropriate adjustments. The prompt for Touch and Go and TacQuad is shown in Figure
Specifically, because the ObjectFolder Real dataset has camera views and touch locations or details may not be visible, we input two visual images and one tactile image simultaneously, with clear indications in the prompt, shown in Figure
Since the TVL. and SSVTP datasets only contain simple phrase-level tactile descriptions, we also use GPT-4o extend the text modality both datasets. We input both visual images and the existing text descriptions simultancously, as shown in Figure g
After generating the annotations using GPT-4o, we conduct simple and rough sampling check to ensure the correctness. We find that as long as the model can correctly recognize the object, the knowledge of object-related physical properties stored in the large model can effectively provide accurate tactile annotations. Errors in the generated annotations may occur when an object's cate- gory is difficult to determine or when occlusions are present. However, such instances are relatively uncommon.
A.8 ROAL- POURING TASK
To test the dynamic perception capability of our method in real-world object manipulation tasks, conduct experiments on real-world task: fine-grained pouring, as shown in Figure 5 In the experiments, we use a 6-DoF UFACTORY xArm robotic arm equipped with a Robotiq 2F-140 gripper. Cartesian space displacement commands are generated robot arm must rely entirely on tactile feedback to pour out 60g of small beads from a cylinder that initially contains 100g of beads. The robot arm can select one the three actions to perform based on the real-time tactile feedback: pouring, waiting, or retracting. The action step size is 56 0.25%. We train the model through imitation learning and collect the training data using keyboard. As both rotating the cylinder and pouring out the small beads lead to continuous variations in pressure on the sensors, the model must analyze the fine- fine-graing grained changes between tactile images to determine the appropriate pouring speed and the right moment retract the cylinder. This task is typically performed using multi-modal data (Li et all 2022). making particularly challenging for models that rely solely tactile perception.
policy frequency of 5 Hz. The
17

---

Published as a conference paper at ICLR 2025
Focus the object being touched by tactile sensor first two images. The two images are shot from two distinct viewpoints. The third image comes from tactile sensor, reflecting the detailed deformations when touching the object. Please provide a description the tactile properties (exact location of the touch, materials at the contact point, texture, roughness level, hardness level, and more than these properties), using series of adjectives. Please do not use ambiguous adjectives like 'moderate' or 'medium'. Ensuring your response does not exceed 60 words, counting punctuation separate words.
Output without tactile image: The tactile contacts the upper center of clear polycarbonate box lid. The material is smooth, glossy, and transparent, featuring hard, durable surface. The texture sleek, with no visible roughness or irregularities. providing a nearly frictionless The hardness level is high, indicative robust plastic material.
Output with tactile image: The tactile propertics indicate contact the smooth, transparent polycarbonate lid, characterized by its high hardness and rigidity. The surface designed with bubble-like texture feels slightly raised and evenly distributed. The texture appears consistent, lacking significant roughness, while retaining firm yet slightly flexible quality at the touchpoint.
Figure 7: Prompt and raw output for OF Real. Given that OF Real includes two camera views and some touch locations or details may be obscured, we imput visual images along with one tactile image simultaneously. If the tactile image is not provided as input, there is possibility of producing incorrect annotations (marked in red).
Focus the object being touched the tactile sensor. Please provide description of the tactile properties (exact location of the touch, materials at the hardness level, more these properties), using series of adjectives. already known the object (phrase-leve level descriptions), Please do not use ambiguous adjectives like 'moderate' 'medium", Ensuring your response does not exceed 60 words, counting punctuation as separate words.
point, texture, roughness level,
Output: The touch is the upper edge, the tactile sensor contacts leather. The texture is smooth, with slight grain. The roughness low, and the hardness low. The material feels soft, yielding minimally pressure, with consistent, uninterrupted surface.
Figure 8: Prompt and raw output for TVL and SSVTP. Given that these datasets only contain phrase-level tactile descriptions, input the visual image and the phrase-level descriptions to gen- erate more detailed tactile descriptions.
A.9 ABLATION STUDY
To investigate the impact of each module in AnyTouch, as well as the individual contributions of the vision and text modalities in multi-modal alignment, we conduct ablation studies on the four downstream datasets. The experimental results are shown in Table 6 We observe performance decline when the paired vision and text modalities are excluded, highlighting the importance of aligning with these paired modalities to narrow the sensor gaps and achieve a comprehensive tactile perception capability. We also find that the performance decline caused by removing the visual modality is greater than removing text. However, this does not necessarily indicate that the visual modality is more important, removing the visual modality results in more significant reduction
18

---

Published as a conference paper at ICLR 2025
Table 6: The impact of modalities and modules in AnyTouch on static perception capabilities.
TAG Material
Feel Grasp
OF 1.0
OF 2.0
Model
Material
Material
AnyTouch
80.82
80.53
49.62
76.02
w/o Text Modality
75.91([4.91)
78.93((1.60)
48.87(10.75)
75.52(10.50)
w/o Vision Modality
74.55()).27)
77.30
48. 1,051 75.22(10.88 80)
w/o Text in TacQuad
80.10((0.12)
80.19()).34)
49.21(10.41)
75.91((0.11)
w/o Stage
78.34((2.48)
78.62((1.91)
48.75(10.87)
76.08(10.06)
w/o Stage
(8.64((12.18)
12.39(18.14)
46.50(13.12) 73.09((2.93)
w/o Cross-Sensor Matching
80.54((0.28)
79.43((1.10)
49.25((0.37) 75.80(10.22)
w/o Dynamic Perception
77.93((2.89)
79.28((1.25)
48.62(11.00) 75.70(10.32)
w/o Universal Sensor Tokens 80.79((0.03)
79.03((1.53) 48.40((1.22) 75.40((0.62)
in data during the aligning. We also remove the text from the TacQuad dataset we proposed to validate the effectiveness of the text in the dataset. Although the TacQuad data is relatively small compared to the total dataset, making it unlikely to significantly impact performance when modified, we observe consistent decline the model's 's performance on downstream tasks after removing the text modality. This demonstrates the important role of the text modality in our dataset as bridge that helps reduce the gap between sensors.
When we remove cross-senson matching and universal sensor tokens from Any AnyTouch, we observe a performance decline primarily on the datasets from unseen sensors. is important to note that the sensors in these datasets are not included in the positive sample pairs for cross-sensor matching, indicating that this task has even greater potential. This demonstrates that both strategies can en- hance the model's generalization to unseen sensors. After removing the entire stage 2, observe a significant performance decline. On OF 2.0, performs even worse than CLIP, which has never encountered tactile data. This result is consistent with the improvement OF 2.0 when removing stage and the analysis in Section 5.3 indicating that learning semantic-level features is crucial for achieving comprehensive tactile perception and cross-sensor generalization. Nevertheless, learning pixel-level features in stage is still meaningful for the seen sensors. In addition, we also observe a consistent decline in performance after removing the joint training for dynamic perception, indicat- ing that integrating dynamic perception can indeed enhance static perception capabilities.
A.10 CROSS-SENSON GENERATION
To more comprehensively demonstrate the value and impact of the dataset we proposed, we conduct cross-sensor generation experiments on the fine-grained spatio-temporal aligned data. Specifically, we trained models to generate aligned DuraGel images from GelSight Mini images, and to recon- struct the 20x20 force fields captured by the Tac3D sensor from DIGIT and GelSight Mini data. We compared the performance of our model with the T3 model, which used more training data than ours (3.08M compared to our 2.48M) for pretraining. Specifically, for generating Duragel images, we constructed a GAN network based on VIT, using T3 or AnyTouch as the encoders for the dis- criminator and generator, similar to ViTGAN (Lee et al. 2022). A ViT-based decoder is then used to generate images across sensors. For the force field generation of Tac3D, due to its low resolution, we treat it as a regression task and use an MLP to reconstruct the force field based on the features extracted by the encoder. Both networks can effectively evaluate the quality of the encoder's 's tactile representations. To further ensure fairness, we also removed the overlapping portions of the coarse- grained aligned data from the training data that overlapped with this dataset. Note that Tac3D is an unseen sensor for both of the models. We use mean square error (MSE) (L) between the generated data and the ground truth as the metric. The results shown in Table 7 indicate that our method outper- forms T3 in terms of generation quality, both for cross-senson generation of vision- images and for foree fields captured by the unseen Tac3D. This demonstrates the effectiveness of our method and the value of the dataset, and supports our motivation to obtain a unified tactile multi-sensor representation that is applicable variety of tasks and sensors.
19

---

Published as a conference paper at ICLR 2025
Table Performance comparison with T3 on the cross-sensor generation task using the fine- greengr spatio-temporal aligned data.
Mean Square Error (L)
Model
Training Data
GelSight Mini
GelSight Mini
DIGIT
DuraGel
+ Tac3D
Tac3D
T3 AnyTouch
3.08M
0.2261
0.0167
0.0155
2.48M
0.2159
0.0151
0.0144
A.11 DISCUSSION ON FRAME NUMBER
In the real world, the complete process of touching object can take several seconds or even tens of seconds. Ensuring the model can comprehend an entire tactile video presents a significant challenge. Current large-scale video understanding models, such as Video-LLaMA (Zhang et al. 2023), often process tens or even hundreds frames as input, encoding them into tokens. However, this comes at the cost of generating very long token sequences, which significantly increase computational overhead and inference time. The tactile modality is frequently used in fine-grained manipulation tasks that demand high real-time performance, which imposes strict requirements the model's inference speed. As a result, models that rely on long frame sequences are challenging to apply in real-time dynamic perception tasks. Moreover, since touch actions are typically performed high speeds, even a sequence of three continual frames (equivalent to 0.1 seconds for DIGIT sensor with a frequency of approximately 30Hz) can exhibit noticeable changes. We anticipated these challenges and, as result, chose to use sequence of three continual frames as the input format for tactile videos. This approach also enables the understanding of longer videos by selecting multiple 3-frame segments and either concatenating or summing their features, similar to ImageBind (Girdhar) iet al. 2023). Using more frames may lead to better perception performance, but this is essentially a trade-off between performance and both computational cost and reference speed.
A.12 DISCUSSION ON OTHER TACTILE SENSORS
Tactile perception is not limited to images. Some tactile properties, such temperature and torque, are difficult to obtain from tactile images alone, requiring the use of other types of tactile sensors. This issue presents challenges from both hardware and algorithmic perspectives.
From a hardware perspective, an ideal tactile sensor should be capable of gathering various types of tactile information, effectively integrating multiple existing tactile sensors into single unit. This may be very challenging. and a more practical solution might involve equipping different fingers of robotic hand with different types of sensors. This would allow for the simultaneous collection of diverse tactile data, maximizing the range of information captured.
From algorithmic perspective, when rision-based tactile sensors replaced with other types of tactile sensors (e.g., tactile sensor arrays), the multi-sensor data alignment method proposed in this paper can still be applied. Aligned data can then be used perform alignment or distill knowledge from the visuo-tactio model to models for other types of tactile sensors. For lower- resolutino tactile sensors, the aligned data can facilitate tactile super-resolution learning, enabling knowledge transfer from vision-basee tactile sensor models to enhance their performance.
If both dision-based tactile sensors and other tactile sensors (e.g., those capturing temperature or other non-visual properties) are used simultaneously. a possible approach is to fuse their outputs into a unified, comprehensive tactile feature. This enriched representation can then be aligned with other modalities in a unified manner.
A.13 LIMITATIONS AND FUTURE WORK
In this section, we discuss some potential limitations of our work and propose corresponding solu- tions for future work:
Compared to all the training data, the scale of the TacQuad dataset we have currently collected is still somewhat limited. Capturing the immense variety of object types within
20

---

Published as a conference paper at ICLR 2025
single dataset is challenging in limited amount of time. Fortunately, the coarse- -aannee- spatial aligmment data collection method we propose has the potential to scale up, as data collection can be performed manually without the need for precise alignment. Fine-grat mennegg data collection can also be expanded by replicating the calibration platform and increasing manpower. We plan to grow our team scale up the dataset and enhance object diversity in future work.
The types of sensors considered are relatively limited. We have made every effort to col- lect all available vision-based tactile sensors around us, yet were only able include four different types. Moreover, we did not explore the differences between individual sen- of the same type address issues such gel damage. Moving forward, we aim to expand our dataset and increase the diversity of sensors through collaborative data collec- tion across multiple laboratories.
The scope of tasks for dynamic tactile perception is currently limited. In this work, validated the dynamic perception capabilities of our model on single real-world manip- ulation task: pouring. We hope to explore more challenging and interesting dynamic per- ception tasks in future work. Additionally, beyond real-world manipulation tasks, studying tactile video understanding- particcllllr fine-grained dynamic tactile understanding that includes direction and action descriptions-is also an interesting direction to explore.
21


Reviews:
[{"review_id": "BCJJbyJB3u", "reviewer": "Reviewer_DUex", "summary": "The author introduces an aligned multi-sensor tactile dataset to unify tactile data. The data across four optical-based sensors. Also, the author proposes a unified static-dynamic multi-sensor representation learning. The author also implements real world experiments to test their framework.", "questions": "1. For those four types of sensors, how many sensors you chose for each type? This is important because the tactile image can be very different even the same types of sensor. The dataset could be more diverse if more sensors for each type are used.\n\n2. [line 189]. The author mentions that collecting fine-grained aligned tactile data is very costly. Please specify the reasons.\n\n3. [line 431]. Incorporating data from more sensors leads to a performance drop. This may potentially reflect the robustness of the mode is not good enough. Using a single sensor for each type may hard to extract effective features of this type of sensor and only focus on some specific details of a single sensor.\n\n4.[line 488-498] The performance of UniTouch and UltraTouch is similar, it is hard to claim the advantages of the current method.\n\n5. Please mention real-world tasks in more detailed in the main paper. The reader will be confused about the task and the meaning of \"mean error\" before reading the appendix.\n\n6. It would be better if the author have experiments to prove the effectiveness of the text in this dataset. \n\n7. [line 450-453] Please explain more details for this.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The idea of aligned multi-sensor dataset, improve generalization across different sensors.\n(2) Capturing both static and dynamic tactile features. This is novel to make the system have comprehensive tactile information understanding.\n(3) The dataset labeling process involves language model. This could improve the efficiency of the dataset collection.\n(4) Using t-SNE visualization is clear to show the impact of components on representation space.\n(5) I recognize the authors' effort to collect large volume dataset of tactile", "weaknesses": "Major Weaknesses:\n1. Although the author's idea of the aligned multi-sensor dataset, the author focuses to optical-based tactile sensors only, which simplifies the problem with similar tactile modalities(tactile image).\n2. In sec 5.2, the conclusion of the material classification task seems contradict to the motivation to build the aligned multi-sensor dataset.\n3. The experiment results are not convincing in Section 5.5.\n4. Some components of methods are not proved effective in the experiment.\n4. No limitations and future work are described in the main paper.\n\nMinor Weaknesses:\n\n1. The presentation needs to be improved. Readers may get confused for some point before reading the supplementary material\n\n2. The author use too many colors to represent different sensor types in the main text, which may not be good for reading.\n\n3. Some conclusions according to experiment results are not reasonable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ACahdUawaA", "reviewer": "Reviewer_TLvC", "summary": "This paper proposes UltraTouch, a model that learns unified static-dynamic representations for multiple tactile sensors with different architectures. To train UltraTouch, a dataset called TacQuad is introduced by collecting multimodal data from four different tactile sensors. UltraTouch captures both pixel-level and semantic-level features, which are shown to be transferable and effective through experiments on multiple downstream tasks.", "questions": "1. Referring to my first concern in Weaknesses section, I'm curious about the computational cost of training with tactile videos. It's common that the frame rate of the video being limited due to data loading/ computation limits, but only using 3 frames still seems too few to me. My guess is that using more frames may lead to better dynamic perception ability, and I'm wondering if an ablation experiment on this parameter can be possible?\n\n2. Besides TacQuad, some of the other datasets also contain tactile videos, such as TAG and OF Real. I wonder if they are already treated as videos during the training of UltraTouch, if not, maybe it can further improve the model performance if they are used as videos.\n\n3. I'm curious about the scalability of the calibration platform used for data collection. Is it easy for other researchers to build a similar platform to scale up the current dataset?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The collected dataset TacQuad, especially the fine-grained spatio-temporal aligned data can be a significant contribution to the community of tactile sensing. None of the prior works has collected tactile videos with shared speed and depth using multiple sensors. Although not fully explored in this work, I think this dataset and its data collection program will potentially benefit future research if being available to public.\n\n2. The proposed UltraTouch framework, to the best of my knowledge, is the first model that learns the dynamic tactile representations through pretraining. This is a considerable improvement upon the prior works that learn unified tactile representations, which are all limited to static images. Future works that leverage touch for fine-grained manipulation tasks can potentially benefit from this framework.\n\n3. The performance of UltraTouch model on the existing material classification/ grasp stability prediction seems promising, which is outperforming the previous pretraining frameworks significantly in most experiments. \n\n4. The proposed real-world pouring task demonstrates the effectiveness of learning dynamic representations (Table 4).", "weaknesses": "1. As is shown in A.4, only 3 frames are used for each tactile video clip. This seems to be a surprisingly small number of frames, and my concern is that this might not be sufficient for extracting some of the fine-grained tactile information (e.g., hardness). One possible ablation experiment would be training the model with different video length and evaluate the performance. See Questions section for more details.\n\n2. The paper proposes a novel calibration platform for collecting fine-grained spatio-temporal aligned data, and they claim that this can be used for fine-grained tasks such as cross-sensor generation. However, it seems that they don't explore much on the potential of this data, i.e., most of the downstream tasks are performed on static perception, and the only pouring task is not closely related to this part of data. It would be interesting to see if any more impressive tasks can be performed using the proposed dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6odlM2SHuG", "reviewer": "Reviewer_dEU8", "summary": "This paper presents a novel and comprehensive representation learning framework for tactile data that addresses (1) V-T-L modalities alignment, (2) capturing dynamic information in tactile modality, and (3) generalization across different sensors together. It constructs a multi-sensor dataset called TacQuad and a 2-stage multi-task representation learning framework, equipped with pixel-level MAE, semantic-level multi-modal alignment, and cross-sensor matching techniques. With extensive experiments, the authors show this representation helps in some downstream tasks under various settings (seen and unseen sensors, different training dataset combinations, and real robot experiments).", "questions": "1. Regarding Fig 7 and 8, is there a significant gap between GPT-4o outputs for OF-Real and TVL/SSVTP, given you didn't use tactile data for the latter? If so, will this affect your training? If not, do you think GPT-4o is making use of tactile input, given it may not be capable of analyzing images from tactile sensors? Can you show some raw output examples?\n\n2. I noticed your \"full model\" removed Feel and OF 2.0 from the dataset. Is it on purpose? If the full model sees the Feel dataset, will its performance in Table 2 be lower than 80.53, between 80.53 and 82.3, or higher than 82.3? Similarly for OF 2.0. The claim that more is not always better itself is interesting, but I just wonder can it be simply due to the lack of some specific dataset? Your current experiment setting cannot rule out this possibility. \n\n3. Sec 5.4 explained the performance drop on material classification, but what about grasp? \n\n4. The tables are a bit hard to read. Is the first row in Table 1 from CLIP as explicitly said in Tables 2 and 3? In Table 2, the colorful detailed dataset configuration makes it hard to find what to compare before carefully reading Sec 5.4. \n\nMinor:\n1. Citation format. Parenthesis missing in lots of scenarios.\n2. Line 797 missing space.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Well-motivated and well-written. Authors observed the difference between tactile sensing and vision, and creatively designed corresponding modules to enhance the representation. \n\n2. Clear visualization. Fig 2 works as a nice abstract of the entire framework.", "weaknesses": "1. In Tables 2 and 3, the average performance of the TAG+Feel+YCB-slide+OF 2.0 model is better than the full model with the TacQuad dataset. This observation\n  - Questions the necessity of your TacQuad dataset, and \n  - Poses a new question: how do you choose the size & scope of the dataset? Contrastive learning has requirements on the dataset size, while too much data will, in turn, make it \"lose focus\". \n\nPlease add more discussion and experiments on this issue, and figure out either the detailed reason (i.e. which dataset is bad for some reason) or how to find a good combination of datasets. \n\n2. Given you claimed the model has the ability to extract sensor-agnostic features from unseen sensors, it would be more convincing to show some downstream task performance on the new sensor, aside from qualitatively showing t-SNE visualization.\n\n3. Some other results are not convincing enough as well. In Fig 4 / Sec 5.3, DuraGel samples are better clustered than baselines, but some samples from GelSight and DIGIT, which are previously seen, are far from each other and mixed with other samples. Is it due to some difficulties in tuning the corresponding weight for loss?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NYyYiVaOSv", "reviewer": "Reviewer_oW9a", "summary": "This paper present a novel framework called UltraTouch and a large dataset TacQuad collected using 4 different tactile sensors. The UltraTouch framework has a multi-stage training in pixel level and semantic level. In stage 1, the authors use MAE to train on static and dynamic tactile images/videos. In stage 2, the authors do contrastive learning on multi-modal aligning and cross-sensor matching to understand semantic-level sensor-agnostic tactile properties. The authors also answer research questions by testing on downstream tasks like material classification, grasp prediction, and pouring.", "questions": "- In data collection process, how do you localize the contact position in world coordinate? How to ensure the contact position is the same for different sensors?\n\n- How many corrections do you have to make when use GPT to generate descriptions? How well does it understand the data?\n\n- Are you using the same sensor throughout the entire data collection process? I feel like most tactile sensors now are not very robust, especially the gel pad, which will degrade over time when it touches objects for too many times. Have you replaced gels during the collection process? Will the data be different after replacement? \n\n- Even the tactile sensors are of the same type. The difference between each individual sensor is huge. See the pixel-wise difference of DIGIT in Fig 7. in this paper [1]. Combined with my previous question, if you have use multiple sensors of the same type during collection and training, I'm curious how this will affect the performance.\n\n- Missing space in line 797 between TAG and includes\n\n[1] Bhirangi, Raunaq, et al. \"AnySkin: Plug-and-play Skin Sensing for Robotic Touch.\" arXiv preprint arXiv:2409.08276 (2024).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Strengths of this paper includes:\n\n- They collect and open-source a large tactile dataset called TacQuad, which makes good contribution to the multi modal robot learning community since the tactile data are far from enough compared with vision data. \n\\\n\\\n- The framework is novel. The idea of learning a unified tactile representation is very interesting and the authors study how tactile aligns with other modalities like vision and text, and how different sensors match with each other. \n\\\n\\\n- The experiments are comprehensive. The authors conduct extensive experiments on multiple downstream tasks like material classification, grasp prediction, and pouring, which demonstrate Ultratouch, learning unified representations, has advantages over other models.", "weaknesses": "Weaknesses of this paper include:\n\n- Though vision based tactile sensors are very popular in robot learning community, it might not be the ultimate solution that people agree on. Tactile perception are more than images. Humans feel temperature, texture, shape, vibrations, etc. when interacting with objects. This is still an open question and is out of scope of this paper. But it will be interesting to see if how other tactile sensors align or match with the vision-based tactile sensors.\n\\\n\\\n- I think there can be some study on the effect from the stage 1, the pixel level learning, on the downstream tasks. How does learning the pixel-level details help the downstream tasks? I think material classification and grasp prediction might not need that level of detail to achieve good performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)