Sparsh: Self-supervised touch representations for vision-based tactile sensing
Authors: Carolina Higuera
Akash Sharma
Chaithanya Krishna Bodduluri
Taosha Fan
Patrick Lancaster
Mrinal Kalakrishnan
Michael Kaess
Byron Boots
Mike Lambeta
Tingfan Wu
Mustafa Mukadam
Venue: CoRL 2024 (2024)
Keywords: Tactile sensing
Pre-trained representations
Self-supervised learning
TL;DR: General touch representations trained with self-supervision, evaluated on a new touch-centric benchmark, outperform custom task and sensor specific models

Abstract: In this work, we introduce general purpose touch representations for the increasingly accessible class of vision-based tactile sensors. Such sensors have led to many recent advances in robot manipulation as they markedly complement vision, yet solutions today often rely on task and sensor specific handcrafted perception models. Collecting real data at scale with task centric ground truth labels, like contact forces and slip, is a challenge further compounded by sensors of various form factor differing in aspects like lighting and gel markings. To tackle this, we turn to self-supervised learning (SSL) that has demonstrated remarkable performance in computer vision. We present Sparsh, a family of SSL models that can support various vision-based tactile sensors, alleviating the need for custom labels through pre-training on 460k+ tactile images with masking and self-distillation in pixel and latent spaces. We also build TacBench, to facilitate standardized benchmarking across sensors and models, comprising of six tasks ranging from comprehending tactile properties to enabling physical perception and manipulation planning. In evaluations, we find that SSL pre-training for touch representation outperforms task and sensor-specific end-to-end training by 95.1% on average over TacBench, and Sparsh (DINO) and Sparsh (IJEPA) are the most competitive, indicating the merits of learning in latent space for tactile images. Project page: https://sparsh-ssl.github.io

Sparsh: Self-supervised touch representations for vision-based tactile sensing
Carolina Higuera 122 Akash Sharma 1,* Chaithanya Krishna Bodduluri', Taosha Fan', Patrick Lancaster', Mrinal Kalakrishnan', Michael Kaess Byron Boots?, Mike Lambeta', Tingfan Wu', Mustafa Mukadam'
FAIR at Meta, University of Washington, Carnegic Mellon University Eqqual contribution
visusication
T3i Force estination DDR (a6))
T2: Sip detection
30
Digit
T&: Pass
T6: Bead mane
entimetion
Group
T4
GatSight
TS: Testile rocagniiins
T4: Grosp stubility
Sparh (M00)
Soanh (2R00)
(DWN0r))
GelSight
Conton
(VUER))
We present Sparsh, family of general touch representations, and TacBench, standardized benchmark of six touch-centric tasks ([T1J-[T6]) covering prominent problems vision-basee tactile sensing. We find Sparsh pre-trained with self-supervision on a dataset of 460k+ tactile images can generalize across many tasks (right) and sensors (left) outperforming task and sensor specific models (E2E). Performance in the plot (middle) with task decoders using 33% labeled data (except [T6] that 50%).
Figure 1:
Abstract: In this work, we introduce general purpose touch representations for the increasingly accessible class of vision- based tactile sensors. Such sensors have led to many recent advances in robot manipulation they markedly complement vision, yet solutions today often rely on task and sensor specific handcrafted perception models. Collecting real data at scale with task centric ground truth labels, like contact forces and slip, is a challenge further compounded by sensors of various form factor differing aspects like lighting and gel markings. To tackle this we turn self-supervised learning (SSL) that has demonstrated remarkable performance in computer vision. We present Sparsh, family of SSL models that support various vision-based tactile sensors, alleviating the need for custom labels through pre-training on 460k+ tactile images with masking and self-distillation in pixel and latent spaces. We also build TacBench, to facilitate standardized benchmarking across sensors and models, comprising of six tasks ranging from comprehending tactile properties to enabling physical perception and manipulation planning. evaluations, we find that SSL pre- treiiiriiig for touch representation outperforms task and sensor-specific end-to-end training by 95.1 %% average over TacBench, and Sparsh (DINO) and Sparsh (IJEPA) are the most competitive, indicating the merits of learning in latent space for tactile images. Project page:
ssl. .ii.gt itttbbioo//
Keywords: Tactile sensing, Pre-trained representations, Self-supervised learning
1 Introduction
Touch comes before sight, before speech. In today's AI landscape, this Margaret Atwood quote is playing out in reverse despite touch being crucial modality for humans to physically interact
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

with the world. Touch provides direct window into information like forces and contact during hand-object interactions, enabling dexterity. Vision-based tactile sensors [1, 2, 3, 4] have emerged as the leading form factor capable of capturing images of physical interactions at the sensor-object- environment interface, often inaccessible through vision. These images contain properties such as contact geometry, texture, and forces and have been leveraged across tasks like insertion [5, 6]. pushing [7]. grasping [8], localization [9], and pose and shape estimation [10, 11].
The prevailing approach to incorporating models using labeled data [6, 12, 13, 14] to estimate useful states. However, this can be inefficient and results in repeated effort across different type of sensors like GelSight 2017 [I] (with markers) and DIGIT [3] (without markers) or different variety of tasks. For example, feature extractors trained on GelSight with markers may not transfer other sensors, and encoders optimized for texture recognition [15] may not be suitable for tasks that require reasoning about forces slip [16].
based tactile sensors in robot tasks is to train custom
Supervision for building large general models is prohibitive as collecting large scale real world data with ground truth labels is challenging. For instance, properties like forces [17] and slip [18] require careful and expensive instrumentation in lab settings, while other properties like tracking deformations [19] or extrinsic contact [6] can be infeasible. To address this fragmentation in the literature across custom solutions, there is a need for touch representations that are broadly applicable to many tasks and many sensors, along with benchmark of standardized tasks useful in measuring progress. Taking inspiration from self-supervised learning (SSL) methods in computer vision, extend these approaches to the tactile sensing domain and build benchmark for evaluation (Figure 1).
In this work, we introduce a family of touch representations for vision-based tactile sensors trained with SSL. Specifically, we provide a recipe to adapt masking-based objectives from computer vision to the tactile domain, and train general-purpose touch encoders by curating a new Touch-Slidk dataset and existing datasets of tactile images (Figure 2), namely YCB-Slide [9], Touch-and-Go [20], and ObjectFolder [21]. Pulling together additional unlabeled data points from the existing datasets train our models total of 460k+ tactile images. Finally, we construct TacBench, benchmark consisting of six touch-centri tasks that cover the space of relevant problems on tactile properties such as force estimation and slip detection, on perception such as pose estimation and grasp stability, and on robot manipulation such as policies for solving a bead maze. Our contributions are as follows:
1. General touch representations, Sparsh pre-trained with SSL on 460k+ tactile images,
2. TacBench benchmark of standardized tasks to evaluate touch representations and models, and
3. Curation of new & existing datasets, unlabeled for SSL and labeled for benchmarking.
In evaluations on TacBench, we find that Sparsh with SSL pre-training yield on average 95. 1% improvement over task and sensor specific end-to-end models under limited labeled data budget (33%-50% of the collected amount) for any task. Additionally, we find Sparsh (DINO) and Sparsh (IJEPA) be the most competitive outperforming Sparsh (MAE), indicating the merits of learning in latent space over pixel space for tactile images.
2 Related work
Self-Supervised Learning with its success in natural language processing and computer vision, has become the new learning paradigm. In the last three years, a variety of general-purpose frame- works [22, 23, 24, 25] have been proposed for learning representations. We refer to [26, 27] for a comprehensive survey on SSL frameworks and their categorization based on pretext tasks and learning algorithms. In Appendix B, we expand on Masked Image Modeling (MIM), self-distillation, and Embeddin Predictive Architecture (JEPA), as we explore them in this study.
Traditionally, tactile sensing has relied on preprocessing tools like marker tracking and finite element method models to extract contact properties, such as shear forces [16, 28], dense normal estima- tion [29, 28], and contact area prediction [30]. From a learning perspective, a trend is to use custom encoder architectures tailored for specific tasks and sensors, which are either pre-trained or trained end-to-end [31, 12, 32, 13, 14, 6, 9]. Nevertheless, there is an increasing interest in representation learning for vision-based tactile sensors. For instance, MAE has shown effectiveness material classification and texture recognition [33]. Fine-tuning convolutional encoders for Bio RoboSkin, and GelSight performs well on fabric decomposition tasks [15]. Even ncarest-neighbor retrieval over

---

for Representation Learning
Sparch (MAE) Generation
Sparsh (DINO / DiNOv2)
(d) Sparch (IJEEPA V-JEPA)
Sei I-diiilation
Join Sern-Ennrddinng Predictive
(Sarwt, al. 3022)
DIGIT (L00) imaga))
Runckar Mlashing
Slakk Minking
Non Bork ReckyReeeeTee Malk
(ours) DVGIT (380k
Fneoder (MIT)
Student Encoder (VIT)
Teacher Encoder (VIT)
Contect Ensoder (VTT)
Target
(W])]
B+A-
head (MLP)
DIND head
Touchandd-Ge Go. (Yang, 2072)
Docodar
Centering
Recoratraction
(VIT)
Object (Gao, 2022)
Saforax
G4S gle Ma. piu eagri?"
Figure (a) We curate new existing datasets vision-based tactile sensors to train touch representations by adapting state-of-the-an SSL vision methods to tactile domain, namely (b) Masked Autoencoder (MAE) [23],
(c) DINO/DINOv2 [48, 491, and (d) Image/Vidco Joint- sentt-Emeddinng Predictive Architecture (JEPA) [50, 51]. sample more images than reported in
'Without need for labels
[20] and Object Folder [21].
pretrained representations, for the XELA [34] uskin sensor, can enable some success dexterous manipulation [35]. Crucially, the current state of standardization in learning touch representations and the wide variety of tactile sensors available, has made it challenging to develop and share pre-trained models across the research community in this domain.
Another direction is exploring the alignment of visual and tactile modalities in latent space using multimodal datasets [20, 36, 37, 38, 8, 39] and techniques like contrastive coding and cross-sensory retrieval [40, 41, 42]. yielding promising results tasks like material classification, grasp stability, and tactile-drived image stylization. However, current approaches [40, 43, 44, 42] primarily focus on texture and visual properties and overlook physical contact properties, such forces, slippage, and poses, which are essential for dexterous manipulation.
The works closest and concurrent to ours are T3 [45] and UniT [46]. T3 trains sensor-specific encoders to capture shared latent information through a shared trunk, using both the MAE objective and labeled task-specific data as supervision. UniT' VQGAN [47] model with a patch-based diseriminator for representation learning only for GelSight Mini (markers). On the other hand, we introduce family of models trained with the latest SSL. algorithms for the three most commonly used families of tactile sensors: DIGIT, GelSight 2017, and GelSight Mini. Similar to T3 and UniT, we evaluate touch representations for policy learning, however we also introduce a standardized benchmark to comprehensively evaluate representations and their ability to solve several relevant touch-centric tasks along tactile properties, physical perception, and manipulation planning.
Touch representations via self-supervised learning
Current approaches incorporating vision-based tactile sensors in robotic tasks lean custom task and sensor specific solutions. As highlighted in the introduction, this can be inefficient, and there is growing need for
touch representations that can more broadly useful. We envision the following guiding principles for such general touch representations: (i) provide performance benefits across many tasks including real-time robot manipulation, (ii) generalize across multiple types of sensors built similar operating principle, like vision-based tactile sensors, and (iii) improve performance by leveraging computation and diverse data scale without the need for manual labels. Self- supervieee learning (SSL) is promising in this regard, as it offers data-agnostic objectives based on wide-reaching concepts such as -yyystss to train generalist models. This motivates the question of whether vision techniques such as masked image modeling (MIM) [23, 50] and self-distillation [48, 49] can be extended to the domain vision-based tactile sensors.
To this end, inspired from advances in self-supervised learning (SSL) in computer vision, we introduce Sparsh, family of touch representation trained with SSL across multiple sensors such as DIGIT [3],
3

---

GelSight 2017 (with markers [1]) and GelSight Mini (without markers). The tactile domain, however, imposes several challenges that impede a straightforward application of SSL approaches from vision towards touch representations.
Tactile sensors inherently provide local information; thus, images can be ambiguous when observed independently and can vary across grasp forces, materials, and shapes. Therefore, investigate the optimal space for training SSL encoders. Specifically, we are interested in the efficiency of pixel reconstruction, latent reconstruction, or clustering approaches to learning representations in the presence of aforementioned ambiguities. We hypothesize that latent reconstruction and clustering could be more efficient in learning representations, as they focus model capacity away from fine reconstruction details [52]. Tactile images contain distractors, such as markers and light placement variations, which can significantly vary due to manufacturing discrepancies. To increase robustness distractors, we perform background subtraction for both DIGIT and GelSight Mini (markerless). This process provides the model with reference to no-contact, which conveys static shear information when a perpendicular force is applied to the elastomer. We find empirically that background subtraction helps models generalize across the same type of sensor.
To address the scarcity of labeled and even unlabeled data in the tactile domain that limits the training of large encoder models, we curate together new and existing vision-based tactile sensor datasets [9, 20, 21], totaling ~661k 1k images as illustrated in Figure 2. 70% or 462. 7k images are used for SSL pre-training which is an order of magnitude larger than any prior work on touch representations [33, 40, 43] (the rest are held out for monitoring training using online probes).
Tokenizing tactile images appropriately for SSL is important because many tasks such as slip detection and relative pose estimation require temporal reasoning. For SSL methods that operate on images, concatenate two tactile images with a temporal stride of 5 samples across the channel dimension, 4 4-5 + E 1 x For sensor operating at 60FPS, this corresponds to inference window of approximately 80 ms, the reaction time that humans need to adjust the grip force when detecting partial slip [53]. For SSL methods that operate on video (e.g. V-JEPA), we generate clips with frames at [t. - 2, t 4, f 6] E 14xhxxx3 corresponding to an inference window of ~ 100 ms. Currently Sparsh is limited by data streaming rates, and not by inference time, as the models support inference rates of upto 112FPS (measured on an Nvidia RTX3080). See Appendix C for additional details model architectures and training.
4 TacBench: Tactile sensing benchmark
We introduce TacBench, collection of touch-centric tasks, and labeled datasets for standardized evaluation for bsseed tactile sensing. We compile data for all tasks from various sensors to evaluate the generalization of representations, These tasks are categorized under three main questions.
Do the representations comprehend tactile properties? Tactile sensing informs finger-object contact interaction properties like forces and slip that are crucial for robot manipulation. In Section evaluate learned representation
estimating instantaneous normal and shear forces [T1] and visualizing foree fields [T1A] [16, 28. 29, 17], and detecting slip [T2] [16, 54, 55, 18].
2. Do the representations enable perception? Tracking and accumulating slip states is essential for tasks like finger-gaiting and in-hand reorientation [56, 57J. In Section 6, we evaluate the ability of the representations to track SE(2) pose changes the object relative to the sensor [T3] [10], prediction
of the stability of a grasp [T4] [8], and textile recognition [T5] [58].
Do the representations enable manipulation planning? Pre- Pre-trained representations can provide tactile features manipulation policy, improving training efficiency and test performance by eliminating the need to extract the states from raw sensor data. In Section 7, we design bead maze [T6] manipulation problem as illustrated in Figure (c), where the robot using tactile sensing is tasked to move a bead along curved wire.
Evaluation protocol. We adopt a frozen evaluation procedure with an encoder-decoder architecture. Specifically, we freeze the pre-trained Sparsh encoder weights and train the parameters of attentive decoder [51, 59] to assess what touch representations have captured from self-supervised pre-training alone. All tasks in the benchmark, except force field visualization and policy learning, train an

---

attentive decoder containing a cross- coss-attention module and two-layer MLP using labeled datasets from Table 3. We also include an end-to-end (E2E) baseline with identical model capacity where the same encoder and decoder probe are initialized with random weights and all the parameters (both encoder and decoder) are trained. Further, we train downstream decoders with different amounts of labeled data to evaluate task performance under progressively low labeled data regimes.
In the following sections, we describe the design, metrics and results of each task in TacBench. Additional details are provided in Appendix D, and ablations with unfrozen Sparsh encoder, encoder model sizes, and few-shot cross-sensor transfer are provided in Appendix E.
5
Comprehending tactile properties
5.1 [T1] Force estimation
Task. Force estimation is defined as the prediction of 3-axis normal and shear forces applied on the sensor's elastomer. Figure 3(a) shows our data collection setup. We use three different indenter shapes collect force-labeled data: hemisphere, sharp, and flat. Our dataset contains 75k time- aligned samples of 3-axis force measurements, end-effector poses, and tactile images from DIGIT at 60fps and GelSight at 25fps. We train the decoder using normalized force measurements scaled between 1-1, 1], supervised using L1 loss and optimized using Adam until convergence. We compare performance using the average root mean squared error (RMSE) across all three axes.
Results. GelSight Mini images are of higher resolution (HD) compared to DIGIT (320 240) resulting in smaller contact regions against the background. For this reason, we observe that when sufficient supervised data is available for DIGIT, it is possible to train a model from scratch to achieve high accuracy, but for GelSight Mini the end-to-end model does not perform well. Figure 4 (i)-(ii) shows across the board that our frozen Sparsh representations can estimate forces with low error. Specifically, we find Sparsh (DINO) to be robust even when access to labeled data is sparse, a common scenario in tactile sensing. Additional details are in Appendix D.3.
5.2 [T1A] Force field visualization
We qualitatively evaluate the representations for rendering normal and shear force fields to understand sensor-object interactions. Although obtaining a shear field for sensors with markers nowadays is trivial via marker tracking [54], is challenging and underexplored for markerless sensors. We train CNN decoder using the caasemble--ussin approach for dense predictions [60] unsupervised, since we do not have access to ground truth for markerless sensors. We frame normal field estimation as depth estimation [61] and shear field estimation as optical flow [62, 63, 64, 65]. Figure 4 (vi) shows visualizations for the top-performing model Sparsh (DINO) in [T1] that provides directional information about the relative motion of the contact patch. For instance, sliding motion (a, c, e, f), torsional slip (b), and divergence field upon contact (d). Additional details are in Appendix D.4.
5.3 [T2] Slip detection
Task. Shear and slip are closely related. Using the same setup as force estimation, we collect strokes where a hemispherical probe slides over the sensor, producing trajectories with both sticking and slipping samples. Slip is labeled using the friction cone model with empirically estimated static friction coefficient (see Appendix D.5). The dataset, with notable imbalance between no-slip and slip classes, contains 125k samples with 13% slip instances. We train two decoders: one for slip detection and another for normalized force changes (A) as we find that predicting the two correlated quantities jointly enhances slip detection. The MLP decoders use cross-entropy for slip detection and mean absolute error for A force regression, reserving 25k samples for evaluation.
We report F1 score instead of accuracy due to the imbalance in the slip labels in the dataset. Figure 4 (iii)-(vv illustrates the advantages of frozen Sparsh features trained under a JEPA paradigm for slip detection, particularly challenging for DIGIT sensor, even when using only 1% of the training dataset. In particular, Sparsh (VJEPA) achieves the highest Fl score among the models. Although all models detect slip from the 80 ms history of tactile data, Sparsh (VJEPA) benefits from a detailed temporal perspective, its encoder processes a video clip with four frames spanning this window. Sparsh backbones also show better performance than the E2E model when labeled training data is significantly reduced. Additional details are in Appendix D.5.
Results.
5

---

Tedtle (Pip))
Piste-ee
Fone Scese
(c)
Hespod
Figure 3: Real labeled data collection setup for TacBench tasks (a) [T1] Force estimation and [T2] Slip detection, (b) [T3] Pose estimation, and (c) [T6] Bead maze.
Enabling physical perception
6.1 [T3] Pose estimation
Task. Estimating object pose changes can help tasks such as tracking object drift for in-hand trans- lation [66], rotation [56. 57J, and pushing [7], among others. Given that tactile images capture local changes between sensor-object, we evaluate Sparsh representations to estimate SE( trans- formations of the object relative to the sensor. Figure 3 (b) illustrates the data collection procedure. The dataset consists of time-synchronized pairs of DIGIT observations 2, Rxxxux3 and object poses T, SE(3). T, are then preprocessed to produce relative pose changes the sensor gel as S à Ar,, Ay, A0) E SE((). We follow the eegression-by-clasiffiee
paradigm for this task [10, 66]. Relative object poses are binned into grid, capturing translations with resolution of +55m and rotations with resolution of 22 For each degree-of-freedde (DOF), we train a head to predict probability distribution over the discretized grid using cross coss-entropy loss and Adam optimizer.
Results. Multiclass accuracy reveals that E2E approaches perform well with ample data, but drastically decline when labeled data is reduced, as shown in Figure 4 (v). Small datasets make it difficult to distinguish between close categories, such as orientation changes from [0.5 1.0%] to [1.0%, 2.0%). Pre-trained representations, however, maintain good performance even with only a third of the data. In low data scenarios, decoders using tactile representations often revert to extreme values, reducing estimation resolution and accuracy. Additional details and examples are provided in Appendix D.6.
6.2 [T4] Grasp stability
Task. Grasp stability is well-studied in the tactile sensing literature for parallel jaw grippers [40, 67, 68, 69]. We evaluate whether representations aid in predicting grasp success given short history of lactile images from a single finger. Specifically, we take inspiration from [8] and adapt the Feeling of Success dataset. Each sample consists triplet of tactile images corresponding to 'before'. 'during', and 'after' grasping a set of objects. The dataset consist 64% successful grasps and 36% failed grasps. We pass the SSL model the 'before' and 'during' as tactile history. Since [8] does not specify official train/test split, create our randomized split with all objects, using approximately 8k grasps for training and the remaining 1.3k grasps for evaluation.
Results. Training with the full dataset, all models achieve similar accuracy. Sparsh (IJEPA) or Sparsh (VJEPA) reach ~ 80% classification accuracy, surpassing results from [8] that combined tactile and vision modalities as shown in Figure 4 (vii). Our model, relying solely on touch from single finger, shows competitive performance even with only 33% and 10% of the data. However, with just 80 training samples, performance drops significantly. More details are in Appendix D.7.
6.3 [T5] Textile recognition
Task. Dision-based tactile sensors are broadly used for material property recognition, since their compliant gel and high resolution cameras make them effective at discriminating different materials by surface texture [58, 39]. Specifically, we take the task definition from [58] and adapt the Clothing Dataset. The dataset consist of 4467 short video clips (10-25 frames), of robot with a GelSight 2017 (markers) grasping several types of textile (20 classes), such as leather, cotton, polyester, etc. We follow the train-test split provided in the metadata of the dataset.

---

(TA) ulimation DIGIT (1) [T] calimatice Gd5igh
[T22 Sip sstacion DIGIT [TI] Sis ddustion GaBigi
[01TT]] Poxe axiinaiin
15
014
17%
(+] [TI4] Foree 644 viaptration (DHGT)
(44 [T4] Grepp stupiitty (1)
[+4] Toxtls swagnitinn (1)
(x4 (T9) Bead More
Spant (MA6]
Spark (DNO))
Sparch (DIMDw))
Spanth (JEEAA
Sparth (NUERA
Calarce
Figure 4: Summary of results comparing Sparsh and E2E [T1]-CT6] tasks in TacBench across varying amounts of labeled data. Pre-training with SSL yields general touch representations that work across several tasks and sensors outperforming task and sensor specific models particularly under limited labeled data budget.
Results. Training an E2E specialist model for textile recognition using the full dataset can be chal- lenging, as noted in [58]. By leveraging pre-trained touch representations, as shown in Figure 4 (viti) the performance of the task can be significantly improved, even when training with only 10% of the labeled data. Sparsh (MAE) particularly effective, as it heavily relies on pixel-level features (see Appendix D.8). Additionally, we evaluate the cross-sensory ability of the representations, finding that with few samples (10-shot) Sparsh quickly adapts the downstream task DIGIT (see Appendix E.3).
7 Enabling manipulation planning 7.1 [T6] Bead maze
Task. The bead maze is children's toy to enhance fine motor skills. We adapt this task to robot policy learning, where the goal is to guide bead from end to another following the path of the wire (maze). Given small history of tactile images (..., 2,-1, 2t), and robot proprioception (...,qt (t). we train a policy to predict changes in joint angles as actions a (Aq,, Aq++1. Ag E IR7, to make progress on this task. This task is fundamentally tactile- focused, as the robot needs to react to resistance encountered by changes in the maze pattern and the subtle local movement of the bead are difficult to perceive from vision even when not occluded by the hand. A prior version of the bead maze task has been explored in robotics relying solely on tactile feedback [70]. In setup, illustrated in Figure 3 (c), assume that the robot starts with initial stable grasp. We collect a dataset of 50 demonstrations on different maze patterns with a mix of VR-based and manual kinesthetic- based teleoperation, corresponding to total of 3434 training pairs of tactile images and robot joint angles. Since we are training policies with real data, we use diffusion policy [71] for this task as it is one of the leading behavior cloning methods. For tactile observation conditioning, we replace the vision encoder in Diffusion Policy with the pre-trained Sparsh encoder.
Results. We evaluate Sparsh (DINO) and Sparsh (IJEPA) for policy learning, as these representations exhibit the best performance across the rest of the benchmark. For completeness, we also consider Sparsh (MAE), and E2E which trains a tactile encoder and policy end-to-end. Due to covariate shift [72] in behavior cloning, prediction errors can accumulate over time; therefore, we report position error between the predicted trajectory and a demonstration trajectory from held-out maze sequence over small chunks of 3em followed by the robot corresponding 15 timesteps of action predictions. Figure 4 (ix) shows the position error over access to different number of demonstrations for training. We find that Sparsh (DINO) and Sparsh (IJEPA) produce significantly (a differennee of ~ ~166) lower trajectory errors compared to training the policy E2E.
Additionally, we evaluate real rollouts of the learned policies (using all 50 demonstrations) over a set of 10 randomized novel starting locations on the maze. In Figure (ix) we report distance traversed (in cm) before failure. We find that policies using Sparsh representations outperform E2E by ~20-53%. We note that given the high precision nature of this task and the considerations for real system deployment for the policy, none of the models succeeds in completing the full maze on real
7

---

robot rollouts. We expect that increasing the diversity of training data with different maze patterns will highlight the generalist capabilities of touch representations, and that temporal ensembling will aid in improving the smoothness of the policy [73]. Additional details are in Appendix D.9.
8 Discussion
Summary. We present Sparsh, family of general touch representations trained with self supervisin for vision- based tactile sensors. We learn general purpose, curated, unlabeled dataset of 460k+ samples from DIGIT, GelSight 2017, and GelSight Mini sensors. We evaluated five SSL approaches (see Figure 2) comparing their performance against task and sensor specific models through TacBench, a benchmark of six touch-centric tasks designed to assess the content and quality of the representations. Our results indicate that Sparsh representations are performant across various sensors and tasks capturing tactile properties, and enhancing physical perception and manipulation planning.
representations from
Analysis. Overall Sparsh excels on all tasks. In particular, we find Sparsh (DINO) is well suited for physics-based tasks like force and pose estimation, while Sparsh (IJEPA) performs better at touch semantic understanding like slip state, stability of a grasp, and textile recognition. On average Sparsh (DINO) outperforms Sparsh (IJEPA) by 5.6% across the benchmark. Both models perform similarly in bead maze test demonstrations, which require implicit knowledge of shear forces and slip. However, this did not translate to real robot performance due to lack of force control and system-level confounding variables not captured during training. These include the high precision required to keep the bead in place, the impossibility of error recovery once grip is lost, and trajectory drift due to local decision-makinn rollout performance is due to the narrow task domain setting that leads to overfitting, a trend similarly observed when studying pre- pre-trainee trainediee vision models [71. 73, 74].
Specialist policies or models trained from scratch exhibiting better robot
Learning touch representations in latent space is more advantageous than in pixel space, as these representations can filter out and generalize over noise lighting differences. Tasks traditionally challenging for markerless sensors (like DIGIT and GelSight Mini), such as shear force (and field) estimation and slip detection, become solvable with our general touch representations. On average, Sparsh achieves a 95.1% improvement compared to an end-to-end approach when all models have access to only 33% - 50% of the labeled dataset per downstream task. Using as little as 10% or 1% of the labeled data for force estimation and slip detection still yields acceptable results (e.g. force error below 0. with Sparsh (DINO)). Fine-tuning Sparsh encoders is another method of assessing the quality of pre-trained representations. We provide in Appendix E. experiments with partial and full fine-tuning. Notably, models pre-trained in latent space perform better in downstream tasks when fully fine-tuned, especially in regression tasks like force and pose estimation. In contrast, partial fine-tuning offers minor improvements, aligning closely with the performance of frozen models. We also evaluate Sparsh decreasing the model capacity, finding the biggest impact in performance for regression- like tasks when training with limited amount of labeled data (see Appendix E.2).
Sparsh is significant step towards a general pre-trained backbone for vision-based tactile sensors. Our aim is to enable efforts to compile larger tactile datasets that include additional vision-bart based tactile sensors and leverage the benefits of scaling up SSL backbones, as seen in computer vision and natural language processing. TacBench serves as an initial benchmark for evaluating these representations, and additional tasks can be incorporated based on the needs of the tactile sensing and manipulation community. For instance, further exploration of pre-trained touch representations in tactile policy learning, tracking dynamic object properties like changing mass during pouring.
Limitations. Open-source tactile datasets we considered in this study predominantly feature discrete contact interactions. We helieve that incorporating data rich in shear interactions can further improve the representations. We do not ablate the length of tactile image history for learning the representations. Such ablations could provide guidance on improving their quality for downstream tasks. Our bead maze policies with pre-trained touch representations deployed on the real robot are only able to complete the maze partially before compounding error leads to the bead falling out of the fingers. Further research is needed to understand how to effectively leverage pre-trained touch representations in behavioral cloning for robot manipulation tasks.
8

---

Acknowledgments
The authors thank Ishan Misra, Mahmoud Assran for insightful discussions on SSL for vision that informed this work, Changhao Wang, Dhruv Batra, Jitendra Malik, Luis Pineda, Tess Hellebrekers for helpful discussions on the research, and Ishan Misra, Homanga Bharadhwaj, Tarasha Khurana, Rosario Scalise for feedback on the paper draft.
References
[1] W. Yuan, S. Dong, and E. Adelson. GelSight: High-resolution robot tactile sensors for estimating geometry and force. Sensors: Special Issue on Tactile Sensors and Sensing, 17(12):
2762 - 2782, November 2017.
[2] E. Donlon, S. Dong. M. Liu, J. Li. E. Adelson, and A. Rodriguez. GelSlim: A high- resolution, compact, robust, and calibrated tactile-sensing finger. In 2018 JEEE/RS In- ternational Conference on Intelligent Robots and Systems (IROS), pages 1927-1934, 2018.
doi:10.1 109/TOOS 2018.8593661.
[3] M. Lambeta, P.-W. Chou, S. Tian, B. Yang. B. Maloon, V. R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, D. Jayaraman, and Calandra. Digit: A novel design for low- cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3):3838-8855 2020. doi: 10. R.0/
[4] F. Lepora, Lin, B. Money-Coomes, and Lloyd. Digitac: A digit-tactip hybrid tactile sensor for comparing low-cost high-resolution robot touch. IEEE Robotics and Automation Letlers, 7(4):9382-9388, 2022.
[5] S. Dong, D. K. Jha, D. Romeres, Kim, D. Nikovski, and A. Rodriguez. Tactile- rl for insertion: Generalization objects of unknown geometry. 2021 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 6437-6443, 2021. URL
https://api... www.nticscaaaoaarrrr.. waap:rorrg/////oooeeeeeee 233004667.
[6] Higuera, Ortiz, H. Qi, L. Pineda, B. Boots, and M. Mukadam. Perceiving extrinsic contacts from touch improves learning insertion policies. arXiv preprint arX1v:2309.16652. 2023.
[7] J. Lloyd and N. Lepora. Goal-driven robotic pushing using tactile and proprioceptive feedback. IEEE Transactions Robotics, 3812):1201-12123 2022. doi:10. 109/TRO.2021. 3104471.
[8] Calandra, Owens, M. Upadhyaya, W. Yuan, J. Lin, H. Adelson, and Levine. The feeling of success: Does touch sensing help predict grasp outcomes? arXiv preprint arXiv: 2017.
S. Suresh, Z. Si, Anderson, M. Kaess, and M. Mukadam. MidasTouch: Monte-Carlo inference over distributions across sliding touch. In Proc. Conf. Robot Learning, CoRL, Auckland, NZ, Dec. 2022.
[10] M. Bauza. A. Bronars, and Rodriguez. Tac2Pose: Tactile object pose estimation from the first touch. The International Journal of Robotics Research, 42(11): 2023. doi: 10. 10..1117//00 110111177777783322211199999
URL 1 /tt.://............///////331113
[11] S. Suresh, H. Qi, T. Wu, T. Fan, L. Pineda, M. Lambeta. J. Malik, M. Kalakrishnan, Calandra, M. Kaess, Ortiz, and M. Mukadam. Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation. In arXiv preprini arXiv:2312. Dec. 2023.
[12] A. Church, Lloyd, Hadsell, and N. F. Lepora. Deep reinforcement learning for tactile robotics: Learning type braille keyboard. IEEE Robotics and Automation Letters, 5(4): 6145-6152, 2020.

---

[13] H. Xu, Y. Luo, S. Wang, T. Darrell, and R. Calandra. Towards learning to play piano with dexterous hands and touch. In 2022 /EEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10410-10416. IEEE, 2022.
[14] Y. Lin, J. Lloyd, A. Church, and N. F. Lepora. Tactile gym 2.0: Sim-to-real deep reinforcement learning for comparing low-cost high-resolution robot touch. IEEE Robotics and Automation Letters, 7(4):10754-1771. 2022.
[15] B. Zandonati, R. Wang, R. Gao, and Y. Wu. Investigating vision foundational models for tactile representation learning. arXiv preprint arXiv:2305.00522 2023.
[16] W. Yuan, R. Li, M. A. Srinivasan, and E. H. Adelson. Measurement of shear and slip with a gelsight tactile sensor. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 304-311, 2015. 10.11009/AA22211...7770
[17] Z. Lu, Z. Liu, X. Zhang, Y. Liang, Y. Dong, and T. Yang. 3d force identification and prediction using deep learning based on gelsight-structures sensor. Sensors and Actuators A: Physical, 367:115036 2024.
[18] C. Bulens, N. F. Lepora, Redmond, and B. Ward-Cherrier. Incipient slip detection with a biomimetic skin morphology. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8972-8978, 2023. 1.t:::..111///00555555551111 110010000000R0555550000000.111
[19] Y. Du, G. Zhang, Y. Zhang, and M. Wang. High-resolution 3-dimensional contact defor- mation tracking for fingervision sensor with dense random color pattern. IEEE Robotics and Automation Letters, 6(2):2147-2154, 2021. doi: 10.1 100.111000R...2213330666
[20] F. Yang, C. Ma, J. Zhang, J. Zhu, W. Yuan, and A. Owens. Touch and go: Learning from human-collected vision and touch. arXiv preprint arXiv:2211.12495 2022.
[21] Gao, Z. Si, Y.-Y. Chang. S. Clarke, Bohg, L. Fei-Fei, W. Yuan, and J. Wu. ObjectFolder 2.0: A multisensory object dataset for sim2real transfer. In CVPR, 2022.
[22] J.-B. Grill, F. Strub, Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your latent-a new ap- proach to self-supervised learning. Advances in neural information processing systems, 33: 21271-21284 2020.
[23] K. He, X. Chen, S. Xie, Yi P. Dollár, and R. Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16000-16009, 2022.
[24] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In H. D. III and A. Singh, editors. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020. URL https::/proceedingg/ v11 1w/111/cheee 9/chen20j. html.
mlr. wwr.ppees/vve /vs
[25] X. Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 15750-15758, 2021.
[26] U. Ozbulak, H. Lee, B. Boga, E. T. Anzaku, H. min Park, A. Messem, W. D. Neve, and
J. Vankerschaver. Know your self-super learning: A survey on image-based generative and discriminative training. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: htttps://openreview.nneee www.:net/oooriieo (iarummmdddM22541udd Survey Certification.
[27] Balestriero, M. Ibrahim, V. Sobal, A. Morcos, Shekhar, Goldstein, Bordes, A. Bardes, G. Mialon, Y. Tian, A. Schwarzschild, A. G. Wilson, J. Geiping, Q. Garrido, P. Fernandez, A. Bar, H. Pirsiavash, Y. LeCun, and M. Goldblum. A cookbook of self self-supervised learning, 2023.
10

---

[28] I. H. Taylor, S. Dong, and A. Rodriguez. GelSlim 3.0: High-resolution measurement of shape, force snip
slip in compact tactile-sensing finger. In 2022 International Conference on Robotics
and Automation (ICRA), pages 10781-10787 2022. doi: (00.1199/CRAA4469992220200
1832.
[29] D. Ma, E. Donlon, S. Dong, and A. Rodriguez. Dense tactile force estimation using gelSlim and inverse FEM. 2019 International Conference on Roboties and Automation (ICRA), pages 5418-5424. IEEE, 2019.
[30] M. Lambeta, H. Xu, Xu, P.-W. Chou, S. Wang, T. Darrell, and R. Calandra. PyTouch: A machine learning library for touch processing. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 13208-13214. IEEE, 2021.
[31] J. Hansen, F. Hogan, D. Rivkin, D. Meger, M. Jenkin, and G. Dudek. Visuotactile-rt: Learning multimodal manipulation policies with deep reinforcement learning. In 2022 International Conference on Robotics and Automation (ICRA), pages -3344. IEEE, 2022.
[32] Y. Chen, M. d. Merwe, A. Sipos, and Fazeli. Visuo-tactile transformers for manipulation. In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 2026-2040. PMLR, 14-18 Dec 2023. URL https::t/p/ootet http:::////ccceiingss. mlr wttp:/ssss.m...eseeeeeeeee pptppssess//ceeeemeeee
html.
[33] G. Cao, J. Jiang, D. Bollegala, and S. Luo. Learn from incomplete tactile data: Tactile representation learning with masked autoencoders. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10800-10805. IEEE, 2023.
[34] T. P. Tomo, M. Regoli, A. Schmitz, L. Natale, H. Kristanto, S. Somlor, L. Jamone, G. Metta, and S. Sugano. A new silicone structure for uskin-4 a soft, distributed, digital 3-axis skin sensor and its integration on the humanoid robot icub. IEEE Robotics and Automation Letters, (13):254-2559) 2018.
[35] I. Guzey, B. Evans, S. Chintala, and L. Pinto. Dexterity from touch: Self-supervised pre- training of tactile representations with robotic play. In 7th Annual Conference Robot Learning, 2023. URL https://openreviees httpe/iwwwww..eeeeeeeee mee//forumidd=Ex0eetttt
[36] R. Gao, Y. Dou, H. Li, T. Agarwal, J. Bohg, Y. Li, L. Fei-Fei, and J. Wu. The object- folder benchmark: Multisensory learning with neural and real objects. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 17276-17286, 2023.
[37] Gao, Y.-Y. Chang, Mall, L. Fei-Fei, and Wu. Objectfolder: A dataset objects with implicit visual, auditory, and tactile representations. In CoRL, 2021.
[38] Gao, Z. Si, Y.-Y. Chang, S. Clarke, Bohg, L. Fei-Fei, W. Yuan, and J. Wu. ObjectFolder 2.0: A multisensory object dataset for sim2real transfer. In Proceedings of the IEEE/CVF conference computer vision and pultern recognition, pages 1599-106088 2022.
[39] L. Fu, G. Datta, H. Huang, W. C.-H. Panitch, J. Drake, J. Ortiz, M. Mukadam, M. Lam- beta, R. Calandra, and K. Goldberg. A touch, vision, and language dataset for multi- modal alignment. In Forty-first International Conference on Machine Learning. 2024. URL https://operreview wwt.rreewwwweeeeeeeeeeeeeee nett/
[40] F. Yang, C. Feng, 7.. Chen, H. Park, Wang, Dou, Zeng, Chen, R. Gangopad- hyay, A. Owens, al. Binding touch everything: Learning unified multimodal tactile representations. arXiv preprint arXiv:2401. 18084, 2024.
[41] S. Yu, K. Lin, A. Xiao, J. Duan, and H. Soh. Octopi: Object property reasoning with large languaee models. arXiv preprint 4rX1v:2405.02794, 2024.

---

[42] V. Dave, F. Lygerakis, and E. Rueckert. Multimodal visual-tactile representation learning through self-suuprvised contrastive pre-training, 2024.
[43] Y. Dou, F. Yang. Y. Liu, A. Loquercio, and A. Owens. Tactile-augmented radiance fields. arXiv preprint arXiv:2405.0455. 2024.
[44] J. Kerr, H. Huang, A. Wilcox, Hoque, J. Ichnowski, R. Calandra, and K. Goldberg. Self- supervised visuo-tactile pretraining locate and follow garment features. In Robotics: Science and Systems, 2023.
[45] J. Zhao, Y. Ma, L. Wang, and E. H. Adelson. Transferable tactile transformers for representation learning across diverse sensors and tasks. arXiv preprini arXiv:2406. 33440, 2024.
[46] Z. Xu, R. Uppuluri, X. Zhang, C. Fitch, P. G. Crandall, W. Shou, D. Wang, and Y. She. Unit: Unified tactile representation for robot learning. arXiv preprint urXiv:2408. 06481, 2024.
[47] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[48] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pages 9650-9660, 2021.
[49] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. DINOv2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. 2023.
[50] M. Assran, Q. Duval, I. Misra, Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 15619-15629, June 2023.
[51] A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas. V-JEPA: Latent video prediction for visual representation learning. 2023.
[52] Y. LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1), 2022.
[53] A. Zangrandi, M. D' Alonzo, C. Cipriani, and G. Di Pino. Neurophysiology of slip sensation and grip reaction: insights for hand prosthesis control of slippage. Journal of neurophysiology. 126()):477-497 2021.
[54] S. Dong, W. Yuan, and E. H. Adelson. Improved gelsight tactile sensor for measuring geometry and slip. In 2017 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 137-144. IEEE, 2017.
[55] F. Veiga, H. van Hoof, J. Peters, and T. Hermans. Stabilizing novel objects by learning to predict tactile slip. In 2015 TEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5065-5072, 2015.
[56] H. Qi, B. Yi, Y. Ma, S. Suresh, M. Lambeta, R. Calandra, and J. Malik. General In- Object Rotation with Vision and Touch. In Conference on Robot Learning (CoRL), 2023.
[57] M. Yang, C. Lu, A. Church, Y. Lin, C. Ford, H. Li, E. Psomopoulou, D. A. Barton, and N. R. Lepora. Anyrotate: Gravity-invariant in-hand object rotation with sim-to-real touch. arXiv preprint arXiv:2405.0739/ 2024.
[58] W. Yuan, Y. Mo. S. Wang, and E. H. Adelson. Active clothing material perception using tactile sensing and deep learning. 2018 IEEE International Conference Robotics and Automation (ICRA), pages 1-8, 2017. URL https: / //apsi////pp..e semanticscholarrr wwrg/CorppusDD. 3543735
12

---

[59] X. Chen, M. Ding, X. Wang, Y. Xin, S. Mo. Y. Wang, S. Han, P. Luo, G. Zeng, and J. Wang. Context autoencoder for self-supervised representation learning. International Journal of Computer Vision, 132(1):208-22) 223, 2024.
[60] R. Ranftl, A. Bochkovskiy, and V. Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference computer vision, pages 12179- 12188, 2021.
[61] Godard, O. Mac Aodha, M. Firman, and G. Brostow. Digging into self-supervised monocular depth prediction. October 2019.
[62] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934-8943, 2018.
[63] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107. 14795, 2021.
[64] L. Liu, Zhang, He, Y. Liu, Y. Wang, Y. Tai, D. Luo, C. Wang. J. Li, and F. Huang. Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation. In IEEE Conference on Computer Vision and Puttern Recognition(CVPRR 2020.
[65] R. Jooschkkwski,, A. Stone, J. T. Barron, A. Gordon, K. Konolige, and A. Angelova. What matters unsupervised optical flow. arXiv preprint urXiv:2006.04902 2020.
[66] S. Kim, A. Bronars, P. Patre, and A. Rodriguez. Texterity-tactile extrinsic dexterity: Simulta- neous tactile estimation and control for extrinsic dexterity. arXiv preprint arXiv:2403.00049 2024.
[67] S. Kanitkar, H. Jiang, and W. Yuan. Poselt: visual-tactile dataset of holding poses for grasp stability analysis. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 71 71-78. IEFE, 2022.
[68] Z. Si, Zhu, A. Agarwal, S. Anderson, and Yuan, Grasp stability prediction with sim-to- real transfer from tactile sensing. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7809-7816, 2022. 11011001099R005571222222299999 1863.
[69] R. Kolamuri, Z. Si, Y. Zhang, A. Agarwal, and W. Yuan. Improving grasp stability with rotation measurement from tactile sensing. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6809-6816, IEEE, 2021.
[70] F. R. Hogan, J.-F. Tremblay, B. H. Baghi, M. Jenkin, K. Siddiqi, and G. Dudek. Finger- STS: Combined proximity and tactile sensing for robotic manipulation. IEEE Robotics and Automation Letters, 10855008661 2022.
[71] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[72] Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-22) 2244,, 2000.
[73] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705 2023.
[74] N. Hansen, Z. Yuan, Y. Ze, T. Mu, A. Rajeswaran, H. Su, H. Xu, and X. Wang. On pre- training for visuo-motor control: Revisiting a learning-from -crrttcratcchhcccccccoccccc baseline. arXiv preprint arXiv:2212.057444 2022.
13

---

[75] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:21 1.07832, 2021.
[76] 7. Xie, Z. Zhang, Y. Cao, Y. Lin, Bao, Z. Yao, Q. Dai, and H. Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9653-9663, 2022.
[77] H. Bao, Dong, S. Piao, and Wei, Beit: Bert pre-training of image transformers. arXiv preprint urXiv:2106.08254 2021.
[78] P. Gao, T. Ma, H. Li, Z. Lin, J. Dai, and Y. Qiao. Convmae: Masked convolution meets masked autoencoders. arXiv preprint arXiv:2205.088 2208.038.. 2022.
[79] W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng. Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, et al. Image foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022.
[80] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 02.55003.23.,... 2015.
[81] Tarvainen and Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi- supi-vspe deep learning results, Advances in neural information processing systems, 30, 2017.
[82] Q. Xie, M.-T. Luong. Hovy, and Q. V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
[83] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.119299 2020.
[84] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli. data2vec: A general framework for self-supervised learning in speech, vision and language. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1212-131. PMLR, 17-23 Jul 2022. URL htt.httts//prrcceddnngg mlr. ptt..pr/sss/v/////// baevski 22a. html.
[85] Z. Fei, M. Fan, and J. Huang. A-JEPA: Joint-embedding predictive architecture can lis- ten. ArXiv, 0bss22311.15830 2023. URL https://api. stw.aapp.ssmmannccchholara orw.llarrror/Coppsstt.00. 265456289.
[86] A. Bardes, J. Ponce, and Y. LeCun. MC-JEPA: A joint-embedding predictive architecture for self-supervised learning of motion and content features, 2023.
[87] A. Saito and J. Poovvancheri. Point-JEPA: A joint embedding predictive architecture for self-supervised learning point cloud. 2024. URL https: www.:essssnnnnnccsoor.ooror org/ CorpusID: 269362564.
[88] Z. Tong, Y. Song, Wang, and L. Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078-10093, 2022.
[89] S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control. In K. Chaudhuri, S. Jegelka, L.. Song, Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 77359-17377 PMLR, 17-23 Jul 2022. URL https: htthtss:: httpps/:///poeeennn....om mlr wwwpppeesvvvveee hwtpp/psppp/////// html.
14

---

[90] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, pages 416-426. PMLR, 2023.
[91] T. Xiao, I. Radosavovic, T. Darrell, and Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173; 2022.
[92] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: universal visual representa-
tion for robot manipulation. arXiv preprint arXiv:2203. 126001, 2022.
[93] Y. J. Ma, Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.000300 2022.
[94] S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language- driven representation learning for robotics. In Robotics: Science and Systems (RSS), 2023.
[95] S. Wang, M. Lambeta, P.-W. Chou, and R. Calandra. TACTO: A fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors. IEEE Robotics and Automation Letters (RA-L), 72):33930-33377 2022. ISSN 2377-3766. doi:10.1109/LRA... 101111999 2022.3 146945. 08456.
URL
[96] Z. Si and W. Yuan. Taxim: An example- baasdee--asedd simulation model for gelsight tactile sensors. arXiv preprint arXiv:2109.04022 2021.
[97] D. F. Gomes, P. Paoletti, and S. Luo. Beyond flat gelsight sensors: Simulation of optical tactile sensors of complex morphologies for sim2real learning. 2023. URL https: roboticsprocest sebboicsproceeding.. otg/rss19/p0333 pdf.
[98] Z. Si, G. Zhang, Q. Ben, B. Romero, Z. Xian, C. Liu, and C. Gan. DIFFTACTILE: A physics- based differentiable tactile simulator for contact-rich robotic manipulation. In The Twelfth International Conference on Learning Representations, 2024. URL / net net/forum? i id=
[99] Z. Chen, S. Zhang, S. Luo, F. Sun, and B. Fang. Tacchi: A pluggable and low computational cost elastomer deformation simulator for optical tactile sensors. JEEE Robotics and Automation Letters, 8(3):1239-124) 1246, 2023. doi: 10. 109/LR..2023333370000
[100] W. Chen, Xu, F. Xiang, X. Yuan, H. Su, and R. Chen. General-purpose sim2real protocol for learning contact-rich manipulation with masker-based visuotactile sensors. IEEE Transactions on Robotics, 40: 10:1550-1526, 2024.
[101] Y. Zhao, X. Jing, K. Qian, D. F. Gomes, and S. Luo. Skill generalization of tubular object manipulation with tactile sensing and sim2real learning. Robotics and Autonomous Systems, 160:104321, 2023. ISSN 0921- dotthtpss/doouorr///000000.....0 10432. URL https://ww/ www.:.scieneeireec..... wwt.///enndeeeeee..oeecc//ee/ere wtt.c//sccnccrrrrrree//////////0
[102] Si, 7. Zhu, A. Agarwal, S. Anderson, and W. Yuan. Grasp stability prediction with sim-to- real transfer from tactile sensing. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1809-7816. IEEE, 2022.
[103] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020.
[104] T. Darcet, M. Oquab, Mairal, and P. Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. URL https: // //openreview ew. nett/forum2id-22d033331.
[105] S. Shao, Z. Pei, X. Wu, Z. Liu, W. Chen, and Z. Li. lebins: Iterative elastic bins for monocular depth estimation. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Appendix
A Contributions
The contributions of the authors are follows.
Carolina Higuera led the execution of the project, designed the experiments, the data collection protocol, implemented some of the SSL pretraining methods and downstream tasks, did bug fixes, code reviews, performance evaluations, and wrote the paper.
Akash Sharma led the execution of the project, was involved in experiment design, implemented core functionality including SSL pretraining, conducted performance evaluations and benchmarking, did bug fixes, code reviews, and wrote the paper. Krishna Bodduluri collected and curated labeled data for downstream tasks such as force estimation and slip classification.
Taosha Fan implemented SSL pretraining and evaluation infra, did bug fixes, code reviews, advised on the evaluations, provided feedback and helped revise the paper. Patrick Laneaster helped with hardware setup for the bead maze task, provided feedback and helped revise the paper.
Mrinal Kalakrishnan advised on the project, supported the research team. Michael Kaess advised A.S., provided feedback evaluations and the paper.
Byron Boots advised C.H., provided feedback on evaluations and the paper.
Mike Lambeta provided insights on tactile sensing and informed model input design choices, provided feedback on experiment design and evaluations.
Tingfan Wu helped design data collection protocols, steered project engineering, provided feedback on experiment design and evaluations, and helped revise the paper.
Mustafa Mukadam led the project, set the vision and research direction, steered the team and provided guidance on all aspects of the project from approach evaluations, and helped write the paper.
B Broader related work
Self Supervised Learning. We detail recent developments in musking-based self supervipe learning approaches.
Masked Image Modeling (MIM) is the strategy of corrupting a data sample by significantly masking a portion of the sample and training a model recover the missing portion, conditioned on the corrupt sample. It has become prominent framework in SSL with the success of [23, 75]. An important design consideration here is the output space of the model for supervision, which can be either raw pixels [23, 76] or an alternative representation space [77, 78, 79, 75]. While training Masked auto-encoders is simple, these models are comparatively sample inefficient during training [50].
Self-distillation [80] the idea of training two (usually identical) networks such that a student network leams predict the output representations of teacher [81] network via a small predictor network when observing augmentations of the same data sample. It has been shown to improve performance significantly even in the case of abundant data [82]. While degenerate constant representations is concern, a common strategy is to stop gradient backpropagation [25] through the teacher network and employ momentum based weight updates [22]. A concrete instance is DINO [48] utilizing ViTs [83] as the student & teacher encoder networks. More recently DINOv2 [49] improved downstream performance significantly by combining self-distillation and MIM.
Joint-Embedding Predictive Architectures (JEPA) [52] share similarities with MIM, as both rely on masking. However, the JEPA framework conceptually prescribes two key changes: a) information restoration in a latent representation space, rather than in input space (pixels or tokens) b) prediction of latent embedding conditioned on the masking parameters. This framework has had success across various modalities, including audio [84, 85), images [50, 86), and pointelouds [87]. Notably, in this paper consider masking strategies from I-JEPA [50] and V-JEPA [51]. I-JEPA utilizes spatial block-masking strategy and V-JEPA utilizes tube- masking [88] with varying aspect ratios for learning representations efficiently in latent space circumventing decoding unnecessary pixel-level details.
16

---

Representation learning in robotics. Pretraining models for multi-task capability has become popular recently, especially after the success of self-supervised learning (SSL.) in computer vision tasks like object classification, segmentation, depth estimation, and image generation. These tasks, while typically tested on computer vision datasets, are also very common in roboties. The idea of using these pre-trained representations for robot learning was initially explored in [89]. showing that pre-trained visual representations can sometimes even be better than using ground-truth state representations for training control policies.
Generative SSL via masked image modeling (MIM) [90, 91] has shown successful transfer of pre- trained representations from in-the-wild data to real-robot scenarios, enabling basic motor skills such as reaching, pushing, and picking. Furthermore, many other works investigate contrastive learning approaches to learning general visual representations in roboties [92, 93, 44]. These methods usually employ pixel reconstruction objective based time-contrastive objective focus contrasting video clips leveraging natural language for video-language alignment.
The field has been moving towards finding general-purposs purpose representations that work well across a wide range of problems in robot manipulation learning. Voltron [94], is a framework for language- driven visual representation learning for robotics that combines both masked auto-encoding and contrastive learning techniques, focusing on multi-task performance. This model is trained to learn representations that capture both low-level spatial reasoning and high-level semantic understanding by using language supervision from human videos.
Tactile sensor simulation. Multiple simulators have been proposed for vision- based tactile sen- sors such as [95, 96, 97, 98, 99] with the hope of sim2real generalization of learned policies [100]. However, many of these methods are either limited to marker-based tactile sensors [100], or narrow tasks [101, 102). Certain other methods [40] also leverage simulated data to train multi-modal repre- sentations. However, in general we find that tactile simulators are still unable to model shadows, as well as real-world per- sensor-instancee discrepancies, hampering their potential use for representation learning.
C Touch representation and self-supervision details
To ensure fair evaluation of all models, our SSL algorithms are largely adapted from official MAE, I-JEPA, V-JEPA, DINO, DINOv2 codebases.
C.1 Training details
We train all models on 8 Nvidia A-100 (80G) GPUs. In addition to training losses, to monitor training progress, we rely on online probes. Specifically, we find that for joint embedding predictive architectures, the training losses are not indicative of model convergence during optimization; therefore, proxy metrics such as reconstruction quality are helpful. For methods, we utilize DPT [103] based decoders to decode the tactile representations back into tactile images. See Figure for some examples of tactile reconstructions from Sparsh embeddings. All encoder models trained for 150 epochs. We use Adam optimizer and use a linear rampup followed by cosine schedule as the learning scheduler. Further, we find that tuning momentum value well as the weight decay factor was important in observing training convergence without collapse. Additional information of hyperparameters is detailed in Table
Arch.
EMA decay
LR
Batch size
Sparsh (MAE)
ViT-B/14
N/A
lc-4
100
Sparsh (DINO)
VIT-B/14
0.998
le-4
150
Sparsh (IJEPA)
VIT- B/14
0.996
6.25e-4
150
Sparsh (VJEPA)
VIT-B/14
0.996
6.25e-4
150
Table 1: Training hyperparameters for Sparsh models. All models for 150 epochs with optimizer AdamW, a weight decay cosine schedule from 0.04 to 0.4, and a learning rate warmup of 30 epochs.).
C.2 Architecture details
All encoder models Vision Transformers (ViT) [83]. Although the main encoder models use ViT-B/14 as the standard architecture, following [50] small ViT as the predictor network.
17

---

Ground Truth
Reconstruction Probe
Sparsh (MAE)
Sparsh (DINO)
Sparsh (UEPA)
Sparsh (VJEPA)
Figure 5: Visualization of reconstructed tactile images using the online probe to monitor SSL training of Sparsh models.
Sparsh (MAE)
Sparsh (DINO)
Sparsh (IJEPA)
Sparsh (VJEPA)
N. parameters
86254848
86255616
86386944
86537472
FPS
104
112
112
60
Table 2: Number of parameters and inference time for Sparsh hackbomes
All the models are pretrained without [cls] token. For DINO, which decodes the [cls] token into classes, we repurpose ViT registers [104] to predict classes. In Table we report the number of parameters for each encoder and their respective inference times.
Tactile images with stride of 5 i.e., + I hxx concatenated along the channel dimension before the background is removed and reshaped to 224 224 for ViT processing. We choose stride of 5 as consecutive images are similar due to high sensor sampling rates, and to match the slip detection window in humans. Ablating the effect of the input image and patch resolution may be important for better performance and is left for future work.
C.3 Dataset splits
We use three available datasets for training Sparsh, namely YCB-Slide [9], Touch-and-Go [20] and Object Folder [37]. The YCB-Slide dataset consist of human sliding interactions with 10 YCB objects. Each object has 5 trajectories, with around 3500 frames each from DIGIT sensors with different optical characteristics (180k frames in total). For each object, we dedicate four trajectories for training and the last for validation. Touch-and-Go consists of discrete human contact interactions with in-the-wild objects, using a GelSight sensor. It consist of 140 videoclips and plain files with labels for the frames with clear contact. We use all frames (220k) in the videoclips since we do not rely on labeled data for SSL training, from which 70% is used for training and the remaining for validation. The data used from ObjectFolder consist of 81k frames of robot discrete contact interactions with objects controlled setting. We also use a train/val split of 70/30.
To complement the dataset, collected Touch-Slide with additional human sliding interactions toy-kitchen objects with the DIGIT sensor. We use 9 objects, shown in Figure and collected trajectories for each, generating 180k frames in total.
For all downstream tasks we use tactile data from real sensors/hardware (DIGIT, GelSight17, and GelSight Mini) that were not seen during Sparsh SSL training. Under our problem formulation, this allows to investigate generalization of Sparsh new sensor instances (consider the case of swapping out sensor from robotic hand due to wear-off).
18

---

Figure 6: Set of objects for collecting sliding contact trajectorics in the Touch-Slide dataset.
Similarly, all objects used for downstream tasks were not used for SSL training. For example, [T1] and [T2] tasks use a real robot arm to slide the sensor elastomer (DIGIT, GelSight Mini) against an indenter to collect force-labeled data. [T4] uses an open source dataset for grasp stability [4], which
includes data from real robot grasping over 100 unique objects using an unseen GelSight17 with printed markers during SSL training. Similarly, [T6] uses DIGIT data collected from real hardware, robot pulling and moving bead along the wire. Note that none of the data used for learning representations comes from this kind of object-robot hand interactions.
C.4 Short summary of SSL methods
In this paper, we consider three SSL paradigms, namely Sparsh (MAE), Sparsh (DINO), and Sparsh (IJEPA) & Sparsh (VJEPA).
Sparsh (MAE) is based on the principle of masked image modeling, where an encoder model tasked with learning the contextual representations of substantially masked images, such that it enables reconstruction of the masked regions via lightweight decoder. We use ViT encoder and decoder for Sparsh, and the MAE loss corresponds simply L. reconstruction loss:
[MAE
Iurget
Ieccnn
(1)
Sparsh (DINO) is based on the principle self-distillation between two identical networks, where student network learns track the output predictions of EMA teacher network. Cross-entropy loss is employed between the predictions of the student and teacher network, both of which consume different crops of the same input. Specifically, feature representations from each branch are passed through MLP head, producing probability vector over an arbitrarily chosen number of classes. These scores are normalized to produce p, and p. for the student and teacher respectively.
DINO = --En Pt log Ps
Sparsh (IJEPA) and Sparsh (VJEPA) share similarities with both masked image modeling and self- distillation. Here, we employ two identical networks termed context and target networks. The context network corresponds to a student network, which is tasked to predict the features from a EMA target network (teacher network), through a small predictor network. In this case, L2 loss over features is used enforce similarity between the two branches. Specifically, the context network observes M global masks of an image produce contextual features, which then passed through predictor to predict target network features of B, local crops of the same image Sy,- On the other hand, the target network consumes local crops of the image to produce 8y5
Gjepn ΣΣΣN
1|Sy
Su,
(3)
iEM jEB;
D
TacBench tasks and evaluation details
D.1 Labeled datasets
See Table for details on labeled data curation for TacBench tasks.
D.2 Probe details
The parameters of the model updated via EMA (target encoder for Sparsh (IJEPA) and Sparsh (VJEPA), teacher network for Sparsh (DINO) and Sparsh (DINOV2), encoder from Sparsh (MAE)) are fixed and used for evaluation. The features are pooled via attentive pooling for tasks that require global representations, such as slip detection, resultant force estimation, and classification tasks. For tasks that require dense reasoning, use DPT decoders [103] decode patch representations into full input resolution quantities such normal and shear force fields, and reconstructed tactile images. See Figure 7 for visual illustration of the probe architectures.
19

---

Task
Dataset
Sensor
Size
Collector
Label
Shear load
DIGIT
75k
Robot
3-axis force
estimation
(indenter: sphere, flat, sharp)
GelSighi Mini
75k
Robol
3-axis force
[T2] Slip detection
Shear load (indenter: sphere)
DIGIT
125k
Friction cone
[T3] Pose estimation
Object sliding
DIGIT
49k
Human
Object pose SE((2
[T4] Grasp stability
Feeling of Success [8]
GelSight 2017
9.3k
Robot
Success (yes/no)
[T5] Textile recognition
Clothing Dataset [58]
GelSight 2017
120k
Robot
Textile ID
[T6] Bead maze
Demonstrations
DIGIT
34k
Human
Joint angles
Table Datasets TacBench evaluating representations on downstream tasks.
(a)
(SSL) Pretralted
Attentive Probe
0.5
MLP
Sparsh Encoder (VIT)
0.2
N
1.2
0.55
0.18
N
0.8
Gigtt Images
laps Embesking
Learned Query
(b)
Pretrained
DPT Probe
Fusion
Emsecdrggee
Reassemblegs
Fusion
Encoder (VIT)
Embbdding,
Reassembicg
Fusion
Embedding,
Reassemble
Fusion
Hest
DIGIT Images
Output Normal and Shear Force
Figure 7: (a) Attentive probe architecture consists of cross attention layer followed by linear layer regress
resultant output quantities such resultant force slip state (b) Dense Prediction Transformer (DPT) [103] consists of multiple reassemble fusion layers decode features from intermediate layers the Sparsh backbones to produce dense outputs such as normal and shear fields
We follow attentive probing[51, 59] to assess the capabilities tactile representations on the bench- mark, as this approach allows us determine what representations capture from self-supervision alone. For most tasks except force field visualization and policy learning in the benchmark, freeze Sparsh and train cross-attention module (hyperparameters in Table 4) followed by light 2 2-layer MLP probe supervised, using the labeled dataset for each task.
Parameter
Setting
Embedding dimension
768
N heads
12
MI.P ratio
4.0
Depth
Layer normalization
Yes
Table 4: Attentive pooling hyperparameters used for evaluation protocol of representation in downstream tasks.
D.3 [T1] Force estimation
After attentive pooling, the tactile features with 768 dimensions are passed to a 2-layer MLP with 192 and units respectively. to get the 3-axis force estimations. Two independent force decoders are trained using DIGIT and GelSight Mini data respectively, using the sharp and sphere probe data during training and the flat indenter data for testing. The target forces normalized be +1.0 and scaled back after prediction. We train the force decoder using Adam optimizer with 1e-4 learning rate.
20

---

(DIND)
RNISE (F.)
RMISE (F,)
RMBE (F]
E2E
Teages
Figure 8: Friction cone of test data and RMSE (mN) for force estimation task with DIGIT sensor.
DIGIT. In Table 5 report the average RMSE 25k samples of unseen DIGIT data for the force estimation task. We report metrics for each Sparsh model and the E2E approach, under four different budgets of training data. We also provide 95% confidence interval to ground the error ranges of each model.
In Figure 8 we plot the friction cone from the test data, where the colormap represents the error in mN for each axis. Note that EZE exhibit larger errors (around 500mN) for the tangential component and they are more predominant as the normal force increases. In contrast, the top model Sparsh (DINOv2) estimates with low error (< 100mN) in general across the whole range of tangential and normal forces.
Model
Full dataset (50k)
1 1/3 dataset
1/10 dataset
1/100 dataset
39.34
61.42
98.22
187.51
E2E
[39.21, 39.48]
[61.12, 61.72]
[97.61, 98.84]
[185,51, 188.51]
36.61
45.96
58.55
115.39
Sparsh (MAE)
[36.51, 36.71]
[45.80, 46.12]
[58.31, 58.79)
[1 14.69, 1 116.09]
36.09
44.03
51.89
97.95
Sparsh (DINO)
[36.01, 16.
[43.87, 44.19]
[51.69, 52.10]
[97.66, 98.52]
29.31
26.85
37.66
185.86
Sparsh (DINOv2)
[29.14, 29.46]
[26.70, 26.99]
[37.45, 37.86]
[184.94, 186.78]
40.27
60.04
86.57
130.37
Sparsh (IJEPA)
[40.16, 40.38] 39.38
[59.72, 60.34]
[86.06, 87 .08
[129.59, 131.
56.34
76.11
130.83
Sparsh (VJEPA)
[39.30, 39.47]
[56.07, 56.62]
[75.67, 76.55]
[130.29, 131.38]
Table 5:
Root Mean Squared Error (mN) and 95% confidence interval for force cstimation with DIGIT data. All models were evaluated on flat indenter data over 25k test samples.
GelSight. In Table 6 we report the average RMSE over 25k samples of unseen GelSight data and the corresponding 95% confidence interval. Notice from Figure 9 that the majority of errors are localized around the dynamic shear region. worth noting that the errors associated with Sparsh (DINO) remain below 150NN, whereas E2E exhibits higher errors, particularly in the estimation of normal forces.
D.4 [T1A] Foree field visualization
Since rendering the force field is dense prediction task, we do apply the attentive probing protocol. Instead, follow DPT [60], training CNN encoder with massemble--usion modules at layers 2,5,8,11 of the Sparsh encoder progressively upsample the representations to obtain
21

---

Model
Full dataset
1/3 dataset
1/10 dataset
1/100 dataset
57.21
59.09
57.43
82.42
E2E
[56.44, 57.98]
[58. 15, 60.04]
[56.44, 58.42]
[80.98, 83. 83.86]
22.72
23.28
33.56
78.98
Sparsh (MAE)
[22.27, 23.17]
[22.83, 23.72]
[33.04, 34.08]
[77.74, 80.21]
20.25
23.79
32.17
53.43
Sparsh (DINO)
[19.85, 20.65]
[23.40, 24.18]
[31.67, 32.67]
[52.69, 54, 14]
37.30
37.79
45.86
105.95
Sparsh (DINOv2)
[36.71, 37.88]
[37.22, 38.37]
[45.14, 46.59]
[104.28, 107.62]
27.91
35.20
44.93
91.81
Sparsh (IJEPA)
[27.37, 28.44]
[24.57, 35.82]
[44.13, 45.73)
[90.76, 92.86]
33.26
34.07
42.35
80.36
Sparsh (VJEPA)
[32667, 33.84]
[33.39, 34.75]
[41.60, 43.10]
[79.26, 81.47]
Table 6: Root Mean Squared Error (mN) and 95% confidence interval for force estimation with GelSight Mini data, All models were evaluated on flat indenter data over 25k test samples.
Sparsh (CINO)
RASE (%g)
RMSE (K)
E2E
Figure 9: Friction cone of test data and RMSE (mN) for force estimation task with GelSight sensor.
a fine- nuueggained prediction of the force field. After the reassemble-fusion modules, we attach two task-specific task head, for normal and shear field prediction.
Since for markerless dision-based sensors it is not trivial to get ground truth of the force field, turn to unsupervised learning. Depth estimation and optical flow are analogous to the estimation of normal and shear force fields, areas where the computer vision community has proposed several unsupervised methodologies [62, 63, 64, 65, 61]. We borrow ideas of unsupervised monocular depth estimation, where from two tactile images Ie and It-ns we learn pose estimator for getting the transform between frames. With the sensor intrinsic K, we map image I, from pixel space to camera plane, translate estimated depth D,, apply transform from to t n, and transform back to image plane to get 7,-,. We supervised based on the reprojection error, MSE between I,-,. and predicted 1,-n- To reconstruct the shear field, we transfer ideas from unsupervised optical flow, where warp the features of image I, to J,-n based on the estimated flow and compute a photometric consistency loss that encourages the estimated flow(shear) to align image patches with a similar appearance. This loss is a linear combination of the Charbonnier loss and the structural similarity (SSIM) between I:-n and I,-n. We also add a smoothness loss that acts as a regularization term, encouraging the shear field to align the boundaries with the visual edges in the tactile image. In Figure 10 we show snapshots of the normal and shear field predictions during sliding trajectories of the DIGIT sensor on YCB and spherical probe objects.
22

---

Canrp
Torm
(+)
(x)
(b)
Norral Foor (pomskad
efgh
Figure 10: Normalized tactile flow (unitless) visualizations using Sparsh (DINO). Top row shows predicted force field for four key-frames from a representative YCB-Slide trajectory and bottom row shows interaction with the spherical probe. Arrows represent the tangential forces, while the colors depict the normal forces. These visualizations provide directional information about the relative motion the contact patch. For instance (a) shows torsional motion resulting from rotating along the edge, (b, d) show sliding the edge, (e) shows diverging field when making contact with spherical probe, and (f, g, h) show forces produced by sliding the probe top-down.
(b)
Slip over time
Friction Cone
Mecca Bohor Art
Tel 2 seg Somgere 11
(Digh)
Probe
Hecapod
Figure 11: (a) Data collection setup for [T1] Force Estimation and [T2] Slip Detection. The Mecca Robot Arm with DIGIT Gelsight is pressed against static prohe with random normal force. arm slides sensor over the prohe which induces shear forces. (b) Slip states over one representative stroke. When the sensor is pressed against the probe the normal force increases. The gel sensor initially resists sliding due to friction. but gives in, which results in slight drop in normal force while the magnitude of shear force increases.
D.5
[T2] Slip detection
To collect labeled slip data we perform a normal/shear load test. Using a firmly affixed hemispherical probe on a flat surface, a robot presses the DIGIT sensor toward the probe, applying random normal forces of up to 5N. Upon reaching the target normal force, the robot slides the probe 2mm capture the shear profile with F/T
randomly selected position the sensor surface, allowing sensor. To label slip, rely the friction cone to identify samples the sticking and slippage regions. A description of the procedure is illustrated in Figure 11.
As cluded to in Section 3, Sparsh's inference window is approximately 80 milliseconds. This appropriate since this duration matches the reaction time needed by humans to adjust the grip foree when detecting partial slip [53]. We train two heads: one for slip detection and the other for the estimation of normalized force change (A). We find empirically that training both heads simultancously improves slip detection, given their high correlation. The MLP probes trained with cross-entropy for slip detection and mean absolute error for A force regression as loss functions. Our dataset comprises 125k samples, with only 13% corresponding to slip instances. We reserve 25k samples for evaluating model performance.
Table 7 provides Fl-score metrics for all models under different amounts of training data. Sparsh (VJEPA) outperforms all models, even when trained under low data regimes. In Figure 12 contrast the predictions over time for sample trajectory between Sparsh (VJEPA) and E2E models trained
23

---

Sparsh (VJEPA)
Farion Bonday Ao
1/
20
Cound
Facton Bondlyy No
Figure 12: Contrast between Sparsh (VJEPA) and EZE for test trajectory with spherical probe sliding the DIGIT sensor. Sparsh (VJEPA), though trained only on 33% of the data, can detect slip accurately, which correlated with ahility to estimate changes normal and shear forces.
with 33% of the data. Note that for Sparsh (VJEPA) the errors are around the friction boundary, where the probe is starting to slide. Also, it is worth noticing that a poor estimation of changes in shear and normal forces reflected the accuracy of distinguishing between slip and no-slip. In Figure 13, we illustrate a failure case for Sparsh (VJEPA), its results do align with the ground truth. However, it is important note that slip labeling is prone to errors due to its reliance coefficient of friction. Despite the inaccuracies in the friction boundary for this trajectory, Sparsh (VJEPA) successfully detects the slip samples.
experimental
Model
Full dataset
1/3 dataset 1/10 dataset
1/100 dataset
E2E
0.767
0.238
0.299
0.214
Sparsh (MAE)
0.783
0.818
0.691
0,269
Sparsh (DINO)
0.685
0.561
0.548
0,489
Sparsh (DINOv2)
0.687
0.601
0.561
0,243
Sparsh (IJEPA)
0.776
0.791
0.775
0.726
Sparsh (VJEPA)
0.820
0.828
0.800
0.760
Table 7: Performance of models on slip detection task under different budgets of training data. We use Fl score metric, given that it ensures the model accurately identifies slip events without favoring the majority class. high score indicates effective and reliable slip detection.
D.6
[T3] Pose estimation
We collect dataset trajectories with time-synchronized pairs of object pose measurements and sensor observations using Allegro hand equipped with DIGIT sensors each finger, mounted on robot arm. The object was placed on table and with the palm facing downward, against it with the fingertips (see Figure 3). We manually perturbed the object's 's pose by sliding and rotating it under the Allegro fingertips. The pose of the object was tracked using ArUco tags. Given ground truth object pose measurements in the world frame, we preprocess them into relative pose change (Ax, Ay, A0) E SE(2) in the sensor frame.
pressed
Since follow regression-be basoonn-b--casaa classssiiiiattonn approach, discretize the range of motion for each degree of freedom into multiple intervals in Log-uniform space. This allows to achieve better

---

Sparsh (VJEMA)
8m0
8:31
Figure 13: Failure case where the ground truth does not reflect slip since relies experimental coefficient of friction. Despite the inaccuracies in the friction boundary for this trajectory. Sparsh (VJEPA) successfully detects slip samples.
data distribution across all classes, as most pose changes are concentrated around zero. The strategy of clasiffication-regression is also commonly explored for monocular depth estimation [105].
After attentive pooling, the features are passed to three heads, one for each degree of freedom. Each head is an MLP with two layers, which outputs the probability distribution over 11 classes (pose change bins). In Figure 14 we present the binning well as the confusion matrices test data for each degree of freedom, comparing E2E, Sparsh (DINO) and Sparsh (IJEPA) for pose estimation when trained on 33% of the available labeled data. Note that Sparsh can accurate distinguish pose changes in a low data regime, while a conventional task-specific approach struggles discerning the differences between adjacent bins, and finally tends to default to zero or maximum relative pose change, losing resolution in estimation.
Figure 15 shows test trajectory over time with its ground truth labels. The colors the plot represent the class agreement between the pose decoders trained with Sparsh (DINO) (using 33% of the data) and the ground truth. Darker colors indicate no error, while brighter colors indicate greater misclassification. In Table we report for each model accuracy in pose estimation over 630 test samples and 95% confidence interval.
Model
Full dataset
1/3 dataset
1/10 dataset
1/100 dataset
0.812
0.245
0.162
0.162
E2E
[0.811, 0.813]
[0.244, 0.247]
[0.160, 0. 0.164]
[0.160, 0.164]
0.896
0.719
0.417
0.223
Sparsh (MAE)
[0.896, 0.897J
[0.718, 0.721]
[0.414, 0.420]
[0.221, 0.225]
0.913
0.834
0.460
0.242
Sparsh (DINO)
[0.912, 0.914]
[0.832, 0.836]
[0.457, 0.461]
[0.240, 0.245]
0.665
0.565
0.411
0.210
Sparsh (DINOv2)
[0.658, 0.673]
[0.559, 0.570]
[0.408, 0.415]
[0,209, 0.211]
0.851
0.601
0.323
0.212
Sparsh (IJEPA)
[0.850, 0.852]
[0.599, 0.603]
[0.321, 0.325]
[0.210, 0.215]
0.856
0.648
0.368
0.228
Sparsh (VJEPA)
[0.854, 0.857]
[0.646, 0.651]
[0.367, 0.370]
[0.225, 0.231]
Table 8: Accuracy and 95% confidence interval for pose estimation task following the regression-hy-
classification paradigm. Relative pose between object and ring finger. Metrics computed over 630 test samples.
D.7 [T4] Grasp stability
We use the Feeling of Success dataset [8], which contains data from a pair of GelSight sensors (with markers) attached to a jaw gripper (left and right fingers). The goal is to determine the success or the failure of the grasp attempt.
We pass to the SSL model the 'before' and 'during' tactile history. We create our randomized split with all objects, using approximately 8k grasps for training and the remaining 1.3k grasps for
25

---

E2E
144.04
Sprsi (D140)
8.00
BSC
Spants (UEPAA)
440
140
P4.05
644
$4.8
24.00
Figure 14: Confusion matrix test data for AT,, AT,, Yaw for E2E, Sparsh (DINO) and Sparsh (IJEPA) trained on 33% of the available labeled data. The test dataset consist of 630 samples.
evaluation. Using attentive probing, we freeze Sparsh and train a 2-layer MLP with two output units for grasp success classification.
In Table 9 report the accuracy for binary classification to compare the performance of the models across different training budgets, including a 95% confidence interval. Figure 16 shows the confusion matrices on test samples for E2E, Sparsh (DINO) and Sparsh (IJEPA) trained on a 33% of labeled data.
D.8 [T5] Textile recognition
This tasks allows to study the capabilities of the representations for semantic understanding of the contact, as in recognizing the type of textile that is being touched by the sensors. We use the task definition and the data set introduced in [58]. This data set contains 4467 short video clips (10-25 frames), of robot with a GelSight (markers) mounted parallel gripper grasping several types of clothing, across 20 textile classes, such as leather, cotton, polyester, etc.
We follow the train-test split provided in the metadata of the dataset. Using attentive probing, we freeze Sparsh and train a 2-layer MLP with 20 output units for textile classification. In Table 10 and Figure 17(c) we report the accuracy for multiclass classification, comparing the performance of the models in different training budgets.
D.9 [T6] Bead maze
The goal in bead maze is to guide the bead along the wire, as shown in Figure 3. We don't 't rely on vision for hand-eye coordination, making the task fundamentally tactile since forces in the fingers indicate whether the bead is moving smoothly or encountering resistance. In our setup, we use
26

---

A 1x
(50.-30)
(mmn)
0.0. 2.0
[20. 1.0)
[80, 5.0)
>=5.
120.1100
(mm)
+1.0.
10.1, 0.1)
10 11)
[1.0 2.0)
[200 100
[30, 5.0)
<:2.0
220, 1.0)
(10)
(Dagness)
0.2)
[0.22 -0.01
[0.03.
>-120
No
Worse misclasalication
Figure 15: Ground truth relative pose classes for Tr. T3 and Yaw for test trajectory. The colormap represents the class agreements between the ground truth and the pose decoder, with darker colors indicating error and brighter colors indicating greater misclassification.
Model
Full dataset
1/3 dataset
1/10 dataset
1/100 dataset
0.784
0.725
0.682
0.478
E2E
[0.783, 0.785]
[0.722, 0.728]
[0.680, 0.684]
[0.472, 0.482]
0.815
0.696
0.764
0.466
Sparsh (MAE)
[0.813, 0.817]
[0.691, 0.702]
[0.761, 0.768]
[0.461, 0.471]
0.780
0.706
0.773
0.473
Sparsh (DINO)
[0.777, 0.782]
[0.702, 0.710]
[0.772, 0.775]
[0.467, 0.479]
0.770
0.770
0.699
0.543
Sparsh (DINOv2)
[0.767, 0.771]
[0.768, 0.772]
[0.697, 0.701]
[0.539, 0.546]
0.802
0.782
0.768
0.598
Sparsh (IJEPA)
[0.800, 0.804)
[0.779, 0.784]
[0.766, 0.770]
[0.597, 0.601]
0.809
0.702
0.743
0.523
Sparsh (VJEPA)
[0.805, 0.813]
[0.700, 0.704]
[0.740, 0.746]
[0.519, 0.527]
Table 9:
Accuracy and 95% confidence interval for grasp stability classification over different budget sizes of training data, using Feeling of Success dataset. Results over 1.3k grasps.
a Franka arm with a robotic hand mounted on the wrist and DIGIT sensors on the fingers. To collect demonstrations for training the policy, we start the task with the bead grasped between the thumb and index fingers and move the arm guide the bead along the wire. We collect 30 demonstrations different maze patterns with mix of VR-based and manual kinesthetic- teleoperation, corresponding to total of ~44 training pairs of tactile images and robot joint angles.
For training the policy, we adapt Diffusion Policy [71] to our problem setting. Given a small history of tactile images (..., Zt-1. 2 and robot proprioception (..., (t), we train the policy to predict changes in joint angles as actions A (Aq, Aqu+1, ...)): Aq E R instead of position control. Following the guidelines in Diffusion Policy, we use an observation horizon of and action prediction horizon of 8. We adhere to the official implementation for policy architecture and training hyper-parameters For conditioning tactile input, modify the CNN encoder from Diffusion
27

---

S (DINO)
Sporth (UEPA)
Figure 16: Confusion matrix test data for grasp stability, comparing E2E, Sparsh (DINO) and Sparsh (IJEPA)
trained on 33% the available labeled data. The test dataset consist of 1.3k grasps.
Model
Full dataset
1/3 dataset
1/10 dataset
1/100 dataset
E2E
01.477
0.365
0.373
0.171
Sparsh (MAE)
0.599
0.588
0.527
0.330
Sparsh (DINO)
0.527
0.520
0.463
0,264
Sparsh (DINOv2)
0.544
0.536
0.469
0,288
Sparsh (IJEPA)
0.506
0.478
0.399
0.217
Sparsh (VJEPA)
0.580
0.545
0.507
0.285
Table 10: Accuracy textile classification over classes using GelSight with markers dataset under different budget of labeled data. Results over 26k tactile images, where accuracy of chance is 0.05.
Policy and replace it with Sparsh backbones with fixed parameters. For training an end-to-end policy, the encoder corresponds ViT-Base encoder with randomly initialized weights.
For each method, evaluate the learned policies on the maze and we measure distance traversed (in cm) before failure. In Table 11, we report mean and variance of distance traversed comparing Sparsh (pre-trained only and pre-trained then fully fine-tumed) against E2E. All models use 50 demonstrations for training the policy via imitation learning. We find that policies using Sparsh representations outperform E2E by ~ failure cases across methods are due to the bead getting stuck on the maze or the bead falling out of the robot hand. While prior work such as diffusion policy suggest that frozen pre-trained models may hurt imitation learning due to domain mismatch, in this application. Leveraging pre-trate pre-trained models in imitation learning is an active area of research, however these results demonstrate the impact of Sparsh touch representations for robot applications.
set of 10 randomized novel starting locations
53%. Most
do not observe significant gains from fine-tuning
(cm) Sparsh (DINO)
Sparsh (IJEPA)
Sparsh (MAE)
E2E
Pre-trained
10.80 3.68
3.1
10.2 = 4.9
6.70 1.67
Fine-tuned
8.45 - 3.21
10.02 + 5.37
11.25 = 3.85
6.70 + 1.67
Table 11: Mean and variance distance traversed (in em) before failure policies based on Sparsh and E2E. Results over 10 randomized novel starting locations on the bead maze.
In Table 12 we report to position error of E2E, Sparsh (DINO) and Sparsh (IJEPA) with respect to test demonstrations on an unseen maze, highlighting the fidelity of Sparsh (DINO) and Sparsh (IJEPA) to follow a similar trajectory. Nevertheless, this doesn't necessarily transfer to real-world performance, since the locality of the observations and predictions make the errors in the adjusted joint angles compound fast, which results in unforeseen collisions and the subsequent lose of the grasp. In overfitting setting, training policy for single maze, policies using Sparsh (DINO) and Sparsh (IJEPA) able to complete almost 30% of the maze the real robot. However, it is expected an specialist policy trained end-to-end to perform better in the overfitting setting. Experimentally, we found than an E2E policy trained for single maze able to complete almost 80% of the maze running on the real robot.
In Table 13 summarize the performance of Sparsh across the benchmark. We find that with respect to E2E approach, with Sparsh Sparsh (DINO) and Sparsh (IJEPA) are in general the best models across the board, showing the
achieve improvement of 98. 75% average.
28

---

benefits of learning touch representations in latent space. Sparsh (MAE), which relies on pixel space supervision, is still comperitive, although it was not evaluated on the policy task.
Model
Full dataset
1/2 dataset
1/10 dataset
8.46
7.14
9.80
Sparsh-(E2E)
[7.61, 9.32]
[6.26, 8.05]
[8.78. 10.82]
5.54
5,98
5.71
Sparsh-(DINO)
[4.90, 6.17]
[5.29, 6.67]
[5.13,6.29]
5.47
5.72
5.46
Sparsh-(UJEPA)
[4.82. 6.13]
[5.05, 6.40]
[4.82, 6.10]
Table 12: Position error (mm) and 95% confidence interval for the Bead Maze task. We compare the ground truth trajectory from test demonstration in unseen against the compounded trajectory from the predicted delta joint angles from each policy.
Task
Best SSL vs E2B
DINO IJEPA
MAE Best
VJEPA Best
Force cstimation (DIGIT)
28.31%
26.67%
-4.88%
-27.96%
Force estimation (GelSight)
59.74%
32.41%
1.72%
-64.23%
Slip detection
242.70%
29.08%
-1.21%
0.00%
Pose estimation
235.89%
37.91%
13.81%
22.33%
Grasp stability
5.14%
8.45%
10/17%
-7.83%
Bead maze
19. 19%
5..6%
Average
98. 75%
8.91%
5.57%
21.17%
Table 13: Performance of Sparsh across TacBench and comparison between SSL approaches.
E Sparsh ablations
E.1 TacBench evaluations via fine-tuning
Fine-tuning the Sparsh encoders is another method of assessing the quality of pre-trained representa- tions. Fine-tuning can potentially enhance performance in downstream tasks when the pre-trained model lacks relevanlevant information.
We evaluated both the full and partial fine-tuning of Sparsh on TacBench. In full fine-tuning, all encoder parameters are updated through task supervision. In partial fine-tuning, we update only the last transformer block of the encoder. Figure 17 shows the fine-tuning results in the benchmark with varying amounts of labeled data. Notably, models pre-trained in latent space (DINO, I-JEPA, V-JEPA) perform better in downstream tasks when fully fine-tuned, especially in regression tasks like force and pose estimation. For example, Figure I 17(a) illustrates that errors in force estimation are significantly lower with full fine-tuning, even with only 33% and 10% labeled data. Full fine-tuning also enhances performance classification tasks such as slip detection, grasp stability, and textile classification, as shown in Figures 17(b,c). Adding in-domain data the encoder reduces performance gaps in the benchmark between Sparsh (DINO), Sparsh (IJEPA), and Sparsh (VJEPA). However, this method is less effective for the Sparsh (MAE) model, which is trained in pixel space. We hypothesize that MAE weights are potentially more brittle when compared to other SSL models which enjoy a wider basin of minima due to weight updates via exponential moving average. In contrast, partial fine-tuning offers minor improvements, aligning closely with the performance of frozen models.
E.2 Sparsh ViT-small and performance
We train Sparsh for all SSL approaches decreasing the model capacity by using transformer ViT- small. This let us study the effect of the dimensionality of the touch representations downstream tasks, from 768 with ViTbase to 384 with ViTsmall. We follow the same training procedure explained in Appendix C.
We evaluate Sparsh-vitsmall across TacBench. In Figure 17 we report the performance of each task for different budgets of labeled data following the attentive probing protocol. Reducing the dimensionality of the representations do plays important role for tasks. Regression-lie like tasks such [T1] Force estimation (see Figure 17a) exhibit decrease in performance when reducing the
29

---

capacity of the encoders. specially when the downstream tasks needs to be trained under limited number of labeled data. For instance, Sparsh (DINO) increases the force estimation error by 74% for DIGIT and 50.3% for GelSight Mini when using representations from Sparsh-vitsmall and training the downstream tasks with 33% of labeled data. The decrease in performance is also observe in regression-by sarriiionnn---lssssi byclassiffication tasks, as in [T3] Pose estimation. With Sparsh-vitsmall all models perform very similar but losing 20% accuracy even when training the downstream tasks with the full labeled dataset. Nevertheless the performance is still better than an E2E model with vitbase encoder.
In general for classification tasks in the benchmark like [T2] Slip detection, [T4] Grasp stability and [T5] Textile recognition, there is no major effect of reducing the capacity of the encoder. The drop in
performance is only significant when training the downstream task with the lowest amount of training data, 1 1% in our experiments.
Sparsh VIT base (froen fnatures)
Sparsh VIT base (finsttrrnng all ancoder params)
Sparch VIT base (line-tuning last. block encoder)
Sparsh VIT small (fross features)
(0 [T1] Force extmaatin- DIGIT (4)
400
00052 (mnk)
600
-00
100
100%
100%
(6) [13] Force estimation Geliight Mini (4)
DU0SE [mN]]
1:00
60-
40
10%
33%
100%
30%
33%
100%
175
10%
33%
100%
1%
10%
33%
100%
EZE
Sparsh (MAE)
Sparsh (UINO)
Sparsh (UEPA)
Sparsh (VJEPA)
Chance
(a) Additional evaluations for TI force estimation.
Sparsh ViT-base (frozen features)
Sparsh ViT-bose
Sparch VIT base (fine tuning last block encoder)
Sparsh ViT-small (frczen fectures)
(fne tuning encoder params)
(ii) [TR) Slip detection DIGIT (1)
0.3
0.2
100%.
100%
(1)) [T2) Slip detectionn GalSight Mini (1)
F1500m
0.4
10%
35%
100%
10%
335
100%
15
10%
33%
100%
15k
10%
35%
100%
22E
Sparsh (MAE]
Sparsh (DINO)
Sparch (UEPA)
Sparsh (VJEPA)
Charce
(b) Additional evaluations for T2 slip detection.
30

---

Spansh ViT-base (frozen festures)
Sparsh ViT-base (Fine encoder porams)
Sparsh ViT-baae (fine block encoder)
Sparsh ViT-imall (frozen features)
(+) [Ta) Posa actimation IIGIT (1)
1.0
0.8
0.6
0.2
146
10%
33%
100%
186
10%
35%
100%
19
10%
33%
100%
156
10%
33%
100%
(v) [T4] Grasp stabiitt GolSight17
0.0
Sccure
0.6
15
136
1055
35%
33% 102%
(vi) [T6]
GelSight17 (1)
0.5
64
0.2
1%
10%
33%
100%
196
10%
33%
100%
155
10%
33%
100%
15%
10%
33%
100%
Sparsh (MAE)
Sparch (DINO)
Sparsh (UEPA)
Spans (VJEPA)
(c) Additional evaluations the perception tasks, T3 pose estimation (top), grasp stability (middle) and T6 textile classification (bottom).
Additional evaluations of Sparsh representations TacBench. We compare frozen Sparsh ViT-base (most left), Sparsh fully and partially fine-tuned (middle) and finally (most right) Sparsh ViT-small gauge the effect of reducing the dimensionality of the representations.
Figure 17:
E.3
Sparsh and cross-sensory representation
Since Sparsh is trained on multiple GelSight-lik data, we investigate whether SSL training enables cross-sensory representations
it helps downstream tasks trained for sensor quickly adapt another. To study this, we use a baseline the decoder trained for [T5] Textile recognition, which was supervised with labeled data from GelSight with markers.
We collect new data using a DIGIT sensor for 10 out of the 20 textiles. Our dataset contains 11 samples for both training and testing. We load the trained decoders for [T5] using Sparsh (DINO) and E2E and perform zero-shot evaluation as well 1-shot, 5-shot, and 10-shot training and subsequent evaluation using the DIGIT data. Table 14 reports accuracy 110 samples of test data. Zero-shot evaluation with DIGIT performs close to chance, while with very few samples (10-shot) Sparsh (DINO) classifier quickly adapts and significantly outperforms E2E. This experiment empirically demonstrates the value of cross-sensor representations.
7ero-shot
I-shot
5-shot
10-shot
Sparsh (DINO) E2E
9.1
19.1
28.2
61.8
3.6
0.0
15.5
10.9
Table Accuracy of æ-shot evaluation of [T5] Textile recognition on DIGIT data to study how Sparsh facilitates cross-sensory adaptation to the dowsntream task.
31


Reviews:
[{"review_id": "nT0hiqAzL3", "reviewer": "Reviewer_brZ3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents a self-supervised learning (SSL) approach to touch representation learning for vision-based tactile sensors, addressing challenges in task-specific models and data collection. The authors introduce Sparsh, a suite of three SSL models (MAE, DINO, and IJEPA) pre-trained on over 460k tactile images from three types of visuo-tactile sensors using masking and self-distillation, eliminating the need for custom labels. They also develop TacBench, a standardized benchmarking platform with five tasks. Results evaluated by splitting datasets into training/test sets show that SSL pre-training outperforms end-to-end training by an average of 98.7% on TacBench task datasets, with Sparsh (DINO) and Sparsh (IJEPA) models demonstrating superior performance.", "summary_of_recommendation": "The work addresses the interesting problem of visuo-tactile feature learning for sample-efficient task learning. However, some critical details about how to use the learned representation for task learning are missing. Additional experiments demonstrating how the proposed method performs in manipulation planning would highlight its potential impact.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "See Review above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "noZlF5PKUB", "reviewer": "Reviewer_EZmQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces Sparsh, a family of self-supervised learning (SSL) models that support various vision-based tactile sensors. It addresses the challenge of task and sensor-specific models by pre-training on a large dataset of over 460,000 tactile images, enabling general-purpose touch representations without custom labels. The paper also presents TacBench, a standardized benchmark with five touch-centric tasks to evaluate these models. Evaluations show that Sparsh models significantly outperform task-specific end-to-end training, with Sparsh (DINO) and Sparsh (IJEPA) demonstrating the most competitive performance. Additionally, the paper curates a unified dataset from new and existing sources, enhancing the scope and applicability of tactile sensing research.", "summary_of_recommendation": "The paper presents a practical approach for touch representations. The idea is interesting to the community and the presentation is well organized. Hence, I recommend accepting the paper.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Issues:\n1.\tAre the results presented in Figure 1 (middle) the same as those in Figure 4? What is the amount of labeled data in Figure 1? This information should be included in the caption as the outperforming only exists under a limited labeled data budget.\n2.\tAs a benchmark, the end-to-end (E2E) approach is not well explained, and the reviewer failed to find the differences between E2E and the proposed Sparsh models in the paper. In lines 160-162, it seems E2E has identical model capacity. What are the differences? \n3.\tIn line 148, predicting the stability of a grasp should be T4 instead of T2.\n4.\tIn line 277, what is behavior closing?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nd7EuDj9GL", "reviewer": "Reviewer_B6vw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents Sparsh, a self-supervised learning model family for vision-based tactile sensors, addressing the limitations of task-specific handcrafted models. By pre-training on over 460k tactile images with masking and self-distillation in pixel and latent spaces, Sparsh eliminates the need for custom labels. The authors introduce TacBench, a benchmarking suite for evaluating tactile properties and manipulation across different sensors and models. They found out that DINO and I-/V-JEPA are the most dominant approaches.", "summary_of_recommendation": "The paper tries to solve a completely reasonable problem and it can also prove to be very effective in this domain. However, certain parts of the paper requires revision as described in the Weaknesses. As of now, I am inclined towards Weak Reject but I  am open to reconsider my score if the authors provide a satisfactory rebuttal addressing these concerns.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "**Questions:**\n1. Is any sensor-specific detail provided to your encoder during training, similar to the approach in [1]? If not, why not? Do you think it can improve the training and generate sensor dependent representations?\n2. In the results section (Section 6.2), it is mentioned that classification relies solely on single finger data. How is it possible to predict grasping success or failure from just this data? Does this imply that the dataset has some inherent linear relationship with grasping success, such as greater deformation correlating with higher success rates?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Before Rebuttal**

Strengths. The paper is well motivated and easy to follow (albeit with some issues in the method section), with a clear technical contribution. There is strong evidence that the proposed method for self-supervised learning of tactile representations is superior to end-to-end learning. There are extensive experiments across multiple sensors and multiple challenges in tactile sensing. The paper also introduces a new benchmark, TacBench.

Weaknesses. Some aspects of the method are difficult to follow. Experiments are limited to standard machine learning evaluations on test/train datasets, with a lack of experiments studying downstream policy learning on realistic tasks. There is insufficient information on the learned features and how they are used for task learning. It is unclear if the learned representations would transfer well to novel objects. There is a lack of depth in experiments regarding the different sensor types, such as generalisation to novel sensors or sensor bias in the dataset.

---

**After Rebuttal**

Following the reviews, authors replied to the reviewers and addressed some of the concerns raised. Authors also provided a revised paper, including some new experiments. These responses and new material were received well by the reviewers, with two of the reviewers then upgrading their recommendation. Following the rebuttal, the AC and reviewers had a discussion, where all three reviewers now recommend acceptance of the paper. There is agreement that, whilst the paper lacks in distinct algorithmic novelty, the experiments, results, and new benchmark, are sufficiently strong that this paper would be interesting and useful to those in the CoRL community working on tactile sensing.
Decision: Accept