Sparsh: Self-supervised touch representations for vision-based tactile sensing

Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 08:11:10

Abstract: In this work, we introduce general purpose touch representations for the increasingly accessible class of vision-based tactile sensors. Such sensors have led to many recent advances in robot manipulation as they markedly complement vision, yet solutions today often rely on task and sensor specific handcrafted perception models. Collecting real data at scale with task centric ground truth labels, like contact forces and slip, is a challenge further compounded by sensors of various form factor differing in aspects like lighting and gel markings. To tackle this, we turn to self-supervised learning (SSL) that has demonstrated remarkable performance in computer vision. We present Sparsh, a family of SSL models that can support various vision-based tactile sensors, alleviating the need for custom labels through pre-training on 460k+ tactile images with masking and self-distillation in pixel and latent spaces. We also build TacBench, to facilitate standardized benchmarking across sensors and models, comprising of six tasks ranging from comprehending tactile properties to enabling physical perception and manipulation planning. In evaluations, we find that SSL pre-training for touch representation outperforms task and sensor-specific end-to-end training by 95.1% on average over TacBench, and Sparsh (DINO) and Sparsh (IJEPA) are the most competitive, indicating the merits of learning in latent space for tactile images. Project page: https://sparsh-ssl.github.io

Main paper: 
 introduction: 
 Touch comes before sight, before speech. In today's AI landscape, this Margaret Atwood quote is playing out in reverse despite touch being a crucial modality for humans to physically interact 8th Conference on Robot Learning (CoRL 2024), Munich, Germany.with the world. Touch provides a direct window into information like forces and contact during hand-object interactions, enabling dexterity. Vision-based tactile sensors [1, 2, 3, 4] have emerged as the leading form factor capable of capturing images of physical interactions at the sensor-objectenvironment interface, often inaccessible through vision. These images contain properties such as contact geometry, texture, and forces and have been leveraged across tasks like insertion [5, 6] , pushing [7] , grasping [8] , localization [9] , and pose and shape estimation [10, 11] .The prevailing approach to incorporating vision-based tactile sensors in robot tasks is to train custom models using labeled data [6, 12, 13, 14] to estimate useful states. However, this can be inefficient and results in repeated effort across different type of sensors like GelSight 2017 [1] (with markers) and DIGIT [3] (without markers) or different variety of tasks. For example, feature extractors trained on GelSight with markers may not transfer to other sensors, and encoders optimized for texture recognition [15] may not be suitable for tasks that require reasoning about forces or slip [16] . Supervision for building large general models is prohibitive as collecting large scale real world data with ground truth labels is challenging. For instance, properties like forces [17] and slip [18] require careful and expensive instrumentation in lab settings, while other properties like tracking deformations [19] or extrinsic contact [6] can be infeasible. To address this fragmentation in the literature across custom solutions, there is a need for touch representations that are broadly applicable to many tasks and many sensors, along with a benchmark of standardized tasks useful in measuring progress. Taking inspiration from self-supervised learning (SSL) methods in computer vision, we extend these approaches to the tactile sensing domain and build a benchmark for evaluation (Figure 1 ).In this work, we introduce a family of touch representations for vision-based tactile sensors trained with SSL. Specifically, we provide a recipe to adapt masking-based objectives from computer vision to the tactile domain, and train general-purpose touch encoders by curating a new Touch-Slide dataset and existing datasets of tactile images (Figure 2 ), namely YCB-Slide [9] , Touch-and-Go [20] , and ObjectFolder [21] . Pulling together additional unlabeled data points from the existing datasets we train our models on a total of 460k+ tactile images. Finally, we construct TacBench, a benchmark consisting of six touch-centric tasks that cover the space of relevant problems on tactile properties such as force estimation and slip detection, on perception such as pose estimation and grasp stability, and on robot manipulation such as policies for solving a bead maze. Our contributions are as follows: 1. General touch representations, Sparsh pre-trained with SSL on 460k+ tactile images, 2. TacBench a benchmark of standardized tasks to evaluate touch representations and models, and 3. Curation of new & existing datasets, unlabeled for SSL and labeled for benchmarking. In evaluations on TacBench, we find that Sparsh with SSL pre-training yield on average 95.1% improvement over task and sensor specific end-to-end models under limited labeled data budget (33%-50% of the collected amount) for any task. Additionally, we find Sparsh (DINO) and Sparsh (IJEPA) to be the most competitive outperforming Sparsh (MAE), indicating the merits of learning in latent space over pixel space for tactile images. 
related work: 
 Self-Supervised Learning with its success in natural language processing and computer vision, has become the new learning paradigm. In the last three years, a variety of general-purpose frameworks [22, 23, 24, 25] have been proposed for learning representations. We refer to [26, 27] for a comprehensive survey on SSL frameworks and their categorization based on pretext tasks and learning algorithms. In Appendix B, we expand on Masked Image Modeling (MIM), self-distillation, and Joint-Embedding Predictive Architecture (JEPA), as we explore them in this study.Traditionally, tactile sensing has relied on preprocessing tools like marker tracking and finite element method models to extract contact properties, such as shear forces [16, 28] , dense normal estimation [29, 28] , and contact area prediction [30] . From a learning perspective, a trend is to use custom encoder architectures tailored for specific tasks and sensors, which are either pre-trained or trained end-to-end [31, 12, 32, 13, 14, 6, 9] . Nevertheless, there is an increasing interest in representation learning for vision-based tactile sensors. For instance, MAE has shown effectiveness at material classification and texture recognition [33] . Fine-tuning convolutional encoders for BioTac, RoboSkin, and GelSight performs well on fabric decomposition tasks [15] . Even nearest-neighbor retrieval over * Without need for labels we can sample more images than reported in Touch-and-Go [20] and Object Folder [21] .pretrained representations, for the XELA [34] uskin sensor, can enable some success in dexterous manipulation [35] . Crucially, the current state of standardization in learning touch representations and the wide variety of tactile sensors available, has made it challenging to develop and share pre-trained models across the research community in this domain.Another direction is exploring the alignment of visual and tactile modalities in latent space using multimodal datasets [20, 36, 37, 38, 8, 39] and techniques like contrastive coding and cross-sensory retrieval [40, 41, 42] , yielding promising results in tasks like material classification, grasp stability, and tactile-driven image stylization. However, current approaches [40, 43, 44, 42] primarily focus on texture and visual properties and overlook physical contact properties, such as forces, slippage, and poses, which are essential for dexterous manipulation.The works closest and concurrent to ours are T3 [45] and UniT [46] . T3 trains sensor-specific encoders to capture shared latent information through a shared trunk, using both the MAE objective and labeled task-specific data as supervision. UniT is a VQGAN [47] model with a patch-based discriminator for representation learning only for GelSight Mini (markers). On the other hand, we introduce a family of models trained with the latest SSL algorithms for the three most commonly used families of tactile sensors: DIGIT, GelSight 2017, and GelSight Mini. Similar to T3 and UniT, we evaluate touch representations for policy learning, however we also introduce a standardized benchmark to comprehensively evaluate representations and their ability to solve several relevant touch-centric tasks along tactile properties, physical perception, and manipulation planning. 
touch representations via self-supervised learning: 
 Current approaches incorporating vision-based tactile sensors in robotic tasks lean on custom task and sensor specific solutions. As highlighted in the introduction, this can be inefficient, and there is a growing need for general-purpose touch representations that can be more broadly useful. We envision the following guiding principles for such general touch representations: (i) provide performance benefits across many tasks including real-time robot manipulation, (ii) generalize across multiple types of sensors built on a similar operating principle, like vision-based tactile sensors, and (iii) improve performance by leveraging computation and diverse data at scale without the need for manual labels. Self-supervised learning (SSL) is promising in this regard, as it offers data-agnostic objectives based on wide-reaching concepts such as analysis-by-synthesis to train generalist models. This motivates the question of whether vision techniques such as masked image modeling (MIM) [23, 50] and self-distillation [48, 49] can be extended to the domain of vision-based tactile sensors.To this end, inspired from advances in self-supervised learning (SSL) in computer vision, we introduce Sparsh, a family of touch representation trained with SSL across multiple sensors such as DIGIT [3] ,GelSight 2017 (with markers [1] ), and GelSight Mini (without markers). The tactile domain, however, imposes several challenges that impede a straightforward application of SSL approaches from vision towards touch representations.Tactile sensors inherently provide local information; thus, images can be ambiguous when observed independently and can vary across grasp forces, materials, and shapes. Therefore, we investigate the optimal space for training SSL encoders. Specifically, we are interested in the efficiency of pixel reconstruction, latent reconstruction, or clustering approaches to learning representations in the presence of aforementioned ambiguities. We hypothesize that latent reconstruction and clustering could be more efficient in learning representations, as they focus model capacity away from fine reconstruction details [52] . Tactile images contain distractors, such as markers and light placement variations, which can significantly vary due to manufacturing discrepancies. To increase robustness to distractors, we perform background subtraction for both DIGIT and GelSight Mini (markerless). This process provides the model with a reference to no-contact, which conveys static shear information when a perpendicular force is applied to the elastomer. We find empirically that background subtraction helps models generalize across the same type of sensor.To address the scarcity of labeled and even unlabeled data in the tactile domain that limits the training of large encoder models, we curate together new and existing vision-based tactile sensor datasets [9, 20, 21] , totaling ∼661k images as illustrated in Figure 2 . 70% or 462.7k images are used for SSL pre-training which is an order of magnitude larger than any prior work on touch representations [33, 40, 43] (the rest are held out for monitoring training using online probes).Tokenizing tactile images appropriately for SSL is important because many tasks such as slip detection and relative pose estimation require temporal reasoning. For SSL methods that operate on images, we concatenate two tactile images with a temporal stride of 5 samples across the channel dimension, I t ⊕ I t-5 → x ∈ R h×w×6 . For a sensor operating at 60FPS, this corresponds to an inference window of approximately 80 ms, the reaction time that humans need to adjust the grip force when detecting partial slip [53] . For SSL methods that operate on video (e.g. V-JEPA), we generate clips with 4 frames at [t, t -2, t -4, t -6] ∈ R 4×h×w×3 corresponding to an inference window of ∼ 100 ms.Currently Sparsh is limited by data streaming rates, and not by inference time, as the models support inference rates of upto 112FPS (measured on an Nvidia RTX3080). See Appendix C for additional details on model architectures and training.4 TacBench: Tactile sensing benchmarkWe introduce TacBench, a collection of touch-centric tasks, and labeled datasets for standardized evaluation for vision-based tactile sensing. We compile data for all tasks from various sensors to evaluate the generalization of representations. These tasks are categorized under three main questions.1. Do the representations comprehend tactile properties? Tactile sensing informs finger-object contact interaction properties like forces and slip that are crucial for robot manipulation. In Section 3 (c), where the robot using tactile sensing is tasked to move a bead along a curved wire.Evaluation protocol. We adopt a frozen evaluation procedure with an encoder-decoder architecture. Specifically, we freeze the pre-trained Sparsh encoder weights and train the parameters of an attentive decoder [51, 59] to assess what touch representations have captured from self-supervised pre-training alone. All tasks in the benchmark, except force field visualization and policy learning, train an attentive decoder containing a cross-attention module and a two-layer MLP using labeled datasets from Table 3 . We also include an end-to-end (E2E) baseline with identical model capacity where the same encoder and decoder probe are initialized with random weights and all the parameters (both encoder and decoder) are trained. Further, we train downstream decoders with different amounts of labeled data to evaluate task performance under progressively low labeled data regimes.In the following sections, we describe the design, metrics and results of each task in TacBench.Additional details are provided in Appendix D, and ablations with unfrozen Sparsh encoder, encoder model sizes, and few-shot cross-sensor transfer are provided in Appendix E.5 Comprehending tactile properties 5.1 [T1] Force estimation Task. Force estimation is defined as the prediction of 3-axis normal and shear forces applied on the sensor's elastomer. Figure 3 (a) shows our data collection setup. We use three different indenter shapes to collect force-labeled data: hemisphere, sharp, and flat. Our dataset contains 75k timealigned samples of 3-axis force measurements, end-effector poses, and tactile images from DIGIT at 60fps and GelSight at 25fps. We train the decoder using normalized force measurements scaled between [-1, 1], supervised using L 1 loss and optimized using Adam until convergence. We compare performance using the average root mean squared error (RMSE) across all three axes.Results. GelSight Mini images are of higher resolution (HD) compared to DIGIT (320 × 240) resulting in smaller contact regions against the background. For this reason, we observe that when sufficient supervised data is available for DIGIT, it is possible to train a model from scratch to achieve high accuracy, but for GelSight Mini the end-to-end model does not perform well. Figure 4 (i)-(ii)shows across the board that our frozen Sparsh representations can estimate forces with low error. Specifically, we find Sparsh (DINO) to be robust even when access to labeled data is sparse, a common scenario in tactile sensing. Additional details are in Appendix D.3. 
[t1a] force field visualization: 
 We qualitatively evaluate the representations for rendering normal and shear force fields to understand sensor-object interactions. Although obtaining a shear field for sensors with markers nowadays is trivial via marker tracking [54] , it is challenging and underexplored for markerless sensors. We train a CNN decoder using the reassemble-fusion approach for dense predictions [60] unsupervised, since we do not have access to ground truth for markerless sensors. We frame normal field estimation as depth estimation [61] and shear field estimation as optical flow [62, 63, 64, 65] . Figure 4 (vi) shows visualizations for the top-performing model Sparsh (DINO) in [T1] that provides directional information about the relative motion of the contact patch. For instance, sliding motion (a, c, e, f), torsional slip (b), and divergence field upon contact (d). Additional details are in Appendix D.4.Task. Shear and slip are closely related. Using the same setup as force estimation, we collect strokes where a hemispherical probe slides over the sensor, producing trajectories with both sticking and slipping samples. Slip is labeled using the friction cone model with an empirically estimated static friction coefficient (see Appendix D.5). The dataset, with a notable imbalance between no-slip and slip classes, contains 125k samples with 13% slip instances. We train two decoders: one for slip detection and another for normalized force changes (∆) as we find that predicting the two correlated quantities jointly enhances slip detection. The MLP decoders use cross-entropy for slip detection and mean absolute error for ∆ force regression, reserving 25k samples for evaluation.Results. We report F1 score instead of accuracy due to the imbalance in the slip labels in the dataset. Figure 4 (iii)-(iv) illustrates the advantages of frozen Sparsh features trained under a JEPA paradigm for slip detection, particularly challenging for DIGIT sensor, even when using only 1% of the training dataset. In particular, Sparsh (VJEPA) achieves the highest F1 score among the models. Although all models detect slip from the 80 ms history of tactile data, Sparsh (VJEPA) benefits from a detailed temporal perspective, as its encoder processes a video clip with four frames spanning this window. Sparsh backbones also show better performance than the E2E model when labeled training data is significantly reduced. Additional details are in Appendix D.5. Estimating object pose changes can help tasks such as tracking object drift for in-hand translation [66] , rotation [56, 57] , and pushing [7] , among others. Given that tactile images capture local changes between sensor-object, we evaluate Sparsh representations to estimate SE(2) transformations of the object relative to the sensor. Figure 3 (b) illustrates the data collection procedure.The dataset consists of time-synchronized pairs of DIGIT observations z t ∈ R h×w×3 and object poses T t ∈ SE(3). T t are then preprocessed to produce relative pose changes on the sensor gel as S t-1 t ≜ (∆x, ∆y, ∆θ) ∈ SE(2). We follow the regression-by-classification paradigm for this task [10, 66] . Relative object poses are binned into a grid, capturing translations with a resolution of ±5mm and rotations with a resolution of ±2 • . For each degree-of-freedom (DOF), we train a head to predict probability distribution over the discretized grid using cross-entropy loss and Adam optimizer.Results. Multiclass accuracy reveals that E2E approaches perform well with ample data, but drastically decline when labeled data is reduced, as shown in Figure 4 (v). Small datasets make it difficult to distinguish between close categories, such as orientation changes from [0.5• , 1.0 • ] to [1.0 • , 2.0 • ].Pre-trained representations, however, maintain good performance even with only a third of the data. In low data scenarios, decoders using tactile representations often revert to extreme values, reducing estimation resolution and accuracy. Additional details and examples are provided in Appendix D.6. 
[t4] grasp stability: 
 Task. Grasp stability is well-studied in the tactile sensing literature for parallel jaw grippers [40, 67, 68, 69] . We evaluate whether representations aid in predicting grasp success given a short history of tactile images from a single finger. Specifically, we take inspiration from [8] and adapt the Feeling of Success dataset. Each sample consists of a triplet of tactile images corresponding to 'before', 'during', and 'after' grasping a set of objects. The dataset consist of 64% successful grasps and 36% failed grasps. We pass to the SSL model the 'before' and 'during' as tactile history. Since [8] does not specify an official train/test split, we create our randomized split with all objects, using approximately 8k grasps for training and the remaining 1.3k grasps for evaluation.Training with the full dataset, all models achieve similar accuracy. Sparsh (IJEPA) or Sparsh (VJEPA) reach ∼ 80% classification accuracy, surpassing results from [8] that combined tactile and vision modalities as shown in Figure 4 (vii). Our model, relying solely on touch from a single finger, shows competitive performance even with only 33% and 10% of the data. However, with just 80 training samples, performance drops significantly. More details are in Appendix D.7.Task. Vision-based tactile sensors are broadly used for material property recognition, since their compliant gel and high resolution cameras make them effective at discriminating different materials by surface texture [58, 39] . Specifically, we take the task definition from [58] and adapt the Clothing Dataset. The dataset consist of 4467 short video clips (10-25 frames), of a robot with a GelSight 2017 (markers) grasping several types of textile (20 classes), such as leather, cotton, polyester, etc. We follow the train-test split provided in the metadata of the dataset. Training an E2E specialist model for textile recognition using the full dataset can be challenging, as noted in [58] . By leveraging pre-trained touch representations, as shown in Figure 4 (viii) the performance of the task can be significantly improved, even when training with only 10% of the labeled data. Sparsh (MAE) is particularly effective, as it heavily relies on pixel-level features (see Appendix D.8). Additionally, we evaluate the cross-sensory ability of the representations, finding that with few samples (10-shot) Sparsh quickly adapts the downstream task to DIGIT (see Appendix E.3).7 Enabling manipulation planning 7.1 [T6] Bead maze Task. The bead maze is a children's toy to enhance fine motor skills. We adapt this task to robot policy learning, where the goal is to guide a bead from one end to another following the path of the wire (maze). Given a small history of tactile images (. . . , z t-1 , z t ), and robot proprioception (. . . , q t-1 , q t ), we train a policy to predict changes in joint angles as actions a ≜ (∆q t , ∆q t+1 , ...); ∆q ∈ R 7 , to make progress on this task. This task is fundamentally tactilefocused, as the robot needs to react to resistance encountered by changes in the maze pattern and the subtle local movement of the bead are difficult to perceive from vision even when not occluded by the hand. A prior version of the bead maze task has been explored in robotics relying solely on tactile feedback [70] . In our setup, illustrated in Figure 3 (c), we assume that the robot starts with an initial stable grasp. We collect a dataset of 50 demonstrations on different maze patterns with a mix of VR-based and manual kinesthetic-based teleoperation, corresponding to a total of ∼34k training pairs of tactile images and robot joint angles. Since we are training policies with real data, we use diffusion policy [71] for this task as it is one of the leading behavior cloning methods. For tactile observation conditioning, we replace the vision encoder in Diffusion Policy with the pre-trained Sparsh encoder.Results. We evaluate Sparsh (DINO) and Sparsh (IJEPA) for policy learning, as these representations exhibit the best performance across the rest of the benchmark. For completeness, we also consider Sparsh (MAE), and E2E which trains a tactile encoder and policy end-to-end. Due to covariate shift [72] in behavior cloning, prediction errors can accumulate over time; therefore, we report position error between the predicted trajectory and a demonstration trajectory from a held-out maze sequence over small chunks of 3cm followed by the robot corresponding to 15 timesteps of action predictions. Figure 4 (ix) shows the position error over access to different number of demonstrations for training. We find that Sparsh (DINO) and Sparsh (IJEPA) produce significantly (a difference of ∼16%) lower trajectory errors compared to training the policy E2E.Additionally, we evaluate real rollouts of the learned policies (using all 50 demonstrations) over a set of 10 randomized novel starting locations on the maze. In Figure 4 (ix) we report distance traversed (in cm) before failure. We find that policies using Sparsh representations outperform E2E by ∼20-53%. We note that given the high precision nature of this task and the considerations for real system deployment for the policy, none of the models succeeds in completing the full maze on real robot rollouts. We expect that increasing the diversity of training data with different maze patterns will highlight the generalist capabilities of touch representations, and that temporal ensembling will aid in improving the smoothness of the policy [73] . Additional details are in Appendix D.9. 
discussion: 
 Summary. We present Sparsh, a family of general touch representations trained with self-supervision for vision-based tactile sensors. We learn general-purpose, cross-sensor representations from a curated, unlabeled dataset of 460k+ samples from DIGIT, GelSight 2017, and GelSight Mini sensors. We evaluated five SSL approaches (see Figure 2 ) comparing their performance against task and sensor specific models through TacBench, a benchmark of six touch-centric tasks designed to assess the content and quality of the representations. Our results indicate that Sparsh representations are performant across various sensors and tasks capturing tactile properties, and enhancing physical perception and manipulation planning.Analysis. Overall Sparsh excels on all tasks. In particular, we find Sparsh (DINO) is well suited for physics-based tasks like force and pose estimation, while Sparsh (IJEPA) performs better at touch semantic understanding like slip state, stability of a grasp, and textile recognition. On average Sparsh (DINO) outperforms Sparsh (IJEPA) by 5.6% across the benchmark. Both models perform similarly in bead maze test demonstrations, which require implicit knowledge of shear forces and slip. However, this did not translate to real robot performance due to lack of force control and system-level confounding variables not captured during training. These include the high precision required to keep the bead in place, the impossibility of error recovery once grip is lost, and trajectory drift due to local decision-making. Specialist policies or models trained from scratch exhibiting better robot rollout performance is due to the narrow task domain setting that leads to overfitting, a trend similarly observed when studying pre-trained vision models [71, 73, 74] .Learning touch representations in latent space is more advantageous than in pixel space, as these representations can filter out and generalize over noise or lighting differences. Tasks traditionally challenging for markerless sensors (like DIGIT and GelSight Mini), such as shear force (and field) estimation and slip detection, become solvable with our general touch representations. On average, Sparsh achieves a 95.1% improvement compared to an end-to-end approach when all models have access to only 33% -50% of the labeled dataset per downstream task. Using as little as 10% or 1% of the labeled data for force estimation and slip detection still yields acceptable results (e.g. force error below 0.1N with Sparsh (DINO)). Fine-tuning Sparsh encoders is another method of assessing the quality of pre-trained representations. We provide in Appendix E.1 experiments with partial and full fine-tuning. Notably, models pre-trained in latent space perform better in downstream tasks when fully fine-tuned, especially in regression tasks like force and pose estimation. In contrast, partial fine-tuning offers minor improvements, aligning closely with the performance of frozen models. We also evaluate Sparsh decreasing the model capacity, finding the biggest impact in performance for regression-like tasks when training with limited amount of labeled data (see Appendix E.2).Sparsh is a significant step towards a general pre-trained backbone for vision-based tactile sensors.Our aim is to enable efforts to compile larger tactile datasets that include additional vision-based tactile sensors and leverage the benefits of scaling up SSL backbones, as seen in computer vision and natural language processing. TacBench serves as an initial benchmark for evaluating these representations, and additional tasks can be incorporated based on the needs of the tactile sensing and manipulation community. For instance, further exploration of pre-trained touch representations in tactile policy learning, or tracking dynamic object properties like changing mass during pouring.Limitations. Open-source tactile datasets we considered in this study predominantly feature discrete contact interactions. We believe that incorporating data rich in shear interactions can further improve the representations. We do not ablate the length of tactile image history for learning the representations. Such ablations could provide guidance on improving their quality for downstream tasks. Our bead maze policies with pre-trained touch representations deployed on the real robot are only able to complete the maze partially before compounding error leads to the bead falling out of the fingers. Further research is needed to understand how to effectively leverage pre-trained touch representations in behavioral cloning for robot manipulation tasks.Representation learning in robotics. Pretraining models for multi-task capability has become popular recently, especially after the success of self-supervised learning (SSL) in computer vision tasks like object classification, segmentation, depth estimation, and image generation. These tasks, while typically tested on computer vision datasets, are also very common in robotics. The idea of using these pre-trained representations for robot learning was initially explored in [89] , showing that pre-trained visual representations can sometimes even be better than using ground-truth state representations for training control policies.Generative SSL via masked image modeling (MIM) [90, 91] has shown successful transfer of pretrained representations from in-the-wild data to real-robot scenarios, enabling basic motor skills such as reaching, pushing, and picking. Furthermore, many other works investigate contrastive learning approaches to learning general visual representations in robotics [92, 93, 44] . These methods usually employ a pixel reconstruction objective based on a time-contrastive objective or focus on contrasting video clips leveraging natural language for video-language alignment.The field has been moving towards finding general-purpose representations that work well across a wide range of problems in robot manipulation learning. Voltron [94] , is a framework for languagedriven visual representation learning for robotics that combines both masked auto-encoding and contrastive learning techniques, focusing on multi-task performance. This model is trained to learn representations that capture both low-level spatial reasoning and high-level semantic understanding by using language supervision from human videos.Tactile sensor simulation. Multiple simulators have been proposed for vision-based tactile sensors such as [95, 96, 97, 98, 99] with the hope of sim2real generalization of learned policies [100] . However, many of these methods are either limited to marker-based tactile sensors [100] , or narrow tasks [101, 102] . Certain other methods [40] also leverage simulated data to train multi-modal representations. However, in general we find that tactile simulators are still unable to model shadows, as well as real-world per-sensor-instance discrepancies, hampering their potential use for representation learning.To ensure fair evaluation of all models, our SSL algorithms are largely adapted from official MAE, I-JEPA, V-JEPA, DINO, DINOv2 codebases.We train all models on 8 Nvidia A-100 (80G) GPUs. In addition to training losses, to monitor training progress, we rely on online probes. Specifically, we find that for joint embedding predictive architectures, the training losses are not indicative of model convergence during optimization; therefore, proxy metrics such as reconstruction quality are helpful. For all methods, we utilize DPT [103] based decoders to decode the tactile representations back into tactile images. See Figure 5 for some examples of tactile reconstructions from Sparsh embeddings. All encoder models are trained for 150 epochs. We use AdamW optimizer and use a linear rampup followed by a cosine schedule as the learning scheduler. Further, we find that tuning momentum value as well as the weight decay factor was important in observing training convergence without collapse. Additional information of hyperparameters is detailed in Table All encoder models are Vision Transformers (ViT) [83] . Although the main encoder models use ViT-B/14 as the standard architecture, following [50] we use a small ViT as the predictor network. All the models are pretrained without a [cls] token. For DINO, which decodes the [cls] token into classes, we repurpose ViT registers [104] to predict classes. In Table 2 we report the number of parameters for each encoder and their respective inference times.Tactile images with a stride of 5 i.e., I t ⊕ I t-5 ∈ R h×w×6 are concatenated along the channel dimension before the background is removed and reshaped to 224 × 224 for ViT processing. We choose a stride of 5 as consecutive images are similar due to high sensor sampling rates, and to match the slip detection window in humans. Ablating the effect of the input image and patch resolution may be important for better performance and is left for future work.We use three available datasets for training Sparsh, namely YCB-Slide [9] , Touch-and-Go [20] and Object Folder [37] . The YCB-Slide dataset consist of human sliding interactions with 10 YCB objects. Each object has 5 trajectories, with around 3500 frames each from DIGIT sensors with different optical characteristics (180k frames in total). For each object, we dedicate four trajectories for training and the last one for validation. Touch-and-Go consists of discrete human contact interactions with in-the-wild objects, using a GelSight sensor. It consist of 140 videoclips and plain files with labels for the frames with a clear contact. We use all frames (220k) in the videoclips since we do not rely on labeled data for SSL training, from which 70% is used for training and the remaining for validation. The data used from ObjectFolder consist of 81k frames of robot discrete contact interactions with objects in a controlled setting. We also use a train/val split of 70/30.To complement the dataset, we collected Touch-Slide with additional human sliding interactions on toy-kitchen objects with the DIGIT sensor. We use 9 objects, shown in Figure 6 and collected 5 trajectories for each, generating 180k frames in total.For all downstream tasks we use tactile data from real sensors/hardware (DIGIT, GelSight17, and GelSight Mini) that were not seen during Sparsh SSL training. Under our problem formulation, this allows us to investigate generalization of Sparsh to new sensor instances (consider the case of swapping out a sensor from a robotic hand due to wear-off). Similarly, all objects used for downstream tasks were not used for SSL training. For example, [T1] and [T2] tasks use a real robot arm to slide the sensor elastomer (DIGIT, GelSight Mini) against an indenter to collect force-labeled data.[T4] uses an open source dataset for grasp stability [4] , which includes data from a real robot grasping over 100 unique objects using an unseen GelSight17 with printed markers during SSL training. Similarly, [T6] uses DIGIT data collected from real hardware, a robot pulling and moving a bead along the wire. Note that none of the data used for learning representations comes from this kind of object-robot hand interactions.In this paper, we consider three SSL paradigms, namely Sparsh (MAE), Sparsh (DINO), and Sparsh (IJEPA) & Sparsh (VJEPA).Sparsh (MAE) is based on the principle of masked image modeling, where an encoder model is tasked with learning the contextual representations of substantially masked images, such that it enables reconstruction of the masked regions via a lightweight decoder. We use a ViT encoder and decoder for Sparsh, and the MAE loss corresponds simply to a L2 reconstruction loss:EQUATIONSparsh (DINO) is based on the principle of self-distillation between two identical networks, where a student network learns to track the output predictions of a EMA teacher network. Cross-entropy loss is employed between the predictions of the student and teacher network, both of which consume different crops of the same input. Specifically, feature representations from each branch are passed through a MLP head, producing probability vector over an arbitrarily chosen number of classes. These scores are normalized to produce p s and p t for the student and teacher respectively.EQUATIONSparsh (IJEPA) and Sparsh (VJEPA) share similarities with both masked image modeling and selfdistillation. Here, we employ two identical networks termed context and target networks. The context network corresponds to a student network, which is tasked to predict the features from a EMA target network (teacher network), through a small predictor network. In this case, a L 2 loss over features is used to enforce similarity between the two branches. Specifically, the context network observes M global masks of an image to produce contextual features, which are then passed through a predictor to predict target network features of B i local crops of the same image ŝyj . On the other hand, the target network consumes local crops of the image to produce s yj .EQUATIOND TacBench tasks and evaluation details D.1 Labeled datasets See Table 3 for details on labeled data curation for TacBench tasks.The parameters of the model updated via EMA (target encoder for Sparsh (IJEPA) and Sparsh (VJEPA), teacher network for Sparsh (DINO) and Sparsh (DINOv2), encoder from Sparsh (MAE)) are fixed and used for evaluation. The features are pooled via attentive pooling for tasks that require global representations, such as slip detection, resultant force estimation, and classification tasks. For tasks that require dense reasoning, we use DPT decoders [103] to decode patch representations into full input resolution quantities such as normal and shear force fields, and reconstructed tactile images. See Figure 7 for a visual illustration of the probe architectures. DIGIT. In Table 5 we report the average RMSE over 25k samples of unseen DIGIT data for the force estimation task. We report metrics for each Sparsh model and the E2E approach, under four different budgets of training data. We also provide a 95% confidence interval to ground the error ranges of each model.In Figure 8 we plot the friction cone from the test data, where the colormap represents the error in mN for each axis. Note that E2E exhibit larger errors (around 500mN) for the tangential component and they are more predominant as the normal force increases. In contrast, the top model Sparsh (DINOv2) estimates with low error (< 100mN) in general across the whole range of tangential and normal forces.Full dataset (50k) 1/3 dataset 1/10 dataset 1/100 dataset Table 5 : Root Mean Squared Error (mN) and 95% confidence interval for force estimation with DIGIT data. All models were evaluated on flat indenter data over 25k test samples.GelSight. In Table 6 we report the average RMSE over 25k samples of unseen GelSight data and the corresponding 95% confidence interval. Notice from Figure 9 that the majority of errors are localized around the dynamic shear region. It is worth noting that the errors associated with Sparsh (DINO) remain below 150mN, whereas E2E exhibits higher errors, particularly in the estimation of normal forces.Since rendering the force field is a dense prediction task, we do not apply the attentive probing protocol. Instead, we follow DPT [60] , training a CNN encoder with reassemble-fusion modules at layers 2,5,8,11 of the Sparsh encoder to progressively upsample the representations to obtain a fine-grained prediction of the force field. After the reassemble-fusion modules, we attach two task-specific task head, for normal and shear field prediction.Since for markerless vision-based sensors it is not trivial to get ground truth of the force field, we turn to unsupervised learning. Depth estimation and optical flow are analogous to the estimation of normal and shear force fields, areas where the computer vision community has proposed several unsupervised methodologies [62, 63, 64, 65, 61] . We borrow ideas of unsupervised monocular depth estimation, where from two tactile images I t and I t-n , we learn a pose estimator for getting the transform between frames. With the sensor intrinsic K, we map image I t from pixel space to camera plane, translate estimated depth D t , apply transform from t to t -n, and transform back to image plane to get Ît-n . We supervised based on the reprojection error, MSE between I t-n and predicted Ît-n . To reconstruct the shear field, we transfer ideas from unsupervised optical flow, where we warp the features of image I t to I t-n based on the estimated flow and compute a photometric consistency loss that encourages the estimated flow(shear) to align image patches with a similar appearance. This loss is a linear combination of the Charbonnier loss and the structural similarity (SSIM) between I t-n and Ît-n . We also add a smoothness loss that acts as a regularization term, encouraging the shear field to align the boundaries with the visual edges in the tactile image. In Figure 10 we show snapshots of the normal and shear field predictions during sliding trajectories of the DIGIT sensor on YCB and spherical probe objects. To collect labeled slip data we perform a normal/shear load test. Using a firmly affixed hemispherical probe on a flat surface, a robot presses the DIGIT sensor toward the probe, applying random normal forces of up to 5N. Upon reaching the target normal force, the robot slides the probe 2mm to a randomly selected position on the sensor surface, allowing us to capture the shear profile with a F/T sensor. To label slip, we rely on the friction cone to identify samples on the sticking and slippage regions. A description of the procedure is illustrated in Figure 11 .As eluded to in Section 3, Sparsh's inference window is approximately 80 milliseconds. This is appropriate since this duration matches the reaction time needed by humans to adjust the grip force when detecting partial slip [53] . We train two heads: one for slip detection and the other for the estimation of normalized force change (∆). We find empirically that training both heads simultaneously improves slip detection, given their high correlation. The MLP probes are trained with cross-entropy for slip detection and mean absolute error for ∆ force regression as loss functions.Our dataset comprises 125k samples, with only 13% corresponding to slip instances. We reserve 25k samples for evaluating model performance.Table 7 provides F1-score metrics for all models under different amounts of training data. Sparsh (VJEPA) outperforms all models, even when trained under low data regimes. In Figure 12 we contrast the predictions over time for a sample trajectory between Sparsh (VJEPA) and E2E models trained with 33% of the data. Note that for Sparsh (VJEPA) the errors are around the friction boundary, where the probe is starting to slide. Also, it is worth noticing that a poor estimation of changes in shear and normal forces is reflected in the accuracy of distinguishing between slip and no-slip. In Figure 13 , we illustrate a failure case for Sparsh (VJEPA), as its results do not align with the ground truth. However, it is important to note that slip labeling is prone to errors due to its reliance on an experimental coefficient of friction. Despite the inaccuracies in the friction boundary for this trajectory, Sparsh (VJEPA) successfully detects the slip samples. We collect a dataset of trajectories with time-synchronized pairs of object pose measurements and sensor observations using an Allegro hand equipped with DIGIT sensors on each finger, mounted on a robot arm. The object was placed on a table and with the palm facing downward, we pressed against it with the fingertips (see Figure 3 ). We manually perturbed the object's pose by sliding and rotating it under the Allegro fingertips. The pose of the object was tracked using ArUco tags. Given ground truth object pose measurements in the world frame, we preprocess them into relative pose change (∆x, ∆y, ∆θ) ∈ SE(2) in the sensor frame.Since we follow a regression-by-classification approach, we discretize the range of motion for each degree of freedom into multiple intervals in Log-uniform space. This allows us to achieve a better evaluation. Using attentive probing, we freeze Sparsh and train a 2-layer MLP with two output units for grasp success classification.In Table 9 report the accuracy for binary classification to compare the performance of the models across different training budgets, including a 95% confidence interval. Figure 16 shows the confusion matrices on test samples for E2E, Sparsh (DINO) and Sparsh (IJEPA) trained on a 33% of labeled data.This tasks allows to study the capabilities of the representations for semantic understanding of the contact, as in recognizing the type of textile that is being touched by the sensors. We use the task definition and the data set introduced in [58] . This data set contains 4467 short video clips (10-25 frames), of a robot with a GelSight (markers) mounted parallel gripper grasping several types of clothing, across 20 textile classes, such as leather, cotton, polyester, etc.We follow the train-test split provided in the metadata of the dataset. Using attentive probing, we freeze Sparsh and train a 2-layer MLP with 20 output units for textile classification. In Table 10 and Figure 17 (c) we report the accuracy for multiclass classification, comparing the performance of the models in different training budgets.The goal in bead maze is to guide the bead along the wire, as shown in Figure 3 . We don't rely on vision for hand-eye coordination, making the task fundamentally tactile since forces in the fingers indicate whether the bead is moving smoothly or encountering resistance. In our setup, we use Policy and replace it with Sparsh backbones with fixed parameters. For training an end-to-end policy, the encoder corresponds to a ViT-Base encoder with randomly initialized weights.For each method, we evaluate the learned policies over a set of 10 randomized novel starting locations on the maze and we measure distance traversed (in cm) before failure. In Table 11 , we report mean and variance of distance traversed comparing Sparsh (pre-trained only and pre-trained then fully fine-tuned) against E2E. All models use 50 demonstrations for training the policy via imitation learning. We find that policies using Sparsh representations outperform E2E by ∼ 20 -53%. Most failure cases across methods are due to the bead getting stuck on the maze or the bead falling out of the robot hand. While prior work such as diffusion policy suggest that frozen pre-trained models may hurt imitation learning due to domain mismatch, we do not observe significant gains from fine-tuning in this application. Leveraging pre-trained models in imitation learning is an active area of research, however these results demonstrate the impact of Sparsh touch representations for robot applications. In Table 12 we report to position error of E2E, Sparsh (DINO) and Sparsh (IJEPA) with respect to test demonstrations on an unseen maze, highlighting the fidelity of Sparsh (DINO) and Sparsh (IJEPA) to follow a similar trajectory. Nevertheless, this doesn't necessarily transfer to real-world performance, since the locality of the observations and predictions make the errors in the adjusted joint angles to compound fast, which results in unforeseen collisions and the subsequent lose of the grasp. In an overfitting setting, training a policy for a single maze, policies using Sparsh (DINO) and Sparsh (IJEPA) are able to complete almost 30% of the maze on the real robot. However, it is expected an specialist policy trained end-to-end to perform better in the overfitting setting. Experimentally, we found than an E2E policy trained for a single maze is able to complete almost 80% of the maze running on the real robot.In Table 13 we summarize the performance of Sparsh across the benchmark. We find that with respect to an E2E approach, with Sparsh we can achieve an improvement of 98.75% on average. Sparsh (DINO) and Sparsh (IJEPA) are in general the best models across the board, showing the Since Sparsh is trained on multiple GelSight-like data, we investigate whether SSL training enables cross-sensory representations or if it helps downstream tasks trained for one sensor quickly adapt to another. To study this, we use as a baseline the decoder trained for [T5] Textile recognition, which was supervised with labeled data from GelSight with markers.We collect new data using a DIGIT sensor for 10 out of the 20 textiles. Our dataset contains 11 samples for both training and testing. We load the trained decoders for [T5] using Sparsh (DINO) and E2E and perform zero-shot evaluation as well as 1-shot, 5-shot, and 10-shot training and subsequent evaluation using the DIGIT data.  
 
 Appendix:  


Decision: **Before Rebuttal**

Strengths. The paper is well motivated and easy to follow (albeit with some issues in the method section), with a clear technical contribution. There is strong evidence that the proposed method for self-supervised learning of tactile representations is superior to end-to-end learning. There are extensive experiments across multiple sensors and multiple challenges in tactile sensing. The paper also introduces a new benchmark, TacBench.

Weaknesses. Some aspects of the method are difficult to follow. Experiments are limited to standard machine learning evaluations on test/train datasets, with a lack of experiments studying downstream policy learning on realistic tasks. There is insufficient information on the learned features and how they are used for task learning. It is unclear if the learned representations would transfer well to novel objects. There is a lack of depth in experiments regarding the different sensor types, such as generalisation to novel sensors or sensor bias in the dataset.

---

**After Rebuttal**

Following the reviews, authors replied to the reviewers and addressed some of the concerns raised. Authors also provided a revised paper, including some new experiments. These responses and new material were received well by the reviewers, with two of the reviewers then upgrading their recommendation. Following the rebuttal, the AC and reviewers had a discussion, where all three reviewers now recommend acceptance of the paper. There is agreement that, whilst the paper lacks in distinct algorithmic novelty, the experiments, results, and new benchmark, are sufficiently strong that this paper would be interesting and useful to those in the CoRL community working on tactile sensing.

Reviews: review: title: title: Self-supervised representation learning for visuo-tactile sensors
paper_summary: paper_summary: This work presents a self-supervised learning (SSL) approach to touch representation learning for vision-based tactile sensors, addressing challenges in task-specific models and data collection. The authors introduce Sparsh, a suite of three SSL models (MAE, DINO, and IJEPA) pre-trained on over 460k tactile images from three types of visuo-tactile sensors using masking and self-distillation, eliminating the need for custom labels. They also develop TacBench, a standardized benchmarking platform with five tasks. Results evaluated by splitting datasets into training/test sets show that SSL pre-training outperforms end-to-end training by an average of 98.7% on TacBench task datasets, with Sparsh (DINO) and Sparsh (IJEPA) models demonstrating superior performance.
main_review: main_review: **Strengths**

- The paper is well-written and easy to follow.
- It addresses the challenge of learning effective tactile representations for visuo-tactile sensors that are useful for various downstream tasks, including force estimation, slip detection, object pose-change estimation, and grasp stability.
- The results suggest that pretraining with self-supervised objectives helps derive latent features that make downstream task learning more sample efficient.

**Weaknesses**

- It is unclear how the learned features are used for downstream task learning. Details about the decoder architectures for each task are missing. What is the dimensionality of the learned feature representation? Is this dimensionality an important factor?
- It appears that the learned representation is frozen while a decoder head is learned for each task. Is this understanding correct? What happens if the full model is unfrozen and fine-tuned at the end of training? Does task performance improve with additional fine-tuning of the full model?
- The work does not demonstrate that the representation is useful for robotic applications beyond reporting evaluation metrics on train/test splits. It does not address how planners or policies may perform on the tasks. For example, Section 7 reports results for the bead maze task as trajectory errors rather than a measure of task success, such as distance to goal or fraction of path before failure. Using metrics like trajectory errors assumes that the reference trajectory is an optimal solution, but the demonstration collection medium is likely to be suboptimal.
- It is unclear how well the representation generalizes to new objects or new sensors, as evaluation is done by splitting the dataset into training/test sets. Additional generalization analysis would be valuable.
questions: questions: See Review above.
limitations: limitations: 2
review_summary: review_summary: The work addresses the interesting problem of visuo-tactile feature learning for sample-efficient task learning. However, some critical details about how to use the learned representation for task learning are missing. Additional experiments demonstrating how the proposed method performs in manipulation planning would highlight its potential impact.
title: title: Self-supervised touch representation
paper_summary: paper_summary: The paper introduces Sparsh, a family of self-supervised learning (SSL) models that support various vision-based tactile sensors. It addresses the challenge of task and sensor-specific models by pre-training on a large dataset of over 460,000 tactile images, enabling general-purpose touch representations without custom labels. The paper also presents TacBench, a standardized benchmark with five touch-centric tasks to evaluate these models. Evaluations show that Sparsh models significantly outperform task-specific end-to-end training, with Sparsh (DINO) and Sparsh (IJEPA) demonstrating the most competitive performance. Additionally, the paper curates a unified dataset from new and existing sources, enhancing the scope and applicability of tactile sensing research.
main_review: main_review: The paper's main strengths are: 1. The proposed framework is well described and explained with words and figures. The paper employs a comprehensive self-supervised learning (SSL) approach, which is rigorously tested and validated across multiple tasks and sensors. 2. Extensive Evaluation: The use of TacBench, a standardized benchmark, allows for a thorough evaluation of the models across five distinct touch-centric tasks. This ensures the results are robust and generalizable.

The main weaknesses of the paper are:
1. Some parts of the paper are dense with technical terms, which could be challenging for readers who are not specialists in the field. Refer to the list of issues for more detailed questions.
2. The reliance on large datasets and sophisticated models suggests high computational requirements. This paper uses 8 Nvidia A100 GPUs, which might limit accessibility for some researchers. 
3. There is a lack of a more intuitive illustration of the learned tactile representation. Besides the resultant higher performance thanks to the proposed models, further discussions on the learned representation would greatly help contribute to the current understanding of vision-based tactile sensing.
questions: questions: Issues:
1.	Are the results presented in Figure 1 (middle) the same as those in Figure 4? What is the amount of labeled data in Figure 1? This information should be included in the caption as the outperforming only exists under a limited labeled data budget.
2.	As a benchmark, the end-to-end (E2E) approach is not well explained, and the reviewer failed to find the differences between E2E and the proposed Sparsh models in the paper. In lines 160-162, it seems E2E has identical model capacity. What are the differences? 
3.	In line 148, predicting the stability of a grasp should be T4 instead of T2.
4.	In line 277, what is behavior closing?
limitations: limitations: 3
review_summary: review_summary: The paper presents a practical approach for touch representations. The idea is interesting to the community and the presentation is well organized. Hence, I recommend accepting the paper.
title: title: Review
paper_summary: paper_summary: The paper presents Sparsh, a self-supervised learning model family for vision-based tactile sensors, addressing the limitations of task-specific handcrafted models. By pre-training on over 460k tactile images with masking and self-distillation in pixel and latent spaces, Sparsh eliminates the need for custom labels. The authors introduce TacBench, a benchmarking suite for evaluating tactile properties and manipulation across different sensors and models. They found out that DINO and I-/V-JEPA are the most dominant approaches.
main_review: main_review: **Strengths:**
1. The paper addresses an important problem and its initial parts are very well written which introduces the significance of the problem.
2. The authors makes their contribution clear.

**Weaknesses:**
1. **Method Section:** The method section is not adequately written. It is understandable that the approach is utilizing the existing method, but still the authors must mention some details of I/V-JEPA (along with loss function) in the main paper. I would suggest the authors to refer to one of their mentioned paper [1], which could serve as a useful template for enhancing the clarity and depth of the method section (as it also leverages on existing method but a detailed description of the method is provided).

2. **Data Bias:** Given that the datasets are derived from multiple sensors and vary significantly in the number of images, there can be some potential bias. It is important to investigate whether the model shows a preference for certain sensors, thereby learning better from some sensors than others. 

3. **Cross-Sensory representation:** The concept of learning "cross-sensory representations" requires further clarification. Specifically, it is unclear whether there is an actual transfer of representations between sensors. For instance, considering the Touch-and-Go [2] dataset, which primarily contains data collected in the wild using the Gelsight sensor. Can the method recognise in a one-shot scenario with a different type of tactile sensor, such as Digit, when presented with a similar object like Grass? This analysis is essential for understanding the generalizability of the proposed method across different tactile sensors.

**Minor Details:**
1. The image reference in line 717 is missing.

**Suggestions:**
1. I would suggest to include these following recent papers in SSL domain [3,4] and this very recent work which is very similar to yours [5,6].
2. It is hard to see the accuracy and F1 Score from the graphs in the Figure 4. I would highly suggest to include the table with bold text as the best result (as usually done).

**References:**

[1] Yang, Fengyu, et al. "Binding touch to everything: Learning unified multimodal tactile representations." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Yang, Fengyu, et al. "Touch and go: Learning from human-collected vision and touch." arXiv preprint arXiv:2211.12498 (2022).

[3] Kerr, Justin, et al. "Self-supervised visuo-tactile pretraining to locate and follow garment features." arXiv preprint arXiv:2209.13042 (2022).

[4] Dave, Vedant, Fotios Lygerakis, and Elmar Rueckert. "Multimodal visual-tactile representation learning through self-supervised contrastive pre-training." arXiv preprint arXiv:2401.12024 (2024).

[5] Fu, L., Datta, G., Huang, H., Panitch, W. C. H., Drake, J., Ortiz, J., ... & Goldberg, K. (2024). A touch, vision, and language dataset for multimodal alignment. arXiv preprint arXiv:2402.13232.

[6] Zhao, Jialiang, et al. "Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks." arXiv preprint arXiv:2406.13640 (2024).
questions: questions: **Questions:**
1. Is any sensor-specific detail provided to your encoder during training, similar to the approach in [1]? If not, why not? Do you think it can improve the training and generate sensor dependent representations?
2. In the results section (Section 6.2), it is mentioned that classification relies solely on single finger data. How is it possible to predict grasping success or failure from just this data? Does this imply that the dataset has some inherent linear relationship with grasping success, such as greater deformation correlating with higher success rates?
limitations: limitations: 2
review_summary: review_summary: The paper tries to solve a completely reasonable problem and it can also prove to be very effective in this domain. However, certain parts of the paper requires revision as described in the Weaknesses. As of now, I am inclined towards Weak Reject but I  am open to reconsider my score if the authors provide a satisfactory rebuttal addressing these concerns.