Toward Artificial Palpation: Representation Learning of Touch on Soft Bodies
Authors: Zohar Rimon
Elisei Shafer
Tal Tepper
Efrat Shimron
Aviv Tamar
Venue: NeurIPS 2025 poster ()
Keywords: Tactile sensors
Touch processing
Robotics
Self-supervised learning
Primary area: reinforcement_learning

Abstract: Palpation, the use of touch in medical examination, is almost exclusively performed by humans. We investigate a proof of concept for an artificial palpation method based on self-supervised learning. Our key idea is that an encoder-decoder framework can learn a **representation** from a sequence of tactile measurements that contains all the relevant information about the palpated object. We conjecture that such a representation can be used for downstream tasks such as tactile imaging and change detection. With enough training data, it should capture intricate patterns in the tactile measurements that go beyond a simple map of forces -- the current state of the art. To validate our approach, we both develop a simulation environment and collect a real-world dataset of soft objects and corresponding ground truth images obtained by magnetic resonance imaging (MRI). We collect palpation sequences using a robot equipped with a tactile sensor, and train a model that predicts sensory readings at different positions on the object. We investigate the representation learned in this process, and demonstrate its use in imaging and change detection.

Toward Artificial Palpation: Representation Learning of Touch on Soft Bodies
Zohar Rimon"
Elisei Shafer
Tal Tepper
Efrat Shimron
Aviv Tamar
Technion - Isracl Institute of Technology
Abstract
Palpation, the use of touch in medical examination, is almost exclusively performed by humans. We investigate a proof of concept for an artificial palpation method based on self-supervised learning. Our key idea is that an encoder-decoder frame- work can learn a reprexentation from a sequence of tactile measurements that contains all the relevant information about the palpated object. We conjecture that such representation can be used for downstream tasks such as tactile imaging and change detection. With enough training data, it should capture intricate patterns the tactile measurements that go beyond a simple map of forces the current state of the art. To validate our approach, we both develop a simulation environment and collect a real-world dataset of soft objects and corresponding ground truth images obtained by magnetic resonance imaging (MRI). We collect palpation sequences using robot equipped with tactile sensor, and train model that predicts sensory readings at different positions on the object. We investigate the representation learned in this process, and demonstrate its use in imaging and change detection.
Introduction
Palpation, the of touch in medical examination, centuries-old practice that is still important today. In woman's breast cancer a case that motivates our work, a large fraction of cases (over 40% in the US. per Roth al. 201T) discovered by palpation, either via self-examination physician, although screening tests based on mammography (X-ray) and magnetic resonance imaging (MRI) are designed to detect tumors that are not yet palpable. The recommendations of the American Cancer Society [2023) for women to be familiar with how their breasts normally feel and report any changes promptly indicate that tactile information still relevant for early detection.
by
Artificial palpation has the potential to better exploit tactile information by detecting patient-specin temporal changes that physicians typically cannot keep track of, and by improving palpation precision beyond what untrained patients can achieve. Tactile imaging methods (Sarvazyan et al.) 2012) typically involve pressing force sensor array against soft tissue to generate force map, which can then be analyzed either visually or via computer vision algorithms. In contrast, elastography (Sar vazyan al., 2011] infers the elastic properties and stiffneess of soft tissue by measuring variations in ultrasound (US) or magnetic resonance (MR) signals in response to applied forces. Additionally, several studies have investigated the use of tactile sensors for classifying tissue based on stiffness [Jia
ct al. 2013 Nichols and Okamura 2013 Di et al. 2024
Aimed at improving tactile imaging and detection accuracy, we propose go beyond direct stiffness estimation and view palpation as an inference process mechanical structures soft body, given sequence of partial, noisy, tactile force measurements. Indeed, physician performing breast examination tries to infer the existence of lumps, cysts, and other anatomical structures from touch. The medical literature that characterizes, for example, benign masses as 'smooth, soft firm, and
"Correspondence to zohar.rimon eoharrinoneemmpssccmiiiiin. github..om//oham///ttttaaaaaaaap
Code and data available
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

mobile, with well-defined margins' [Klein 2005]. characterization of how different structures give in finger motions rather than stiffness alone.
hints that human palpation relies on more involved
While palpation can be taught and perfected, human touch is a general skill, learned from years of physical interactions and manipulations of objects. We argue that data-driven approach artificial palpation can learn, from touching many different objects, how to interpret a sequence of tactile measurements into corresponding mechanical structures, potentially leading more accurate imaging than currently available.
Motivated by self-supervised learning results in computer vision and natural language processing (He et al., 2022 Devlin et al. 2019], we propose to learn a general, artificial, palpation representation. by predicting future tactile force measurements given sequence of past measurements. If the representation is useful for predicting future forces, must contain relevant information about the object being palpated, and may therefore be useful for tactile imaging, change detection, and classification suspicious findings.
Data collection
Self-supervised learning tactile data
Downstream tasks Image prediction
Modular soft phantoms
Tactile
tactile data
Sequence
Encoder
soft-body representation
Change detection
Tactile prediction
sensor pose
Decoder
Robot
GT images
MRI
Palpation
Figure 1: Proof of concept system for learning artificial breast palpation. Left: we fabricate soft objects and palpate them using a tactile sensor mounted on robot arm. We also obtain MRI scans of the objects as ground truth object models. Middle: we train an encoder-decoder neural network to predict the tactile measurements at given positions from a sequence of previous measurements. Right: we use the learned representation to train model for tactile imaging, and perform change detection based on predicted images. In principle, by replacing the phantoms with human subjects, system can be used for clinical studies.
We present proof of concept system that includes object fabrication techniques, data collection routines, model training, and an evaluation protocol of tactile imaging and change detection for soft bodies. Our fabricated objects include lumps with various sizes and shapes, resulting in over 1150 possible object configurations. We collect data using a robotic manipulator with tactile sensor tip programmed palpate the phantom, and train a neural representation by learning to minimize tactile force prediction error. To evaluate the utility the representation for tactile imaging, we sean the phantoms in an MRI to generate ground truth object models, and train an additional neural network to predict this model from the palpation representation. To evaluate change detection, we use the predicted models to evaluate change in the size of the lump, and compare with human evaluations. We find that our learned representation contains relevant information about the position and shape the lump, and yields tactile images that are arguably easier to interpret than map of forces, which can be used for change detection at a level comparable humans.
novel soft breast phantoms with modular component that can
While clearly only a prototype, we argue that these results provide promising direction for an artificial palpation system that uses human MRI scans and tactile data learn tactile imaging.
2
Related Work
Learning tactile representations: work in this area focused almost exclusively on rigid bodies, and applications to object manipulation and shape reconstruction. Guzey al. (2033 considered

---

instantancous tactile measurements, and applied BYOL [Gril et al. 2020] to multiple taxel readings
arranged as an image. Hiiuuera et al. [2024] considered vision-based tactile sensors (Lambeta et al.)
2020], proposed several self-supervised learning representations, and a benchmark for evaluating
them in various robotic manipulating tasks. More recently, Feng et al. [2025] collected matched
tactile data for four different sensors, and proposed an encoder-decoder framework for images and
short (3-frame) videos from vision-based tactile sensors. Qi et al. [2023] represented a sequence of tactile, proprioceptive, and visual observations of rigid body, by learning to reconstruct simulated 3D point-clouds of the object. The representation was further used for learning in-hand object rotation and shape reconstruction. Focusing rigid-object shape reconstruction and pose estimation, ZZao et al. 2023] estimate pose and shape with neural networks, from data of manually pressing a rigid object with visuo-tactile sensor, while Suresh et al. [024] used neural fields to learn the pose and 3D shape of an object rotated by a robotic hand. Neither of these methods are applicable to estimating the properties of soft objects.
Robotic Palpation and Cancer Diagnosis Khanna and Shah [024] provide recent survey in the
context of breast cancer diagnosis; we highlight several studies most relevant to work. JJenkinson
ct al. 2023b] develop a radial robotic mechanism for breast palpation, while Syrymova et al. 2025 investigate a purpose-built tactile glove. [Scimeca et al. [2022] palpate a soft silicone object with hard inclusions of 3 different sizes, and classify the size of the inclusion by projecting instantaneous measurements their first principal component, and fitting a Gaussian density to the projected measurements over time per inclusion size. Our work is similar in spirit but significantly larger in scale, both in the data and the learning models. Di al. 2024 used data from vision-basse tactile sensor to classify both prostate phantoms and real prostate tissue ex-vivo according their hardness, by fine-tuning video masked auto-encoder applied sequences of images.
Several studies explored how to define the robot's motion during palpation. SScimeca et al. 2022 investigated various motions that revolve around a point in space. (annni et al. [2022] used deep movement primitives for robotic breast palpation, while et al. [2025] explored a shared autonomy scheme between human tele-operator and computerized control. In this work we focus on simple linear motions and leave the investigation of more complex movements to future work.
In terms of phantom fabrication, most of the literature concerns fabrications suitable for MRI and ultrasound imaging Ustbas al., 2018 Keenan et al. 2016]. We are not aware of standard methods for fabricating phantoms for breast palpation.
3 Method
We propose a data-driven approach for soft-body tactile palpation. We first formulate our inference problem. We then use self-supervised learning approaches to learn a representation of the palpated object (Section B.2 Finally, we use this representation learn downstream tasks with limited amount of supervised data (Section 3.3
3.1 Formulation
We model the artificial palpation problem as follows. A tac- tile sensor is a rigid body that is controlled be at time t E o,..., in pose T, E The sensor interacts with a soft body M, resulting in k-dimensional force reading ft E R We consider M to represent all the structural and mechanical properties of the body, which determine the force on the sensor, and note that in all practical situations M is unknown. Subse- quently, the sensor is moved the next pose T4+1 by the controller, and the next reading is obtained. In addition, we assume access to some observation of the soft body, denoted T, for example, an MRT scan. In this work we do not consider how to control the sensor, and assume that some palpation motion controller is available. Also, we assume that the force sensor is noisy, but we do not model the noise in any explicit form.
fuo
In the tactile imaging problem, our goal is to use the poses and force readings To. fu..... Ir, fr to predict the observation I. the change detection problem, we are given readings from two bodies M, M', which may or may not be different, and our goal is to determine if M = M'.
For simplicity consider single sensor; our formulation trivially extends multiple sensors.
3

---

Sequence Encoding
Force (b)
Prediction
21
Force Decoder
Sequence Encoder
FLB
FLE
FLE
Booale
Eorale
K
A
x K
Figure 2: Representation Learning: A sequence of tactile measurements and poses is encoded by first encoding every measurement+posee by a force-location encoder (FLE), and then encoding the sequence by GRU. (b) The decoder predicts tactile measurement at time f from the representation at time la and the pose at time Ky.
3.2 Representation Learning
To address the inference problems described above, propose learning based approach. We first explain the data structure, and then our learning algorithm.
We collect data from N bodies M,,..., Mr. For each body M,, we collect tactile data D; {at, fi fi) by a palpation controller as described above. addition, for the first Nj bodies we have corresponding observation I,, i N₁.
Our working hypothesis is that collecting large amounts of observations (c.g., MRI scans) is more difficult than collecting palpation data, that is, N2 < N. Following prior works (Devlin et al. 2019 He et al. 2022]. we design our approach to first use self-supervised learning techniques to pretrain representation from the large amount of palpation data, and later use the pretrained representation to effectively learn downstream tasks such as tactile imaging.
We use an encoder-decoder architecture to learn a representation in a self-supervised manner. The key idea is that the encoder is trained to map sequence of measurements {z), fi] )-- from body M; to d--dimensional vector representation 2₁ Rdd that contains the information in the measurements about Mj. Thus, at the last time step T, the representation contains information about the complete trajectory, and can be used for downstream tasks. We next detail our architecture and training objective.
Encoder Our encoder maps between a sequence of measurements sequence of latent representa- tions. We first use force-location encoder (FLE) to encode each step separately: F FLE(fe, MLP())= Pε(x.).
E [1, TJ. The forces are encoded via a two-layer MLP, and the locations are encoded with sinusoidal positional encoding (PE). Afterwards, we use sequence encoder to
produce sequence of embeddings of the same length as the input sequence {at) We choose gated recurrent unit (GRU) [Cho et al., 2014] encoder as it can easily scale to long sequences, and acts information bottleneck in the encoding process
Decoder We hypothesize that if the representation informative for predicting force measurements, it should contain relevant information about the body being palpated. Therefore, we structure our force decoder (FD) to predict the force reading
pose Ze based on the representation 2,: PF(*))) R Yt, t' E []. For L' > I, this corresponds to predicting future sensor readings, while for 1' < this means recalling past measurements. We note that similar idea of predicting both past and future measurements was proposed by Zintgraf et al.
FD (xe) )= M MML.MMMPPPC
2021 for decision making partially observed domains.
In simulation experiments that are not reported here, we also investigated other architectures such as transformer based masked autoencoders [He et al. 2022 but obtained similar results.

---

Training Objective We use the mean squared enror (MSE) reconstrcctin loss between the predicted and true forees to train our encoder-decoder model. Since predicting all the forces from all the timestamps has an O (T²) complexity, applying it to long sequences is challenging. Instead, we uniformly subsample reconstruction steps for calculating the loss, as described next for data from single body M:
K
Crea
FD 24ge 2ig
2KK
where (1x {(x] are K < T uniform samples without replacement from [). and for each E [1, K], K" ) are also K" < uniform samples without replacement from [1. 7] After calculating all representations 2....... 2T during the encoder's 's forward pass, a decoder forward pass and loss calculation can be done in parallel for all indices k, k' and has complexity only (K' K'). In
our implementation we set K K' = 64.
3.3 Tactile Imaging
We hypothesize that the representations z learned as described in Section 3.2 are useful for predicting more general properties of the body M than forces. In particular, we focus on the tactile imaging problem, and propose to predict the observation I from the representation of a complete palpation sequence 27.
We use flow matching Lipman et al. 2024], conditioned on the vector representation, to map 27 128 128 image where each pixel take one of 3 different values (to be described later). During inference, we draw single noise sample to generate an image, as the uncertainty at time step is very small (the standard deviation over results from different noise samples two orders of magnitude smaller than the expectation), full technical details are in Section E This architecture can easily be modified handle more general images, or even 3-dimensional volumetric images. We train the imaging network separately from the pretrained tactile representation, using a standard cross-entropy loss for each pixel in the image.
For change detection based on the predicted images, we consider two different methods. The first compares between the pixel values of images obtained from palpation trajectories of two bodies directly. This method, which reported in the supplementary material, works well when the image predictions are relatively accurate, however, we first evaluated the lump size from the predicted images, and detected change based on the predicted lump size (see Section
obtained in our simulation results. For real world results,
Results
We next present our experimental results. We begin with introducing a new simulated environment for palpation, in which we investigate our design choices and quantify the scale of data required to obtain meaningful results. then detail our real-world data collection and experiments. We provide supplementary visualizations in zoharri.... izobarri.githul io artificial-pall palpationn and our code can be found in the supplementary material.
4.1 The PalpationSim Simulator
We introduce PalpationSim simulation environment be used mock of the real palpation learning setup. We designed PalpationSing to be both lightweight, quick, and easy to visualize and interpret, yet indicative of our real-world domain.
Our main component is 2-dimensional finite element method (FEM) simulation of a soft semi- circular body with the option of having harder lump element inside, depieted in Figure 3a The body is composed of triangular elements with linear elasticity, and we model the different hardness
*We release Palpation insiim as part our open- source code. While several physical simulators that
popular in the robotics literature can simulate soft objects [oooorov et al. 2012 | The Isaac Lab Project Developers) 2025). we did not find any versatile enough for our simulation environment at the time of writing. On the other hand, desired much faster and lightweight solution than full blown commercial FEM packages.
5

---

(a)
(b)
0.6
17.0000
New
0.4
1.44
0.4
0.2
421 sel-oup
0.2
02
4/0 22f sup
cemmuttton
0.0
0.0
200
1600 2200
40
80
120
200
400
500
1600
2100
-40
50
120
200
400
800
1600 3200
TTwrring
Thaining Thats
TTeining Triats
(d)
Figure 3: PalpationSim simulator and simulation results. (a) A 2-dimensional finite-element model of a round sensor pressing on soft object with harder lump inside: insert shows the forces on the sensor. (b+c) A ground truth image of the body (b), and a predicted image (c). (d-f) Image prediction results: (d) with and without self-supervised pretraining. (e) with and without permutation augmentation, (f) with different number of trajectories per trial. See text for details.
of the body and lump by different values of Poisson's ratio and Young's 's modulus [Lubliner and Papadopoulos, 2016). The bottom vertices of the semi-cirele are fixed in place, while the other vertices are free to move. A tactile sensor is modeled using 16 2-dimensional points arranged in circle, with the center of the circle denoting its pose (here, is a 2-dimensional position instead of 6-dimensional pose). Each point, when inside a triangular element, applies a spring-like repulsion force the nearest edge of the triangle. Given z, find the positions of all the vertices by minimizing the energy of the system using Adam [Kingma and Ba 2014). and measure the forces on each sensor proportionally to their penetration into the body. Note that the simulation is quasi-static per sensor position x; we measure the forces after the vertices have stabilized in a steady state. When move the sensor, we measure the steady state of the system for each sensor position along the way (we warm-start the energy minimization with the previous solution for faster optimization). This design choice corresponds
slow motion of the sensor in the real world.
The motions available to the probe Zi...., ØT are linear trajectories that "press" on the soft body at various positions. The body observations / are 2-dimensional 128 x 128 images where we discretize each pixel value into 3 classes: background, body, and lump, as depicted in Figures 3bBc 3c
Data Collection We first randomly sample Np random bodies with different lump locations (if any), sizes, and variations the Young modulus and Poisson ratio for each finite element. For each body, collect Nirial 2 "trials", where each trial contains sequence of Neruj trajectories pressing on the body from uniformly distributed angles. Between trials, we randomly modify the Poisson's 's ratio and Young's 's modulus for each finite element, and, in 10% of the models, increase the size of the lump, for our change detection task. Full details for the data collection appear in Appendix B.2
Simulation Results We focus the following two questions: (1) how important is self- pretraining, and (2) how do results scale with the number of models and the number of trajectories collected from each model. In addition, we report a simple and useful augmentation.
Figures FiguresBbJandd demmonstttee our image prediction results. In the following, we treat each pixel in the predicted image classification problem, and report the F1 score for the complete image [Manning et al., 2008), which we found to correlate well with visual image quality. We report results for different sizes of the data, per the total number of trials in the dataset (specifically, we train the models on a subsample of trials and corresponding images to report each data point the figures). In Figure
3d we compare the image prediction using pretrained representation (trained on the full data), as

---

(a)
(b)
(c)
Figure 4: Modular Breast Phantom Design. (a) The insert has an octagonal 3D-printed base, a soft-silicone skin, and is filled with polyvinyl acetate slime. A soft silicone "lump" is embedded within and attached to the base. (b) The shell has two layers of soft silicone skin, with slime in between, where the bottom layer is attached to 3D-printed base with octagonal hole. The insert can be positioned in possible orientations inside the shell. (c) An assembled iiner+insert and standalone insert. The bar-code labels allow automatically record the component types and orientations using an overhead camera.
described in Section 3 with supervised learning method that directly predicts the image from sequence of measurements. Interestingly, pretraining improves results for all sizes of data, and is more dramatic when less data is available, showing the benefit of pretraining with large amounts of only tactile data. In Figure 31 we show the scaling with respect to number of trajectories from each body used during pretraining: intuitively, performance plateaus when the trajectories sufficiently 'cover' the body. Finally, in Figure 3e we show that a simple augmentation of randomly changing the order of trajectories from the same trial during pretraining dranatically improves performance; we used this augmentation in all other reported results.
To conclude, the simulation study allowed dimension real world study, and obtain well-performing working point for our learning algorithms. We next report our real world results.
4.2 Real- World Results
Our goal is a proof-of-concepp for artificial palpation. To this end, we designed an experimental procedure that, while artificial and simplistic in nature, addresses some of the realistic challenges in obtaining tactile measurements and ground truth imaging for breast palpation.
4.2.1 Benchmark Curation
Modular Breast Phantoms: At the core of our experiments is novel breast phantom design (see Figure that is composed of two modular parts, an outer shell and an insert, which can be assembled together in multiple orientations collect data from a variety of bodies. Each part is composed of soft silicone skin, and a polyvinyl acetate slime filling, mimicking the breast fat tissue. We add lump of certain shape inside the insert made also of soft silicone. While our phantoms are clearly not anatomically accurate depictions of human breasts, we posit that they reflect a realistic range of tactile sensations present in breast palpation, as we verified with several physicians and breast oncologists. We provide detailed description of the phantom and instructions for reproducing our design in Appendix A We fabricated 6 shells with different slime characteristics and inserts with 8 different lumps positions and shapes, each in 3 different sizes, ranging from 8[mm] to 14[mm] diameter, which can be placed inside the shell possible different phantom combinations that we can collect data with.
different orientations, resulting in over 1150
Automatic Palpation Data Collection: We use single XELA uSkin tactile sensor Tomo et al. 2018 which has 30 3-dimensional force sensors, and records data at 85 Hz
To collect data in consistent and repeatable manner, nipulator with the tactile sensor attached to its end effector, as depicted in Figure TO We control the arm using hybrid force-motion controller [Lynch and Park 2017) that applies a downward motion of the tip and a vertical force of 3.8N at a fixed orientation, implemented as a modification of the default force controller in panda-py
Franka Emika Panda robotic
Elsner 2023 Such downward motion particular
'We opted for the uSkin based on preliminary comparison with a Digit sensor [Lambeta et al, 2020). reported in detail in Section A..I. We found the vision-bass Digit perform poorly inside the soft material, which conjecture is due the low spatial frequency of relevant forces, which the Digit is not optimized for.
7

---

y position lasting for 5 seconds is termed a 'poke". We execute pokes on a preset matrix of 110 positions, and for each poke we record both the sensor readings, the robot end effector poses, and their corresponding timestamps. In total, collecting data from a single phantom takes 20 minutes of robot time. Collecting data from ~ 550 phantom combinations resulted total of 60K pokes, and ~ 30M instantaneous sensor readings. To mitigate time-varying bias in the Xela sensor, each measurement sequence was normalized relative its first sensor reading.
Ground Truth Phantom MR Images: We generate ground truth labels for tactile imaging by
scanning our phantoms in an MRI system. The motivation for this is threefold: (1) while we have 3D CAD designs for the phantom molds, manufacturing process of the silicone skin has inaccuracies due to its manual nature; (2) the MRI scans, while not realistic due to the simple structure of our phantoms, are still susceptible to real-world challenges of noise and variations in positions and shape due to the softness of the phantom, making for a more challenging prediction problem; and (3) for a future human study, collecting ground truth data using MRI is viable approach.
We scan our inserts using a 3T MRI (Siemens Prisma) system with a 64-channel coil. The acqui- sition protocol involved a volumetric T2-weighted SPACE (fast spin echo [Bernstein et al.) 2004]) acquisition. The parameters were turbo factor 270, resolution of 0.8 0.8 0.9, field of view was 220 x 200 x 86.4mm, TR of 3200mns, and TE of 412ms. The scans were accelerated by 4x using a standard undersampling technique. The images were reconstructed on the scanner using the GRAPPA algorithm (Griswold et al. [002). They were then imported to external servers for training the networks. Since in our inserts lumps have similar height, for simplicity we predict a horizontal image slice instead of complete 3D model (full details are in Section II.
4.2.2 Results
Tactile Imaging: In Figure we show several imaging results and their ground truth counterparts; a more extensive demonstration is provided in Section G For comparison, visualize map of forces from the same data by taking the maximal recorded vertical force across sensors each location. Evidently, learning- based approach aggregates the scattered force measurements into coherent object, as appears in the data, leading to a tactile image that is arguably more interpretable. For interpreting image quality, in addition to the F1 score, we consider two measures that evaluate the lump prediction accuracy: its area in the image (henceforth, size), and the position of its center of mass (CoM). In Table [ we provide quantitative results, showing that our method (Image Pred.)
Table 1: Lump Size error, Center-of-Me -fases (CoM) error and Fl score. For prediction methods, we report standard deviation of the sample mean across 5 random seeds. For the average prediction, report standard deviation across samples.
Method
Size Error [%]] J
CoM Error [mm] +
Fl Score [%] 1
Image Pred.
23.0 + 2.1
2.0
74.4 0.1
Image Pred. (0.5x Data)
20.9 + 0.9
0.1
65.3 + 0.1
Image Pred. (0.25x Data)
41.6 + 3.2
6.4
43.9 2.6
MLP Pred.
10.5 + 0.5
2.9 0.1
Force Map Image Pred.
46.6 2.8
5.3 0.1
47.3 1.3
Average Pred.
51.9 + 47.5
12.1 + 4.1
achieves 23% error in lump size and 2.4mm error in lump location. In comparison, the average errors for predicting the average lump area and location (Average Pred.) are 52% and 12.1mm, respectively. Therefore, our representation is clearly informative about the lump properties. We also trained multilayer perceptron (MLP) predictor from our pretrained representation x yy CoM position and lump size (MLP Pred.). The lump size MLP was more accurate than the image predictor on size prediction, with 10.5% error. This shows that the representation contains even more information than is extracted by the image predictor, probably due to insufficient image data. We also evaluated a method that uses the moo-learrable) force map as a representation, with the same flow matching
'We scan inserts instead of the complete phantoms for time and cost reasons we scan several inserts at once, and rotate the insert images digitally, obtaining seans for almost 200 different bodies. While removing the shell makes for slightly easier prediction task, we found it challenging enough to obtain meaningful insights
8

---

image predictor as we used with our representation (Force Map + Image Pred.), see Section for complete details. This method yielded significantly worse results, which strengthens the importance of our self-supervised learning approach.
As our approach is data-driven, hypothesize that increasing the amount of data should improve the tactile imaging results. To support this, we repeat the self-supervised pre-training and MRI prediction with 0.5x and 0.25 less training data. As can be seen in Table decreasing the amount of training data significantly worsens image prediction performance.
Data Scaling: Compared to non-learning ap- proaches, our data-oriented paradigm benefits as we train on more data (as shown in Table I). Still, collecting more labeled data (i.e. tactile measurements, coupled with an MRI scan) is costly. This problem will be even more signif- icant for real human data. This issue raises question does our approach scale with more unsupervised data only, without more labels?
0.5 Sup. Data
1x Sup. Data
0.5x Rep.
65.3 + 0.1
7
Data
1x Rep. Data
72.7
7.14
Table 2: F1 score of our proposed tactile imaging approach with varying amount of unsupervised and supervise data. We report standard deviation of the sample mean across random seeds.
To test this we trained model with 50% su- pervised data and 100% unsupervised data (and vice versa), the results are shown in Table [ Adding unsupervised data alone resulted in over 11% increase the FI score, while adding the supervised data on top of contributed only an additional 2%. In the context of a future study with human data, this result hints that costly labels can be replaced with casier to obtain self-supervised training.
Change Detection: To ground our results and highlight the task difficulty, we performed a small- scale human study for the change detection task. We asked 15 participants to touch the phantom and detect whether or not we changed the insert to one with a larger lump at the same position (full technical details regarding the human study can be found Section C). Each participant repeated the experiment ~ 4 times, resulting in a sample size of ~ 60. We compared with classification based on our lump size prediction for the same sample that was presented a recall of 0.82 and ailse-llam-raas of 0.32. is remarkable that these results were obtained with the Xela uSkin, for which the spatial density of sensors is almost two orders of magnitude smaller than the human finger (Corniani and Saal 2020). To further appreciate the task's difficulty, we provide video in the supplementary material that visualizes how the lump moves inside the slime when touched.
the participants. We obtained (FAR) of 0.19, better than the human recall of 0.62 and FAR
To further demonstrate the information content in our learned representation, we show in Sfction]oof the supplementary material that it can be used to reliably predict the shell ID) information that available in the insert images. Thus, the representation is informative enough to capture small artifacts in the manufacturing process (all shells are manufactured in the same way), effectively distinguishing between the shells with an accuracy of 99.6 + 0.7.
Shell Classification:
5 Discussion
Our results indicate that with enough data, neural network trained to process tactile measurements yields a representation that is informative about the palpated body. Importantly, the performance of our method improves with additional data, without changing the hardware unlike conventional tactile imaging methods that are not learning-base based. How much, and what kind of, data is required to scale our approach to be clinically relevant?
Tactile measurements depend on the body being palpated, the sensor type, and the sensor movement. Relating to findings in other modalities such as vision and language, we predict that collecting massive tactile data from multiple sensors on general objects (not necessarily soft phantoms) may lead to foundation models for touch processing that can be fine-tuned to specific palpation tasks. The large dataset collected here can contribute collective data collection effort [O' (O'Neill et al.] 2024 Bell and Shimron 2023]. One interesting question is whether language, which has played key role

---

(a)
(b)
(c)
(d)
Figure 5: Tactile imaging with real data. Columns show: (a) 3D CAD design, (b) Ground-truth MRI image slice, (c) Predicted images using our method, (d) Force-map visualization, for 3 arbitrary bodies.
in foundation models for vision and robotics, is also important for touch (cf. the manual palpation instructions mentioned in our introduction).
Our data collection protocol requires a robot both for automation, but also for pose estimation. Extending our results to human moving the sensor likely application, requires adjustments of
palpation trajectories we record to more human-like or even apply active sensing (Scimecal
et al. 2022]. while pose estimation has standard solutions such fiducial markers [Fiala] [2055]
Relating to our imaging results, MRI human breast tissue reveals intricate details of small structures like ducts and blood vessels that do not exist in our fabricated models, making them significantly more complex to predict. In addition, the common use contrast agent injections in breast scans to enhance the visibility of cancerous tumors is not accounted for in our current models. Thus, we cannot immediately deduce that our results will generalize to real human data. Nevertheless, as there is correlation between abnormalities in breast tissue and their tactile sensing, we see promise further investigating this direction. Moreover, although a vast body of work has explored deep learning for MRI [Heckel al. 2024), to the best of our knowledge, learning the mapping from sensory data to MR images has not been explored yet, and our work initiates this new line of research.
Finally, in addition to the technological challenges in artificial palpation, there are clinical and sociological challenges. While several clinical trials have been performed with conventional tactile imaging [Bexa Inc. 2024]. we do not yet have information about the sensitivity and specificity of learning based approaches, or palpation using low-cost sensors. On a positive note, the survey by Jenkinson et al. 2023a showed generally positive reaction to automated breast cancer screening.
Acknowledgements
This work received funding from the European Union (ERC, Bayes-RL, Project Number 10104 1250). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. E.S. is Horev Fellow and acknowledges funding support from the Technion's 's Leaders in Science and Technology program and Alon Fellowship. We are grateful for the support of the May-Dahl-Blin Blum Technion Human MRI research center staff and services.
10

---

References
American Cancer Society.
American cancer society
recommendations for the
early detection of breast cancer, 2023. URL https://www cancer org/
org
cwrresstcancee//ccreennng tests--sttt-e-e and ott.rrreeeeaaaaeeeeaaaaaaaaaaaaa
htw.::/eeeeeeeeooooooo-ooeeeeeee for- of breesst cancer.
html
Accessed: 2025-05-07.
Laura C Bell and Efrat Shimron. Sharing data is essential for the future of ai in medical imaging.
Radiology: Artificial Intelligence, 6(1):e233337, 2023.
Matt A Bernstein, Kevin F King, and Xiaohong Joe Zhou. Handbook of MRI pulse sequences. Elsevier, 2004.
Bexa Inc. Evaluation of the bexa breast examination (bbe). httts:://clinicaltrrr goov study/ /NC73/64706666777777 2024. ClinicalTrials. Identifier: NCT06643767. Recruiting. First posted 28 Aug 2024; last update posted 05 Apr 2025.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406. 1078, 2014.
Giulia Corniani and Hannes P Saal. Tactile innervation densities across the whole body. Journal of Neurophysiology, 124(4):1229-13)) 1240, 2020.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171-4186, 2019.
Julia Di, Zdravko Dugonjic, Will Fu, Tingfan Wu, Romeo Mercado, Kevin Sawyer, Victoria Rose Most, Gregg Kammerer, Stefanie Speidel, Richard Fan, al. Using fiber optic bundles to miniaturize vision-based tactile sensors. IEEE Transactions on Robotics, 2024.
Jean Elsner. Taming the panda with python: A powerful duo for seamless robotics program- ming and integration. SoftwareX, 24:101532, 2023. ISSN 2352-7110. doi: https://oi..rg 10.1016// softx 2023 URL lattps://wwzz http///wwwwiiineeirrretoooo com http://wew...eeeeccce/eeee/eeeee aatticcccee pii/$28821102302285
Ruoxuan Feng, Jiangyu Hu, Wenke Xia, Ao Shen, Yuhao Sun, Bin Fang, Di Hu, et al. Anytouch: Learning unified static-dynamic representation across multiple visuo-tactile sensors. In The Thirteenth International Conference on Learning Representations, 2025.
Mark Fiala. Artag, a fiducial marker system using digital techniques. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 05), volume pages 590-596.
IEEE, 2005.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
Mark Griswold, Peter M Jakob, Robin M Heidemann, Mathias Nittka, Vladimir Jellus, Jianmin Wang, Berthold Kiefer, and Axel Haase. Generalized autocalibrating partially parallel acquisitions (grappa). Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 1206):120211210 1210, 2002.
Irmak Guzey, Ben Evans, Soumith Chintala, and Lerrel Pinto. Dexterity from touch: Self-supervised pre-training of tactile representations with robotic play. In Conference on Robot Learning, pages 3142-3166. PMLR, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEFF/CV conference on computer vision and pattern recognition, pages 16000-16009,
2022. 2022.

---

Reinhard Heckel, Mathews Jacob, Akshay Chaudhari, Or Periman, and Efrat Shimron. Deep learning for accelerated and robust mri reconstruction: a review. arXiv preprint arXiv:2404. 15692, 2024.
Carolina
Higuera, Akash Sharma, Chaithanya Krishna Bodduluri, Taosha Fan, Patrick Lancaster, Mrinal Kalakrishnan, Michael Kaess, Byron Boots, Mike Lambeta, Tingfan Wu, et al. Sparsh: Self-supervised touch representations for vision- based tactile sensing. In Conference on Robot Learning, 2024.
George P Jenkinson, Natasha Houghton, Nejra van Zalk, Jo Waller, Fernando Bello, and Antonia Tzemanaki. Acceptability of automated robotic clinical breast examination: survey study. Journal of Participatory Medicine, 15(4):,4 2023a.
George P Jenkinson, Karl Tiemann, Angeliki Papathanasiou, Jonny Bewley, Andrew T Conn, and Antonia Tzemanaki. A robotic radial palpation mechanism for breast examination (iris). In 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pages 337-342. IEEE, 2023b.
Xiaodan Jia, Rui Li. Mandayam A Srinivasan, and Edward H Adelson. Lump detection with gelsight sensor. In 2013 World Haptics Conference (WHC), pages 175-179. IEEE, 2013.
Kathryn E Keenan, Lisa Wilmes, Sheye O Aliu, David Newitt, Ella F Jones, Michael Boss, Karl F Stupic, Stephen E. Russek, and Nola M Hylton. Design of breast phantom for quantitative mri. Journal of Magnetic Resonance Imaging, 44(3):610-619, 2016.
Raghav Khanna and Elizabeth Shah. Robotics in screening, diagnosis and treatment of breast cancer: perspective view. Clinical Breast Cancer, 24(1):17-26, 2024.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint urXiv::412.6900 2014.
Susan Klein. Evaluation of palpable breast masses. American family physician, 71(9):1731-1738, 2005.
Mike Lambeta, Po-Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, et al. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3):3838 384588888888,,,,-8 2020.
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky Q. Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code, 2024. URL
https://arxi>: org.:/xvv.orraaaaooo aa/////s/2/211 06264
Jacob Lubliner and Panayiotis Papadopoulos. Introduction to solid mechanics. Springer. doi, 10: 978-3, 2016.
Kevin M Lynch and Frank C Park. Modern roboties. Cambridge University Press, 2017
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction Information Retrieval. Cambridge University Press, 2008.
Kirk A Nichols and Allison M Okamura. Methods to segment hard inclusions in soft tissue during autonomous robotic palpation. IEEE Transactions on Robotics, 31(2):444-34, 2015.
Nobuyuki Otsu et al. A threshold selection method from gray-level histograms. Automatica, 1 (285-296)23--27, 1975.
Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x- x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892-6903, IEEE, 2024.
Haozhi Qi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik. General in-hand object rotation with vision and touch. In Conference on Robot Learning, pages 2549-2564, PMLR, 2023.
12

---

Mara Y Roth, Joann G Elmore, Joyce P Yi-Frazier, Lisa M Reisch, Natalia V Oster, and Diana L Miglioretti. Self-detection remains key method of breast cancer detection for us women. Journal of women's health, 20(8):1 135-1139, 2011.
Oluwatoyin Sanni, Giorgio Bonvicini, Muhammad Arshad Khan, Pablo C L'ppee-Custodio Kiyanoush Nazari, et al. Deep movement primitives: toward breast cancer examination robot. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 22126-12134 2022.
A Sarvazyan, V Egorov, and N Sarvazyan. Tactile sensing and tactile imaging in detection of cancer. Biosens. Mol. Technol. Cancer Diagn, pages 339-354, 2012.
Armen Sarvazyan, Timothy J Hall, Matthew W Urban, Mostafa Fatemi. Salavat R Aglyamov, and Brian S Garra. An overview of elastography-an emerging branch of medical imaging. Current Medical Imaging, 7(4):255-228) 2011.
Luca Scimeca, Josie Hughes, Perla Maiolino, Liang He, Thrishantha Nanayakkara, and Fumiya lida.
Action augmentation of tactile perception for soft-body palpation. Soft robotics, 9()):800-292,
2022.
Sudharshan Suresh, Haozhi Qi, Tingfan Wu, Taosha Fan, Luis Pineda, Mike Lambeta, Jitendra Malik, Mrinal Kalakrishnan, Roberto Calandra, Michael Kaess, et al. Neuralfeels with neural fields: Visuotactile perception for in-hand manipulation. Science Robotics, 9(96))ead10628, 2024.
Togzhan Syrymova, Amir Yelenov, Karina Burunchina, Nazgul Abulkhanova, Huseyin Atakan Varol, Juan Antonio Corrales Ramon, and Zhanat Kappassov. Breast lump detection and localization with tactile glove using deep learning. arXiv preprint arXiv:2502.15767 2025.
The Isaac Lab Project Developers. Isaac Lab: A unified framework for robot learning. //gi www.coooc.iaae its//sssaccaaaa seseaac-simm/ee Isaaaaaasaaaaaaa 2025. Version 2.0, accessed May 10 2025.
https
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model-based control. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5026-5033 IEEE, 2012. doi: 10.1 1..100//0S.
Tito Pradhono Tomo, Massimo Regoli, Alexander Schmitz, Lorenzo Natale, Harris Kristanto, Sophon Somlor, Lorenzo Jamone, Giorgio Metta, and Shigeki Sugano. A new silicone structure for uskin-a soft, distributed, digital 3-axis skin sensor and its integration on the humanoid robot icub. IEEE Robotics and Automation Letters, 3(3):2584-2511, 2018.
Burcin Ustbas, Deniz Kilic, Ayhan Bozkurt, Mustafa Erkin Aribal, and Ozge Akbulut. Silicone- based composite materials simulate breast tissue be used as ultrasonography training phantoms. Ultrasonics, 88:9-15. 15, 2018.
Jialiang Zhao, Maria Bauzá, and Edward H Adelson. Fingerslam: Closed-loop unknown object localization and reconstruction from visuo-tactile feedback. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8033-8039. IEBE, 2023.
Jingwen Zhao, Leone Costi, Luca Scimeca, and Fumiya lida. Remote robotic palpation with depth-vision-driven aatnmmmmosstimmessiiiiirrrredddd shared control. IEEE Transactions on Robotics, 2025.
Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: Variational bayes-adaptive deep rl via meta-learning. Journal of Machine Learning Research. 22(289):1-39, 2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made in the abstract and introduction are met throughout the paper.
Although some of the goals of the paper are aspirational, we clearly discuss this in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our work are discussed in detail in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of our learning algorithms, control algo- rithms, network architectures, and hyperparameter choices in the supplementary material. We provide detailed explanations on how we generated our data and fabricated our models, including photos illustrating step-by-step instructions. We will make our date and check- points publicly available. We include source code for our simulated experiments in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
15

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code is publicly available. All of the data, including simulated data, MRI images, and palpation data, have been made open-source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guidess (eggli//uuides//ooeebbbiisssssss onlliicy))yy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. //nips.c/ppie httpol/i/ggiidee//oooooeo
wwwiimssssooniii ssionpolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Most of the details regarding the experiments are provided in the main text while other, more technical details, are presented in Sections D and
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All relevant experiments contain error bars, which were calculated by running the experiments with multiple random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The computer resources that were used in the experiments throughout the paper are detailed in Section|K] K
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The paper conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
17

---

Justification: The paper is foundational research proof of concept for touch processing framework that can potentially be used in medicine, and we discussed potential positive societal impacts in Section 5 We do not foresee any meaningful negative societal impacts of the work worth discussing.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: we used the panda-py open source package from Elsner 2023 and credited appropriately in the Results section in the text.
Guidelines:
The answer NA means that the paper does not existing assets.
18

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code and data have been released with proper instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The detailed procedure our human-study, including given instructions and financial compensation, can be seen in Section
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [Yes]
19

---

There are no potential risks incurred by study participants, and an IRB has approved (Section
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
wwwpuriisscc/oooffeeee /ctc//ooeeencce/2255///////////e MLMM
20

---

A Robotic Data Collection Setup
While the simulation setup useful for quick development iterations, it does not mimic the real-world, and the learned representations will fail generalize outside the simulation. To test our approach on a real tactile sensor we designed setup to mimic simplistic palpation examination.
A.1 Xela uSkin and Digit Comparison
We use a single Xela uSkin [omo et al. 2018 which has 30 force sensors, each measuring forces in y, and directions.
In our work, have not made strong assumptions the tactile sensor, and in theory, our learning approach can be used with other sensors. We have tested the behavior Digit sensor (Lambeta) al., 2020]. To test the behavior of Digit compared the uSkin, we performed simple experiment. We first touched a phantom containing an insert without a lump, and then touched an insert with lump exactly top the lump. We repeated this test with the Digit and uSkin, and the results reported in Figure 6 As can be seen in the Figure, when touching the samples with Xela sensor, we get significant measurements when touching the shell, while the digit doesn't seem be affected by it. We tried to measure various objects with the digit sensor and saw that it is very good at detecting surface patterns and is responsive hard surfaces, but very lacking in sensitivity soft objects and depth changes. In an experiment we conducted carlier, we tried the Xela vs. the digit in previous generation of inserts. We easily classified the touching lump no lump with the Xela and completely failed with the digit.
(a)
(b)
(c)
(d)
Figure 6: Comparison of tactile perception pipelines. (a), (b): Xela system - robot and tactile sensor views. (b) shows captures from the Xela visualizer. The top image depicts state of no touching, the middle one touching a phantom without lump, and the bottom one touching lump. (c), (d): Digit system robot and tactile sensor views. (d) Images obtained by the Digit sensor. The sensor positions are the same as in (b).
21

---

(c)
(b)
Figure 7: In (a): Section view of the shell and insert assembly. In green, insert silicone skin. In blue, shell silicone skin. In dark gray, PLA bases for the insert and shell. Voids are filled with slime. Note that holes in the bases are filled with silicone for anchoring. (b) Lump sizes: In white from furthest to closest, 8, 12, and 14mm diameter spheres. In Green ellipsoids with long/short axes of 12/8, 14/8, and 16/12mm In (c), four insert bases, each with 14mm lump a different location. Other lumps (not shown) can be inserted at the same locations.
A.2 Breast Phantoms
For the breast phantoms, we aimed for modular design, which will allow for effective data collection. Our phantoms are built from two parts a shell and an insert. Both shell and insert are made using thin outer layer of silicone (Smooth On Dragonskin FX) filled with "slime" gel matrix made of polyvinyl acetate (PVAc), borax, and water. We shall refer to the outer layer as skin. The skin is attached base plate made of polylactic acid (PLA) plastic using pre-made holes in the base plate. Inside the insert, we have the lump, which is made of the same Dragonskin FX silicone as the outer layer. See Figure 7 for section view of the assembly. The insert has an octagon shapee base plate that fits into the shell's base plate, this is so we can have eight configurations for each insert, lessening the amount of inserts needed. We created 24 inserts in the following configuration: lump locations as seen in Figure 7c with 3 spherical and 3 ellipsoidal lump sizes as seen in Figure 7b
A.3 Phantoms Fabrication
The skin for both the insert, and shell is constructed much like hollow chocolate bunnies or Easter eggs: we pour small amount of silicone into multipart mold and brush all the surfaces (Figure Sa next we close the mold and rotate it until it sets. When we open the mold, we have a hollow shell or insert. During the molding process the silicone enters holes in the base, which serve as anchor points for the silicone as seen in Figure 7a For the shell, the process with silicone ends at this stage. In the case of the insert, have to remove the lump mold (Figures 8b and 8c). pour silicone into the lump skin, and close the hole in the base with a plug (Figure &d).
Following the silicone stage, we then inject the slime using needle through a thick part of the phantom (Figure Seb; this works for both shell and insert. This concludes fabrication of the phantom.
A.4 Transparent Insert
To showcase the task's difficulty, manufactured single transparent insert so that the lump will be visible during handling. As can be seen in Figure 9 the lump moves when touched. This is a realistic phenomenon of the 3D structure that would appear in real tissue.
A.5 Data Collection Setup Design
To collect data in a consistent and repeatable manner, we utilize a Franka Emika Panda robotic arm with 7-DoF. The complete setup, which can be seen in Figure 10 consists of the robotic manipulator, the tactile sensor held as the end effector, camera attached to the manipulator, and the phantom. Both are attached/held using 3d printed adapters. We controlled the panda using panda-py (Flsner) 2023]. For the actual data sampling, we edited the force controller to be force- porce-position controller
22

---

(a)
(b)
(c)
(d)
(e)
Figure 8: Fabrication process for insert. (a) Silicone the mold parts is fully covered. (b) Parts after removal from the mold. (c) using mold as support. (d) Hole filled with silicone. The silicone is colored black for illustration purposes. (e) Injection of slime.
Figure 9: Poke trajectory on a special transparent insert.
order to press the phantom with a specific force at specific point. In order to be organized and automatic as possible, used the camera to scan data matrices (similar to QR codes) that attached to the shells and inserts to automatically record the used items and the insert orientation. Data was recorded for trajectories at 110 points on the phantom. We recorded Xela data 85[H2] 2), robot data at 100 H] and RGB images from the camera at 10[H=]. We collected the data with a fixed force and fixed sensor orientation.
A.6 Data Collection Technical Details
The robotic manipulator first scans the attached data matrices (similar to QR codes) to determine the labels of the phantom and the insert, and also obtain the orientation of the insert inside the phantom. The robot goes to each line in by 2[mm] steps until we sense significant force with the Xela sensor (the sensor output is noisy), and then backtracks 2[mm]. after that we use force/position controller that controls the force in Z and the position æ, y of the end effector for seconds, during that time we record all Xela measurements (8[Hz)),, data from the robot (100[Hz] g, dg. calculated force, lau_J.0_7 T_EPP and RGB images from the realsense camera (10[Hz] currently not use). We considered using the depth capability of the RealSense camera, but it had too much noise in these settings. We add
position (110 trajectories) and starts to descend in a straight
23

---

Figure 10: Our automatic robotic data collection setup consists of breast phantom (including visible QR codes), Franka Emika Panda robotic arm with a mounted Xela uSkin tactile sensor, and an RGB camera.
relevant timestamps to the data. Robot data consists the libfranka variables q, dq joint angles and angular velocity, calculated force force and torque at the end effector calculated using the Jacobian, Ty Measured link-side joint torque sensor signals, and O O_T_FE EE Measured end effector pose in base frame. After testing number of positions/sensor angles/forces, the best scan for our needs, which takes 20 minutes, will consist 110 positions on the phantom with single fixed force 3.8[N] and single sensor orientation.
decided that
B PalpationSim
B.1 Simulation Visualization
In Figure I1 we visualize trajectories from two models in the simulation. As be seen, the lump inside the breast model affects the sensed force, as expected.
B.2 Simulation Data Collection Hyperparameters
The hyperparameters used for the simulation data collection are shown in Table
B.3 Simulation Imaging Results
To visualize the results shown in Section 4 image reconstruction results in the simulation. As we can see our image predictor can reconstruct the model image fairly well from our tactile sequence representation.
we present in Figure 12
20 randomly sampled model-

---

Figure 11: Comparison of selected simulation trajectories for big and small lumps. The simulation interface includes a force graph measured by the probe (including force vectors for easy visualization), the probe in green, and the breast model.
Name
Value
Name
Value
nosse
0.0001
31
0.2
N points
16
82
0.999
Rprobe 0.1
0.001
Scollision
0.01
(b) Adam optimizer hyperparametens
(a) Probe hyperparameters
Name
Value
Name
Value
Moddel
N (1,0.01)
yMuump
0.01
Legrid
0.15
L perimeter
0.1
prtump
0.1
N points
0.001
Pehange
0.1
centeriump
Ring (0.44, 0.55)
Bym
U [0. (0.0027, 0.0033]
Rtump- chaange
U |0.07, 0.15
yym
0.0002
Rlump - no-nnonnheanee
U 00.11, 0.21
Her
U [0.09, 0.11]
N (0.03, 0.01)
Opr
0.01
Riamy
(d) Lump hyperparameters
(c) Breast-model hyperparameters
Table 3: Hyperparameters used for the simulation data collection
B.4 Simulation Change Detection
Since our image reconstruction in simulation accurately predicts the lump shape and location, we can use the image prediction for change detection and localization. As can be seen in Figure given two trials from the same model, we first generate multiple image predictions by taking different permutation of the trial trajectories (we used 10 permutations in practice). Using the multiple prediction, we can generate joint confidence map:
ci
C
(a 1 o2
Where c(i), 01 and 02[i] are the confidence map. first trial standard deviation and second trial standard deviation at pixel i, Vi [1, 128 128), respectively (the standard deviations calculated across the different predicted image of each trial). The confidence calculation behaves such that very high and low standard deviations correspond to confidence of 0 and respectively, and when 2- Finally, each pixel gets score:
an [i] = o2[i] = we get c(i]
s|1]
|u [i]
pa ji]
cld
25

---

Figure 12:
20 random test results of our simulation tactile imaging predictions. The ground-truth model image and our prediction are on the left and right of each column, respectively.
26

---

Figure 13: Our simulation change detection algorithm. (a) Two ground truth model images (b)
Prediction multiple images using the perturbations augmentation from each of the trials (c)
Generation of confidence map (d) Change map generation
Where lsl and μl2 the average pixel value at pixel i, Vi [1, 128 128). Using the scores plot a change score map and localize changes.
201
20
change
127
Lut lubel
Figure 14: Our change prediction confusion matrix, when setting the threshold the change score to 0.1
To aggregate the change map to single score for the two trials, we simply take the mean over all pixels. We report the confusion matrix when setting the threshold to 0.1 in Figure [4
C Human Study Procedure and Results
Procedure The full procedure and given instructions for each participant are detailed below:
For explanation purposes, single insert is put into single shell.
Our research is about detecting breast cancer using artificial palpation. We have breast phantoms, consisting of shells with inserts inside them, where each insert has a lump inside it. All lumps are spherical, and change in diameter and location. There is always single lump inside of the insert. Your task will be to try and detect changes in the lump after a given amount certain amount of time. The lump can only grow larger or stay the same; it cannot change location, nor shrink. The lump will change with 0.5 probablity, otherwise it will stay the same. Please follow these instructions while palpating:
Use only one finger when palpating
You can press however you like and for as long as you want to, but use reasonable force not to tear the phantom
In your answer, state "change" or "no change"
Here, there is a phantom with lump inside, just as an example, is not part of the trial, you can feel get sense the task.
Next, the insert changed, with the smallest possible change in the lump size (to calibrate the participants answers).
27

---

Now we have replaced the insert have larger lump size, such that the change in size is the lowest possible change. You can now palpate to understand what a change might feel like.
Finally, the following steps are repeated ~ 4 times:
1. A completely new insert is put into phantom (with random lump location, orientation, and size)
2. The participant palpates the phantom
3. After the time interval, the lump, with 0.5 probability the insert is replaced with different insert with the same location and orientation, but a larger size, and with 0.5 probability, the insert is taken out and put back in.
4. The participant palpates the phantom again, and their answer is logged together with the experiment metadata.
Results We had 15 participants, and explained above, each participant repeated the experiment ~> 4 times (with different time intervals). The results were 23/34
Humans
Ours
FAR
0.32
0.19
(~> 68%) correct non-changing classifications and 18/29 (~ 62%)ct changing classifications.
Recall
0.62
0.82
Table 4: Change detection recall and
Ethical Concerns The human study was approved by the Technion's 's Institutional Review Board (IRB), approval number 2025-068. There are no dangers to participants. The experiment is equivalent to playing with a squishy toy for several minutes, The materials are safe silicone, and the inner filling (which is not touched by participants) made out of gel that commonly sold as children's 's toy (Borax, water, and glue). There is absolutely no inconvenience of any kind to the participants during the experiment. This experiment allows us to calibrate the performance of our system in comparison to human skills, which is important for understanding the potential of our research. Each participant received financial compensation above the minimum wage. The data recorded is kept private on a secure server, which only contains the names and emails of the participants and answers questions. All files password password-protected. with passwords only known to the relevant researchers. The results we publish summarized statistics, and do not contain any personal information about participants. Each participant interacted directly with one of the researchers in the project. We explained verbally and in written instructions participants that their participation voluntary and that they can withdraw any time with no negative consequences for them.
FAR for human participants and our ap- proach.
D Self-supervised Training Technical Details
Our self-supervised architecture, as shown in Figure 2 is composed of the FLE (Force Location Encoder), sequence encoder, and a force decoder. The FLE is simply adding linear projection for the forces and basic sinusoidal Positional Encoding (PE) for the 6-dim locations (both the projection layer RNN with GRU and with 1024
and the PE are of size 256). The sequence encoder is hidden size. The force decoder first linearly projects the input representation and uses PE on the desired reconstruction to size 1020, and adds both together. Next, a three-layer MLP is used (with 2048, 1024 hidden sizes) to predict the forces at the desired location. An MSE loss (with random indices sampling) is used as shown in Section 3.2
visualization of the results of the reconstruction from the last representation shown in Figure
E
Downstream Tasks Technical Details
To reconstruct the MRI images, we use the pre-trained (and frozen) encoder from the self-supervised step. We aim to learn a mapping from the last-step representation to the model MRI image. To do so, we use vanilla conditional flow matching to reconstruct the 128 x 128 image from the vector representation. After training, in order to sample image, simply draw single noise sample
28

---

wwwmy
Figure 15: Force reconstruction on real data of the 2-axis component of each of the 30 sensors from the representation in the last time step (#N). In blue and red are the and reconstructed forces, respectively.
and map it using the learned conditional mapping an image. For the tactile imaging problem, we randomly split the trials to train and test (with the same split for the representation learning and the imaging). As for the change detection, we split per insert configuration, where all trials from the same configuration are either in train or test (this is done to have negative examples in the change detection task).
F Architecture Ablation
To motivate the chosen architectures, we perform an ablation study, both for the representation learning and for the tactile imaging. For the representation learning architecture tested both an RNN with GRU (GRU) and a two-layer vanilla transformer (TR). For both architectures, we also considered version where reconstruct only these. For the tactile imaging architecture, we considered the flow matching (FM) and series of transposed convolutions (TC). We perform the training for the ablation study using x0.5 of our full training data due to cost and time concerns.
of the input poke trajectories (e.g 20%) masked, and
The results of the ablation study be seen in Table 5 While this ablation study is preliminary, we chose the GRU and flow matching models for the representation learning and tactile imaging, respectively. We leave further investigation into more advanced architecture to future work.
Table Lump Size and Center-of-Mant (CoM) errors. We report standard deviation of the sample mean across 3 random seeds.
Method
Size Error [%]
CoM Error [mm] 1
GRU + FM (Ours)
20.9 + 0.9
3.6 - 0.1
GRU TC
27.9 + 1.8
3.5 0.1
TR + FM
3
6.7 0.3
Masked TR FM
57.9 2.0
9.9
Masked GRU + FM
23.9 - 1.0
4.0 0.1
29

---

G Tactile Imaging Results
To visualize the results shown in Table 1 present in Figure]66400 randomly sampled MRI prediction results. The predictions are accurate, in terms of lump location and size, as was quantitatively shown in [
MRI Scan
Prediction
MRI Scan
Prediction
MRI Scan
Prediction
MRI Scan
Prediction
Figure 16: 40 random test results of our MRI predictions. The ground-truth MRI image and our prediction are the left and right of each column, respectively.
30

---

H Force Map Generation Procedure
Figure 17: Top view of our 3D tactile visualization for a constant insert. Each visualization cor- responds trial taken from the insert different orientation (from rotation clockwise, starting from in the top left corner).
The fact that tactile measurements are not interpretable in their raw form, makes it difficult to debug the data collection process. To this end created visualization tool for tactile data, collected from a single trial (with multiple trajectories). We aggregate the tactile measurements at each location by taking the maximal z force measured by all 30 sensors. The result can seen Figure I7 and an interactive 3D version can be found in the supplementary materials.
(a)
(b)
(c)
Figure 18: Force map generation process. (a) The 3D forces visualization. (b) The KDE image. (c)
The resulting image after taking threshold over the KDE image.
As can be in Figure the generated force map is correlated the lump location. Hence, we hypothesized that can be used as baseline for the tactile- setill--maggigggg problem. As first simple baseline, we take the mean over the forces and locations of the last 10% data points of each trajectory in the 3D map. Next, we use Kernel Density Estimation (KDE) to produce an image from individual measurements. Finally, we take threshold to binarize the image (which accuracy). The result can be seen in Figure 18 As can be seen from the image, although the force map is very informative, it is enough for accurate lump size and CoM prediction.
set to maximize test
The result above illustrates that we cannot directly use the force map to predict the lump properties. To further motivate approach, also show that the force map is an insufficient representation for
31

---

tactile imaging. To do so, we trained an identical flow matching predictor to the one we used on top of our learned representation, but with a flattened force map. The results can be seen in [ Clearly, the (non-learnable) force-map representation underperforms compared to our approach. We have also tried alternative architectures for the prediction, including a convolutional architecture on top of non-flatten force-map representation.
I MRI Image Preprocessing
As explained in the main text, our imaging reconstruction is a classification problem, aiming to reconstruct of the classes background/hoodylumg for each pixel. So, in order to the MRI images as ground-truth, we first had to build a pre- pre-processing tool.
Although MRI data is 3D in nature, since our lumps are manufactured with similar heights, chose to work with a single horizontal slice for simplicity. The procedure for our MRI pre-processing shown in Figure 19, and examples of the final result are shown in Figure 20
MRI Scan
Figure 19: MRI pre- pre-processing procedure. We 3D printed hub, which can hold up to 6 inserts at time for the MRI scan in order to save time and resources. After the MRI scan, we take slice at constant height and use Otsu's 's threshold Otsu et al. 1975 to binarize the image. Finally, we rotate the scan to achieve any desired insert orientation.
Shell Classification
To further show the expressiveness of our extracted representation, we chose another downstream task that cannot be inferred from the tactile imaging output. We aimed to classify the shell based on the tactile sequence. We have trained small MLP on top of the frozen representation with a cross-entropy loss, aiming to classify between the 4 possible shells. Although the shells were manufactured in the exact same way, the representation expressive enough capture small manufacturing artifacts, effectively distinguishing between the shells with an accuracy of 99.6 = 0.7 on test sequences. The confusion matrix can be seen in Figure 21
K
Compute
The PalpationSim simulator runs on CPU. We collected data from it by running the simulation ~ 500 CPUs to parallelize the process, but a single instance is very light-weight. All the training procedures, including the self-supervised phase and the image reconstruction training ran cluster of 12 A4000 GPUs, although each single run needed only a single GPU to run.
32

---

1.3, d15
313
3, 2. d15 3, dIo 5, 0, 6, <13 G, 4, 2. d25 6, d30
12: c1B
12, c5
12, 410
12, d15
14, c13
14, c5
14, d10
14, d15
8, c13
8, c5
d10
8, d15
Figure 20: MRI images used ground-truth, after pre-processing. Each insert was scanned times. Above each trio, we show the first and second radii, the orientation, and location (either diagonal of center and radius from center) of the ellipsoid lump, where, in the case of spherical lump, only signed radius and orientations are shown.
class
38.00
0.00
0.30
0.00
choss
1.00
29:00
0.30
0.00
Predcted
dess
0.00
11.
0.00
Class
0.00
4.00
diss_00
d
class_2
class_3
Thue
Figure 21: Confusion matrix for the shell classification downstream task on test sequences. Other than mistake, the model has successfully classified all sequences.
33


Reviews:
[{"review_id": "aXaLsPKrUY", "reviewer": "Reviewer_1RZ3", "summary": "The paper explores quantifying palpation forces in tissues with motivation for breast cancer imaging and the need to reduce subjectivity as well as detect/quantify subtle changes.\nThey do this via computer-based simulation of lumps within tissues as well as phantom-based experiments showing promising results in predicting MRI slices from aggregated reprsentation of sequence of palpation forces measured by the tactile sensors.\nThey show that self-supervised learning on sequences, (predicting future or past forces from a sequence), can be useful in representation learning and excels in low-data regime. \nThe self-supervised learning is modelled as non-autoregressive sequence-to-sequence prediction task.\nExploration of Recurrent Neural Network-based architecture (specifically, GRU) was done (although transformed-based architecture also resulted in similar results.)", "questions": "The results section could be made more clear. \n\nPlease explain why \"In contrast with the simulated predictions, ..., We do not find the F1 score to be informative for interpreting image quality\"?\n\nIn Table 1, the explanation of what these methods  \"Image Pred\", \"Linear Pred.\", \"Force Map\" mean is scattered throughout the paragraphs. \nIt would have been useful for reader to have these explanations done upfront in one-place including additional row for \"naive baseline\" that predicts the average lump area and CoM for all examples.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is easy to follow (results section could be improved).\n\nThe paper explores quite novel and challenging task of tactile imaging via artificial palpation using force sensors to explore imaging the tumor structure within a tissue, albeit in a simplified setting of simulations and phantoms. \nGiven the rigor, extensiveness of the setup and the obtained data, the work is significant and original.\n\nOn the other hand, method-wise, limited exploration of the representation learning within the self-supervision framework has been done. \nBut given the dataset being made available publicly, the community can build upon the work and explore methodological innovations and improve the performance on the downstream clinical metric, in this case, extracting size and location of the lump, which remains far from clinically permissible.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful review.\n\n\nFirst, following a careful investigation of our data after the submission deadline, we found that our Xela sensor had a time-varying bias in the measurements. Fixing for this bias (by subtracting the first measurement in each trial) dramatically improved our real-world results (e.g., an F1 score improvement from 20% to 65% and real imaging results are very similar to simulated ones). We report these fixed results below, including additional experiments suggested by the reviewers.\n\nRebuttal instructions forbid us from adding images to the anonymous URL in the paper. If the AC agrees, we will update the images on the website - our predictions are now much better, almost as good as in the simulation.\n# Additional Results   \nAll experiments were repeated with 3 random seeds. \n\n## F1 as a Palpation Imaging Metric\nWe thank the reviewer for this question. We did not present the F1 score for two reasons:\n1. Compared to the CoM and Size prediction errors, the F1 score is less intuitive.\n2. In our real-world results, the F1 score was often zero (for a lump prediction without overlap with the ground truth). \n\n\nIn our amended results, following the Xela bias fix, the F1 is informative, and we include it. \n## Sensor Calibration (SC)\nModified results based on subtracting the first measurement in each trial.\n\n| **Method**    | **Size Error [%]** | **CoM Error [mm]** | **F1 Score** |\n| :------- | :-------: | :-------: | :-------: |\n| Average Prediction  |  51.9 ± 47.5 | 10.7 ± 3.6 | N/A |\n| MLP | 24.1 ± 0.2 | 7.6 ± 0.1 | N/A | \n| GRU | 9.5 ± 5.7 | 9.7 ± 1.3 | 19.5 ± 0.3 |\n| MLP + SC |  11.0 ± 1.1   | 3.7 ± 0.1 | N/A | \n| GRU + SC | 27.9 ± 1.8 | 3.1 ± 0.1 | 65.3 ± 0.2 |\n\n## Learning Architecture\nAs requested by multiple reviewers, we have performed an ablation study on the representation learning architecture, as well as the imaging predictor. \nWe tested with a Transformer (TR) [1]  instead of a GRU and a Flow Matching (FM) [3] instead of the Transposed Convolution (TC). \nWe also tried using a masked autoencoder (similar to [2]), where we mask 20% of the trajectories and reconstruct them using the decoder. \n\n| **Method**  | **Size Error [%]** | **CoM Error [mm]** |\n| :-------- | :-------: | :-------: |\n| GRU + TC + SC | 27.9 ± 1.8 | 3.1 ± 0.1|\n| GRU + FM + SC | 20.857 ± 0.9 | 3.2 ± 0.1 |\n| TR + FM + SC | 35.4 ± 0.7  | 5.9 ± 0.3 |\n| Masked TR + FM + SC | 57.9 ± 2.0 | 8.7 ± 0.1 |\n| Masked GRU + FM + SC | 23.9 ± 1.0 | 3.5 ± 0.1 |\n\nThe GRU, together with a Flow Matching predictor, works best. We believe that this experiment, together with the sensor calibration results, shows that the GRU is a reasonable choice. \n\n## Force Map Learnable Baseline\nAs suggested by reviewer SEnm, we added a learnable predictor with the suggested force map as a representation.\n\n| **Method**    | **Size Error [%]** | **CoM Error [mm]** |\n| :------- | :-------: | :-------: |\n| Force Map | 262.9 ± 234.4 | 15.9 ± 13.0 |\n| Force Map + FM | 49.7 ± 2.4 | 5.1 ± 0.1 |\n\nWe can see that our approach outperforms this baseline by a large margin, indicating the effectiveness of our self-supervised representation learning.\n\n## Data Scaling\nAs suggested by reviewer hsJh, to further support the claim of the scalability of our approach, we have collected more data (x1.7 from what was presented in the paper, collected after the submission deadline), and performed two additional experiments:\n1.  Pretrained the self-supervised encoder on the x1.7 data, while training the downstream tasks on the original (x1) data:\n| **Method** | **Size Error [%]** | **CoM Error [mm]** | **F1 Score** | \n| :------- | :-------:| :-------: | :-------: |\n| MLP + SC |  11.0 ± 1.1   | 3.7 ± 0.1 | N/A |\n| GRU + FM + SC | 20.857 ± 0.9 | 3.2 ± 0.1 | 65.3 ± 0.1 |  \n| GRU + FM + SC x1.7 rep data | 18.6 ± 1.5 | 2.4 ± 0.0 | 72.7±0.3 |\n| MLP + SC x1.7 rep data | 11.7 ± 0.4 | 3.1±0.1 | N/A |\n\n2. Pretrained both the self-supervised encoder and downstream tasks on the x1.7 data (as well as x0.5 and x0.25 data, as shown in the paper). The results are as follows:\n| **Method** | **Size Error [%]** | **CoM Error [mm]** | **F1 Score** | \n| :------- | :-------: | :-------: | :-------: |\n| GRU + FM + SC x0.25 data | 99.8 ± 0.1 | 8.7 ± 0.2 | 0.8 ± 0.4 |\n| GRU + FM + SC x0.5 data | 41.6 ± 3.2 | 5.4 ± 0.4 | 43.9 ± 2.6 |\n| MLP + SC x1.7 data | 10.5 ± 0.5 | 2.6 ± 0.1 | N/A |\n| GRU + FM + SC x1.7 data| 23.0 ± 2.1 | 2.1±0.0 | 74.4±0.1 |\n| Force Map + FM x1.7 data | 46.6 ± 2.8 | 4.7 ± 0.1 | 47.3 ± 1.3 |\n\nObserve that additional data helps more in the self-supervised pretraining phase than to downstream tasks. This further validates our approach.\n\n\n# Results Section Clarity\nWe will add a sub-section explaining all of the used baselines in the revised version.  \nWe appreciate the suggestion of adding the naive, average prediction baseline; we agree it might help to ground the presented results. We added this baseline to our updated Table 1 (see above).\nIf the reviewer has any further ideas on how to improve the Results Section, please let us know. \n\n# “...method-wise, limited exploration of the representation learning within the self-supervision framework has been done”\nAs can be seen in the results above, we now include our results from experimenting with different representation learning architectures and imaging predictors.\n\n\n[1] Vaswani et al. Attention Is All You Need, 2017\n\n[2] He et al. Masked Autoencoders Are Scalable Vision Learners, 2021\n\n[3] Lipman et al. Flow Matching for Generative Modeling, 2022"}, {"review_id": "Kz2fqM5NYF", "reviewer": "Reviewer_hsJh", "summary": "This paper proposes a self-supervised learning framework for artificial palpation using tactile sensors on soft objects. The authors design a simulation and collect real-world tactile data with MRI as ground truth. A representation is learned via an encoder-decoder, and its usefulness is demonstrated on imaging and change detection. The motivation and method are well explained, and the visual materials help understanding. However, the downstream tasks are limited, and the model is only tested at small scale, raising questions about real-world applicability.", "questions": "- How can you conclude pretraining helps at scale when only small datasets (x1, x0.5, x0.25) are used?\n\n- Do you plan to add more downstream tasks to validate the learned representations?", "limitations": "- The experiments are done on a small scale and don’t show generalization to real clinical settings.\n\n- The paper presents limited use cases for the learned representation.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- Very well-written introduction and motivation, especially in the medical context.\n\n- The overall pipeline is clear and follows standard representation learning design.\n\n- Simulator visualizations are helpful.\n\n- Experiments show that the model performs better than force-map baselines.\n\nWeaknesses:\n\n- Only two downstream tasks are included, which is limited for a representation learning paper.\n\n- The scale of the data is small. It's unclear how the method would generalize to real human data.\n\n- Some results (e.g., pretraining on small-scale data) are stated strongly without strong support.\n\n- Figure 5(c) is hard to understand and needs a better caption.", "was_revised": true, "final_justification": "The rebuttal strengthens the paper with sensor calibration fixes, architectural ablations, a new force map baseline, and an additional downstream task. These address some of my earlier concerns, though I remain cautious about giving full weight to extensive post-deadline additions.\n\nThe dataset scale is now clear and impressive for this domain. However, the motivation for self-supervised learning could be better articulated—beyond the general observation that more data improves performance, it’s not fully clear why this approach is particularly valuable here.\n\nThe authors’ clarification on the long-term clinical goal is reasonable, and I suggest making this future plan more explicit in the paper. I maintain my borderline accept score.", "rebuttal": "We thank the reviewer for the insightful review.\n\n\nFirst, following a careful investigation of our data after the submission deadline, we found that our Xela sensor had a time-varying bias in the measurements. Fixing for this bias (by subtracting the first measurement in each trial) dramatically improved our real-world results (e.g., an F1 score improvement from 20% to 65% and real imaging results are very similar to simulated ones). We report these fixed results below, including additional experiments suggested by the reviewers.\n\nRebuttal instructions forbid us from adding images to the anonymous URL in the paper. If the AC agrees, we will update the images on the website - our predictions are now much better, almost as good as in the simulation.\n# Additional Results   \nAll experiments were repeated with 3 random seeds. \n\n\n## Sensor Calibration (SC)\nModified results based on subtracting the first measurement in each trial.\n\n| **Method**    | **Size Error [%]** | **CoM Error [mm]** | **F1 Score** |\n| :------- | :-------: | :-------: | :-------: |\n| Average Prediction  |  51.9 ± 47.5 | 10.7 ± 3.6 | N/A |\n| MLP | 24.1 ± 0.2 | 7.6 ± 0.1 | N/A | \n| GRU | 9.5 ± 5.7 | 9.7 ± 1.3 | 19.5 ± 0.3 |\n| MLP + SC |  11.0 ± 1.1   | 3.7 ± 0.1 | N/A | \n| GRU + SC | 27.9 ± 1.8 | 3.1 ± 0.1 | 65.3 ± 0.2 |\n\n## Learning Architecture\nAs requested by multiple reviewers, we have performed an ablation study on the representation learning architecture, as well as the imaging predictor. \nWe tested with a Transformer (TR) [1]  instead of a GRU and a Flow Matching (FM) [3] instead of the Transposed Convolution (TC). \nWe also tried using a masked autoencoder (similar to [2]), where we mask 20% of the trajectories and reconstruct them using the decoder. \n\n| **Method**  | **Size Error [%]** | **CoM Error [mm]** |\n| :------- | :-------: | :-------: |\n| GRU + TC + SC | 27.9 ± 1.8 | 3.1 ± 0.1|\n| GRU + FM + SC | 20.857 ± 0.9 | 3.2 ± 0.1 |\n| TR + FM + SC | 35.4 ± 0.7  | 5.9 ± 0.3 |\n| Masked TR + FM + SC | 57.9 ± 2.0 | 8.7 ± 0.1 |\n| Masked GRU + FM + SC | 23.9 ± 1.0 | 3.5 ± 0.1 |\n\nThe GRU, together with a Flow Matching predictor, works best. We believe that this experiment, together with the sensor calibration results, shows that the GRU is a reasonable choice. \n\n## Force Map Learnable Baseline\nAs suggested by reviewer SEnm, we added a learnable predictor with the suggested force map as a representation.\n\n| **Method**    | **Size Error [%]** | **CoM Error [mm]** |\n| :------- | :-------: | :-------: |\n| Force Map | 262.9 ± 234.4 | 15.9 ± 13.0 |\n| Force Map + FM | 49.7 ± 2.4 | 5.1 ± 0.1 |\n\nWe can see that our approach outperforms this baseline by a large margin, indicating the effectiveness of our self-supervised representation learning.\n\n## Data Scaling\n\n\nTo further support our scalability claim, we have collected more data (x1.7 from what was presented in the paper, collected after the submission deadline), and performed two additional experiments:\n1. Pretrained the self-supervised encoder on the x1.7 data, while training the downstream tasks on the original (x1) data:\n| **Method** | **Size Error [%]** | **CoM Error [mm]** | **F1 Score** | \n| :------- | :-------: | :-------: | :-------: |\n| MLP + SC |  11.0 ± 1.1   | 3.7 ± 0.1 | N/A |\n| GRU + FM + SC | 20.857 ± 0.9 | 3.2 ± 0.1 | 65.3 ± 0.1 |  \n| GRU + FM + SC x1.7 rep data | 18.6 ± 1.5 | 2.4 ± 0.0 | 72.7±0.3 |\n| MLP + SC x1.7 rep data | 11.7 ± 0.4 | 3.1±0.1 | N/A |\n2. Pretrained both the self-supervised encoder and downstream tasks on the x1.7 data (as well as x0.5 and x0.25 data, as shown in the paper). The results are as follows:\n| **Method** | **Size Error [%]** | **CoM Error [mm]** | **F1 Score** | \n| :------- | :-------: | :-------: | :-------: |\n| GRU + FM + SC x0.25 data | 99.8 ± 0.1 | 8.7 ± 0.2 | 0.8 ± 0.4 |\n| GRU + FM + SC x0.5 data | 41.6 ± 3.2 | 5.4 ± 0.4 | 43.9 ± 2.6 |\n| MLP + SC x1.7 data | 10.5 ± 0.5 | 2.6 ± 0.1 | N/A |\n| GRU + FM + SC x1.7 data| 23.0 ± 2.1 | 2.1±0.0 | 74.4±0.1 |\n| Force Map + FM x1.7 data | 46.6 ± 2.8 | 4.7 ± 0.1 | 47.3 ± 1.3 |\n\nObserve that additional data helps more in the self-supervised pretraining phase than to downstream tasks. This further validates our claims.\n\n## Additional Downstream Task\nAs suggested by the reviewer, we added an additional downstream task: to predict the phantom shell that we used (4 different phantoms, each with slightly different gel mixture inside, and minor shape variations due to our manufacturing process). This information is not contained in the imaging task, as we only reconstructed the image of the insert, not of the outer phantom shell. \nWe achieved a near-perfect classification accuracy of 99.6 ± 0.7, showing the representation contains effective information regarding the slight variations of phantom stiffness and shape (although all were produced in the same manner).\n\nAnother potential task that our data supports is 3D reconstruction of the MRI scans (vs. our 2D slice results). \n\nIf the reviewer has any further ideas for possible downstream tasks, we would be happy to incorporate them as well.\n\n# “The scale of the data is small”\nThe reviewer claims, “The scale of the data is small”. We are not sure what the reviewer means - small compared to what exactly? Our data is several orders of magnitude larger than previous datasets for soft objects that we are aware of (see Section 2 in the paper), and consists of ~550 different phantom-insert combinations with more than one hundred MRI scans for ground truth and 30 million tactile reading samples. \n\nNevertheless, to further support our claims, we ran additional experiments on an even larger-scale dataset (collected after the submission deadline), which can be seen in the Data Scaling results above.\n\n\n# Generalization to real clinical setting\nWe agree, this study does not validate the relevance of our new technology to real humans, but only takes a step in that direction (as discussed in depth in our Discussion Section). \nA study on humans requires several hundred patients who were tested positive for breast cancer, have not yet undergone surgery, have had an MRI scan recently done or planned, and are willing to undergo one or more artificial palpation exams.\nSuch a study takes several years to complete, and involves many costs: financial, logistical, and emotional (to the patients). It is part of our research plan (we’re already talking with a large hospital to support it), but before we must ready our technology: how to scan using a human operator, how to improve the imaging results, what are the accuracies that we expect to measure, how to make the scanning convenient as possible, how to make anatomically accurate phantoms, etc. This submission is the first of many steps in this project, and we believe the current stage is interesting to the ML community to improve the technology. In any case, we believe that real human findings will be a better fit for a medical journal than NeurIPS.\n\nThe contributions of the current study include:\n1. An extensive dataset, phantom manufacturing process, and simulation environment that can be used to improve the learning algorithms;\n2. Promising preliminary results on tactile imaging and change detection;\n3. Specific capability measurements for this technology (CoM and size errors), which can be improved upon in future studies.\n\n\n\n\n# “How can you conclude pretraining helps at scale when only small datasets (x1, x0.5, x0.25) are used? “\nSince we didn’t have more data, we proved our scalability claim using two results:\n1. Simulation result - In Figure 3d, we show that pretraining on a vast amount of self-supervised data helps in downstream tasks, especially when a little amount of supervised data is available.  \n2. Real data result - at the time of submission, we had collected data from ~550 phantom-insert combinations. We obviously couldn’t prove what would happen if more data were to be collected, but to hypothesize over it, we showed that if we were to train on x0.5 and x0.25 of the amount of data, we would get a serious deterioration in performance (as can be seen in Table 1). This demonstrates the trend of improvement vs. data size.\n\n\nTo further support this claim, we have now collected more data (x1.7 from what was presented in the paper) and performed two additional experiments, demonstrating similar trends. The results can be seen in the Data Scaling results above. \n\n\n# “Figure 5(c) is hard to understand and needs a better caption”\nThe force map generation procedure is depicted in Appendix G. We acknowledge that it is not sufficiently clear in the main text and will incorporate a dedicated subsection within the results section to describe all the used baselines in detail.   \n\n[1] Vaswani et al. Attention Is All You Need, 2017\n\n[2] He et al. Masked Autoencoders Are Scalable Vision Learners, 2021\n\n[3] Lipman et al. Flow Matching for Generative Modeling, 2022"}, {"review_id": "iS6zlhKejc", "reviewer": "Reviewer_SEnm", "summary": "The paper presents a pipeline to train tactile representations from trajectories of poking either a simulated or real breast phantom with embedded lumps of different sizes and locations. For this, the authors built a custom FEM-based simulator, PalpationSim as well as an automated data collection pipeline in the real world based on a Franka Panda arm and a Xela uSkin sensor. They furthermore introduce the manufacturing process of artificial breast phantoms for data collection. \n\nThe tactile representations are learned in a self-supervised way by predicting the next or previous tactile measurement of a poking trajectory. The paper shows that in the FEM-based simulated environment, the learned tactile representations are very informative for downstream tasks such as predicting a map of the breast phantom (based on MRI images). The authors also present results on data collected with real breast phantoms and show a comparison to human performance with a human subject study.", "questions": "How can the simulation results and the real world results be related? Which conclusions from the simulation can be used for the real world? \nThe authors state that the simulation was developed among other things to evaluate the amount of data necessary to obtain meaningful results. However, this objective does not seem to be met, as the authors describe severe overfitting of the image prediction network on real data, however, with the simulated data this is not the case. It would make the paper more impactful if it was more clear, which findings of the simulation can be transferred to the real world, as the simulated experimental results are significantly better than the one on the real phantom.\n\nIn the real-world experiments, the authors conduct a baseline experiment on the raw force maps that does not involve any learning, but instead a thresholding operation and report this to be much worse. A better baseline in my opinion would be to compare the learning-based method on the force maps as a representation, instead of the self-supervised learned one. This would also give conclusions about whether pretraining helps for the real-world experiments and allow a real assessment of the main claim of the paper, namely that the new representations are more informative than force maps\nOther design choices, such as indenter shape or number of force sensors in the simulation, should be justified and investigated. \n\nThe data collection procedure in simulation and the real world seems to differ significantly. In simulation, the indenter is approaching the soft body from a normal direction, while in the real world, the sensor is pressed down from the z direction, leading the different approaching angles relative to the surface at every location. How does this major difference impact the comparability of the results?\n\nIn the method formulation, a tactile sensor is described as a rigid body. However, many of the cited vision-based tactile sensors are soft. How high does the difference in material properties have to be, to assume that the tactile sensor is a rigid body? Would it change the theoretical background if the sensor were a soft body? The paper is presenting a more general theoretical framework in the beginning, therefore it would be interesting under which constraints this holds. \n\nThe architecture choices of using a GRU over a transformer-architecture are not transparent to me. Earlier results are mentioned but not reported. According to the manuscript, the GRU was chosen as it scales more easily to long sequences, while transformers are known to be better at processing longer sequences without the problem of vanishing gradients. Reporting the initial comparison results and explaining the architecture choice would be interesting. \n\nOn page 7 in the footnote, what is meant by the low spatial frequency of relevant forces, which the Digit is not optimized for? The Digit sensor provides a very high resolution spatial image, but no forces without calibration. \n\nThe manuscript references Section D in the Appendix for the Human subject study, but it should be Section C\n\nMinor points: \n\nClarity could be improved by giving more assertive names to the sensor readings vs the additional external observation of the body I (e.g. the MRI scan). As reading and observation are often used interchangeably, this was confusing for me. Calling the MRI Scan ground truth or something similar could resolve this.\n\nThis is a very minor point, but the use of the word \"slime\" seems inappropriate. The Oxford Dictionary defines slime as \"​any unpleasant thick liquid substance\". As I do not think you want to convey judgment about whether the substance, representing human breasts, is pleasant or unpleasant to touch, the word \"gel\" or another substitute could be considered.\n\nComments about the answer to the Ethics question:\nThe paper includes a human subject study with 10 participants, but does not mention whether this study was approved by the IRB or any other ethics board. This should be clarified.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The paper tackles the challenging task of perceiving softness differences by tactile sensing, and is clearly structured and written (besides a few issues detailed below).\n- While the representation learning method employs established encoder-decoder architectures with sequence models (GRU, MLPs, positional encoding), its application appears novel in the domain of tactile representation learning for soft body perception.\n- A new way of manufacturing breast phantoms with different layers and embedded lumps of various sizes is introduced, enabling systematic experimentation.\n- Extensive results on a custom FEM-based simulation are presented to support the main claim that the learned representations can capture intricate patterns in tactile measurements beyond a simple force map.\n\nWeaknesses:\n- It is questionable whether the promising results obtained in simulation transfer effectively to the real world, as in-depth experiments addressing this gap are missing.\n- As a consequence, the significance and practical impact of the findings remain unclear.\n- The machine learning architecture itself is not novel, so the empirical findings are more important", "was_revised": true, "final_justification": "After considering the authors’ thorough rebuttal and new experimental results, I am satisfied that my main concerns have been addressed. The authors have provided convincing additional evidence, clarified key methodological choices, and outlined clear plans for addressing outstanding issues. In light of these improvements and the overall contribution of the work, I have updated my score and recommend acceptance.", "rebuttal": "We thank the reviewer for the insightful review.\n\n\nFirst, following a careful investigation of our data after the submission deadline, we found that our Xela sensor had a time-varying bias in the measurements. Fixing for this bias (by subtracting the first measurement in each trial) dramatically improved our real-world results (e.g., an F1 score improvement from 20% to 65% and real imaging results are very similar to simulated ones). We report these fixed results below, including additional experiments suggested by the reviewers.\n# Additional Results   \nAll experiments were repeated with 3 random seeds. \n\n\n## Sensor Calibration (SC)\nModified results based on subtracting the first measurement in each trial.\n\n| **Method**    | **Size Error [%]** | **CoM Error [mm]** | **F1 Score** |\n| -------- | ------- | ------- | ------- |\n| Average Prediction  |  51.9 ± 47.5 | 10.7 ± 3.6 | N/A |\n| MLP | 24.1 ± 0.2 | 7.6 ± 0.1 | N/A | \n| GRU | 9.5 ± 5.7 | 9.7 ± 1.3 | 19.5 ± 0.3 |\n| MLP + SC |  11.0 ± 1.1   | 3.7 ± 0.1 | N/A | \n| GRU + SC | 27.9 ± 1.8 | 3.1 ± 0.1 | 65.3 ± 0.2 |\n\n## Learning Architecture\nAs requested by multiple reviewers, we have performed an ablation study on the representation learning architecture, as well as the imaging predictor. \nWe tested with a Transformer (TR) [1]  instead of a GRU and a Flow Matching (FM) [3] instead of the Transposed Convolution (TC). \nWe also tried using a masked autoencoder (similar to [2]), where we mask 20% of the trajectories and reconstruct them using the decoder. \n\n| **Method**  | **Size Error [%]** | **CoM Error [mm]** |\n| -------- | ------- | ------- |\n| GRU + TC + SC | 27.9 ± 1.8 | 3.1 ± 0.1|\n| GRU + FM + SC | 20.857 ± 0.9 | 3.2 ± 0.1 |\n| TR + FM + SC | 35.4 ± 0.7  | 5.9 ± 0.3 |\n| Masked TR + FM + SC | 57.9 ± 2.0 | 8.7 ± 0.1 |\n| Masked GRU + FM + SC | 23.9 ± 1.0 | 3.5 ± 0.1 |\n\nThe GRU, together with a Flow Matching predictor, works best. We believe that this experiment, together with the sensor calibration results, shows that the GRU is a reasonable choice.\n\n## Force Map Learnable Baseline\nAs suggested by the reviewer, we added a learnable predictor with the suggested force map as a representation.\n\n| **Method**    | **Size Error [%]** | **CoM Error [mm]** |\n| -------- | ------- | ------- | \n| Force Map | 262.9 ± 234.4 | 15.9 ± 13.0 |\n| Force Map + FM | 49.7 ± 2.4 | 5.1 ± 0.1 |\n\nWe can see that our approach outperforms this baseline by a large margin, indicating the effectiveness of our self-supervised representation learning.\n\n## Data Scaling\nAs suggested by reviewer hsJh, to further support the claim of the scalability of our approach, we have collected more data (x1.7 from what was presented in the paper, collected after the submission deadline), and performed two additional experiments:\n1. Pretrained the self-supervised encoder on the x1.7 data, while training the downstream tasks on the original (x1) data:\n| **Method** | **Size Error [%]** | **CoM Error [mm]** | **F1 Score** | \n| -------- | ------- | ------- | ------- |\n| MLP + SC |  11.0 ± 1.1   | 3.7 ± 0.1 | N/A |\n| GRU + FM + SC | 20.857 ± 0.9 | 3.2 ± 0.1 | 65.3 ± 0.1 |  \n| GRU + FM + SC x1.7 rep data | 18.6 ± 1.5 | 2.4 ± 0.0 | 72.7±0.3 |\n| MLP + SC x1.7 rep data | 11.7 ± 0.4 | 3.1±0.1 | N/A |\n2. Pretrained both the self-supervised encoder and downstream tasks on the x1.7 data (as well as x0.5 and x0.25 data, as shown in the paper). The results are as follows:\n| **Method** | **Size Error [%]** | **CoM Error [mm]** | **F1 Score** | \n| -------- | ------- | ------- | ------- |\n| GRU + FM + SC x0.25 data | 99.8 ± 0.1 | 8.7 ± 0.2 | 0.8 ± 0.4 |\n| GRU + FM + SC x0.5 data | 41.6 ± 3.2 | 5.4 ± 0.4 | 43.9 ± 2.6 |\n| MLP + SC x1.7 data | 10.5 ± 0.5 | 2.6 ± 0.1 | N/A |\n| GRU + FM + SC x1.7 data| 23.0 ± 2.1 | 2.1±0.0 | 74.4±0.1 |\n| Force Map + FM x1.7 data | 46.6 ± 2.8 | 4.7 ± 0.1 | 47.3 ± 1.3 |\n\nObserve that additional data helps more in the self-supervised pretraining phase than to downstream tasks. This further validates our self-supervised representation learning approach.\n\n# Simulation Design\nWe emphasize: the simulation is mainly for quick iteration on design choices in the learning algorithm and data collection setup. Since the simulation is 2D and quite simplistic, *it is not meant for sim2real transfer*, and we do not expect the results in the real world to be the same as those in the simulation.\nOur real world data collection and training required significant time and resources, and it was important to establish general guidelines from simulation beforehand: the number of different objects required (hundreds), the length of palpation trajectories (tens of samples), the number of trajectories per model (tens), the factors of variation in the objects (lump size and location). In the future, we believe the simulator will be useful for (1) investigating different representation learning architectures with much larger data, (2) investigating active learning for data collection, among other ideas. \nInsights that we validated that transfer from simulation:  (1) Other than the sensor force measurement and location dimensions, our self-supervised learning and downstream predictors are the same; (2) We show in the new Data Scaling results above that the simulation scaling law, presented in Figure 3d, is also valid in the real world. \nTherefore, we see the simulation as one of the contributions of the paper. We did not perform an ablation study on the number of sensors on the probe since we cannot change this in our hardware. For the “poke” trajectory's angle, we believe this is a minor detail; an improved data collection policy that we will pursue in future work would rotate the probe in various angles like a human. If the reviewer insists, we can also incorporate this study in the revised version.  \n# “A better baseline in my opinion would be to compare the learning-based method on the force maps as a representation”:\nWe thank the reviewer for the excellent idea. \nThe results of a learning-based method on the force maps as a representation can be seen in the additional results above. Note that the advantage of self-supervised learning is evident both from these results but also from Figure 3d - we improve the representation just from self-supervised interactions, without image labels. This is not possible in a non-learned representation like the force maps.\n# “In the method formulation, a tactile sensor is described as a rigid body”\nFor all practical purposes, both the Xela sensor and a common vision-based tactile sensor can be assumed to be rigid, as they are several orders of magnitude harder than the phantoms in our experiments. This can clearly be seen for the Xela sensor in the videos on the supplementary project website.\nTo further support this, we measured the deformation of different sensors under 5N force (the largest force applied during our measurements). The Xela sensor was deformed by 0.19+-0.04 [mm], and a Digit vision-based sensor [4] was deformed by 0.18+-0.02 [mm].\nWe are not aware of significantly softer tactile sensor products. This may be an interesting future direction to explore.\n# Representation Learning Architecture\nGRUs (or any RNNs for that matter) memory consumption depends linearly in the sequence length, compared to transformers, which scale quadratically. Since in our case the sequences are quite long (up to 50,000 sensor readings per sequence), this became a significant issue and required significant computing resources to run.\nWe ran an experiment showing that GRU also outperformed a vanilla Transformer in this setting (and other architectures), which can be seen in the results above. We acknowledge that there is still plenty to experiment with in this direction, but hypothesise that the data is more important than the representation learning architecture in this setting (as evident from our sensor calibration and data scaling results above).\n# “On page 7 in the footnote, what is meant by the low spatial frequency of relevant forces, which the Digit is not optimized for? The Digit sensor provides a very high resolution spatial image, but no forces without calibration.”\nThe digit is excellent at capturing high frequency variations in the image, for example, when pressing on a coin or cloth, the digit can identify the small changes in pressure that the coin indentation or cloth fibers apply. However, when pressing inside the soft phantom, a relatively similar pressure is applied on all the image pixels, and the sensor is effectively saturated. See Figure 6d for an example. Nevertheless, extending our approach to use gelsight sensors in some way is an exciting direction for future study.\n# Human-Study Ethical Concern\nThe experiment required each participant to touch something like a squishy toy for several minutes and answer a question. There was absolutely no danger or discomfort involved. \nWe have now obtained IRB approval. Since our institution does not allow retroactive approval, we will redo the experiment (under exactly the same conditions).\nIn any case, this preliminary study was meant to give an impression of the difficulty of the task. A more extensive evaluation will be required to fully calibrate the performance of this technology, which is deferred to future work.\n\n\n\n\n# Clarity and Writing Comments\nWe thank the reviewer for carefully reading our paper and finding these issues.\nWe will make the following changes in the revised version:\n1. Fix the references for the Human subject study in the Appendix.\n2. Change the notation used for sensor reading and MRI scans (will indeed call the MRI scans ground truth as suggested).\n3. Change “slime” to “gel” throughout the text.\n\n[1] Vaswani et al. Attention Is All You Need, 2017\n\n[2] He et al. Masked Autoencoders Are Scalable Vision Learners, 2021\n\n[3] Lipman et al. Flow Matching for Generative Modeling, 2022\n\n[4] Lambeta el al. DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation, 2020"}]

Author rebuttal:

Meta-review: This paper describes a preliminary study designed to investigate the feasibility of using palpation data collected with a force sensor on a robotic arm to build a representation of a soft object that could be used to provided clinically relevant data such as the size and location of possible tumors or changes in a embedded mass over time that could be indicative of tumor growth.

The paper investigates the efficacy of the representations built from the collected data using a neural representation by comparing predictions made with the model to MRI images which are used as ground truth. It also presents results on simulated data evaluating the utility of building a representation model vs building a model solely based on the data acquired from a particular simulated phantom.

Some of the papers strengths include:

A reasonably detailed 2D simulation experiment which provides some initial evidence for the approach under controlled settings where ground truth is known.

A fairly extensive data set collected on a custom designed phantom where the position of the inserted tumor was varied and the force sensor and MRI data are collected.

Preliminary results which support the claims made in the paper regarding the possible utility of this palpation modality for tumor sensing.

Some weaknesses include:

The lack of data on human subjects since that was beyond the scope of this preliminary study.

Concerns regarding IRB approval for a small human subject experiment which was designed to provide a baseline of a humans ability to make decisions based on palpation.


As a preliminary study this paper provides some initial evidence about the utility of exploring palpation data for anatomical analysis. The representation learning techniques applied are fairly standard but the domain is novel and the authors present arguments for the design choices they have made.

The authors provided a detailed rebuttal that addressed most of the reviewers concerns about data set size and design decisions for their approach to representation learning. They also provided a path to address concerns related to the use of human subjects by stating that the experiments could be re-run with IRB approval and by addressing in the document the questions raised by the ethics reviewers.
Decision: Accept (poster)