Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper
Authors: Xinyue Zhu
Binghao Huang
Yunzhu Li
Venue: NeurIPS 2025 poster ()
Keywords: Visuo-Tactile Sensing
Fine-Grained Manipulation
Scalable Data Collection
Primary area: reinforcement_learning

Abstract: Handheld grippers are increasingly used to collect human demonstrations due to their ease of deployment and versatility. However, most existing designs lack tactile sensing, despite the critical role of tactile feedback in precise manipulation. We present a portable, lightweight gripper with integrated tactile sensors that enables synchronized collection of visual and tactile data in diverse, real-world, and in-the-wild settings. Building on this hardware, we propose a cross-modal representation learning framework that integrates visual and tactile signals while preserving their distinct characteristics. The learning procedure allows the emergence of interpretable representations that consistently focus on contacting regions relevant for physical interactions. When used for downstream manipulation tasks, these representations enable more efficient and effective policy learning, supporting precise robotic manipulation based on multimodal feedback. We validate our approach on fine-grained tasks such as test tube insertion and pipette-based fluid transfer, demonstrating improved accuracy and robustness under external disturbances. Our project page is available at \url{https://binghao-huang.github.io/touch_in_the_wild/}.

Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper
Xinyue Zhu!,"
Binghao Huang
Yunzhu Li'
Columbia University
'Equal contribution
Abstract
Handheld grippers are increasingly used to collect human demonstrations due to their ease of deployment and versatility. However, most existing designs lack tactile sensing, despite the critical role of tactile feedback in precise manipu- lation. We present portable, lightweight gripper with integrated tactile sors that enables synchronized collection of visual and tactile data in diverse, real-world, and in-the-wild settings. Building on this hardware, we propose a cross-modal representation learning framework that integrates visual and tac- tile signals while preserving their distinct characteristics. The learning proce- dure allows the emergence interpretable representations that consistently focus on contacting regions relevant for physical interactions. When used for down- stream manipulation tasks, these representations enable more efficient and ef- fective policy learning, supporting precise robotic manipulation based multi- modal feedback. We validate our approach on fine-grained tasks such as test tube insertion and pipette-based fluid transfer, demonstrating improved accu- racy and robustness under external disturbances. Our project page is available at https: httpb/inghoohuuuee github. io htt//ouch_iin.. the wild/. Keywords:
Sensing, Fine-Graines Manipulation, Scalable Data Collection
Hardware Store
Teat Tube Collection
Pencil Inseriion
Reorient
Reorient
Fish-Eye Camera
Makerspice
Flnid
Whiteboard Erasing
Tactile Seasers
Campus
Lefi Tactile
Right Tactile
In-the-Wild Visuo-Tactile Data Collection
(b) Visuo-Tacce Tactile Policy Rollout
Figure 1: (a) Our portable handheld gripper enables synchronized collection of visual and tactile data, supporting large-scale data collection in the wild. (b) We introduce multimodal representation learning framework that fuses visual and tactile inputs support fine-grained downstream manipulation tasks.
Introduction
Humans naturally rely on both vision and touch when interacting with the physical world. Whether inserting key into lock or adjusting a pipette during lab experiment, tactile feedback plays critical role in guiding precise motor actions, especially in situations where visual information may
39th Conference Neural Information Processing Systems (NeurIPS 2025)

---

be unreliable due to occlusion, poor lighting, or dynamic backgrounds. While vision provides global, semantic context, touch offers local, high-resolution feedhack about contact and force. The integration of these two modalities is fundamental to effective manipulation in everyday environments.
Recent advances in handheld grippers have made it easier collect human demonstrations "in the wild." However, most existing systems focus exclusively on visual sensing, largely neglecting tactile feedback. This gap limits their usefulness for capturing the fine- mre-grained, contact-rich strategies humans use in real-world tasks. Moreover, relying on vision alone makes these systems vulnerable to environmental variability, whereas tactile sensing offers a complementary and robust signal that is invariant to lighting conditions and camera viewpoint.
Two key challenges have prevented widespread visuo-tactile data collection in the wild: (i) Portable tactile hardware. Most existing tactile sensors are bulky, rigid, or not robust enough. For example, soft bubble sensors [1] are too large for handheld use, while other optical based sensors [2] are often bulky or less robust for long-term in-the-wild use, making them less suitable for mobile or outdoor deployment. (ii) Learning from multimodal data. Tactile and visual signals differ significantly in scale and in the nature of collected information. Tactile inputs are local and physical, while vision encodes broader spatial context. Learning effective representations that integrate both modalities -atttiii--pptccularly from large-scale, unstructured datasets- an open challenge.
To address these issues, we present a Portable Visuo-Tactile System for large-scale data collection and multimodal policy learning in real-world settings. Our contributions are threefold: (1) A lightweight, handheld visuo-tactile gripper. We integrate flexible piezoresistive tactile sensors into a soft, handheld gripper to enable portable, visuo-tactile, in-the-wild data collection. The system captures human manipulation demo trajectories across a wide range of environments, both indoors and outdoors. (2) multimodal representation and policy learning framework. We introduce masked autoencoding approach that uses cross-attention to jointly learn from visual and tactile inputs while preserving characteristics. This design enables policies to interpret fine-grained tactile feedback more effectively, leading to improved sample efficiency and manipulation accuracy. (3) An in-the-wild multimodal manipulation dataset. We curated a diverse dataset of over 2.6 million visuo-tactile pairs, comprising more than 2,700 demonstrations across 43 manipulation tasks 12 indoor and outdoor environments. This dataset supports effective visuo-tactile pretraining, and the resulting encoder significantly improves downstream policy learning. We committed to open-sourcing the dataset.
We validate our system on fine-grained robotic manipulation tasks in the real world, such as test tube insertion and pipette-based fluid transfer, demonstrating successful policy transfer and robustness to environmental disturbances. Our results underscore the promise of portable visuo-tactoo tactile platforms in bridging the gap between human demonstrations and robot learning in complex, real-world settings.
2 Related Works
Scalable multi-sensory data and learning. Reinforcement and imitation learning have driven major advances in robotic manipulation [3-9], yet progress remains limited by the lack of tactile sensing and large-scale visuo-tactile datasets. Simulation can partially address data scarcity, but simulated tactile signals often diverge from real-world contact dynamics, reducing transfer effectiveness [5, 10-12].
This makes scalable real-world visuo-tactile data collection increasingly important. Acquiring multi- sensory data at scale, however, is difficult. Each additional modality- RGB aadeo-addo hardware complexity, synchronization overhead, and environmental constraints [13-18]. Conse- quently, most existing datasets confined structured indoor environments [19]. To address this, recent work has shifted focus toward large-scale visuo-tactile pretraining, which leverages existing data to learn shared representations that generalize across sensors and manipulation tasks [20, 21]. These approaches demonstrate that pretraining can reduce task-specific data demands and better align visual and tactile modalities, laying the foundation for scalable multimodal learning.
However, tactile sensing is most valuable in uncontrolled, in-the-wild settings, where vision may degrade due to poor lighting background clutter [22, 23], while contact forces remain stable. Prior "in-the-wild" systems have demonstrated strong vision-only performance on simple tasks, but they

---

Fish Eye Camera
Top FPC (1tmm)
XAm 850
Sensitiee
Film (mm)m)))
Fish-Eye Camera
Bottom FPC "gurr)
Tactile Sensors
Reading Bourd (23Hz)
Flexible Tactile
Sensor
(a) Data Collection in the
(b) Portable Visuo-Tactile Data Collection Gripper
(c) Robot Setup
Figure 2: (a) Multimodal data collection in the wild using our portable aisuottatile system, with example tactile signals from both fingers. (b) Close-up of the handheld gripper, equipped with flexible tactile sensors and fisheye camera for synchronized visuo-tactile capture. Robotic setup for downstream tasks, featuring an XArm 850 with the same sensor configuration.
overlook the complementary benefits of touch [24, 25]. To bridge this gap, propose a portable handheld visuo-tactile system that combines fisheye RGB camera with a lightweight, flexible tactile array. This setup enables synchronized, large-scale collection of visual and tactile data across diverse, unstructured environments, Leveraging this system, we build visuo-tactio tautlee dataset and train a masked reconstruction encoder that (i) accurately recovers tactile signals, and (ii) enhances downstream policy learning through visuo-tactile representations. In contrast to previous methods that directly merge vision and touch in unified 3D space [19], our approach enables scalable cross-moda representation learning, fostering stronger correlations between visual and tactile modalities for fine-grained manipulation.
Visuo-tactile manipulation. Tactile feedback is essential for human manipulation, especially when vision is occluded or ambiguous [26, 27]. Likewise, tactile sensing enables robots to perform more adaptive and precise manipulation. Consequently, there growing interest in combining vision and touch enhance robotic manipulation [9, 19, 28-38]. Much of this prior work relies on optical tactile sensors, which image surface deformations infer contact geometry and texture [15, 33, 39-43]. While these sensors provide rich signals, they are typically rigid, bulky, and less robust for long-term in-the-wild use, limiting their applicability in portable or unstructured settings.
In contrast, we use thin, flexible tactile sensors [19, 44, 45] that directly measure force distributions over the contact surface. Embedded in soft robotic fingers, these sensors provide consistent, object- agnostic representations that are easier to generalize and better suited for scalable, real-world learning. Although such sensors have been explored in lab setting [19, 46], large-scale, wild visuo-tactile datasets with synchronized RGB and tactile data capturing physical interactions have not previously been available. Our work fills this gap by collecting-an -nd publicly releasing-& a diverse, real-world visuo-tactile dataset. It spans a wide range of tasks and environments, laying the foundation for scalable multimodal learning and robust policy development. We acknowledge the concurrent project ViTaMIn [47], which introduces a portable visuo-tactile data collection system and a cross-modal learning framework for contact-rich manipulation. While our efforts share similar goals, our system design, representation learning strategy, and task focus differ in several key aspects. We appreciate their contribution and encourage readers consult their work for complementary insights.
3
Visuo-Tactile Data Collection System
3.1 Scalable Flexible Tactile Sensors
As shown in Fig. 2 (a), we embed thin, matrix based tactile pads into the soft, fin-shaped fingers of our handheld gripper. The sensor architecture builds on the triple-layer design from ViTae [19], adapted to fit the geometry of the adaptive fin-shaped gripper [24]. Each tactile pad consists of piezoresistive sensing layer sandwiched between two flexible printed circuits (FPCs). To accommodate the elongated, flexible fins, we introduce two key modifications: (1) Higher spatial
3

---

resolution. Capturing contact patterns along the finger's length requires denser spatial sampling. The stainless-steel electrodes used in 3D-ViTac limit both resolution and signal stability. By replacing them with FPC electrodes, we achieve uniform trace pitch, improved robustness, and a per-pad resolution of 12 x 32 taxels, each measuring a 2 2mmm2 area. This allows us to capture fine-grained, dynamic contact signals. (2) Rapid, scalable fabrication. The use of FPCs enables tool-free assembly. Each pad can be fabricated in under five minutes and mounted the gripper in additional two, supporting scalable deployment for large-scale tactile data collection.
3.2 Portable Multi-Modal Sensing System
In-the-wild large-scale data collection. To enable real-world visuo-tactile data collection at scale, we design a compact and ergonomic handheld gripper that integrates both sensing modalities. Each tactile pad connects
custom Arduino-based PCB, with two boards neatly housed beneath the gripper's ' palm (Fig. 2 (b)). The full handheld unit- unit-including batteries batteries-weighs weighs approximately 962 g, making it comfortable for prolonged use.
At the firmware level, we optimize the serial protocol to stream each 12 x 32 pad at 23 Hz and synchronize with the visual information from the fish-eye camera. Tactile frames are timestamped directly on the microcontroller and transmitted over USB host device (e.g., a laptop or any portable Ubuntu system). The battery-powert handhag-sized system is easily deployed in grocery stores, outdoor markets, and other unstructured, in-the-wild environments (Fig. (a)), enabling high-throughput and scalable visuo-tactile data collection.
Multi-modal data synchronization. Precise aligmment between vision and touch is essential for learning effective visuo- tactile representations. Although both the tactile system and the GoPro camera operate above 20 Hz (e.g., tactile sensors 23 Hz, GoPro Hero at 60 Hz), aligning their data streams poses challenges due clock drift and limited timestamp precision on the camera.
We address this with a hardware-free synchronization strategy: (i) Video stream. Before each demonstration, a QR code displaying the current host time is shown to the camera, refreshed at 30 Hz. (ii) Tactile stream. Tactile data is published via ROS2 at 23 Hz, with each packet carrying a host-clock timestamp. (iii) Post- past-prceessing. During offline processing, we decode the QR code sequence from the video, recover exact host timestamps for each frame, and align them with the tactile data using the shared clock reference. This procedure yields tightly aligned visual and tactile recordings-withed the need for external synchronization hardware- hardware-enabling accurate multimodal supervision for downstream learning.
4 Visuo-Tactile Representation and Policy Learning
To perform precise manipulation, robots must effectively integrate both visual and tactile signals. RGB images provide global, semantic context-such as object identity and workspace layout- while tactile signals offer local, contact-rich feedback that is often occluded from vision [48]. Because these modalities follow different statistical distributions, learning unified yet modality- specific representations for effective cross-modal reasoning remains challenging. We propose two-stage learning framework (Fig. 3) that first learns joint visuo-tactile representation via masked tactile reconstruction, and then integrates the learned representation into a diffusion policy [8] for downstream manipulation tasks.
4.1 Problem Formulation
Let DD pretrain = ((I,T)) be large-scale dataset of synchronized RGB-tactile frame pairs, where I E 322224x224 is RGB image captured from wrist-mounted fisheye camera, and T R 1x24332 is a tactile image composed of vertically stacked fingertip sensor readings. The goal is to learn multimodal encoder Es, trained via self-supervised learning, that fuses these two modalities into a joint representation Zfusion = (T,)) that preserves modality-specific structure to support downstream manipulation tasks.
We divide the learning into two stages: Stage 1: Pretrain Es using masked autoencoding objective that reconstructs full tactile images from partially observed tactile input and corresponding visual

---

Pretrain
Downstream Task
Student Lounge
Chipotle
Hardware Store
Student Lounge
Kitcbea
Mikerspace
Campus
Visuo- Dactilactile Encoder
Robot Proprioception
Relative E. Pose, Grippe: Width
Diffusion Policy
Makerspuce
Reorientation
Irsertion
RGB Image
CLIP
CrossAtton (aci
Decoder
Masked
Layer CNN
Tastile
CroaAlian
Tactile
Tokens
Reconstrucion
Result
Method Overview Our Two-Stage Pipeline. Stage pretrain a visuo-tactile encoder cross modal reconstruction using a large-scale dataset collected across diverse indoor and outdoor environments. Stage 2: The pretrained encoder is combined with robot proprioception to condition diffusion policy for downstream tasks such object reorientation and insertion.
Figure 3:
frames. Stage 2: Use the pretrained encoder within a diffusion policy to learn manipulation behaviors from human demonstrations collected using our portable visuo-tactile gripper.
4.2 Stage Visuo-Tactile Representation Learning
While prior work often relies on contrastive learning to align embeddings from different modali- ties [37, 47, 47,
49], such objectives tend to suppress the fine-grained, geometry-sensitive signals captured by tactile sensors. Instead, we adopt a masked autoencoding objective [50], which reconstructs miss- ing tactile regions conditioned on partially observed tactile input and visual context. This formulation encourages the encoder retain tactile-specific information while leveraging vision for inference.
Formally, we jointly optimize the encoder Eg. and decoder De via
(Ø" 2b4) arg min B(r,Ty) - D (E
where Es is our visuo-tactile encoder and Do is the tactile reconstruction decoder.
Tactile encoder. Each tactile reading consists of two fingertip arrays, each of shape 12 x 32, stacked vertically form a x x tactile image. We then apply fixed colormap to convert this single-channel map into 3-channel RGB tactile image. This image divided into non-overlapping patch grid. During training, we randomly mask 60-80% of the
4 x patches, resulting in patches in 95% of samples using learnable token Tmaski the remaining 5% are shown in full. The masked tactile input is processed by 3-layer CNN to produce d-dimensional embedding Stacs where d = 768.
Vision encoder. The RGB image is processed by ViT-B/16 encoder initialized from CLIP [51]. We finetune all layers with a learning rate of 3 10-5, and extract the final [CLS] token as the d-dimensional visual embedding Zing.
Cross-modal fusion. To integrate the tactile and visual features, we apply two rounds of multi-head cross-attention (MHAttn):
Frac MHALLn(Q = Ztan: K Zinng V Zimg)
LayerNorm "'
ztac7
(2)
4/ng MHAttn(Q Zing: K 2/ V
2
LayerNorm "
2/m-
(3)
We concatenate the updated embeddings to obtain the fused representation:
Ffusio [2(aci Afngl 2d
(4)
5

---

Tactile reconstruction decoder. The fused feature iusion is passed through a two-layer MLP followed by a sigmoid activation to produce the reconstructed tactile image 7 E R1x21x32 i.e.,
T (2(stusio)) We use full-image reconstruction loss Ssaget (o, v)) = T A which encourages both local contact inference and global structural understanding.
Stabilization via EMA. We maintain a target encoder updated via exponential moving average (EMA) of the online weights with a decay factor of 0.9995. The tactile CNN and cross-attention layers are optimized with learning rate of 1 10- while the CLIP backbone finetuned with 3 x 10-5 The EMA encoder is used for checkpointing and attention visualization.
4.3 Stage 2: Policy Learning via Behavior Cloning
Once the visuo-tactile encoder Egg is pretrained, it integrated into conditional diffusion policy for downstream manipulation tasks.
Observation space. At each timestep I, the robot first receives raw sensory inputs (Jr, Tr, pt), where I and T; are the RGB and tactile images, and denotes the proprioceptive state (e.g., end-effector pose, gripper width). I, and T, are then passed through the pretrained encoder to produce the visuo- tactile embedding 24 = Es It, T.). The diffusion policy is then conditioned on or = (2z, Pt), i.e. the fused visuo-tactile embedding together with proprioception.
Diffusion policy. We adopt conditional difffusion policy [8] that predicts noise instead of regressing actions directly. The model outputs é = ca(af or. k), where at is the noisy action at step k, o, the conditioning observation, and E the predicted noise. During training, k is uniformly sampled and Gaussian noise cf is added to the ground-truth action a.- The loss is the MSE between added and predicted noise:
Gauge Ee, [lle, 2
(5)
At inference, actions are initialized as at ~N 1) and iteratively denoised:
a 1-1 = aaat y co on k)) + (0),
(6)
where a, 7, a are hyperparameters.
Training details. We use Diffusion Policy's convolutional U-Net [52] with DDIM inference [53]. The model is conditioned on the fused visuo-tactile embedding and two consecutive proprioceptive observations. All encoder components including the CLIP backbone, tactile CNN, and cross- attention layers-are finetuned during this stage using a learning rate of 3 x 10-5,
inncouents-incluuinng
5 Experiments
In this section, we address key questions about the role of touch in fine-grained manipulation and the influence of different pretraining strategies on downstream performance. Specifically, we ask: (1) How does touch enable fine-grained manipulation? (2) How does visuo-tactile pretraining improve policy robustness?
(3) How do pretraining variations affect task performance?
5.1 Large-Scale Tisuo-Tactile Data and Pretraining
To enable effective visuo-tactile pretraining, curated diverse dataset of over 2.6 million visuo- tactile pairs collected from 12 indoor and outdoor environments. This dataset comprises more than 2,700 demonstrations and spans 43 manipulation tasks. We categorize the data into three groups: (1) the four core tasks presented in this paper, (2) other indoor tasks that enhance distributional diversity, and (3) over 30 in-the-wild tasks designed to capture complex, worldworld scenarios. The distribution of demonstrations across these three categories is shown in Fig. 4.
We evaluate the learned visuo-tactile representations through two complementary qualitative analyses. First, we test cross modal reconstruction by providing partially masked tactile and RGB inputs and measuring the recovery of missing tactile signals, assessing whether the model captures meaningful visuo-tactile associations in both in- and duitt--distiibutiin settings. Second, we visualize self- attention maps from the final ViT layer to examine whether the model consistently attends to contact-relevant regions in RGB images across diverse scenarios. Fig. shows qualitative results from four representative tasks, including two in-distribution and two out-of- examples.

---

Data Distribution by Categories
Test Tube Collection
Fluid Trmeder
Poneil Insertion
Whitebourd Emeing
Main Tasks (Indoor)
38.2%
36.7%
Wriling
Dynamic Tossing
Hex Key Inceriion
Pop Insertion
Ohter Tasks (Indoor)
25.1%
In-the- Wild Tasks (Outdoor)
Plase Orgamization
Throwing
Placing
Categorics
Main Tasks Indoor
Other Indoor
In-the-Wild Tasks
Outdoor
Figure 4: Pretraining Data Distribution. Our dataset comprises over 2,700 demonstrations, split across three categories: (I) the four core tasks introduced in this paper. (2) other indoor tasks to broaden the data distribution, and (3) in-the-wild tasks collected in diverse outdoor environments. We include representative examples from each category highlight the variety both complexity and environmental context.
RGB Image (Input)
Masked Tactile (Input)
Vision Self Atten
Predicted Tactile
Ground Truth Inpul)
(Output lleutraap)
(Rsonntrraction
In- In-Distribution Set
Right R Tacolle
Factile
Right
Left Tactll
Right
Left foctile
Right I Tacele
Tastile
Right Tucill
Lefi Taczile
Hight Jactile
RGB Image (Input)
Masked Tactile (Input)
Vision Self Atten (Output leeutmap)
Predicted Tactile (Reconstruction Result)
Ground Truth Tac (Not Imput)
Out-of-Distriutiio Sct
Left Toctile
Right Encnile
Leh Tastile
Right Tucille
Left Thotlle
Right Tactile
Let Tocille
Righ Tacalee
Left Tactile
Right Tactile
Left Tactlle
Right Tactile
Figure 5: Qualitative Results of Pretraining. We show four examples illustrating our pretrained encoder's
tactile reconstruction performance and VIT self-attention heatmaps. The encoder accurately reconstructs tactile images for both in- and out-of-distribution inputs, while the vision module consistently attends to the gripper-contact region, independent of background object familiarity.

---

5.2 Experimental Setup
We evaluate our multi-modal sensing and learning system on four challenging real-world robotic tasks. Representative snapshots of each task are shown Fig. 6. Below are the basic descriptions and evaluation metrics for all tasks:
(1) Task Requiring In-Hand State Information
Test Tube Collection. The robot must pick test tube from a box, reorient it in-hand using the test tube rack, and precisely insert it into the test tube rack. Evaluation Metric: The task is considered successful the test tube is fully inserted into the test tube rack without being dropped or broken.
Pencil Insertion. The robot needs insert pencil into sharpener. Since the pencil is initially grasped upright, the robot must first reorient it before performing precise insertion. Evaluation Metric: The task is considered successful if the pencil is accurately inserted into the sharpener.
(2) Task Requiring Fine-Grained Force Information
The robot a pipette to transfer fluid between containers, requiring precise pressure extract liquid without dropping it. It then moves above the target container and gently squeezes to release the fluid. This task demands fine-grained force control. Evaluation Metric: The task considered successful if the fluid is transferred to the second container without spilling.
Fluid Transfer.
Whiteboard Erasing. The robot uses soft eraser to remove two strokes of text from the whiteboard. must apply the right amount of pressure erase the marker ink without exceeding force limits that could damage the system. The task requires consistent and controlled force throughout. Evaluation Metric: The task is considered successful if all visible marker ink is removed from the whiteboard.
Text Tobe Collection
(a) Grasp the Tube
(b) Extrinsic in In-Hand Reorient
(c) Complete Reorient
(d) Insert
(a) Move Toward Sharpenet
(b) Extrinsic In Ha-Hand Reomeent
(c) Complete Reorient
(d) Insert
Pencl Inserion
LD7
Slightly Press and Lift
(b) Release Extract Fluid
Move the Beaker
(e) Expel the Fluid
Fliid Transfer
Mave the Target
(b) First Fimse
(c) Second Frase
(4) Finish
Whitsboard Emainn
Figure 6: Quantitative Results. We evaluate visuo-tactile policy across four fine-graines manipulation tasks. Descriptions and metrics tasks can found in Sec.
In the experiments, we compare our methods with the following baselines.
(1) Vision-Only. This method feeds one RGB image as input to CLIP, and extracts 768-dimensional CLIP embedding. This embedding, along with the robot's proprioceptive information, is then fed into the image-based diffusion policy. We follow the same implementation outlined in Chi et al. [24].

---

(2) Ours w/o Cross-Artention. This method does not employ any rross-tteention between vision and touch. Instead, it processes two tactile images (from the left and right fingers) through 3-layer CNN to produce a 512-dimensional feature vector, which is then simply concatenated with the visual embedding and proprioceptive states as input to the diffusion policy.
(3) Ours w/o Pretraining. This method uses the visuo-tactile encoder proposed in the paper, with the vision backbone initialized from CLIP and the other parts of the joint encoder initialized from scratch (no visuo-tactile pretraining). The resulting embedding is concatenated with proprioceptive inputs to condition the diffusion policy.
(4) Ours w/ Pretraining. This method uses the visuo-tactile encoder proposed in the paper, with weights obtained from visuo-tactile pretraining. Similarly, the output embedding from the visuo- tactile encoder is concatenated with robot proprioception as conditioning for the diffusion policy.
For each of the four tasks, we conduct 20 trials with moderate randomization of the initial robot position and environmental conditions. All policies are trained for 60 epochs, which point they have converged. The results are presented in Table 1.
Tasks Roquiring In- Hand State Information
Test Tube Collection (197 demos)
Pencil Insertion (170 demos)
Modalities
Grasp
Reorient
Insert
Whole Task
Reorient
Insert
Whole Task
Vision-Only
1.00
0.25
0.25
0,25
0.50
0.65
0.45
Ours w/o Cross-Attention
1.00
1.00
0.50
0.50
0.80
0.75
0.70
Ours w/o Pretraining
1.00
1.00
0.70
0.70
0.80
0.80
0.75
Ours w/ Pretraining
1.00
1.00
0.85
0.85
0.95
0.90
0.85
Tasks Requiring Fine GrainGd Force Information
Fluid Transfer (160 demos)
Whiteboard Erasing (105 demos)
Modalities
Acquire
Transfer
Expel
Whole Task
First Erase
Second Erase
Whole Task
Vision-Onil Only
0,95
0.85
0.55
0.55
0.65
0.65
0.55
Ours w/o Cross-Attention
0,90
0.75
0.75
0.70
0.70
0.50
0.45
Ours Pretraining
1.00
0.90
0.80
0.80
0.60
0.75
0.60
Ours w/ Pretraining
1.00
1.00
0.90
0.90
0.70
0.75
0.70
Table 1: Comparison with Baselines. We evaluate our policy over 20 episodes and the best performance for each task is bolded. The numbers in parentheses indicate the number of training demonstrations.
5.3 Analysis and Discussion
Our system enhances handheld gripper by integrating tactile sensing and training large-scale visuo-tactile encoder to further improve manipulation policies. We observe three key benefits from incorporating touch and leveraging pretrained representations.
(1) Tactile feedback provides explicit in-hand state information. In a single-camera setup, visual inputs often suffer from severe occlusions. For example, in the Test Tube Collection task, the vision-only policy relied heavily on the color of the wooden cork to infer orientation. A minor change (such as switching to lighter cork with less distinct features) confused the vision policy and degraded performance in reorientation. The tactile policy, however, remained unaffected by such variations.
(2) Tactile feedback improves detection of critical state transitions. In fine-grained, force-controlled tasks like Fluid Transfer, recognizing transitions between action phases is crucial. The vision-onie policy struggles because visual cues, such as gripper width, appear similar before and after contact, making it hard to tell whether squeezing has ended. Consequently, often skips the "expel water" phase prematurely. In contrast, the tactile policy uses pressure feedback sense subtle force changes, accurately detecting when the water is fully released. This tactile awareness enables smoother phase transitions and improves both accuracy and robustness.
(3) Joint visuo-tactile encoders enable more coordinated use of vision and touch. Naive fusion methods that simply concatenate visual and tactile features, such as the policy without cross-attention, often fail to integrate the two modalities effectively, leading the policy over-rely on one and neglect the other. This imbalance was evident in the Whiteboard Erasing task, where the policy without cross-ate antennattnn applied excessive force to amplify tactile signals, triggering safety stops. In contrast,

---

Ablation on Number Epochs Demos 197)
# Epochs 10
Epochs - 20
Epochs 30
# Epochs 60
1.0
1.0
1.0
1.0
Success Rate
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
wio Pretrinning
0.4
Pectruining
0.4
0.4
0.4
0.2
0.2
Pretesining
0.2
wio Preteainiing
0.2
Pretmining
0.0
0.0
w/ Pectoiising
0.0
Pretniining
0.0
w/ Prcteaining
Grasp
Reorient
Insert
Grasp
Reorient
Insert
Grasp
Reorient
Insert
Grasp
Reorient
Insert
Ablation on Number of Demonstrations (# Epochs 60)
# Demos
30
Demos 60
# Demos 100
Demos 197
1.0
1.0
1.0
Success Rate
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4
wio Prateining
0.4
0.4
0.4
Pretmining
Pactmining
Pretasining
0.2
0.2
0.2
0.2
Pretraning
Pretraining
Pretmining
0.0
0.0
0.0
0.0
Grasp
Reorient
Insert
Grasp
Reorient
Grasp
Insert
Grasp
Reorient
Insert
Figure 7: Ablation Results for Varying Numbers of Training Demonstrations and Epochs for Test Tube Collection Task. Our findings show that the policy with visuo-tactile pretraining consistently outperforms the policy without pretraining, both in low-epoch and low-demonstration regimes.
our jointly trained visuo-tactile encoder coordinates both modalities, enabling force modulation based on visual context and tactile feedback, and reducing failures caused by misuse of either modality.
Pretraining Ablations: Varying Number of Training Demonstrations and Epochs.
To assess pretraining effectiveness, we evaluate Test Tube Collection performance across varying numbers of demonstrations and training epochs, as shown in Fig. 7. We found that pretraining provides significant benefits, particularly in low-data and low-epoch training settings.
(1) Low-Data Regime (Fewer Than 60 Demonstrations). When training with only 30 or 60 demon- strations, the policy without pretraining often becomes stuck after grasping, uncertain how to proceed. In contrast, the policy with pretraining- trajectories and usually only fails during the final insertion step. We believe that pretraining helps the encoder to learn visuo-tactile correlations early on, which enables the downstream policy to focus on learning effective action trajectories.
with just 30 demonstrations--olloon much smoother
(2) Low-Epoch Regime (Fewer Than 60 Epochs). In the low-epoch regime, we observed that the policy without pretraining more sensitive to initial environmental configurations. For instance, when the test tube was placed at steep incline and was difficult grasp, imperfect grasp position had considerable impact on the execution of the reorientation task. The policy without pretraining sometimes over- or under-reoriented, resulting in failure. We believe that in the low-epoch regime, the policy with pretraining benefits from prior knowledge that emphasizes tactile cues, which makes more robust environmental disturbances.
6 Conclusion and Limitations
In this work, we present handheld gripper enhanced with tactile sensing and introduce a large-scale visuo-tactile dataset. We demonstrate its utility by pretraining a visuo-tactile joint encoder and evaluating it on several fine-graid fine-grained manipulation tasks using a single-arm robot equipped with parallel gripper, mirroring the handheld setup. Looking ahead, we aim extend this approach to multi-finger dexterous hands, where tactile feedback enable richer and more dexterous manipulation skills.
10

---

Acknowledgement
This work is partially supported by the DARPA TIAMAT program EIR0001-2449-0000 NSF Award #2409661, Toyota Research Institute (TRI, Sony Group Corporation, Samsung Research America (SRA), Google, Dalus AI, Pickle Robot, and an Amazon Research Award (Fall 2024). This article solely reflects the opinions and conclusions of its authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
References
[1] N. Kuppuswamy, Alspach, A. Uttamchandnni, S. Creasey, Ikeda, and Tedrake. Soft- bubble grippers for robust and perceptive manipulation. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 9917-9924. IEEE, 2020.
[2] W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017.
[3] Z.-H. Yin, B. Huang, Y. Qin, Q. Chen, and X. Wang. Rotating without seeing: Towards in-hand dexterity through touch. arXiv preprint arXiv: 10880, 2023.
[4] B. Huang, Y. Chen, T. Wang, Y. Qin, Y. Yang, N. Atanasov, and X. Wang. Dynamic handover: Throw and catch with bimanual hands. arXiv preprint arXiv:2309.05655 2023.
[5] E. Su, C. Jia, Y. Qin, W. Zhou, A. Macaluso, B. Huang, and X. Wang. Sim2real manipulation on unknown objects with tactile-based reinforcement learning. arXiv preprint arXiv:2403. 12170, 2024.
[6] Y. Wang, G. Yin, B. Huang, Kelestemur, J. Wang, and Y. Li. Gendp: 3d semantic fields for category-level generalizable diffusion policy. In 8th Annual Conference Robot Learning. 2024.
[7] H. Ha, Y. Gao, 7. Fu, J. Tan, and S. Song. UMI on legs: Making manipulation policies mobile with manipulation-centric whole-body controllers. In Proceedings of the 2024 Conference on Robot Learning, 2024.
[8] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 2024.
[9] H. van Hoof, N. Chen, M. Karl, P. van der Smagt, and Peters. Stable reinforcement learning with autoencoders for tactile and visual data. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3928-39344 2016. doi: 10. 10610
7759578.
[10] I. Akinola, J. Xu, J. Carius, D. Fox, and Y. Narang. Tacsl: A library for visuotactile sensor simulation and learning. IEEE Transactions on Robotics, 2025.
[11] S. Wang, M. Lambeta, P.W. Chou, and R. Calandra. Tacto: A fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors. IEEE Robotics and Automation Letters, 7(2):3930-39) 2022.
[12] 7. and W. Yuan. Taxim: An example-hased simulation model for gelsight tactile sensors. IEEE Robotics and Automation Letters, 7()):2361-2688, 2022.
[13] R. Gao, Z. Si, Y.-Y. Chang, Clarke, Bohg, L. Fei-Fei, W. Yuan, and J. Wu. Objectfolder 2.0: A multisensory object dataset for sim2real transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10598-10608 2022.
11

---

[14] F. Yang. C. Feng, Z. Chen, H. Park, D. Wang. Y. Dou, Z. Zeng, X. Chen, R. Gangopadhyay, A. Owens, et al. Binding touch to everything: Learning unified multimodal tactile representa- tions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26340-26353. 2024.
[15] Y. Dou, Yang, Y. Liu, A. Loquercio, and A. Owens. Tactile-augmented radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26529-26539, 2024.
[16] Yang, J. Zhang, and A. Owens. Generating visual scenes from touch. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22070-22080, 2023.
[17] H. Li, Y. Zhang. J. Zhu, S. Wang, M. A. Lee, H. Xu, E. Adelson, L. Fei-Fei, R. Gao, and J. Wu. See, hear, and feel: Smart sensory fusion for robotic manipulation. arXiv preprint 2022.
arXiv:22/2.0388
[18] Z. Liu, C. Chi, E. Cousineau, N. Kuppuswamy, B. Burchfiel, and S. Song. Maniwav: Learning robot manipulation from in-the-wild audio-visual data. arXiv preprint arXiv:2406. 19464, 2024.
[19] B. Huang, Y. Wang, X. Yang, Y. Luo, and Y. Li. 3d-vitac: Learning fine-grained manipulation with visuo-tactile sensing. arXiv preprint arXiv:2410.2409), 2024.
[20] C. Higuera, A. Sharma, C. K. Bodduluri, T. Fan, P. Lancaster, M. Kalakrishnan, M. Kaess, B. Boots, M. Lambeta, T. Wu, et al. Sparsh: Self-supervised touch representations for vision- based tactile sensing. arXiv preprint urXiv:2410.24090, 2024.
[21] J. Zhao, Ma, L. Wang, and E. H. Adelson. Transferable tactile transformers for representation learning across diverse sensors and tasks. arXiv preprint arXiv:2406. /3640, 2024.
[22] C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint 122:: 2023.
[23] C. Finn, Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. In Conference robot learning, pages 357-368. PMLR, 2017.
[24] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel. S. Feng. R. Tedrake, and S. Song. Universal manipulation interface: fn-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402. 10329, 2024.
[25] Z. Wu, T. Wang, C. Guan, Z. Jia, S. Liang, H. Song, D. Qu, D. Wang, Z. Wang, N. Cao, et al. Fast-umi: A scalable and hardware-indeeendent universal manipulation interface. arXiv preprint arXi /9499, 2024.
[26] R. S. Johansson and G. Westling. Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects. Experimental brain research, 56(3):550-566 1984.
[27] Jenmalm and S. Johansson. Visual and somatosensory information about object shape control manipulative fingertip forces. Journal of Neuroscience, 17(11):4486444) 4499, 1997.
[28] Suresh, H. Qi. Wu, T. Fan, L. Pineda, M. Lambeta, Malik. M. Kalakrishnan, Calandra, M. Kaess, J. Ortiz, and M. Mukadam. Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation. Science Robotics, page ad10628, 2024.
[29] Z. Ding, G. Chen, Z. Wang, and L. Sun. Adaptive visual-tactis fusion recognition for robotic operation of multi-materis system. Frontiers in Neurorobotics, 17:1181383, 2023,
12

---

[30] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg. Making sense vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. 2019 International Conference on Robotics and Automation (ICRA), pages 8943-8950. IEEE, 2019.
[31] Y. Chen, M. Van der Merwe, A. Sipos, and N. Fazeli. Visuo-tactile transformers for manipulation. In 6th Annual Conference on Robot Learning, 2022.
[32] Hansen, Hogan, D. Rivkin, D. Meger, M. Jenkin, and G. Dudek. Visuotactile-rl: Learning multimodal manipulation policies with deep reinforcement learning. In 2022 International Conference on Robotics and Automation (ICRA), pages
2022. doi: 10.1 109/
(CR.44699.2022291120099
[33] Y. Yuan, H. Che, Y. Qin, B. Huang, Z.-H. Yin, K.-W Lee, Y. Wu, S.-C. Lim, and X. Wang. Robot synesthesia: In-hand manipulation with visuotactile sensing, arXiv preprint arXiv:2312.01853, 2023.
[34] Y. Qin, B. Huang, Z.-H. Yin, Su, and X. Wang. Dexpoint: Generalizable point cloud rein- forcement learning for sim-to-real dexterous manipulation. In Conference on Robot Learning, pages 594 605. PMLR, 2023.
[35] A. George, S. Gano, P. Katragadda, and A. B. Farimani. Vital pretraining: Visuo-tactile pretraining for tactile and non-tactile manipulation policies. arXiv preprint arXiv:2403. 18898, 2024.
[36] F. Yang, C. Ma, J. Zhang, J. Zhu, W. Yuan, and A. Owens. Touch and go: Learning from human-collected vision and touch. arXiv preprint arXiv:2211. 2022.
[37] V. Dave, F. Lygerakis, and E. Rueckert. Multimodal visual-tactile representation learning through self-supervised contrastive pre-training. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 8013-8020, 2024. doi: 10.1009/CRRA5714722224
10610228.
[38] B. Huang, J. Xu, I. Akinola, W. Yang, B. Sundaralingam, R. O' O'Flaherty, D. Fox, X. Wang, A. Mousavian, Y.-W. Chao, et al. Vt-refine: Learning bimanual assembly with visuo-tactile feedback via simulation fine-tuning. arXiv preprint 2w..:ii511.0193000 2025.
[39] R. Calandra, A. Owens, M. Upadhyaya, W. Yuan, Lin, E. H. Adelson, and S. Levine The feeling of success: Does touch sensing help predict grasp outcomes? arXiv preprint arXiv: 171000551, 2017.
[40] W. Yuan, Y. Mo, Wang, and H. Adelson. Active clothing material perception using tactile sensing and deep learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 4842 4849. IEEE, 2018.
[41] R. Kolamuri, Z. Si, Y. Zhang, A. Agarwal, and W. Yuan. Improving grasp stability with rotation measurement from tactile sensing. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6809 111996816 6816. IEEE, 2021.
[42] R. Calandra, A. Owens, D. Jayaraman, J. Lin, W. Yuan, J. Malik, E. H. Adelson, and S. Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. JEEE Robotics and Automation Letters, 344):3300-3307, 2018
[43] H. Xue, J. Ren, W. Chen, G. Zhang, Y. Fang, G. Gu, H. Xu, and C. Lu. Reactive diffusion policy: Slow-fast visual-tactile policy learning for contact-rich manipulation. arXiv preprint arXiv:2503.0288 0288), 2025.
[44] S. Sundaram, P. Kellnhofer, Y. Li, J.-Y. Zhu, A. Torralba, and W. Matusik. Learning the signatures of the human grasp using scalable tactile glove. Nature, 569(7758):698-7022. 2019.
13

---

[45] R. Feng, J. Hu, W. Xia, T. Gao. A. Shen, Y. Sun, B. Fang, and D. Hu. Anytouch: Learning unified static-dynamic representation across multiple visuo-tactile sensors. arXiv preprint arXiv:2502. 12191, 2025.
[46] A. Bicchi and V. Kumar. Robotic grasping and contact: review. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065), volume 1, pages 348-353 vol.1, 2000. doi:
10. 001100//R0B011200004440000
[47] F. Liu, C. Li, Y. Qin, A. Shaw, J. Xu, P. Abbeel, and R. Chen. Vitamin: Learning contact-rich tasks through robot-free visuo-tactile manipulation interface. arXiv preprint arXiv:2504.06//6, 2025.
[48] Y. Li, J.-Y. Zhu, R. Tedrake, and A. Torralba. Connecting touch and vision via cross-modal prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10609-10618, 2019.
[49] I. Guzey, B. Evans, Chintala, and Pinto. Dexterity from touch: Self-supervised pre- pre-training of tactile representations with robotic play. arXiv preprint arXiv:2303.12076 2023.
[50] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[51] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PmLR, 2021.
[52] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[53] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXi522010.02502, 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction accurately describe the paper's core contributions, including the design of portable visuo-tactile gripper. the proposed cross-modal representa- tion learning framework, and empirical validation on real-world manipulation tasks such as test tube insertion and fluid transfer. The claims are consistent with the presented hardware, learning method, and experimental results, and do not overstate generalization beyond the evaluated settings.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses limitations related to the use of parallel-jaw gripper, which constrains dexterity and limits applicability to more complex manipulation scenarios.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
15

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not present formal theoretical results or proofs. It focuses on system design, representation learning, and empirical evaluation.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: The paper describes the data collection process, task definitions, model architectures, training objectives, optimization settings, and evaluation metrics in sufficient detail to enable reproduction of the reported results by an informed researcher.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
16

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open-souree dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to deseribe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We have released the collected large-scale visuo-tactile dataset and accom- panying code, along with documentation and detailed instructions in the project's GitHub repository, to support reproducibility of the main experiments.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips..
Mhollc//uuide//odeeSuiiisss sooooiiissssoooociiiiiioioo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ wttp///bllcg/gguuuooooooo httpi///ggiiidee/eeeeuuuiiiiiiii wwweiisssssnnliiiooiooooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experiment section specifies task setups, data splits, training procedures, and evaluation protocols. Additional implementation details and hyperparameters provided in the supplemental material.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The reported experimental results do not include error bars or formal statistical significance analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emrrr bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The paper and supplemental material specify the compute modality (GPU- based training), model scales, batch sizes, training epochs, and optimization settings, which together provide sufficient information for reproducing the experiments comparable hardware.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
18

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : //neurrrpsp... cc/ http/publlcclteeeee httpai/e/eehcscciieeeeeeeeee
Answer: [Yes]
Justification:
The research complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
The work is foundational research on robotic manipulation systems and datasets, and does not have a direct or immediate societal impact that requires dedicated discussion broader impacts.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
19

---

Answer:
[NA]
The released data and models focus robotic manipulation and do not pose high risk of misuse.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external datasets, models, and codebases used in the paper are appropriately cited, and their licenses and terms of use are respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces a new large-scale visuo-tactile dataset, which docu- mented in the paper and supplemental material, including data collection procedures, sensor descriptions, and task definitions. The dataset is released via Hugging Face, and detailed instructions for data access and experiment reproduction are provided in the project's GitHub repository.
Guidelines:
The answer NA means that the paper does not release new assets.
20

---

Researchers should communicate the details of the dataset/codeemmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector:
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No human subject research is involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
21

---

Justification:
Large language models are not used as part of the proposed methods or
experiments.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non- components.
Please refer to our LLM policy hattps://neurip: httpp///errrpps.cc/oofeeeeeeeeee wct..///eeeeeeeee/225555555555 for what should or should not be described.
22

---

Supplementary Materials
Contents
Introduction
2
Related Works
3
Visuo-Tactile Data Collection System
3.1
Scalable Flexible Tactile Sensors
3.2
Portable Multi-Modal Sensing System
4
Visuo-Tactile Representation and Policy Learning
4.1
Problem Formulation
4.2
Stage I: Visuo-Tactio Representation Learning
4.3
Stage 2: Policy Learning via Behavior Cloning
5
Experiments
5.1
Large-Seale Visuo-Tactile Data and Pretraining
5.2
Experimental Setup
5.3
Analysis and Discussion
6
Conclusion and Limitations
10
Cross-Sensor Generalization and Consistency
23
Pretraining Ablations
24
B.1 Effect Demonstration Quantity on Pretraining
24
Impact Removing Sekk-Specffic Demonstrations on Pretraining
24
Downstream Task Training Analysis
25
C.1 Impact of Pretraining Training Loss Convergence
25
C.2 Policy Self-Attention Maps Before vs. After Downstream Fine-Tuning
26
Baseline Failure Analysis
26
Training Hyperparameters for Pretraining
28
Detailed Hyperparameters for Downstream Task Training
28
Extended Task Descriptions
28
G.I Details for the Test Tube Collection Task
28
G.2 Details for the Fluid Transfer Task
29
................................
G.3 Details for the Whitehoard Frasing Task
29
G.4 Details for the Pencil Insertion Task
30
A Cross-Sensor Generalization and Consistency
Throughout the project, the visuo-tactile gripper and robot grippers used different pairs of tactile sensors, providing an inherent test of cross-sensor generalization. To further evaluate sensor con- sistency, we replaced the robot's 's tactile sensors with newly fabricated ones that had been used previously in this project and re-evaluated the Test Tube Collection task with the new sensors. As shown in Table 2, the success rates across all subtasks remained nearly identical, showing strong cross- sensor consistency and confirming that the sensors are reliable and interchangeable for downstream deployment.
Test Tube Collection Task
Grasp
Reorient
Insert
Whole Task
Policy Pretraining (Original Sensors)
1.00
1.00
0.85
0.85
Policy w/ Pretraining (New Sensors)
1.00
1.00
0.85
0.85
Table 2: Policy performance with tactile Consistent downstream performance confirms sensor reliability.
23

---

B Pretraining Ablations
B.1 Effect of Demonstration Quantity on Pretraining
To assess the impact of pretraining data, we analyze how the size of the pretraining dataset influences reconstruction and representation quality in our visuo-tactile encoder. As shown in Fig. 8, larger datasets yield lower reconstruction losses and more stable training- Corresponding tactile reconstruc- tions and ViT self-attention maps show that with more data, attention concentrates more precisely on gripper contact regions, and reconstructions become cleaner and more structured, which signals stronger rross-modal representations.
RGB Image (loput)
Test Data MSE Loss VS Pretruining Data Size
0.020
Tost Data MSE Loss
0.015
0.010
0.005
Duta% 10%
30%
50%
100%
Reconstruction Results of Different Pretraining Data Size
Prodicted Tae Vision Self Attn
Left Tactile
Left Tactile
Left Tactile
Right Tactile
Right Tactile
Right Tactile
Masked Tactile (laput)
Ground Truth Tactile
Data2
10%
30%
50%
100%
Figure 8: Impact of Pretraining Dataset Size Cross-Modis Reconstruction. On the left, show representative visuo-tactile example with RGB image. the masked tactile input, and the ground-truth tactile image. On right, the top panel plots test data MSE loss against the amount pretraining demonstrations, and the bottom panel displays reconstructed tactile images alongside their visual self-attention heatmaps for four different dataset amounts. As the amount of pretraining data grows, the test MSE steadily decreases and the reconstructions become notably cleaner and more accurate.
B.2 Impact of Removing Task-Specific Demonstrations Pretraining
To evaluate the robustness and generalization of visuo-tactile pretraining, conducted two experiments where we removed demonstrations for the specific downstream fine-tuning tasks from the pretraining dataset. In the first experiment, all demonstrations for the Test Tube Collection task were excluded, and in the second, all demonstrations for the Whiteboard Erasing task were removed. This setup allowed us to assess how effectively the pretrained visuo-tactile encoder generalizes when trained solely on in-the-wild data, without any task-specific examples included during pretraining.
After each removal, we retrained the encoder on the reduced dataset, fine-tuned downstream policy for the corresponding task, and then evaluated its real-world performance. The resulting pretraining dataset contained approximately 2,500 in-the-wild demonstration videos, compared to the original 2,700.
Qualitatively, across both held-out tasks, we observed that the ViT attention heatmaps, evaluated on both training and held-out tasks, remained focused on the gripper's contact regions, consistent with those obtained using the full pretraining dataset. This indicates that the encoder continued to

---

Test Tube Collection
Pencil Insertion
Training Loss
0.015
0.05
w/o Pretraining
0.017
w/o Pretraining
0.014
w/ Pretraining
w/ Pretraining
0.015
0.013
0.65
0.50
0.012
0.55
0.013
0.70
0.75
0.70
0.80
0.85
0.85
10
20
30
40
50
60
10
20
30
40
50
60
Epoch
Epoch
Figure 9: Downstream Policy Training Loss for Two Tasks. Training loss curves from epoch 5 to 60 for Test Tube Collection and Pencil Insertion tasks. Numbers along each curve indicate rollout success corresponding epochs. Policies with pretraining show faster convergence and consistently higher success rates compared to policies without pretraining.
capture meaningful visuo-tactile correspondences even when the downstream tasks were excluded from pretraining.
Quantitatively, as shown in Table 3, the downstream policies trained with and without the correspond- ing task data in pretraining achieved nearly identical performance. These results demonstrate that the learned visuo-tactile representations generalize effectively to unseen manipulation tasks rather than overfitting to the specific demonstrations present in the pretraining data.
Effect of Removing Task- Sask-Specific Demonstrations From Pretraining
Test Tube Collection
Whiteboard Erasing
Policy
Grasp
Reorient
Insert
Whole Task
First Erase
Second Erase
Whole Task
With Pretraining (Ours)
1.00
1,00
0.85
0.85
0.70
0.75
0.70
Without Task Duta Pretraining
1.00
1.00
0.80
0.80
0.75
0.75
0.75
Table 3: Downstream policy performance when task-specific demonstrations were excluded from pretraining We removed all demonstrations for (I) Test Tube Collection and (2) Whiteboard Erasing from the pretraining dataset. The resulting performance remains comparable to that achieved with the full dataset, indicating that the pretrained visuo-tactile encoder generalizes well even without task-specific data.
C Downstream Task Training Analysis
C.1 Impact Pretraining on Training Loss Convergence
To evaluate the impact of visuo-tactile pretraining on downstream task performance, we analyze both task success rates and training convergence behavior. In addition final success rates for the Test Tube Collection task under low-demonstration and low-epoch regimes with and without pretraining (see section 5 in the main paper), we also track training loss curves for two representative tasks-Test Tube Collection and Pencil Insertion- -iith and without pretraining, shown in Fig. 9.
We plot training loss from epoch 5 to epoch 60 for both settings. Across both tasks, we observe that policies with visuo-tactile pretraining consistently achieve lower loss values throughout training compared to those without pretraining. This indicates faster convergence and improved training stability.
Importantly, even when policies trained with and without pretraining converge to similar loss levels during training, we observe that policies with pretraining consistently achieve higher success rates in real-world rollouts. We attribute this discrepancy to the difference between open-loop training (mea- sured by loss) and closed-loop execution (evaluated via physical rollouts). In real robot experiments, downstream tasks subject to unpredictable variations in object positions, environmental dynamics, and contact conditions. Policies with visuo-tactile pretraining more robust to these variations
25

---

Policy w/ Pretraining
Policy wio Protraining
Figure 10: ViT Self-Attention Maps after Downstream Fine-tuning. The self- attentttenion maps from the VIT module show that policies with pretraining tend to focus on task-relevant contact regions, such as the gripper and object. In contrast, policies without pretraining often attend to background features, like table edges box boundaries, resulting in less stable downstream performance.
because they have learned rely on tactile cues that generalize across scenarios. As result, they exhibit more reliable and consistent behaviors, particularly in out-of-distribution conditions.
C.2 Policy Self-Attention Maps Before vs. After Downstream Fine-Tuning
To further understand the performance gap between policies with and without pretraining observed during real-world evaluation, we analyzed the self-attentio heatmaps from the VIT module in the Test Tube Collection task, as shown in Fig. 10. The policy with pretraining consistently focuses attention on the gripper-ohject contact region, suggesting that it retains spatial priors critical for manipulation- priors learned during large-scale visuo-tactile pretraining. In contrast, the policy without pretraining frequently directs attention to irrelevant background features, such as table edges or nearby objects, likely in an attempt to infer contact indirectly. This misplaced focus can degrade performance, particularly in cluttered or visually diverse environments. These attention patterns provide insight into why the policy with pretraining not only learns more efficiently but also generalizes better across task variations.
D Baseline Failure Analysis
To better understand the challenges involved in fine-grained real-world manipulation, analyze common failure cases across baseline policies for the four core tasks. Fig. 11 presents representative examples of failures for both vision-only policies and policies without cross-attention. These cases highlight the limitations relying solely on visual or static tactile input and emphasize the importance of integrated visuo-tactile feedback.
(a) Test Tube Collection. In the Test Tube Collection task, vision-only policies often struggle to differentiate between the reorientation and insertion phases. As shown in Fig. 1 11(,), a typical failure involves prolonged hesitation when deciding whether to continue reorienting or to proceed with insertion. Even after a successful reorientation, the policy may continue the motion unnecessarily, suggesting uncertainty about the current phase the task.
Although policy without cross-attention transitions decisively between phases, it can become overly dependent on localized tactile cues. This sometimes results in the robot getting stuck during
26

---

Typical Failure Cases of Baselines
(a) Test Tube Collection
(b) Fluid Transfer
(c) Whiteboard Frasing
(d) Pencil Insertion
Long Repeated
Over-extract
redere-eeorent
Miurieated Pencil
Coninitss Fiaid
Failed Insertion
Stucked Reorientation
Unfinished Expel
Destructive Force
Failed Insertion
Figure 11: Failure Cases. We present typical failure cases of baseline methods for all four tasks and analyze the reasons for these failures to highlight the complexity of the tasks and the importance of tactile feedback and pretraining during these steps.
reorientation, where it maintains excessive contact without progressing. Additionally, the policy without cross-attention frequently fails during insertion, likely due insufficient use of visual input for aligning the test tube with the narrow slots of the test tube rack.
(b) Fluid Transfer. Fluid Transfer requires precise, force-sensitive control vision-only policies typically struggle to provide. As illustrated in Fig. 11(b), a common failure during fluid extraction involves excessive squeezing, where the robot either applies too much pressure or continues squeeze for too long. Without tactile feedback, the policy relies entirely on ambiguous visual cues, often resulting in drawing too much fluid. This overfilling hinder subsequent steps, particularly during the expel fluid phase, where the robot may fail to fully release the fluid. The result is an imprecise release and overall task failure.
Failures are also frequent during the expel fluid phase. The policy may skip the expel fluid step entirely, particularly when the fluid is not clearly visible either due an insufficient amount drawn when the second beaker contains nearly water. These behaviors suggest that vision-only policy overly sensitive subtle environmental variations and fails to detect critical task-related cues.
(c) Whitehoard Erasing. Whiteboard Erasing requires consistent force against flat surface. Vision- only policies often fail due to incomplete visual feedback regarding contact between the craser and the board. As result, insufficient pressure is applied, leaving visible marks and leading to task failure (Fig. 11())).
On the other hand, policies without cross-attention tend overcompensate by pressing too hard in an attempt to maximize tactile feedback. This often causes the robot to apply excessive force, triggering safety stops and prematurely terminating the task.
(d) Pencil Insertion. Pencil Insertion requires precise in-hand reorientation of the pencil before the final insertion. Vision-only policies often struggle with this step, particularly when the pencil is held low in the gripper, partially occluded, or only slightly inclined (Fig. I1))). These conditions limit the visibility of the pencil and lead to inaccurate orientation estimates. As result, the policy frequently misjudges the pencil's s orientation, causing insertion failures.
27

---

E
Training Hyperparameters for Pretraining
Parameter
Value
Parameter
Value
Seed
42
Batch size
128
Tactile embed dimension
768
Epochs
Tactile patch size
CLIP learning rate
10
Number of attention heads
Encoder learning rate
x 10
Cross-attention dropout
0.20
Weight decay
2 x 10-3
Decoder hidden dim
768
Warmup ratio
0.10
Scheduler
Linear warmup Cosine
Optimizer
AdamW
annealing
Validation ratio
0.10
Tactile masking ratio
60%-80%
Table 4: Pretraining hyperparameters and training configuration. Tactile masking ratio means random patch masking during pretraining.
F Detailed Hyperparameters for Downstream Task Training
Observation Settings
Optimization Parameters
Image obs. horizon
Optimizer
Adam
Momentum
81 0.95, & 0.999
Proprio. obs. horizon
LRR (action diff.)
LR (pretrained enc.)
10
Action horizon
I.R schedule
Cosine decay
Batch
64
Obs. resolution
224
Train diff. stops
50
Inference steps
Table 5: Training hyperparameters for policy learning. Observation settings on the left: optimization parameters the right.
G Extended Task Descriptions
In this section, provide detailed information on the four tasks described in the main paper. We describe the motions and evaluation metries for each step, highlighting how these steps demonstrate the capabilities of our tactile sensors.
G.1 Details for the Test Tube Collection Task
Initial Positions. At the start of each episode, the robot is positioned near a cluttered box containing a test tube. The test tube is partially visible, with its close end typically protruding slightly outside the box, while the rest embedded among other objects. Several aspects are randomized across episodes: the robot's initial pose, the position and orientation of the test tube, the clutter surrounding the test tube, and the color of the wooden cork. These variations introduce uncertainty in the grasping setup and require precise spatial reasoning and tactile feedback for reliable performance.
Step 1: Grasp the Test Tube. The robot reaches into the box to grasp the partially exposed test tube. Depending on its orientation and surrounding clutter, the robot may need adapt wide range of grasping conditions. This step demonstrates the strength precise manipulation within narrow and constrained environments. Evaluation Metries: The step is considered successful if the robot securely grasps the test tube and prepares to proceed to the next step.
thin, flexible tactile sensor enabling
Step 2: Reorient the Test Tube. After grasping, the robot moves to the test tube rack and uses its edge rotate the test tube by approximately degrees. This reorientation is particularly challenging due the test tube's transparency, which can cause vision-only policies misjudge in-hand orientation and either under- over-rotate. In contrast, tactile feedback provides clear contact signals, enabling reliable reorientation. Evaluation Metrics: The step is successful if the robot completes single,
28

---

correct reorientation. The task fails if multiple attempts are required or if the final orientation remains incorrect.
Step Insert the Test Tube. Finally, the robot attempts to insert the reoriented test tube into a tight-fitting slot the rack. Given the transparent material and narrow tolerance, accurate alignment is critical. Tactile sensing plays a key role in localizing the body of the test tube and guiding the insertion process. Evaluation Metries: The step is considered successful if the test tube is inserted fully and securely into the rack.
G.2 Details for the Fluid Transfer Task
Initial Positions. The robot begins with its parallel grippers positioned around the bulb of pipette, roughly centered between its fingers. Across episodes, several aspects are randomized: the robot's initial pose, the orientation and position of the pipette within the first beaker, the location of both beakers, the amount of water in each beaker, and the water's color and opacity. These variations introduce uncertainty in grasping and perception, making tactile sensing essential for accurately locating the bulb and applying the correct amount of force.
Step 1: Grasp Pipette and Acquire Fluid. The robot first applies gentle pressure to the pipette's bulb to extract water from the first beaker. This step requires fine control of squeezing force- too little and not enough water is drawn, too much and the pipette may slip spill. The robot must also detect when a sufficient amount of liquid has been collected, and then lift the pipette while maintaining stable grasp. preparing for the transfer. Visual feedback is unreliable here due to partial occlusion of the bulb and the small volume of liquid involved. Our tactile sensors provide accurate contact localization and force feedback, enabling the robot to perform this motion precisely and without spillage. Evaluation Metrics: The step considered successful if the robot extracts an appropriate amount of liquid and lifts the pipette without spilling.
Step 2: Transfer to the Second Beaker: With the pipette securely grasped and filled, the robot moves toward the second beaker. It must maintain consistent pressure on the bulb throughout the motion to avoid accidental release or dropping the pipette. This step requires precise motion planning and continuous force regulation, as even small deviations in grip force can cause the fluid to spill. Tactile sensing ensures grip stability, while vision assists coarse positioning. Evaluation Metrics: The step is successful if the robot moves the pipette directly above the second beaker without spilling any water during the transfer process.
Step 3: Expel Fluid. Once positioned above the second beaker, the robot lowers the pipette and squeezes the bulb to release the water. This step requires accurately detecting contact and applying the right amount of force to fully expel the liquid. Vision-only policies often struggle in this scenario due the transparency of both the pipette and the fluid, making difficult to verify whether the liquid has been released. In contrast, our visuo-tactile policy leverages force signals to monitor the release process and prevent premature lifting. Evaluation Metrics: The step is considered successful if all the liquid is transferred to the second beaker and the pipette is lifted cleanly, with no remaining water or unintended spills.
G.3 Details for the Whiteboard Erasing Task
Initial Positions. The robot starts with soft foam brush held vertically in its gripper. Across episodes, several aspects of the setup randomized: the robot's initial pose, the height and position of the whiteboard, and the exact location of the brush within the gripper. These variations introduce uncertainty in aligmment and contact, making tactile sensing critical for successful execution.
Step 1: Erase First Stroke. The robot begins with the foam brush held vertically in its gripper. To prepare for erasing, first rotates its gripper by approximately 90 degrees, reorienting the brush into a horizontal position. It then moves toward the whiteboard and attempts to erase the letter "T" using vertical stroke. The robot must apply slight downward force and rely tactile feedback to confirm contact with the whiteboard surface. This step is challenging because visual input alone
29

---

may not reliably indicate whether the brush has made effective contact, especially in the presence of occlusions or slight misalignments. Evaluation Metrics: The step considered successful the robot fully erases the letter "T" with proper contact and without slipping or losing control of the brush.
Step 2:
Erase Second Stroke. After completing the vertical stroke, the robot reorients the brush vertical position and then performs horizontal stroke crase the letters "ac." Tactile sensing is again crucial not only for confirming contact with the surface, but also for detecting when sufficient pressure is upplied to begin effective erasure. Evaluation Metrics: The step is considered successful the robot cleanly erases the letters "ac" while maintaining stable and continuous control of the brush.
G.4 Details for the Pencil Insertion Task
Initial Positions. The robot begins with each episode holding slightly inclined pencil in its gripper. Several factors randomized across episodes: the pencil's orientation and position within the gripper, the placement and height of the pencil sharpener, the size and alignment of the insertion hole, and the robot's initial pose. These variations introduce uncertainty that requires fine-grained control and perception.
Step 1: Reorient the Pencil. The robot first moves toward the vertical surface of the pencil sharpener and uses it reference reorient the pencil. The goal is to align the pencil so that it becomes parallel to the gripper and ready for insertion. This step is challenging due to the pencil's 's small size and partial occlusion within the gripper, which makes it difficult to estimate its pose using vision alone. Tactile feedback provides critical information for detecting contact and adjusting alignment. Evaluation Metrics: This step considered successful if the pencil is reoriented to be parallel to the robot's gripper, ensuring it properly aligned for insertion.
Step 2: Insert the Pencil. With the pencil correctly oriented, the robot must insert it into the top hole of the pencil sharpener. This step demands precise positioning, fine alignment, and controlled force to avoid missing the hole applying excessive pressure, which could cause the pencil to slip or become jammed. Tactile sensing enables the robot to detect contact with the hole and adjust the insertion motion accordingly. Evaluation Metrics: The step is considered successful if the pencil is inserted smoothly into the sharpener without slipping, jamming, or needing realignment.
30


Reviews:
[{"review_id": "akxRFZncLc", "reviewer": "Reviewer_Tduh", "summary": "This paper presents a complete, end-to-end solution for the challenging problem of learning fine-grained robotic manipulation in unstructured, \"in-the-wild\" environments. The authors begin by designing and implementing a portable, handheld gripper integrated with piezoresistive tactile sensors, which enables the synchronized collection of visual and high-resolution tactile data. Building upon this hardware, the paper proposes a novel cross-modal representation learning framework, centered around a masked tactile reconstruction self-supervised task, designed to effectively fuse global context from vision with local contact details from touch. Finally, the learned multimodal representations are integrated into a Diffusion Policy and validated on a series of challenging tasks where subtle force feedback and control are needed, such as \"Transparent Tube Collection\" and \"Fluid Transfer.\" The experimental results clearly demonstrate the significant superiority of this method compared to vision-only baselines and other simpler fusion approaches.", "questions": "see the weakness", "limitations": "The authors provide a thorough analysis of the potential limitations of their method.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n1. This work accurately identifies a key deficiency in current large-scale teleoperation systems, which is crucial for fine-grained manipulation. The proposed portable visuo-tactile gripper is a highly practical and well-engineered solution. Its lightweight design (approx. 962g), high-resolution flexible tactile sensors, and innovative hardware-free data synchronization scheme together constitute a powerful data collection tool that greatly facilitates community research on contact-rich manipulation tasks.\n    \n2. In contrast to common approaches like contrastive learning or simple feature concatenation, the authors employ a masked auto-encoding objective. This method cleverly forces the model to understand the intrinsic physical patterns of tactile signals and learn how to infer missing tactile information from visual context. As shown in Qualitative Results of Pre-Training(Figure 4), this approach not only reconstructs tactile images accurately but also learns more interpretable features—the vision encoder's attention consistently focuses on the contact regions between the gripper and the object, which is an elegant and effective contribution.\n    \n3. The experimental results in Table 1 are highly persuasive. The proposed method consistently outperforms all baselines across the board. It shows significant improvement not only over the Vision Only approach (achieving an 85% vs. 25% success rate in the tube collection task, for example) but also clearly surpasses a naive CNN Tactile Fusion method and its own architectural variant without pre-training. Furthermore, the ablation study in Figure 6 clearly demonstrates the immense value of pre-training, showing that the pre-trained model achieves far superior performance in low-data and low-epoch training settings.\n    \n4. The authors' commitment to open-sourcing their hardware design, code, and a large-scale visuo-tactile dataset across 43 task is a valuable contribution. This will significantly enhance reproducibility and drive future work in the field, an effort worthy of high praise.\n\nWeaknesses \n\n1. Piezoresistive sensors can experience performance drift under long-term mechanical stress and in varying environmental conditions, such as temperature and humidity. The paper could be strengthened by discussing the long-term durability of the sensors. This would make the hardware contribution more complete and practical.\n    \n2. The paper reports high success rates, but a more in-depth qualitative analysis of failure cases would be beneficial. For example, what is the root cause of the 10% of failures in the \"Fluid Transfer\" task in Table 1? Providing a visual analysis of failure cases (e.g., with corresponding tactile signals) would offer deeper insights.", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1**: Piezoresistive sensors can experience performance drift under long-term mechanical stress and in varying environmental conditions, such as temperature and humidity. The paper could be strengthened by discussing the long-term durability of the sensors. This would make the hardware contribution more complete and practical.\n\n**A1**: We appreciate the reviewer’s thoughtful question regarding long-term durability. We specifically designed our piezoresistive tactile sensors using flexible PCBs to enhance mechanical robustness and ensure consistency across sensor batches. In practice, we have distributed our sensors to external labs, where they have been used continuously for nearly a year without noticeable performance degradation or failure. Regarding environmental factors: \n\n(i) Humidity: We have not observed any performance issues related to humidity during extensive in-the-wild data collection across both indoor and outdoor environments.\n\n(ii) Temperature: Like most piezoresistive materials, the sensing layer can exhibit signal drift under elevated temperatures. While our sensors operate reliably under typical usage conditions, extremely high temperatures may impact performance. Many commercial piezoresistive tactile sensors are rated for operation within the range of –40 °C to 60 °C, and we expect similar properties from our design, as it shares the same material foundation.\n>**Q2**: The paper reports high success rates, but a more in-depth qualitative analysis of failure cases would be beneficial. For example, what is the root cause of the 10% of failures in the \"Fluid Transfer\" task in Table 1? Providing a visual analysis of failure cases (e.g., with corresponding tactile signals) would offer deeper insights.\n\n**A2**: We list the common failure cases observed with the policy with pretraining:\n\n- Test Tube Collection: The policy misaligns the test tube with the rack hole, resulting in collision with the rack edge.\n\n- Fluid Transfer: During the expelling stage, the pipette occasionally fails to fully release the liquid, leaving some water inside.\n\n- Pencil Insertion: The robot reaches the vicinity of the insertion hole but fails to achieve precise alignment, which causes the insertion failure.\n\n- Whiteboard Erasing: The robot may rotate to an incorrect position, misaligning the eraser with the text and leading to incomplete erasure.\n\nWe have already created visualizations that pair failure cases with corresponding tactile signals. While we cannot include external links due to rebuttal policy, we will incorporate representative visualizations in the revised versions."}, {"review_id": "gI8mUvPiIr", "reviewer": "Reviewer_5R2Q", "summary": "This paper presents a portable tactile gripper designed for in-the-wild data collection and manipulation. The authors develop the tactile sensors compatible with parallel grippers and propose a representation learning framework that aligns tactile and visual modalities through masked reconstruction. The system is pretrained on a large dataset of over 500K visuo-tactile pairs collected across 43 diverse tasks and later fine-tuned on four downstream manipulation tasks. The experiments demonstrate clear benefits of using tactile information and using the visuo-tactile pretraining in improving policy robustness and success rates, particularly in contact-intensive and visually ambiguous settings.", "questions": "Can you quantify the contribution of the tactile sensor’s spatial resolution and time synchronization to downstream policy success?\n\nHow does your pretraining compare quantitatively against existing visual-tactile methods such as VITaL or MVT?\n\nDo the vision encoders (in either pretrained or baseline conditions) use CLIP? If so, does this conflate the benefits of vision-language pretraining with visuo-tactile learning?\n\nCan you provide a more detailed visualization or summary of the 43 tasks used for pretraining?", "limitations": "While the system is capable of collecting tactile data in unstructured environments, the method is currently limited to hand-held grippers and parallel jaw designs. It may not generalize to whole-body or multi-fingered manipulation without significant redesign. Additionally, policy learning still requires domain-specific downstream fine-tuning.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\nHardware Contribution. The design and deployment of a compact, vision-aligned tactile sensor for use on handheld grippers is a great engineering achievement. It enables scalable collection of visuo-tactile data in the wild and lowers the barrier for tactile policy learning.\n\nData Collection at Scale. The authors curate a large and diverse dataset spanning 43 tasks across real-world environments, which is beneficial to the community.\n\nCross-Modal Pretraining. The paper makes a compelling case for joint visuo-tactile representation learning via masked reconstruction. The ablations convincingly show the advantages of pretraining for sample efficiency, attention localization, and final task performance.\n\nDetailed Experiments. The evaluation across four long-horizon manipulation tasks is extensive. The attention maps, loss curves, and failure analyses provide valuable insights into how and when tactile feedback contributes to robustness.\n\nWeakness:\n\nSensor Evaluation. While the tactile sensor design is innovative, the paper lacks quantitative evaluation of the sensor’s improvements. More specifically, how does the improved tactile resolution and time synchronization affect manipulation accuracy?\n\nBaselines: Although the paper presents comprehensive ablation experiments. The baselines for visuo-tactile representation learning are relatively weak. Comparisons against recent visual-tactile methods (e.g., Masked Visual-Tactile Pretraining [1], VITaL [2]) would strengthen claims of superiority.\n\n[1] Masked Visual-Tactile Pre-training for Robot Manipulation. \n[2] VITaL Pretraining: Visuo-Tactile Pretraining for Tactile and Non-Tactile Manipulation Policies\n\nVisualization of Tasks: While the paper mentions 43 tasks for pretraining, a more detailed breakdown or visualization of task categories and environments (beyond Figure 5) would help clarify dataset diversity.\n\nCLIP Usage Ambiguity. It is unclear whether the “no pretraining” baseline still benefits from a pretrained CLIP vision encoder. If so, this should be clarified to disentangle the benefit of cross-modal pretraining from that of strong visual initialization.", "was_revised": true, "final_justification": "The authors provide a comprehensive rebuttal and I believe my concerns have been adequately addressed, so I will maintain my acceptance rating.", "rebuttal": ">**Q1**: Sensor Evaluation. While the tactile sensor design is innovative, the paper lacks quantitative evaluation of the sensor’s improvements. More specifically, how does the improved tactile resolution and time synchronization affect manipulation accuracy?\n\n**A1**: We appreciate the reviewer’s positive comments regarding our tactile sensor design.\n\na) Quantitative Results for sensor resolution:\n\nTo assess the impact of tactile spatial resolution on downstream manipulation performance, we perform 20 rollouts using our policy with pretraining, each with reduced-resolution tactile input 6×16 for each finger (original 12×32). Initial conditions are kept consistent, as described in the main paper.\n\n| Task                                     | Grasp | Reorient | Insert | Whole Task |\n|------------------------------------------|-------|----------|--------|-------------|\n| Policy w/ pretraining (ours)             | 1.0   | 1.0      | 0.85   | 0.85        |\n| Policy w/ pretraining (low-res tactile)  | 1.0   | 1.0      | 0.75   | 0.75        |\n\nQualitative Observations: With lower tactile resolution, we observed a higher likelihood of misalignment during the insertion phase. This is likely due to the coarser spatial granularity: the downsampled tactile signal fails to precisely capture the contact location with the test tube, making it more difficult for the policy to localize contact and perform accurate insertions with tight clearance. We also noticed increased hesitation during reorientation, as the reduced resolution introduces uncertainty in state estimation—similar to behaviors observed in vision-only policies. \n\nb) Synchronization:\n\nWe refer to the UMI paper (Chi et al.) which conducted experiments comparing latency matching (synchronized observation) and no latency matching (non-synchronized observations). Their results showed that aligning sensor latency led to approximately a 30% improvement in performance, highlighting the importance of accurate time synchronization in multi-modal policy learning.\n\n>**Q2**: Baselines: Although the paper presents comprehensive ablation experiments. The baselines for visuo-tactile representation learning are relatively weak. Comparisons against recent visual-tactile methods (e.g., Masked Visual-Tactile Pretraining [1], VITaL [2]) would strengthen claims of superiority.\n\n**A2**: We appreciate the suggestion and have conducted thorough comparisons by faithfully reimplementing and adapting M2VTP and VITaL to our pipeline for a fair evaluation. Our method achieves higher success rate and leads to faster convergence and consistently reaches a lower final loss compared to baselines. The following tables show the downstream tasks loss and success rate for ours and two baselines for test tube collection task:\n\n**Downstream Tasks Loss**\n\n| Epoch    | MAE (Ours) | M2VTP     | VITaL     |\n|----------|-------------|-----------|-----------|\n| Epoch 10 | **0.013352** | 0.014486  | 0.014425  |\n| Epoch 20 | **0.012058** | 0.013864  | 0.013885  |\n| Epoch 30 | **0.011490** | 0.013360  | 0.013529  |\n| Epoch 40 | **0.011196** | 0.012967  | 0.013142  |\n| Epoch 50 | **0.010870** | 0.012537  | 0.012802  |\n| Epoch 60 | **0.010569** | 0.012164  | 0.012492  |\n\n**Downstream Tasks Success Rate**\n\n| Task                                     | Grasp | Reorient | Insert | Whole Task |\n|------------------------------------------|-------|----------|--------|-------------|\n| MAE (ours)                                | 1.0   | 1.0      | 0.85   | 0.85        |\n| M2VTP                                      | 1.0   | 0.9      |  0.1   |   0.1      |\n| VITaL                                         | 0.1  | 0.0      | 0.0   | 0.0        |\n\n**Detailed Analysis**: Our analysis shows that these methods are tailored to fundamentally different use cases and are not well-suited to the demands of single-camera, in-the-wild manipulation, which is the focus of our work. \n\na) M2VTP: M2VTP pretrains a masked autoencoder with a shared transformer that processes 224×224 RGB images and tactile signals, embedded into 768D tokens. After partial masking and reconstruction of both modalities, the encoder is used as a feature extractor for reinforcement learning-based dexterous control.\n\n- i) Qualitative Observations: M2VTP achieves strong tactile reconstruction, but even after extensive training and convergence, its image reconstructions remain coarse, particularly in the gripper's central region and corresponding object. This may be due to the diversity of in-the-wild data and the method’s reliance on large image masking.\n\n- ii) Downstream task performance: M2VTP is designed for simulated tabletop tasks with a dexterous hand and uses a transformer-based vision encoder trained from scratch, which limits its capacity and generalization to in-the-wild settings. Its use of heavy image masking and weak focus on visual reconstruction, likely beneficial in settings that prioritize tactile contact, further reduces sensitivity to fine visual details. In our setting, while M2VTP performs well on grasping and reorientation, it completely fails on insertion tasks for transparent objects, which demand precise visual feedback.\n\nb) VITaL: VITaL uses two ResNet-18 encoders for vision and tactile inputs to produce 512-dimensional features. These are projected into a shared space via MLPs and trained using a temporally-aware contrastive loss. Post-training, the encoder is used in imitation learning frameworks.\n\n- i)  Qualitative Observations: VITaL requires roughly 3 times longer training on the same dataset and shows less stable pretraining performance, with noticeably noisier validation curves and larger standard deviations across different data batches.\n\n- ii) Downstream task performance: VITaL is a contrastive learning method tailored for multi-camera setups, where contrastive loss is computed separately for each camera view. It uses ResNet-18 encoders for both vision and tactile inputs, trained from scratch. The vision encoder is 7× smaller than our CLIP ViT-B/16, severely limiting visual capacity, especially under single-camera, in-the-wild conditions. Moreover, using a full ResNet to encode our 12*32 tactile is unnecessarily complex and inefficient. Despite faithfully reimplementing the full pipeline, VITaL was unsuccessful in rollout, as its trajectories were consistently noisy and failed to complete most task steps. The method is fundamentally mismatched to the demands of large-scale, single-camera, real-world manipulation.\n\n>**Q3**: Visualization of Tasks: While the paper mentions 43 tasks for pretraining, a more detailed breakdown or visualization of task categories and environments (beyond Figure 5) would help clarify dataset diversity.\n\n**A3**: We appreciate the reviewer’s interest in the task diversity of our dataset. We have visualized it, but we may not provide figure during rebuttal due to policy. Below is a concise breakdown of the 43 tasks grouped by environment:\n\n**Indoor (Main Tasks)**:  \n\nTest Tube Collection, Fluid Transfer, Whiteboard Erasing, Pencil Insertion\n\n**Indoor (Other Tasks)**:  \n\nWrite on Whiteboard, Pick & Place Circular Objects, Pick & Place Cup, Toss SD Card, Peg Insertion, Hex Key Insertion\n\n**In-the-Wild Tasks**: \n\n- **Chipotle**: Mix Bowl  \n- **Starbucks**: Place Plate, Insert Straw\n- **Makerspace**: Reorient Marker, Place Plastic Thread into Box, Place Glass into Glass Box, Put Tool into Solder Holder, Turn Soldering Machine Button, Plug USB, Organize Paper Rolls, Place Hex Key, Shut LEGO Box Lid, Assemble LEGO Brick, Insert Metal Socket into Set  \n- **Sink**: Hang Cleaning Spray, Reorient Paint Brush\n- **Kitchen**: Open Microwave, Open Cabinet, Plug Cord  \n- **Library**: Sort Books  \n- **Hardware Store**: Hang Product Pads, Organize Canvas Items, Place Foam Brush, Place Wooden Box, Place Glass Cup, Place Vaseline, Move Books  \n- **Campus – Table**: Pour Water, Place Saucer Cup  \n- **Campus – Staircase**: Write Letter “R”  \n- **Grassland**: Throw Squeeze Ball  \n- **Student Lounge**: Hang Public Phone, Grasp Food\n\n>**Q4**: CLIP Usage Ambiguity. It is unclear whether the “no pretraining” baseline still benefits from a pretrained CLIP vision encoder. If so, this should be clarified to disentangle the benefit of cross-modal pretraining from that of strong visual initialization.\n\n**A4**: All vision encoders in our experiments, both in the baseline and visuo-tactile pretraining conditions, use a CLIP backbone. This follows established practice (e.g., UMI) and provides a strong, consistent visual representation across all policies.\n\nWe would like to clarify that this setup does not conflate the benefits of vision-language pretraining (from CLIP) with those of visuo-tactile learning. In our work, “pretraining” refers specifically to visuo-tactile pretraining, where the model aligns visual and tactile inputs through joint representation learning. Both baseline and pretrained models use the same CLIP-initialized vision encoder; the only difference is whether this encoder is further trained with tactile context. This design isolates the contribution of visuo-tactile pretraining: since all models use CLIP, performance gains reflect the value of cross-modal alignment, not the vision backbone alone.\n\n>**Q5**: Can you quantify the contribution of the tactile sensor’s spatial resolution and time synchronization to downstream policy success?\n\n**A5**: Please see our response to **Q1**.\n\n>**Q6**: How does your pretraining compare quantitatively against existing visual-tactile methods such as VITaL or MVT?\n\n**A6**: Please see our response to **Q2**.\n\n>**Q7**: Do the vision encoders (in either pretrained or baseline conditions) use CLIP? If so, does this conflate the benefits of vision-language pretraining with visuo-tactile learning?\n\n**A7**:  Please see our response to **Q4**.\n\n>**Q8**: Can you provide a more detailed visualization or summary of the 43 tasks used for pretraining?\n\n\n**A8**: Please see our response to **Q3**"}, {"review_id": "ZSTWApM6in", "reviewer": "Reviewer_MhLa", "summary": "This paper presents a handheld (UMI-style) gripper system equipped with tactile sensing for collecting visuotactile demonstrations from human operators. The authors also propose a representation learning framework for learning visuotactile representations. The system is evaluated on four contact-rich tasks. While the approach is timely and results are promising, the paper leaves several key concerns unaddressed, and the overall novelty is limited.", "questions": "I would consider increasing the score if these concerns are clearly addressed:\n* How do the authors justify collecting tactile signals through humans that receive no tactile feedback? In that case, how can the demonstrations reflect touch-driven behaviors?\n* Is the encoder fine-tuned during imitation learning, or kept frozen?\n* Are the evaluation tasks included in the pretraining data? Can you report performance when these tasks are held out?\n* Are the hardware files and fabrication details of the handheld gripper going to be released?\n* What limits the tactile sampling rate to 23Hz, and could it be increased?\n* Would a stronger vision-only baseline with data augmentations reduce the observed performance gap?\n* How reusable is the collected tactile data across sensors, especially considering variability in fabrication?", "limitations": "Just about a line is dedicated to one single limitation of the approach. Please consider including some of the above weaknesses to provide readers with a clearer perspective on this work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n* Timely effort to enable large-scale visuotactile data collection using a handheld gripper interface.\n* The system is demonstrated on contact-rich manipulation tasks, showcasing the importance of tactile sensing for representation learning.\n* The authors will release the collected dataset, which should benefit the robotics community.\n\n**Weaknesses:**\n\n* A key issue with handheld grippers is the distribution mismatch between human-operated data collection and downstream robot deployment, which is not discussed or analyzed.\n* The human teleoperator does not seem to receive tactile feedback. This raises concerns about the ability of the demonstrations to reflect touch-driven behavior, especially for tasks claimed to depend on tactile input (e.g., fluid transfer and whiteboard erasing). To further clarify, I agree that representations can still be stronger because of tactile sensing, but I am questioning the presence of touch-reactive behaviors.\n* While the system is well-executed, the overall novelty is limited. The introduction lists 3 contributions. While equipping UMI with tactile sensing (1) is useful, contribution (2) is closely based on [22], and contribution (3) is a fairly standard application of masked autoencoding to visuotactile fusion [1],[2],[3], despite being claimed as novel.\n* The encoder appears to be pretrained on data including the same tasks used for downstream evaluation. This makes it difficult to isolate the impact of pretraining. Holding out evaluation tasks during pretraining would provide a better perspective on generalization.\n\n[1] VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulation. Sun et al., 2024.\n\n[2] The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning. Sferrazza et al., 2023.\n\n[3] Sparsh: Self-supervised touch representations for vision-based tactile sensing. Higuera et al., 2024.", "was_revised": true, "final_justification": "In general, I believe that limited novelty is the main remaining weakness of this work. But I think the paper has considerably improved during rebuttal and the work is overall well executed, so I will raise my score to a borderline accept.", "rebuttal": ">**Q1**: A key issue with handheld grippers is the distribution mismatch between human-operated data collection and downstream robot deployment, which is not discussed or analyzed.\n\n**A1**: We took several steps to ensure consistency between the handheld gripper used for data collection and the robot platform used for deployment.\n\n(i) Camera Observation Alignment: A custom camera mount was built on the robot to match the height and orientation of the camera as it appears on the handheld gripper. During rollouts, we verified that the handheld and robot cameras captured nearly identical views.\n\n(ii) Gripper Width Calibration: We aligned the min/max widths of the handheld and XARM grippers, using linear interpolation to map between them. Calibration was verified both qualitatively (by finger spacing) and quantitatively (by replaying demonstrations).\n\n>**Q2**: The human teleoperator does not seem to receive tactile feedback. This raises concerns about the ability of the demonstrations to reflect touch-driven behavior, especially for tasks claimed to depend on tactile input (e.g., fluid transfer and whiteboard erasing). To further clarify, I agree that representations can still be stronger because of tactile sensing, but I am questioning the presence of touch-reactive behaviors.\n\n**A2**: The handheld gripper includes a built-in spring that provides force-based haptic feedback, allowing the human operator to feel resistance during grasping. This enables the operator to gauge how much force is being applied and whether the object is securely held.\nDuring downstream task data collection in the lab, we also provide real-time visualization of tactile signals on a screen, which the operator actively uses to guide interaction. In the fluid transfer task, where the pipette tip is occluded by water and not visible, the operator uses the tactile feedback to ensure that sufficient force is applied to extract the liquid successfully. Similarly, in the whiteboard erasing task, the operator both feels the contact through the spring mechanism and monitors the tactile signals on the screen to confirm that adequate pressure is being applied.\nMoreover, in the tube reorientation task, we conduct human-intervention experiments during policy execution. When the tube pose is intervened, the policy reorient the tube again. This demonstrates that the learned policy is responsive to tactile input.\n\n>**Q3**: While the system is well-executed, the overall novelty is limited. The introduction lists 3 contributions. While equipping UMI with tactile sensing (1) is useful, contribution (2) is closely based on [22], and contribution (3) is a fairly standard application of masked autoencoding to visuotactile fusion [1],[2],[3], despite being claimed as novel.\n\n**A3**: \n\na) While our handheld device is inspired by [22], our work extends far beyond simply adding tactile sensors. We deliberately chose a piezoresistive tactile sensor for large-scale in-the-wild data collection due to its unique advantages: (i) Lightweight. (ii) Easy integration with soft grippers. (iii) We also hope to highlight another key advantage that may not be obvious at first glance on the **sensor consistency**. We designed and fabricated custom flexible PCBs to ensure sensor consistency, enabling cross-sensor transfer. This means we can collect data with one pair of sensors and successfully rollout policies with another sensor pair. Our design supports scaling to multiple data collection devices with different sensor pairs.\n\nb) While our masked autoencoding approach builds on a standard framework, our architecture is carefully designed to meet the unique challenges of a single-camera, handheld gripper setup. This targeted design enables it to outperform baselines like VITaL and MVT (see Q2 from Reviewer 5R2Q). We believe our approach offers meaningful insights for advancing visuo-tactile representation learning, particularly in the context of large-scale, in-the-wild datasets.\n\nc) Another core contribution of our work is large-scale visuo-tactile data collection in the wild, covering 43 diverse manipulation tasks. The dataset contributes unique value to the community in considering both vision and touch, and also physical interactions in the wild (not just touch as in paper [1]). We believe this scale and diversity largely advance the study of visuo-tactile correlation. \n\n[1] Tactile-Augmented Radiance Fields\n\n\n>**Q4**: The encoder appears to be pretrained on data including the same tasks used for downstream evaluation. This makes it difficult to isolate the impact of pretraining. Holding out evaluation tasks during pretraining would provide a better perspective on generalization.\n\n**A4**: We removed all test tube collection tasks from the pretraining dataset and did additional experiments. This results in a total of ~2500 in-the-wild videos for pretraining (previously ~2700). We pretrained the encoder on this reduced dataset, then finetuned it on the test tube collection task. Finally, we evaluated policy performance.\ni) Qualitative Observations: The attention heatmap on both the training and validation set images remains primarily focused on the gripper region.\nii) Quantitative Result: We conducted policy rollouts for the test tube collection task for both policies. We observed no behavioral inconsistencies between the two policies: \n| Task                                                       | Grasp | Reorient | Insert | Whole Task |\n|------------------------------------------------------------|-------|----------|--------|-------------|\n| Policy w/ pretraining (ours)                               | 1.0   | 1.0      | 0.85   | 0.85        |\n| Policy w/ pretraining (without tube in pretrain data)      | 1.0   | 1.0      | 0.80   | 0.80        |\n\n>**Q5**: How do the authors justify collecting tactile signals through humans that receive no tactile feedback? In that case, how can the demonstrations reflect touch-driven behaviors?\n**A5**: Please see our response to **Q2**\n\n>**Q6**: Is the encoder fine-tuned during imitation learning, or kept frozen?\n\n**A6**: The encoder is fine-tuned during imitation learning. We believe fine-tuning is important because, while the pretrained encoder captures general visuo-tactile relationships, individual tasks often require additional task-specific features from both modalities. To preserve the benefits of pretraining while allowing adaptation, we fine-tune the encoder using a smaller learning rate (3e-5) compared to the diffusion module (3e-4).\n\n>**Q7**: Are the evaluation tasks included in the pretraining data? Can you report performance when these tasks are held out?\n\n**A7**: Please see our response to **Q4**\n\n>**Q8**: Are the hardware files and fabrication details of the handheld gripper going to be released?\n\n**A8**: Yes, we will release the complete hardware setup, including all fabrication details and files, to enable full reproducibility of our handheld gripper system.\n\n>**Q9**: What limits the tactile sampling rate to 23Hz, and could it be increased?\n\n**A9**: The tactile sampling rate is currently limited to 23 Hz due to constraints in our PCB design and firmware. It can be increased through several possible improvements:\n\n(i) **Upgrading MCU** (current Arduino Nano) with a higher-performance microcontroller would directly enhance sensor reading speed.\n\n(ii) **Data compression** for serial communication in firmware using. For instance, instead of individually transmitting long sequences of zero-valued readings, we can encode consecutive zeros compactly (e.g., transmitting “17 zeros” rather than seventeen individual zeros).\n\n(iii) Employing **subsampling and reconstruction** methods by randomly subsampling sensor measurements and reconstructing full tactile frames.\n\n>**Q10**: Would a stronger vision-only baseline with data augmentations reduce the observed performance gap?\n\n**A10**: Our vision-only baseline follows the exact implementation from Chi et al. in the UMI paper. Specifically, we finetune from a pretrained CLIP vision encoder and apply standard image augmentations, including color jittering and spatial cropping, consistent with prior work and shown to be effective.\n\nAll four policies in our comparison use the same vision augmentation setup to ensure fairness. While additional augmentations might offer marginal improvements, we believe they are unlikely to close the performance gap. Most failure cases in the vision-only baseline occur in scenarios where tactile feedback is essential—for example, determining whether a test tube is upright or tilted—especially given its transparency—or extracting the correct amount of fluid with a pipette, which requires precise force control. These challenges are inherently difficult to solve with vision alone.\n\n>**Q11**: How reusable is the collected tactile data across sensors, especially considering variability in fabrication?\n\n**A11**: During data collection and evaluation, we used four tactile sensors across two different grippers and robot arms. All sensors were fabricated in the same batch using an identical production process, and we observed consistent performance across all units.\n\nIn addition to this consistency, our sensor design focuses on capturing structural tactile patterns—such as contact distribution and pressure regions—rather than relying on fine-grained signal details. This makes our sensor inherently robust to minor variations between sensor units, which supports generalization across different hardware setups.\n\n>**Q12 Limitations**: Including some of the above weaknesses to provide readers with a clearer perspective on this work.\n\n**A12**: \n\n- Careful manual calibration is still required to align the physical setups of the handheld and robot grippers. \n\n- We still need to stream tactile data to a host Ubuntu machine using USB. Enabling wireless communication  would improve portability.\n\n- Tactile reading is limited to 23 hz for current PCB Design."}, {"review_id": "cldydOPBwz", "reviewer": "Reviewer_meGV", "summary": "In this paper, the authors first present a data collection hardware for multi-modal data collection, which integrates RGB camera and tactile sensors. Then  they designs a cross-modal representation learning framework to intergrate these modalties through pretraining. The authors evaluate their hardware and method on 4 real-world manipulation tasks, where it outperforms the vision-only baseline. Ablation studies demonstrate the effectiveness of multi-modal pretraining.", "questions": "1. Initial state distribution: The consistency of the initial states of objects across the comparisons is unclear. The authors should explicitly state whether the initial conditions are standardized to ensure fair comparision.\n\n2. Comparision with other multi-modal pretraining methods should be added.\n\n3. The authors use a parallel gripper from UMI as the data collection system. However, it is deployed on XARM gripper, whose design is different. The paper should explain why this discrepancy does not affect performance.\n\n4. The inclusion of an ablation study analyzing the impact of in-the-wild data would strengthen the paper.\n\n5. Tactile Sensor Consistency: The consistency of different tactile sensor instances is not addressed. The authors should clarify whether the same sensor instances are used during data collection and policy deployment.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 1, "strengths_and_weaknesses": "Strengths:\n1. A robot-free data collection system that can collect visual and tactile data simultaneously.\n2. A multi-modality pretraining framework that improves training and sample efficiency.\n3. The paper is well written,and the quality of visualizations is good.\n\nWeakness:\n1. Similarity: The paper structure, including the multi-modal pretraining, task design (tube reoriation, insertion) and ablation studies (w/o pretraining) , are very similar to https://arxiv.org/abs/2504.06156. While it is cited, it is not well described and discussed in the related work.\n\n2. Technical novelty: The novelty of the hardware is very limited. It just intergrates the tactile sensor into UMI. The pretraining of multimodal representation is common and standard design in the community. No new architectures are introduced.\n\nDave V, Lygerakis F, Rueckert E. Multimodal visual-tactile representation learning through self-supervised contrastive pre-training[C]//2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024: 8013-8020.\n\nKerr J, Huang H, Wilcox A, et al. Self-supervised visuo-tactile pretraining to locate and follow garment features[J]. arXiv preprint arXiv:2209.13042, 2022.\n\n3. Title: It is named as \"in the wild\". but it still needs to a laptop to collect the tactile signals. So it is not a stand-alone system.\n\n4. Experiments: While the authors collect data in the environments that are different from the tasks, its usefulness isnot validated in the experiment.", "was_revised": true, "final_justification": "Some of my concerns are addressed. But  the technical novelty does not meet the standard of NeurIPS. And the comparison with the work of Vitamin should be strengthened.", "rebuttal": ">**Q1**: Similarity: The paper structure, including the multi-modal pretraining, task design (tube reoriation, insertion) and ablation studies (w/o pretraining) , are very similar to https://arxiv.org/abs/2504.06156. While it is cited, it is not well described and discussed in the related work.\n\n**A1**: While both projects share the high-level goal of enabling visuo-tactile learning for contact-rich manipulation using portable systems, there are several key differences that distinguish our work:\n\na) Hardware and System Design:\n\nOur work goes beyond integrating tactile sensors into handheld devices. We deliberately chose a piezoresistive tactile sensor for large-scale in-the-wild data collection due to its unique advantages: (i) Lightweight. (ii) Easy integration with soft grippers. (iii) We also hope to highlight another key advantage that may not be obvious at first glance on the **sensor consistency**. We designed and fabricated custom flexible PCBs to ensure sensor consistency, enabling cross-sensor transfer. This means we can collect data with one pair of sensors and successfully rollout policies with another sensor pair. In contrast, the ViTaMIn, they use the same sensor pair, likewise many other optical-based tactile sensors, maintaining the consistency among different pairs of sensors is noticeably harder, limiting their scalability. Our design supports scaling to multiple data collection devices with different sensor pairs.\n\nb) Dataset:\n\nA core contribution of our work is large-scale visuo-tactile data collection in the wild, covering 43 diverse manipulation tasks. The dataset contributes unique value to the community in considering both vision and touch, and also physical interactions in the wild (not just touch as in paper [1]). We believe this scale and diversity largely advance the study of visuo-tactile correlation. \n\n[1] Tactile-Augmented Radiance Fields\n\nc) Downstream Task Design\n\nRegarding downstream tasks, although both our work and ViTaMIn include test tube reorientation, the task complexity differs substantially. In ViTaMIn, the test tube is grasped, reoriented, and the task concludes. In contrast, our setup involves grasping the test tube from within a cluttered container, performing reorientation, and then executing a precise insertion to verify the accuracy of reorientation. This added step demands a higher robustness of policy, making our task setting significantly more challenging.\n\n>**Q2**: Technical novelty: The novelty of the hardware is very limited. It just intergrates the tactile sensor into UMI. The pretraining of multimodal representation is common and standard design in the community. No new architectures are introduced.\n\n**A2**: a) Our work goes beyond integrating tactile sensors into handheld devices. See answer of **Q1**. \n\nb)  While our masked autoencoding approach builds on a standard framework, our architecture is carefully designed to meet the unique challenges of a single-camera, handheld gripper setup. This targeted design enables it to outperform baselines like VITaL and MVT (see **Q6**). We believe our approach offers meaningful insights for advancing visuo-tactile representation learning, particularly in the context of large-scale, in-the-wild datasets.\n\n>**Q3**: Title: It is named as \"in the wild\". but it still needs to a laptop to collect the tactile signals. So it is not a stand-alone system.\n\n**A3**: While our current setup uses a laptop for data logging, this component is not fundamental to the system design. It can be easily replaced with any lightweight, Ubuntu-based computing unit—a common approach in mobile robotic systems operating in the wild. Our focus is on the portable sensing and data collection hardware, which is fully compatible with embedded systems for stand-alone deployment.\n\n>**Q4**: Experiments: While the authors collect data in the environments that are different from the tasks, its usefulness isnot validated in the experiment.\n\n**A4**: To verify the usefulness of in-the-wild data, we removed all test tube collection tasks from the pretraining dataset and did additional experiments. This results in a total of ~2500 in-the-wild videos for pretraining (previously ~2700). We pretrained the encoder on this reduced dataset, then finetuned it on the test tube collection task. Finally, we evaluated policy performance.\n\ni) Qualitative Observations: The attention heatmap on both the training and validation set images remains primarily focused on the gripper region.\n\nii) Quantitative Result: We conducted policy rollouts for the test tube collection task for both policies. We observed no behavioral inconsistencies between the two policies: \n\n| Task                                                       | Grasp | Reorient | Insert | Whole Task |\n|------------------------------------------------------------|-------|----------|--------|-------------|\n| Policy w/ pretraining (ours)                               | 1.0   | 1.0      | 0.85   | 0.85        |\n| Policy w/ pretraining (without tube in pretrain data)      | 1.0   | 1.0      | 0.80   | 0.80        |\n\n>**Q5**: Initial state distribution: The consistency of the initial states of objects across the comparisons is unclear. The authors should explicitly state whether the initial conditions are standardized to ensure fair comparision.\n\n**A5**: We ensured fair comparisons by standardizing initial conditions across all policy evaluations. All four policies were rolled out in the same session using an identical workspace setup. We also recorded and verified overlapping initial RGB frames to confirm state consistency.\n\n>**Q6**: Comparision with other multi-modal pretraining methods should be added.\n\n**A6**: As suggested by the reviewer 5R2Q,  we contact baseline multi-modal pretraining methods M2VTP and VITaL to our pipeline for a fair evaluation. Our method achieves higher success rate and leads to faster convergence and consistently reaches a lower final loss compared to baselines. The following tables show the downstream tasks success rate for ours and two baselines for test tube collection task:\n\n**Downstream Tasks Success Rate**\n\n| Task                                     | Grasp | Reorient | Insert | Whole Task |\n|------------------------------------------|-------|----------|--------|-------------|\n| MAE (ours)                                | 1.0   | 1.0      | 0.85   | 0.85        |\n| M2VTP                                      | 1.0   | 0.9      |  0.1   |   0.1      |\n| VITaL                                         | 0.1  | 0.0      | 0.0   | 0.0        |\n\n**Analysis**: Our analysis shows that these methods are tailored to fundamentally different use cases and are not well-suited to the demands of single-camera, in-the-wild manipulation, which is the focus of our work. Due to length limit, please refer to reviewer 5R2Q **Q2** for the detailed implementation and analysis. \n\n>**Q7**: The authors use a parallel gripper from UMI as the data collection system. However, it is deployed on XARM gripper, whose design is different. The paper should explain why this discrepancy does not affect performance.\n\n\n**A7**: We calibrate the gripper width by aligning the minimum and maximum openings of both grippers and applying linear interpolation. This mapping proved effective in practice. A similar approach was used in the original UMI paper (Chi et al.), where data collected with the UMI gripper was successfully deployed on the WSG gripper. This cross-gripper transfer is a standard and validated solution.\n\n>**Q8**: The inclusion of an ablation study analyzing the impact of in-the-wild data would strengthen the paper.\n\n**A8**: Please see our response to **Q4**\n\n>**Q9**: Tactile Sensor Consistency: The consistency of different tactile sensor instances is not addressed. The authors should clarify whether the same sensor instances are used during data collection and policy deployment.\n\n**A9**: All sensors were fabricated using the same manufacturing process based on flexible PCBs from factory, ensuring high consistency across instances. Thanks to this consistency, policies trained using data collected from one sensor pair can be reliably executed on a different sensor pair during deployment, without additional replacement. This cross-sensor feature is one of the key advantages of our system design for easy transfer cross embodiment and large scale data collection in the wild.\n\nQuantitative Results: \n\nWe processed 60 Hz tactile data from training and recorded 10 Hz tactile signals across 20 successful rollouts. We then compared the statistical distributions of training and rollout signals, confirming their alignment.\n\n### Per-pixel Distribution Statistics\n\n| Split | n       \t| mean   | std\t| min\t| max\t| skew   | kurt   |\n|-------|-------------|--------|--------|--------|--------|--------|--------|\n| Train (Human Demonstrations) | 286,114,560 | 0.0461 | 0.1699 | 0.0000 | 1.0000 | 3.9312 | 14.8065 |\n| Eval (Robot Rollouts) | 1,324,800   | 0.0414 | 0.1523 | 0.0000 | 1.0000 | 4.0988 | 16.8261 |\n—\n\n### Per-episode Distribution Statistics\n\n| Split          \t| mean   | std of means | median | 95%ile  |\n|--------------------|--------|---------------|--------|---------|\n| Train episode-means (Human Demonstrations) | 0.0461 | 0.0035    \t| 0.0465 | 0.0512  |\n| Eval episode-means (Robot Rollouts) | 0.0414 | 0.0036    \t| 0.0409 | 0.0477  |\n\nResults show similar tactile signal levels between train (mean = 0.046, std = 0.170) and eval (mean = 0.041, std = 0.152), with tight episode-level variability (train std = 0.0035; eval = 0.0036). Both distributions exhibit strong right skew and high kurtosis, indicating sparse background with high-intensity contacts—consistent with touch-driven behavior."}]

Author rebuttal:

Meta-review: This paper presents a practical, portable visuo-tactile handheld gripper and a simple two-stage learning recipe (masked tactile reconstruction with cross-modal fusion to diffusion policy) that together deliver clear gains on contact-rich, partially occluded manipulation tasks collected “in the wild.” Reviewers were mixed on technical novelty—arguing the learning pieces are incremental and contemporaneous with related work—but were uniformly positive about the engineering value, the sizeable synchronized RGB-tactile dataset, and the convincing real-robot improvements over vision-only and naïve fusion baselines. The rebuttal strengthened the case by adding stronger baselines (e.g., VITaL, M2VTP) under matched settings, clarifying CLIP usage, and providing useful durability notes; while some risks remain (breadth limited to four tasks on one platform, modest gains on one task, and missing uncertainty estimates), these are addressable in camera-ready. 

I encourage the authors to expand comparisons to concurrent methods, report confidence intervals for success rates, include failure-case visualizations with tactile traces, and summarize calibration/drift guidance. 

Overall, the combination of accessible hardware, real-world data, and consistent empirical improvements makes this a valuable contribution for the community. Recommendation: Accept.
Decision: Accept (poster)