Universal Visuo-Tactile Video Understanding for Embodied Interaction
Authors: Yifan Xie
Mingyang Li
Shoujie Li
Xingting Li
Guangyu Chen
Fei Ma
Fei Yu
Wenbo Ding
Venue: NeurIPS 2025 poster ()
Keywords: Embodied Interaction
Tactile Perception
Video Understanding
Primary area: applications

Abstract: Tactile perception is essential for embodied agents to understand the physical attributes of objects that cannot be determined through visual inspection alone. While existing methods have made progress in visual and language modalities for physical understanding, they fail to effectively incorporate tactile information that provides crucial haptic feedback for real-world interaction. In this paper, we present VTV-LLM, the first multi-modal large language model that enables universal Visuo-Tactile Video (VTV) understanding, bridging the gap between tactile perception and natural language. To address the challenges of cross-sensor and cross-modal integration, we contribute VTV150K, a comprehensive dataset comprising 150,000 video frames from 100 diverse objects captured across three different tactile sensors (GelSight Mini, DIGIT, and Tac3D), annotated with four fundamental tactile attributes (hardness, protrusion, elasticity, and friction). We develop a novel three-stage training paradigm that includes VTV enhancement for robust visuo-tactile representation, VTV-text alignment for cross-modal correspondence, and text prompt finetuning for natural language generation. Our framework enables sophisticated tactile reasoning capabilities including feature assessment, comparative analysis, and scenario-based decision-making. Extensive experimental evaluations demonstrate that VTV-LLM achieves superior performance in tactile reasoning tasks, establishing a foundation for more intuitive human-machine interaction in tactile domains.

Universal Visuo-Tactile Video Understanding for Embodied Interaction
Yifan Xie', Mingyang Li', Shoujie Li', Xingting Li', Guangyu Chen?, Fei Ma Fei Yu², Wenbo Ding
Tsinghua Shenzhen International Graduate School, Tsinghua University Shenzhen Campus of Sun Yat-sen University Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)
Abstract
Tactile perception is essential for embodied agents understand physical attributes of objects that cannot be determined through visual inspection alone. While exist- ing approaches have made progress in visual and language modalities for physical understanding, they fail to effectively incorporate tactile information that provides crucial haptic feedback for real-worh interaction. In this paper, we present VTV- LLM, the first multi-modal large language model for universal Visuo- Tactile Video (VTV) understanding that bridges the gap between tactile perception and natural language. To address the challenges of cross-sensor and cross-modal integra- tion, we contribute VTV150K, comprehensive dataset comprising 150,000 video frames from 100 diverse objects captured across three different tactile sensors (Gel- Sight Mini, DIGIT, and Tac3D), annotated with four fundamental tactile attributes (hardness, protrusion, elasticity, and friction). We develop novel stage training paradigm that includes VTV enhancement for robust visuo-tactile repre- sentation, VTV-text alignment for cross- modsll correspondence, and text prompt finetuning for natural language generation. Our framework enables sophisticated tactile reasoning capabilities including feature assessment, comparative analysis, scenario-based decision making and so on. Experimental evaluations demonstrate that VTV-LLM achieves superior performance in tactile video understanding tasks, establishing a foundation for more intuitive human-machine interaction in tactile domains.
Introduction
Touch is fundamental sensory modality that provides humans with physical information unattainable through vision alone, such as material attributes, surface texture, and compliance. This tactile feedback enables sophisticated physical reasoning and interaction in our environment 223 While recent advances in vision- languae models [ 6 7 8 have demonstrated impressive capabilities in visual reasoning, these models remain fundamentally limited by their inability to perceive tactile attributes. restricting their effectiveness in scenarios requiring physical interaction and reasoning about material characteristics that cannot be reliably inferred from visual cues alone.
Visuo-tactile sensors (9), like GelSight [TO], DIGIT [T] and Tac3D [12], have emerged as promising technologies for capturing tactile information, generating image-like representations that encode physical properties such as pressure distribution, surface geometry, and friction characteristics. However, there remains significant challenge in bridging the domain gap between these tactile representations and natural language understanding. The inherent differences between tactile data
"Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Data Collection
(b) Attribute Annotation
Blrmnss
Preinaka
Siglat
Sranks
DIGIT
Molesady Eoroomable
iazusally Held
TocaD
Kight Ticiic
Objects
Sensors
150,000 Video Frames
Attributes
Question:
Scaiuees object presented the
Anooy lhes ihnce vidkoos: svideo Es, svideo 2>, e) <video, 30, identify the <compartson> <attribute= object after desembing each
cavocado mpp-??
VTV-LLM: modentlly deformablc under fiece with strong proinesion lues modertte dlsticily and modeute Giglion cleraciriatics.
Surface Feature Distinction
WTTALA
Qucetio:: Compuring objects Savacadiumpp sxcleet.npee which one more prseruded?
Fidew/Comparinn Altribute Selection
VTV-LLM: The finst object
bark
Question: Deserbbe thene two vidaos based on their uctile sttributen:
Tactile Scenario Analyis
Question:
Amang these three vodeos: -Jimi szircult board,
inside box
-harkLmp4> b) svclvet.mpp- Which material would he better lining
hold delicate jewelry?
ball augds, idemify the least proiruded object descriting Answer: al.. 4)... Concluson: promoded object is
extremll and strong protnasion
elasticity strong diicioon b) presents modentely
velvermp4
peotrasion with no elssicity and sight fiction, The most suitable oboect b).......
10,000 Question-Ansin -nnseer Pairs
(d) Embodied Interaction
(c) Template Generation
Figure The workflow consists of four key components: (a) Data Collection, which includes 100 diverse objects recorded by 3 different tactile sensors, resulting in 150,000 video frames; (b) Attribute Annotation, where objects are systematically categorized across static and dynamic tactile attributes:
hardness, protrusion, elasticity, and friction; (c) Template Generation, which generates 10,000 question-answer pairs using structured templates for various comparative analyses; and (d) Embodied Interaction, demonstrating VTV-LLM's capability to perform tactile feature assessment, surface feature distinction, tactile scenario analysis and so on. Through this integrated approach, VTV-LLM enables multi-modal reasoning about physical attributes that cannot be determined through visual inspection alone, creating a foundation for more sophisticated buman-maching machine interaction in tactile understanding domains.
captured across various sensor types further complicates this integration, as each sensor produces distinct data formats with varying resolutions and physical property encodings.
Existing research on tactile learning has made progress in representation learning 17 [8], but these approaches often focus either exclusively on static attributes or fail develop comprehensive frameworks that integrate both tactile perception and language understanding. Most critically, they lack the ability ground tactile perceptions natural language descriptions and reasoning, which is essential for human-machin machine communication about physical properties and interactions [19 200. Additionally, the temporal dimension of tactile interactions, which captures how surfaces respond to pressing, sliding, and rotational movements, remains underexplored in current approaches, despite containing crucial information about dynamic material attributes.
13
14
15
16
To address these challenges, we present VTV-LLM, the first multi-modal large language model for universal visuo-tactile video understanding. Our approach treats tactile perception as cross- modal reasoning problem, where tactile videos are aligned with linguistic descriptions to enable sophisticated reasoning about physical attributes. As illustrated in Fig, [d), VTV-LLM supports diverse range of embodied interaction capabilities, from basic tactile feature assessment to complex comparative analyses and ecenario-based decision making. Additionally, we construct the VTV150K dataset, comprising 150,000 video frames collected from 100 common objects across three different tactile sensors. We systematically annotate these videos with four fundamental tactile attributes (hardness, protrusion, elasticity, and friction), creating a structured foundation for tactile reasoning. To bridge the substantial gap between tactile perception and language understanding, we develop three-stage training paradigm: (1) VTV enhancement through optical flow-guide masking to learn

---

robust visuo-tactio representations, (2) VTV-text alignment to establish cross moosll correspondence, and (3) text prompt finetuning to optimize natural language generation about tactile attributes.
Our main contributions can be summarized as follows:
We introduce VTV-LLM, the first multi-modal large language model capable of universal visuo-tactile video understanding, enabling sophisticated embodied reasoning through natural language interaction.
We contribute VTV150K, comprehensive dataset of 150,000 visuo-tactile video frames capturing 100 diverse objects across three tactile sensors, annotated with four fundamental tactile attributes.
We develop a novel three-stage training paradigm that effectively bridges the domain gap between tactile perception and language understanding, providing valuable reference for future cross-modal learning efforts.
2
Related Works
Tactile Perception Tactile perception has evolved significantly from early sensors measuring basic physical properties sophisticated vision-based systems providing high-resolution contact information. Visuo-tactile sensors [9] such as GelSight [0O], DIGIT [11, and Tac3D [12] have garnered widespread attention for their ability capture detailed contact deformations through elastomeric gels and embedded cameras. These sensors have enabled numerous robotic applications including material classification (21). shape reconstruction 22 233, and dexterous manipulation tasks [24] 14]. Recent research has focused on developing representation learning approaches for tactile data, progressing from task-specific models [25] general cerepol-purpose representations using self- supervised techniques like contrastive multi-view coding [21] and masked autoencoders [26). The integration of tactile sensing with other modalities has also emerged as a promising direction, with works like UniTouch (17) dynamically fusing tactile signals with visual and audio data to enhance cross-sensor knowledge transferability, Yu al.
[15] aligned tactile images with vision- lisoon-language models for object property reasoning, and Fu et al. [16] used nouch-vision-lane language model for open-vocabulary classification. Unlike prior works, our method processes visuo-tactile video directly and focuses on sophisticated tactile reasoning.
Self-Supervised Video Representation Learning Self-supervised video representation learning has emerged critical area for developing robust visual features without manual annotations. VideoMAE [27] pioneered this approach by effectively adapting masked autoencoding strategies to the video domain, demonstrating significant performance improvements across various benchmark tasks. Subsequently, VideoMAEV2 [28) enhanced this framework through the introduction of dual masking mechanisms, which substantially improved computational efficiency while maintaining representational power. Recent advancements in this field have focused on sophisticated optimiza- tions along both temporal and spatial dimensions (29 30 31 32 addressing challenges unique video understanding such as motion coherence and long-range dependencies. In the tactile domain, Sparsh [18] explored the ability of different existing self-supervised learning methods to characterize in tactile video. Feng et al. [13] utilized the tube masking strategy to process the tactile video. Our method builds upon these foundations by introducing optical flow-guided masking specifically designed for visuo-tactile videos, which addresses the unique challenges of capturing both spatial deformation and temporal dynamics in tactile interactions.
Multi-Modal Large Language Models Multimodal Large Language Models (MLLMs) have transformed AI research by enabling reasoning across textual and visual modalities. Early efforts integrated LLMs as agents for downstream tasks [33
33). Later approaches focused on parameter- efficient tuning 36 37] and instruction tuning 38 to align visual semantics with language. 41 and diverse sensory inputs [42] our work, we present the first visuo-tactile video large language model to bridge the gap between tactile perception and natural language.
Recent advances have incorporated video processing 40
enabling applications in roboties 43

---

3 Methods
In this section, we first introduce VTVI50K, a large-scale dataset of video-question-anae ansser pairs in 3.1] Subsequently, we present VTV-LLM, the first visuo-tactile video large language model designed for visuo-tactile video understanding and embodied interaction in Sec. 3.2
3.1 VTV150K
Overview Visuo-tactile sensor technologies suffer from inadequate standardization and significant cross-sensor data discrepancies, which substantially impede the transferability of tactile representation models across different sensing platforms. Existing methods [14
18]
45 [3] addressing these
challenges exhibit notable limitations, as they either neglect the integration of both static and dynamic tactile attributes or fail to incorporate comprehensive visuo-tactile video understanding for embodied interaction.
In this work, introduce VTV150K, comprehensive large-scale dataset comprising vidco-question- answer pairs collected across three diverse visuo-tactile sensors, as illustrated in Fig. [a-c.. The dataset construction methodology encompasses three sequential stages: data collection, attribute annotation, and template generation. We will delve into the specifics of these stages.
Data Collection facilitate the grounding of embodied interaction on tactile inputs, we collected a comprehensive dataset comprising 100 common objects. yielding total of 150,000 visuo-tactile video frames, with each video recorded at 20 FPS and resolution of 320x320 pixels.
As illustrated in Fig. Ta). we employed multiple visuo-tactile sensors to ensure style diversity: GelSight mini [10] and DIGIT [I] sensors for capturing high-resolution visuo-tactile information, and Tac3D 12] for measuring deformation force fields. Due to the relatively low resolution of Tac3D, we implemented the cubic spline interpolation algorithm (46) reconstruct more detailed force field representations.
Data collection performed manually to address the challenges associated with properly interacting with irregularly shaped objects. For each object, we systematically captured five visuo-tactile videos across different regions using various sensors. Our data collection process consisted of three sequential interactions: (1) normal pressing against the object surface to capture pressure distribution, (2) rotational movement acquire shear information, and (3) sliding motion to obtain friction characteristics. This multi-interaction approach enables comprehensive tactile information extraction for embodied interaction.
Attribute Annotation To facilitate tactile reasoning, we annotated our dataset across four funda- mental static and dynamic tactile attributes as shown in Fig. [b). Each attribute was categorized into three distinct levels, with harness classified as highly deformable (28%), moderately deformable (33%), and extremely hard (39%); protrusion categorized as absent (41%). moderate (26%). or strong (33%); elasticity measured as none (42%), moderate (30%), or strong (28%); and friction assessed as slight (32%), moderate (25%), or strong (43%). This structured annotation framework enables comprehensive tactile attribute analysis for downstream reasoning tasks.
Template Generation Template generation facilitates the creation of question- answer pairs for model training. We developed multiple problem templates encompassing various reasoning tasks: tac- tile feature assessment, surface feature distinction, texture optimal selection and so on. To instantiate these templates, we systematically integrated diverse visuo-tactile video combinations, comparison operators (e.g., "more", "less", "most", "least"), and attribute selectors to generate a comprehensive dataset of 10,000 question-answer pairs. As illustrated in Fig. [c), our generation process follows hicrarchical framework: selection, filtration, and structured question formulation with corresponding ground-truth annotations. For more comprchensive details about attribute annotation and template generation, please refer to the Supplementary Material ??.
3.2 VTV-LLM
Overview VTV-LLM aims to serve as a multi-modal framework capable of integrating visual- tactile vidco data with large language models to facilitate tactile reasoning for embodied interaction.

---

"What the tactile features the object presented in the video <video start>" "<video>", "svideo end>?"
Prompt:
Tokenizer
T-Prrojctor
VTV-LLM: It presents
Large Language Model (Qwen 2.5)
moderately deformable strong protrusion surface with moderate clasticity and moderate friction.
Visuo-Tacee Video
VTV Encoder
V-Projector
In
(a) VTV-LLM Framework
(b) Multi-Stage Training
Large Language Model
Large Language Model
VTV Decoder
Attribute Classifier
V-Projector
T I-Projector
V-Projector
T-Projector
VTV Encoder
VTV Encoder
VTV Encoder
Text Prompt
Text Prompt
Masked Video Frames
Video Frames
Video Frames
Stage I: VTV Enhancceennt
Stage 2: VTV-Text Alignment
Stuge 3: Text Prompt Finetuning
Figure 2: (a) VTV-LLM framework: A multi-modal system integrating visual-tactile video data with large language models to facilitate tactile reasoning for embodied interaction; (b) Multi-Stage Training: It consists of VTV enhancement, alignment between visuo-tactile video and text, and prompt-based finetuning to generate accurate tactile descriptions.
As illustrated in Fig. aa),, VTV-LLM formulates tactile perception as cross-modal approach to question answering and descriptive generation. By leveraging the rich sensory information inherent visuo-tactile video data, VTV-LLM enhances understanding in scenarios traditionally challenging for standard vision-only models, particularly in applications requiring tactile attribute inference.
At the core of VTV-LLM lies (Qwen 2.5 4 that synthesizes complex multi-modal information from risuo-tactile videos, utilizing world knowledge to generate coherent, human-readable descrip- tions of tactile attributes. In general, a visuo-tactioy video can be mathematically represented sequence of frames V = {T.) where each frame 7, captures both visual and tactile information at timestamp l. Initially, high-dimensional features FVTV are extracted from using a VTV encoder based on ViT-base architecture [47 adapted from VideoMAE [27 28):
EVTV (V) VIT (Patch(4)) + TE()))
where Patch(-) denotes the patch embedding operation and TE(t) represents temporal embeddings. These features are then processed through a visual projector fv-proj consisting of two linear layers with GELU activation function 48 in between produce the visual embedding Ev:
By fv proj (Fvrv) (Fvrv)
W2 GELU(W, Fvrv + b1) by.
where W., W2 are learnable weight matrices and b,, b2 are bias terms. Concurrently, the textual prompt is tokenized and processed through LLM's text projector produce text embedding Er. For effective multi-modal reasoning, introduce special tokens <video start>, <video>, and

---

Keyfrume Selection & Inirialization
Backward Warping & Masking
fo
Ad's
Re
Rr
Visuo-Tactik Video
Reconstructed Video Frames
VTV Encader
VTV Decoder
Optical Flow
Gemerator
Hardness:
Proermion:
1-1
Ch-1mn
M,
M,
Classilier
Elunkiity
Opticul Flow
Musked Video Frumes
Priction:
Figure 3: Training pipeline of VTV enhancement.
<video_ent sede__ndddde to denote the beginning, content and end of the visuo-tactile video in the input sequence. These tokens serve as anchors for the model to properly align visual information with textual understanding during the inference process.
Given these aligned representations, the large language model fLLMM performs reasoning generate a response A describing tactile attributes:
A = (EV,
Qven(Conca(((y;; ET)))
Given the complexity of integrating visuo-tactile information with language representations, implement a staged training approach to develop our framework. As shown in Fig. [b). VTV-LLM adopts three-stage training paradigm encompassing VTV enhancement, VTV-text alignment, and text prompt finetuning. This structured progression enables the model to first learn robust tactile- visual representations, then align them with textual descriptions, and finally optimize response generation, enhancing VTV-LLM's capability for cross-modal understanding and tactile reasoning. In the following, describe each of these stages in detail.
VTV Enhancement Existing multi-modal LLMs predominantly process natural images via modified Vision Transformer (ViT) encoders 477 However, our research addresses visuo-tactile inputs, which exhibit fundamentally different characteristies from natural images, thus necessitating specialized fine-tuni tine-tunigg to extract meaningful representations.
Furthermore, the temporal nature of our video data introduces challenges not present in static images. Unlike images, videos possess an inherent time dimension characterized by temporal redundancy and inter-frame correlations, requiring robust video representation methodologies. While VideoMAE 27 28 offers a powerful masked video autoencoder with an asymmetric encoder-decoder architecture utilizing tube masking, this approach assumes minimal motion across large frame regions. This assumption proves problematic for visuo-tactile videos, which typically exhibit significant motion patterns. Direct application of tube masking to such inputs risks substantial information leakage. wherein the model can trivially reconstruct masked segments using visible tokens from temporally adjacent frames, which is a critical concern in masked video pre-training. To address these limitations, we propose a novel training pipeline specifically designed for visuo-tactile video representation, illustrated in Fig.
Given the visuo-tactile video sequence V = {I E RHxWxCyT (T=- where each frame I, encodes both visual and tactile information at timestamp / with spatial dimensions H W and C channels, we propose selecting the middle frame as the keyframe. This selection is motivated by empirical observations that the middle frame typically exhibits the maximum contact surface area, facilitating more robust optical flow warping in subsequent processing stages. For keyframe mask initialization, conventional binarization approaches [49 significantly degrade the spatial continuity of object surfaces, compromising the fidelity of the reconstructed tactile information. Therefore, we introduce Gaussian mixture model (50) to obtain the keyframe mask. For the keyframe 4g. we formulate probabilistic mask using localized Gaussian functions. We select set of N = [a- HW/ 627 sampling
points (p. 2 distributed across the frame, where (0, controls density and / is the sampling
(m-pi, ) ++(y (y-yyn (yp--,,)) p4,) with
grid size. Each point P: generates a Gaussian kernel Gi y) = exp
2x
scale parameter X. The final keyframe mask defined as M min (I,E/ G.). creating

---

a continuous-valle valued mask that preserves spatial structure while enabling controlled sparsity for subsequent processing.
Additionally, we employ dense motion estimation across the visuo-tactile video V using the RAFT architecture [51]. We compute bidirectional optical flow fields between consecutive frames to capture the continuous deformation patterns throughout the interaction process. For each adjacent frame pair, we define the forward flow field O
(+411) Each flow field O,-4+1 R # XWxx encodes pixel-wise displacement vectors (U.z,) (M.y) V2.y) for every spatial location (x, y), mapping positions from frame I to their corresponding locations in frame I411. The complete of optical flows & for the sequence is formulated as:
Ulo U (0-1-1) 1=k+1
t=0
This bidirectional flow representation tracks visuo-tactile features throughout the interaction, support- ing warping operations and masked frame generation. We apply spatial normalization before flow computation to ensure scale invariance across different sequences.
After that, we utilize the backward warping [52 53 to generate the temporal consistent masking map based on the keyframe and mask the corresponding video frames. The masked visuo-tace Aisuo-tactil frames Vn = {M, ) are fed into the VTV encoder architecture for reconstruction using the mean squared error loss [27 28 We also incorporate an attribute classifier to predict tactile attributes (hardness, protrusion, elasticity, and friction) using the cross-entropy loss (54). Our total loss function combines both the reconstruction loss and the attribute classification loss, enabling simultaneous optimization of visuo-tactile reconstruction quality and tactile attribute classification accuracy.
VTV-Text Alignment In the VTV-text alignment stage, we focus on establishing cross-modal alignment between video and language representations. With the pretrained VTV Encoder from stage 1. we introduce both V-Projector and T- T-Projector modules while keeping the Large Language Model frozen. This stage leverages our initial constructed VTV VTV150K dataset to bridge the representational gap between visual and textual modalities. The V-Projector maps video embeddings from the VTV Encoder into the language model's embedding space, while the T-Projector processes corresponding text prompt representations. By training these projection modules exclusively while freezing other components, we establish foundational cross-modal understanding, enabling the model to associate visual content with appropriate textual descriptions. This alignment is critical for downstream video understanding and description tasks as it creates a shared semantic space between the video frames and natural language.
Text Prompt Finetuning In the text prompt finetuning stage, we enhance the model's capacity to respond accurately to textual prompts about video content by implementing supervised fine-tuning across multiple components. The V-Projector, and T-Projector are jointly fine-tuned along with the LLM. Unlike previous stages where the LLM remained frozen. this stage employs parameter-efficient techniques [37 36] to fine-tune the language model using 10,000 newly generated question-answw answer pairs. These pairs are created using the same template generation approach as our VTVI50K dataset, featuring diverse video understanding tasks. By generating new data rather than reusing subsets, significantly increase training diversity and model robustness. This end-to-end optimization enables the model to generate about video content. The supervised nature of this phase significantly improves the model's ability to comprehend complex video scenes and produce natural language descriptions that align with human expectations. This final stage integrates the previously aligned representations into a cohesive multi-modal understanding system, culminating in enhanced video-language capabilities.
coherent, accurate, and contextually relevant responses to text prompts
Experiments
4.1 Setup
Our experiments utilize the proposed VTVI50K dataset for both training and evaluation protocols. The training process follows our three-stage paradigm: Stage employs multi-sensor visuo-tactile videos with their corresponding attribute annotations for representation learning. For Stage 2 and 3,
7

---

Optimality
Among
a) cammpd <humg4> scamper stumiiy the abject descriting
picietts catrenely and strong peotrusion surface cislicity stroe, fiction. b) procetis
0.mpp (cireulit hearr))
highly defommahic and moderate pretnision surface with modense chunnity and frction 4) prosens extrmmly hand and no protnsson su fioz with no clasticity and slight fricton The wifles d'jez is
Objectt Sensation Correlation
6) -amppP, b)) <humpp a) scauper ideatid softes object after deserihing each onc. You have tacille vodcos of theee deflereut objects a) -aamppp---- b)-lhmppp-. .e) -t.ppp....eee. Assign each video ta. c) ore of these objects listed alplubencally 1) inn rule, 2) liws, Sjaciecui beerd cicut board, by is kiwi and
Question: Aming
Broe
bampá (kiwi)
VTV-LLM:
rules.
Tactile Scenario Analysis
strrlaae samppe <hampp: sumppr. Which object would be better für serping ice odT windslidd?
Question: Deseribe theze
besea
Campl (iron naker)
VTV-LI a) ... b) 4)... The mos sntbble object
Figure 4: Several task examples from the proposed VTVIS0K along with predictions from VTV- LLM.
Table I: Performance comparison of VTV-LLM-7B against seven state-of-thee of-tth-artt methods on the VTVI dataset. The evaluation covers different tasks, with results reported in percentages (%) and the boldface indicates the best performance.
Models
Harelness
Protrusion
Elasticity
Friction
Combined
SHD
SOI
OSC
TSA
Average
GPT-4a
555
34.7
32,6
32.6
18.7
2.1
40.9
38.4
16.6
36.0
28.0
Gemini-2.5PPro-Exp
50
36.2
34.7
39.1
21.0
4.3
42.6
29.4
18.5
40.0
29.5
LLaVA-OnsVisioon-88
57
20.2
0.7
28.2
11.7
30.0
53
30.4
29.7
28.9
18.1
2.1
33.6
29.4
17.2
36.0
25.0
Intrm\VL2.5VLL88
59
18.1
13.7
0.0
24.5
17.9
24.0
17.1
Lad33.
[1]
15.2
21.7
14.4
10.8
0.0
11.4
12.8
7.4
20.0
12.6
Qwen2.5-VL-7B
25.3
15.9
22.9
28.2
30.0
20.6
VTV-LLM-7B (Ours)
73,9
75.0
67.3
56,5
35,6
71.3
57.6
43,2
64.0
60.4
we utilize two independently generated sets of 10,000 question-answer pairs to prevent data leakage between stages. To evaluate model performance, we create separate test set comprising 600 question- answer pairs for novel objects not present in the training data, ensuring comprehensive coverage across various tactile reasoning tasks. Our LLM backbone is based on Qwen 2.5 4 5 experimenting with three model variants (3B, 7B, and 14B parameters). All experiments are conducted on NVIDIA RTX 6000 Ada GPUs. Additional implementation details and hyperparameter configurations provided in the Supplementary Material ??.
4.2 Results
To verify the effectiveness of our VTV-LLM, we compare against two strong proprietary models, such GPT-4o (55) and Gemini-2.8. 1..5-Pro-Eiit -xpp (56), as well as five open-source video-based VLMs, including LLaVA-Ome Vineoiion-77 7B [57]. LLaVA- Video-Qwen2-7B [58], Intern IntermV2..5-V VL2.5-VL-8B [59], VideoLLaMA3-7B 41 and Qwen2.5-Ve LL--TB [60] Since most of the video-based VLM models have parameters around 7B, we only use the VTV-LLM-7B model for fair comparison. To guarantee the robustness of the experimental results, report the average results of the triplicate test with random
seeds. Our first experiment focuses tactile feature assessment, which evaluates the model's ability to perceive and describe physical sensory attributes objects visuo-tactile videos. As illustrated in Fig. d)), when presented with visuo-tactile video and a question prompt, VTV-LLM generates descriptions the four key tactile attributes. The results presented Tab. demonstrate that our method consistently outperforms all bascline models across both individual attribute and combined attribute settings. The performance gap is particularly notable in the combined attribute setting, which attribute to three-stage training paradigm that effectively bridges the domain gap between tactile perception and natural language understanding.
In addition, we conduct high-level tactile reasoning experiments, including surface feature distinction (SFD), surface optimality identification (SOI), object sensation correlation (OSC), and tactile scenario analysis (TSA). SFD involves comparing tactile qualities between objects to determine relative differences, SOI entails analyzing multiple surfaces to determine which exhibits the highest degree of particular quality, OSC aims at relating tactile perceptual information to the identity of

---

VTV-LLM-3B (2.52 s/question)
VTV-LLM-7B (3.28 s/question)
VIV-LLM-14B (5.42 s'question)
18
75.0
60
Accuracy (%)
16.1
46.1
30
15.0
M2
20.1
Elasticity
Combined
Average
Tasks
Figure 5: Performance comparison of VTV-LLM the different parameters.
Table 2: Ablation study VTV encoder settings
Table 3: Ablation study three-stage training
using the VTV-LLM--B model.
paradigm settings using the VTV-LLM-7B model.
Settings
SFD
SOI
OSC
TSA
Average
Settings
SFD
SOI
OSC
TSA
Average
VidcoMAE (wlo train)
37.5
29.7
8.5
16.0
w/o stage
58.1
50.0
35.2
60.0
50.8
VideoMAB (w/ train)
52.4
46.1
28.3
38.0
41.2
w/o stage
50.8
42.3
29.0
52.0
43.5
Ours (w/o
62.2
48.7
40.1
55.0
51.5
Same dataset
61.4
53.8
33.9
58.0
51.7
Ours
71.3
57.6
43.2
64.0
39.0
Ours
71.3
57.6
43.2
64.0
59.0
particular real-world object, and TSA addresses applying haptic knowledge real-world situations that require physical reasoning. It is worth noting that the TSA task is not included in the training set. The qualitative results presented in Fig. (d) and Fig. demonstrate that VTV-LLM can generate reasonable outputs. The quantitative experimental results in Tab. further confirm that VTV-LLM achieves superior performance across these complex reasoning tasks, highlighting its potential for embodied interaction.
4.3
Ablation Studies
LLM Backbone To examine the effect of model scale visuo-tactile understanding, compare different parameter sizes of our LLM backbone. Fig, 5lsows performance results for VTV-LLM using three Qwen 2.5 variants (3B, 7B, and 14B parameters). We observe consistent performance improvements with increasing model size. This improvement is most significant for complex reasoning tasks like TSA, indicating larger models better integrate cross-modal information. However, larger models also require substantially more computation time during inference.
VTV Encoder We conduct an ablation study our VTV encoder design shown in Tab. 2 Bascline VideoMAE (27 28) without training achieves only 22.9% average performance, while training with our VTV150K dataset improves it to 41.2 41.2%. Our method without the attribute classifier reaches 51.5%, showing the effectiveness of our optical flow-guided masking strategy. The full method with the attribute classifier further improves to 59.0%, confirming that joint reconstruction and attribute classification significantly enhances tactile understanding.
Three-Stage Training Paradigm Tab. validates our three-stage training paradigm through ablation studies. Removing stage 2 (VTV-text alignment) drops average performance to 50.8%, while omitting stage 3 (text prompt finetuning) causes a steeper decline to 43.5%. Using identical datasets across stages also underperforms at 51.7%, confirming that independent datasets for each stage significantly improve model robustness.

---

Conclusion
In this work, we presented VTV-LLM, the first multi-modal large language model for universal visuo- tactile video understanding, We contributed VTVI50K, a comprehensive dataset of visuo-tactile videos across multiple sensors, and developed novel three-stage training paradigm that effectively bridges the gap between tactile perception and natural language. Experimental results demonstrate that VTV-LLM consistently outperforms tfae-of-thee-a methods across various tactile reasoning tasks, establishing foundation for more intuitive
interaction in embodied domains.
References
[1] Qiang Li, Oliver Kroemer, Zhe Su, Filipe Fernandes Veiga, Mohsen Kaboli, and Helge Joachim Ritter. A review of tactile information: Perception and action through touch. IEEE Transactions on Robotics, 36(6):1619-160) 1634, 2020.
[2] Müge Cavdan, Katja Doerschner, and Knut Drewing. Task and material properties interac- tively affect softness explorations along different dimensions. IEEE Transactions Haptics, 14(3):603-60 614, 2021.
[3] Mudassir Ibrahim Awan, Waseem Hassan, and Scokhee Jeon. Predicting perceptual haptic attributes of textured surface from tactile data based on deep cnn-lstm network. In Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology, pages 1-9, 2023.
[4] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.1660, 2023.
[5] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
[6] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
[7] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14455-14465, 2024.
[8] Yifan Xie, Jingge Wang, Tao Feng, Fei Ma, and Yang Li. Ccis-diff: A generative model with sta- ble diffusion prior for controlled colonoscopy image synthesis. arXiv preprint arXiv:2411.12198, 2024.
[9] Shoujie Li, Zihan Wang, Changsheng Wu, Xiang Li, Shan Luo, Bin Fang, Fuchun Sun, Xiao- Ping Zhang, and Wenbo Ding. When vision meets touch: A contemporary review for visuotactile sensors from the signal processing perspective. IEEE Journal of Selected Topics in Signal Processing, 2024.
[10] Wenzhen Yuan, Siyuan Dong, and Edward H Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(1))2762,, 2017.
[11] Mike Lambeta, Po-Wes Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, et al. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3):3838-3845. 2020.
[12) Lunwei Zhang, Yue Wang, and Yao Jiang. Tac3d: A novel vision-based tactile sensor for measuring forces distribution and estimating friction coefficient distribution. arXiv preprint arXiv:2202.06221, 2022.
[13] Ruoxuan Feng. Jiangyu Hu, Wenke Xia, Ao Shen, Yuhao Sun, Bin Fang, Di Hu, et al. Anytouch: Learning unified static-dynamic representation across multiple visuo-tactile sensors. In The Thirteenth International Conference Learning Representations, 2025.
10

---

[14] Jialiang Zhao, Yuxiang Ma, Lirui Wang. and Edward Adelson. Transferable tactile transformers for representation learning across diverse sensors and tasks, In 8th Annual Conference on Robot Learning, 2024.
[15] Samson Yu. Kelvin Lin, Anxing Xiao, Jiafei Duan, and Harold Soh. Octopi: Object property reasoning with large tactile- lantuae models. arXiv preprint arXiv:2405.02794 2024.
[16] Letian Fu, Gaurav Datta, Huang Huang, William Chung-Ho Panitch, Jaimyn Drake, Joseph Ortiz, Mustafa Mukadam, Mike Lambeta, Roberto Calandra, and Ken Goldberg. touch, vision, and language dataset for multimodal alignment. In International Conference on Machine Learning, pages 14080-144 14101. PMLR, 2024.
[17] Fengyu Yang, Chao Feng, Ziyang Chen, Hyoungseob Park, Daniel Wang, Yiming Dou, Ziyao Zeng, Xien Chen, Rit Gangopadhyay, Andrew Owens, et al. Binding touch to everything: Learn- ing unified multimodal tactile representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 00340,-6333,,, 2024.
[18] Carolina Higuera, Akash Sharma, Chaithanya Krishna Bodduluri, Taosha Fan, Patrick Lancaster, Mrinal Kalakrishnan, Michael Kaess, Byron Boots, Mike Lambeta, Tingfan Wu, et al. Sparsh: Self- supervise touch representations for Aision-based tactile sensing. In 8th Annual Conference on Robot Learning. 2024.
[19] Neha Sunil, Shaoxiong Wang, Yu She, Edward Adelson, and Alberto Rodriguez Garcia. Visuo- tactile affordances for cloth manipulation with local control. Conference on Robot Learning, pages 1596-1606. PMLR, 2023.
[20] Ying Zheng, Lei Yao, Yuejiao Su, Yi Zhang, Yi Wang, Sicheng Zhao, Yiyi Zhang, and Lap-Pui Chau. A survey of embodied learning for object-centric robotic manipulation. arXiv preprint arXiv:2408. 1 1537,, 2024.
[21] Fengyu Yang, Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, and Andrew Owens.
Touch and go: learning from human-collecte vision and touch. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 8081-8103, 2022.
[22] Ruohan Gao, Yen- Yu Chang, Shivani Mall, Li Fei-Fei, and Jiajun Wu. Objectfolder: dataset of objects with implicit visual, auditory, and tactile representations. In Conference on Robot Learning, 2021.
[23] Ruohan Gao. Zilin Si, Yen- Yu Chang, Samuel Clarke, Jeannette Bohg, Li Fei-Fei, Wenzhen Yuan, and Jiajun Wu. Objectfolder 2.0: A multisensory object dataset for sim2real transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10598- 10608, 2022.
[24] Hao Li, Yizhi Zhang, Junzhe Zhu, Shaoxiong Wang, Michelle A Lee, Huazhe Xu, Edward Adelson, Li Fei-Fei, Ruohan Gao. and Jiajun Wu. See, hear, and feel: Smart sensory fusion for robotic manipulation. In Conference on Robot Learning, pages 1368-1378. PMLR, 2023.
[25] Wenzhen Yuan, Yuchen Mo, Shaoxiong Wang, and Edward H Adelson. Active clothing material perception using tactile sensing and deep learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 4842-4849. IEEE, 2018.
[26] Justin Kerr, Huang Huang, Albert Wilcox, Ryan Hoque, Jeffrey Ichnowski, Roberto Calandra, and Ken Goldberg. Self-supervised visuo-tactile pretraining to locate and follow garment features. arXiv preprint arXiv:2209. 13442, 2022.
[27] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 15:10088-1009, 2022.
[28] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 14549-14560, 2023.
11

---

[29]
Mohammadreza Salehi, Michael Dorkenwald, Fida Mohammad Thoker, Efstratios Gavves, guidedd masked video modeling. In European Conference on Computer Vision, pages 293-312. Springer, 2024.
Cees GM Snoek, and Yuki M Asano. Sigma: Sinkhorn Sinhhon-guided
[30] Bingkun Huang, Zhiyu Zhao. Guozhen Zhang, Yu Qiao, and Limin Wang. Mgmae: Motion guided masking for video masked autoencoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13493-13504, 2023.
[31] Yunze Liu, Peiran Wu, Cheng Liang, Jumxiao Shen, Limin Wang, and Li Yi. Videomap: Toward scalable mamba-based video autoregressive pretraining. arXiv preprint arXiv:2503.12332, 2025.
[32] Gensheng Pei, Tao Chen, Xiruo Jiang, Huafeng Liu, Zeren Sun, and Yazhou Yao. Videomac: Video masked autoencoders meet convnets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
2024.
[33] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving tasks with chatgpt and its friends hugging face. Advances Neural Information Processing Systems, 36:381 06::3115422 154-38180 2023.
[34] Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888-1188 11898, 2023.
[35] Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model use tools via self self-instruution. Advances in Neural Information Processing Systems, 36:71995-72007, 2023.
[36] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[37] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhanc- ing lasiun-language understanding with advanced large language models. In ICLR, 2024.
[38] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances neural information processing systems, 36:34892 34916, 2023.
[39] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306. 2024.
[40] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video- Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 12585-12602, 2024.
[41] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprini arXiv:2501. 13106, 2025.
[42] Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. arXiv preprint 240.022000006666, 2024.
[43] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-lb: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.078e.
2024.
[44] Joshua Jones, Oier Mees, Carmelo Sferrazza, Kyle Stachowicz, Pieter Abbeel, and Sergey Levine. Beyond sight: Finetuning generalist robot policies with heterogeneous sensors via language grounding. arXiv preprint 2rXiv:2501.04693 2025.
12

---

[45] Samanta Rodriguez, Yiming Dou, Miquel Oller, Andrew Owens, and Nima Fazeli. Touch2touch: Cross-modal tactile generation for object manipulation. arXiv preprint arXiv:2409.0826%. 2024.
[46] Sky McKinley and Megan Levine. Cubic spline interpolation. College of the Redwoods, 45()):004--100, 1998.
[47] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, G Heigold, S Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2020.
[48] Dan Hendrycks and Kevin Gimpel. Gaussian eтror linear units (gelus). arXiv preprint 16xiv:160.008411 2016.
[49] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[50] Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, 741(659- 663):3, 2009.
[51] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt II 16, pages 402-419. Springer, 2020.
[52] Joi Shimizu, Heming Sun, and Jiro Katto. Forward and backward warping for optical flow-based frame interpolation. In 2022 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), pages 082-086. IEEE, 2022.
[53] Sung In Cho and Suk-Ju Kang. Extrapolation- video retargeting with backward warping using image-to-warping vector generation network. IEEE Signal Processing Letters, 27:446- 450, 2020.
[54] Anqi Mao, Mehryar Mohri, and Yutao Zhong. Cross-entropy loss functions: Theoretical analysis and applications. In International conference Machine learning, pages 23803-23828. PMLR, 2023.
[55] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
(56) Google DeepMind. pennini----ppp-prrriee--------- http://rweeeeeettttttttttee////e /ttttps:/aai.googeee... den.gleeddd/gemnii.eee api/ docs/modeelsHgemiin p. 2025.
[57] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint wwXiv:2408.03326 2024.
[58] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713.. 2024.
[59] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint 11205.2412.05271, 2024.
[60] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We claim the contribution and scope in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the Supplemental Material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the details to reproduce the experimental results in both methods section and Supplemental Material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: Although no link is available for our work now, we are strongly intended to share work in public when the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpollliigggddde/ooooooo SoweeeeeeeooeSbubiiieoo ssionpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We present the experimental details in both experiments section and Supple- mental Material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
No statistical significance involved in our work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information on the computer resources both experiments section and Supplemental Material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp///eaurii..cccooobb.e/ttte
Answer: [Yes]
Justification: Our work conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: discuss limitations in the Supplemental Material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credit all used assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper introduces the new dataset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
This paper describes the usage of LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neurip::: htt.p//s././//ooeeeeeee//////// for what should or should not be described.
20


Reviews:
[{"review_id": "yal4uSn3xW", "reviewer": "Reviewer_P8wL", "summary": "This paper introduces the Visuo-Tactile Video (VTV) dataset, which comprises 150K frames of tactile videos annotated with attributes. Additionally, the author proposes the VTV-LLM framework, which facilitates reasoning in embodied interactions. To train the VTV-LLM framework, the author utilizes the VTV dataset and introduces a three-stage training process for the model.", "questions": "Please refer to weakness section.", "limitations": "Yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strength**\n1. Introductin of VTV dataset that include attribute annotation and tactile Q&A \n2. Demonstrates the effectiveness of the proposed VTV-LLM and three-step training strategy through experiments on tactile understanding tasks.\n\n**Weakness**\n1. I fail to comprehend the advantage of video over tactile. What is the advantage of tactile video compared to tactile images? I believe tactile video doesn’t offer significant differences over time, potentially resulting in redundant frames.\n\n2. Additionally, since the network processes video as input, the number of videos becomes more crucial than the number of frames in the dataset. Furthermore, it’s unclear whether annotations are applied to videos or frames. For instance, what does 150,000 video frames represent? I think including frames per second (FPS) would help understand the dataset size or the number of videos. 150,000 frames at 60 FPS could equate to just 40 minutes of video, which is not a substantial amount.\n\n3. For VTV-LLM, it’s unclear why tactile video is necessary for tactile understanding. A single image would suffice for the task proposed by the task author. Video input only increases the model’s size. To demonstrate the necessity of video, a task that shows temporal tactile information should be introduced.", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for the constructive comments. We provide our feedbacks as follows.\n\n**Q1: I fail to comprehend the advantage of video over tactile. What is the advantage of tactile video compared to tactile images? I believe tactile video doesn’t offer significant differences over time, potentially resulting in redundant frames.**\n\n**A1**: Thanks for this comment. Regarding your question about the advantages of tactile video over tactile images, I believe there is a conceptual misunderstanding that needs clarification. Tactile videos indeed provide crucial temporal information in physical interactions that static tactile images cannot capture, which is essential for understanding dynamic tactile attributes of objects.\n\nFirst, our research design includes three sequential interaction modes: normal pressing, rotational movement, and sliding motion. This design is not arbitrary but based on the physical principles of tactile perception. As clearly described in lines 129-131 of our paper: *\"Our data collection process consisted of three sequential interactions: (1) normal pressing against the object surface to capture pressure distribution, (2) rotational movement to acquire shear information, and (3) sliding motion to obtain friction characteristics.\"* Each interaction mode reveals different physical properties of objects: pressing reveals hardness and elasticity, rotation captures shear characteristics, and sliding obtains friction information. **These dynamic interaction processes cannot be represented in a single static image.**\n\nMore importantly, **tactile videos can capture the deformation patterns and recovery behaviors of objects during interactions.** For example, when we press an elastic object, its surface deforms and gradually returns to its original state after release. This temporal change pattern directly reflects the object's elastic properties. As stated in lines 41-43 of our paper: *\"the temporal dimension of tactile interactions, which captures how surfaces respond to pressing, sliding, and rotational movements, remains underexplored in current approaches, despite containing crucial information about dynamic material attributes.\"* **This temporal dimension contains crucial dynamic material attribute information that static methods cannot acquire.**\n\nFrom a technical implementation perspective, our proposed optical flow-guided masking method is specifically optimized for the motion characteristics of tactile videos. Lines 186-191 of our paper explain why directly applying existing tube masking methods would be problematic: *\"While VideoMAE offers a powerful masked video autoencoder... this approach assumes minimal motion across large frame regions. This assumption proves problematic for visuo-tactile videos, which typically exhibit significant motion patterns. Direct application of tube masking to such inputs risks substantial information leakage.\"* **Our method, through dense motion estimation and backward warping techniques, can better handle significant motion patterns in tactile videos, which is a technical challenge that doesn't need to be considered when processing static images [1].**\n\nExperimental results also confirm the importance of temporal information. Our ablation study (Table 2) shows that using our temporally-aware method achieves a 36.1% improvement in average performance over the VideoMAE baseline without training (from 22.9% to 59.0%). This significant performance improvement directly demonstrates the value of temporal information in tactile videos. **If tactile videos were truly just collections of redundant frames, we would not observe such significant performance improvements.**\n\nFinally, from an application perspective, real robotic interactions require understanding how objects behave during dynamic contact processes. Static tactile images can only provide contact information at a specific moment and cannot guide robots on how to adjust grasping force or manipulation strategies. **The dynamic feedback provided by tactile videos enables robots to adjust their operational behavior in real-time, which is indispensable for complex physical interaction tasks [2].** Our VTV-LLM is designed precisely to address this need for dynamic tactile understanding and natural language interaction.\n***\n**Q2: Additionally, since the network processes video as input, the number of videos becomes more crucial than the number of frames in the dataset. Furthermore, it’s unclear whether annotations are applied to videos or frames. For instance, what does 150,000 video frames represent? I think including frames per second (FPS) would help understand the dataset size or the number of videos. 150,000 frames at 60 FPS could equate to just 40 minutes of video, which is not a substantial amount.**\n\n**A2**: Thank you for this important clarification question regarding our dataset specifications. We appreciate the opportunity to provide more detailed information about VTV150K.\n\n**Our annotations are applied directly to videos, not individual frames.** Each video represents a complete tactile interaction sequence of 5 seconds, capturing the full temporal dynamics of object contact. The 150,000 video frames represent the total frames across all videos in our dataset, with each video recorded at **20 FPS** and a resolution of **320×320 pixels**. This means our dataset contains **1,500 complete tactile interaction videos** (150,000 frames / 100 frames per video = 1,500 videos), providing **125 minutes** of total video content focused on tactile understanding.\n\n**When comparing VTV150K to existing tactile-text annotated datasets [3] [4], our dataset represents the largest collection of visuo-tactile video frames with natural language annotations currently available.** The Touch100k dataset [3] contains 100,147 touch-language-vision entries but focuses primarily on static tactile properties with limited temporal dynamics. The PHYSICLEAR dataset [4] includes 408 tactile videos from 74 objects using only GelSight sensors, which is significantly smaller in scale than our 1,500 videos from 100 objects.\n\nMoreover, a crucial distinction of VTV150K is its multi-sensor coverage. **While existing datasets [3][4] typically focus on single sensor types, our dataset incorporates three widely-used commercial tactile sensors: GelSight Mini, DIGIT, and Tac3D.** This multi-sensor approach addresses a critical gap in the field, as real-world tactile applications often require cross-sensor generalization. The diversity across sensor types, combined with our comprehensive object selection (100 diverse everyday objects), provides a more robust foundation for developing universal tactile understanding models.\n\nFinally, the video-based nature of our dataset is particularly important for capturing dynamic tactile attributes like elasticity and friction, which cannot be adequately represented through static frames alone. **Each 5-second video at 20 FPS provides sufficient temporal resolution to capture the complete interaction cycle including initial contact, deformation, and recovery patterns essential for understanding material properties.** This temporal richness distinguishes VTV150K from frame-based approaches and enables learning of sophisticated tactile dynamics that are crucial for embodied interaction tasks.\n***\n**Q3: For VTV-LLM, it’s unclear why tactile video is necessary for tactile understanding. A single image would suffice for the task proposed by the task author. Video input only increases the model’s size. To demonstrate the necessity of video, a task that shows temporal tactile information should be introduced.**\n\n**A3**: Thank you for this comment. In fact, the question raised in **Q3** is similar to that in **Q1**, both focusing on the necessity of tactile video understanding. For specific explanations, please refer to **A1**.\n\nMoreover, to further demonstrate the value of temporal modeling in tactile perception, we adopted Octopi [4]'s data reading method (selecting 5 frames) and trained using the identical dataset and pipeline. **The results shown in the table below demonstrate the importance of tactile video understanding, especially for dynamic attributes.**\n\n|                    | Hardness | Protrusion | Elasticity | Friction | Combined | SFD  | SOI  | OSC  | TSA  | Average |\n|--------------------|----------|------------|------------|----------|----------|------|------|------|------|---------|\n| Selecting 5 frames | 62.6     | 60.7       | 41.3       | 31.7     | 10.8     | 58.1 | 47.5 | 27.1 | 51.0 | 43.4    |\n| Ours               | **73.9**     | **75.0**       | **67.3**       | **56.5**     | **35.6**     | **71.3** | **57.6** | **43.2** | **64.0** | **60.4**    |\n\n***\n[1] Zhao, Jialiang, et al. \"Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks.\" Conference on Robot Learning. PMLR, 2025.\n\n[2] Zhao, Zihang, et al. \"Embedding high-resolution touch across robotic hands enables adaptive human-like grasping.\" Nature Machine Intelligence (2025): 1-12.\n\n[3] Cheng, Ning, et al. \"Touch100k: A large-scale touch-language-vision dataset for touch-centric multimodal representation.\" Information Fusion (2025): 103305.\n\n[4] Yu, Samson, et al. \"Octopi: Object Property Reasoning with Large Tactile-Language Models.\" Robotics: Science and Systems. 2024."}, {"review_id": "qskEVon2iJ", "reviewer": "Reviewer_5L3e", "summary": "The paper proposes a universal Visuo-Tactile Video Large Language Model(VTV-LLM) to address the gap in integrating tactile perception into LLMs. The authors collect a large-scale tactile video dataset, which contains 150K video frames with 4 fundamental tactile attributes. A three-stage training paradigm is proposed to train the VTV-LLM, which includes a VTV enhancement for robust visuo-tactile representation, a VTV-text alignment for cross-modal correspondence, and a text prompt finetuning for natural language generation. After training, the VTV-LLM can execute various tactile reasoning tasks.", "questions": "The annotation focuses on four fundamental tactile attributes. Are there plans to expand the dataset to more complex tactile features, such as texture and temperature?\nIs the VTV-LLM possible for edge-device deployment?\nHow to mitigate the adverse effects of the LLMs' hallucination issue?", "limitations": "The dominating limitation is the real-world feasibility of the LLM-based tactile reasoning model in real-time scenarios.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "【Strengths】\n1. The first work to integrate visual, tactile, and language modalities for universal tactile understanding.\n2. A valuable and comprehensive dataset encompasses a diverse range of objects, sensors, and tactile attributes, which can be used for training tactile-related models.\n3. The three-stage paradigm guarantees effective cross-sensor and cross-modal integration, enhancing tactile reasoning accuracy. \n4. The trained model performs well on the tactile reasoning task and lays the groundwork for the embodied domain.\n\n【Weaknesses】\nLimitation of the dataset: a small number of tactile sensors (GelSight, DIGIT, Tac3D) and limited fundamental attributes.\nReal-world feasibility concerns: The computational overhead of the LLM restricted its application in real-time embodied interaction scenarios.", "was_revised": true, "final_justification": "Accept", "rebuttal": "We thank reviewer for the constructive comments. We provide our feedbacks as follows.\n\n**Q1: Limitation of the dataset: a small number of tactile sensors (GelSight, DIGIT, Tac3D) and limited fundamental attributes.**\n\n**A1**: Thanks for this comment. For the sensors, **our three selected sensors are the most common commercial sensors, representing the main technological approaches in current visuo-tactile sensing (GelSight and DIGIT for capturing high-resolution information, and Tac3D for measuring deformation force fields), while existing datasets [1][2] typically focus only on single sensor types.** As stated in lines 72-74, these sensors have garnered widespread attention in robotic applications, such as material classification, shape reconstruction, and dexterous manipulation tasks, demonstrating their importance in the visuo-tactile field.\n\nMoreover, our method successfully addresses the challenge of cross-sensor data discrepancies, and the robustness test in Supplementary Material A.4.4 demonstrates good generalization performance across different data distributions, which is itself an important technical contribution.\nTo further verify the effectiveness, **we also conducted a zero-shot experiment on the XENSE sensor (a variant of the GelSlim sensor [3]).** In the experimental setup, we use the XENSE sensor to perform the same data collection steps for 20 objects as in the original experiment and generated 100 templates for each task. Then, the trained VTV-LLM-7B model is used for zero-shot inference. Related results can be found in the following table, it proves that **although our method is trained on only three kinds of sensor data, it generalizes well for the other sensors.**\n|                                     | Hardness | Protrusion | Elasticity | Friction | Combined | SFD  | SOI  | OSC  | TSA  | Average |\n|-------------------------------------|----------|------------|------------|----------|----------|------|------|------|------|---------|\n| VTV-LLM-7B (for original 3 sensors) | 73.9     | **75.0**       | **67.3**       | **56.5**     | **35.6**     | 71.3 | **57.6** | **43.2** | 64.0 | **60.4**    |\n| VTV-LLM-7B (for XENSE sensor)       | **77.0**     | 71.0       | 55.0       | 52.0     | 26.0     | **78.0** | 55.0 | 39.0 | **69.0** | 58.0    |\n\nFor the limited fundamental attributes, our four selected attributes (hardness, protrusion, elasticity, friction) are based on fundamental theories of tactile perception [4]: **hardness and elasticity for reflecting the mechanical deformation properties of materials, protrusion for capturing surface geometric texture information, and friction for characterizing surface interaction dynamics.** As stated in lines 22-23: \"*This tactile feedback enables sophisticated physical reasoning and interaction in our environment,*\" our attribute selection covers the core dimensions of tactile perception.\nMoreover, our methodology and technical framework have good scalability and can easily integrate new sensor types and attribute dimensions.\n***\n**Q2: Real-world feasibility concerns: The computational overhead of the LLM restricted its application in real-time embodied interaction scenarios.**\n\n**A2**: As shown in Figure 5, we provide three different parameter-scale model variants (3B, 7B, 14B), allowing selection based on computational resource constraints of specific application scenarios. **The VTV-LLM-3B model significantly reduces computational overhead while maintaining reasonable performance, providing a feasible solution for resource-constrained robotic platforms.**\n\nMoreover, Supplementary Material A.4.5 demonstrates our deployment testing in real robotic systems, using a UR5 collaborative robot with Robotiq-85 gripper integrated with GelSight Mini sensor. The experiments prove the system's operability in real environments. This practical validation demonstrates that **despite computational overhead, the current system can effectively operate on real robotic platforms.**\n\nFinally, many important embodied interaction applications do not require millisecond-level real-time response, including material property assessment and quality control. As the first work in this field, our main contribution lies in demonstrating the feasibility of visuo-tactile-language multimodal understanding and establishing an evaluation benchmark for visuo-tactile video.\n\n***\n**Q3: The annotation focuses on four fundamental tactile attributes. Are there plans to expand the dataset to more complex tactile features, such as texture and temperature? Is the VTV-LLM possible for edge-device deployment? How to mitigate the adverse effects of the LLMs' hallucination issue?**\n\n**A3**: Our four selected fundamental attributes (hardness, protrusion, elasticity, friction) cover the core dimensions of tactile perception. As described in lines 133-139, these attributes are systematically categorized into three levels, providing a comprehensive foundation for tactile attribute analysis in downstream reasoning tasks. We plan to expand to more physical properties in the next stage of work, such as temperature, sharpness and so on.\n\nFigure 5 shows our three model scales (3B, 7B, 14B). The VTV-LLM-3B model significantly reduces parameter count while maintaining reasonable performance, making edge deployment possible.\n\nFor the hallucination issue, our three-stage training paradigm (Figure 2(b)) establishes reliable cross-modal correspondence by gradually aligning visuo-tactile representations with textual descriptions, reducing the possibility of hallucination generation. Moreover, unlike open-ended image description, our tasks are based on clearly defined physical properties (hardness, protrusion, elasticity, friction). **This structured output space naturally limits the scope of hallucination generation.**\n***\n[1] Cheng, Ning, et al. \"Touch100k: A large-scale touch-language-vision dataset for touch-centric multimodal representation.\" Information Fusion (2025): 103305.\n\n[2] Yu, Samson, et al. \"Octopi: Object Property Reasoning with Large Tactile-Language Models.\" Robotics: Science and Systems. 2024.\n\n[3] Taylor, Ian H., et al. \"Gelslim 3.0: High-resolution measurement of shape, force and slip in a compact tactile-sensing finger.\" 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022.\n\n[4] Stefani, Antonio Luigi, et al. \"Signal processing for haptic surface modeling: A review.\" Signal Processing: Image Communication (2025): 117338."}, {"review_id": "9SY22SbyaX", "reviewer": "Reviewer_Q6gv", "summary": "This paper proposes VTV-LLM, the first multi-modal large language model that supports universal visuo-tactile video understanding. The author constructed VTV150K dataset, covering 150k frames of videos from 100 common objects collected by three sensors, annotated with four tactile attributes: hardness, convexity, elasticity, and friction. To achieve efficient multi-modal fusion, the paper designed a three-stage training paradigm: visual tactile representation enhancement, video and text alignment, and text prompt fine-tuning.", "questions": "Is it suitable for real-world dexterous manipulation tasks that demand real-time performance, or is it limited to tactile reasoning applications?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strength:**\n\n1. This work represents the first systematic attempt to develop a dedicated masking strategy specifically for tactile video data.\n2. The dataset proposed in this work represents good contribution to the field.\n\n\n**Weeknesses:**\n\n1. As one of the similar works to Octopi [1]—the first tactile video understanding model—this paper should provide a detailed comparative analysis highlighting both its differences from and advantages over Octopi. Furthermore, compared to models like TVL [2] that only accept single-frame tactile image inputs, how much benefit does video-based tactile understanding actually provide? A rigorous quantitative comparison would better demonstrate the value of temporal modeling in tactile perception.\n2. This work proposes an optical flow-guided video reconstruction strategy for tactile videos, which specifically masks deformation regions. However, unlike natural videos, tactile videos exhibit highly localized and fine-grained deformations. Masking 90% of the area may leave insufficient effective information in the remaining regions, potentially causing the mask modeling to overfit. The authors should provide examples of masked images and reconstructed results, and conduct an ablation study on masking ratios to validate the approach. Directly transferring parameters from natural video models may not be optimal for tactile video reconstruction.\n3. A key potential of combining tactile sensing with large language models is the ability to achieve zero-shot tactile perception and understanding. To highlight the generalization capability and practical value of their approach, the authors should include experimental results that demonstrate the model’s performance in zero-shot scenarios.\n4. I commend the authors for their efforts in collecting the dataset and conducting detailed annotations. However, the dataset’s limited scale—covering only 100 objects with relatively simple interaction patterns—may restrict its applicability and generalization potential in real-world embodied interaction settings.\n5. The proposed dataset incorporates Tac3D sensor data with a 20×20 resolution force field alongside high-resolution tactile images.  However, the paper does not clearly explain how this data is combined with high-resolution tactile images. Figures 1 and 7 display two different input forms of Tac3D data (interpolated images and force fields), but it remains unclear which one is actually used—or whether these represent different task settings. What is the size after interpolation? Additionally, is the Tac3D data processed into image-form inputs rather than the original point-cloud-like format? If so, might this approach fail to fully leverage the fine-grained capabilities of Tac3D data in perceiving force?\n6. The claim that the method \"maintains strong performance across these heterogeneous input modalities\" is not sufficiently convincing when relying solely on Figure 7. More comprehensive experimental results should be provided to substantiate this conclusion.\n\n\n[1] Yu S, Lin K, Xiao A, et al. Octopi: Object property reasoning with large tactile-language models[J]. arXiv preprint arXiv:2405.02794, 2024.\n[2] Fu, Letian, et al. \"A Touch, Vision, and Language Dataset for Multimodal Alignment.\" *International Conference on Machine Learning*. PMLR, 2024.", "was_revised": true, "final_justification": "This paper proposes VTV-LLM, the first multi-modal large language model that supports universal visuo-tactile video understanding. The author constructed VTV150K dataset, covering 150k frames of videos from 100 common objects collected by three sensors, annotated with four tactile attributes: hardness, convexity, elasticity, and friction. To achieve efficient multi-modal fusion, the paper designed a three-stage training paradigm: visual tactile representation enhancement, video and text alignment, and text prompt fine-tuning. During the discussion, most of raised concerns have been addressed. The reviewer believes this work presents incremental improvements on both the model and the dataset, so finally recommending BA.", "rebuttal": "We thank reviewer for the constructive comments. We provide our feedbacks as follows.\n\n**Q1: ...this paper should provide a detailed comparative analysis highlighting both its differences from and advantages over Octopi...**\n\n**A1**: Compared to Octopi, our VTV-LLM offers significant improvements in three key areas. First, our VTV150K dataset is **four times larger**, containing 150,000 video frames across 100 objects versus Octopi's PHYSICLEAR dataset. Second, we use **three different tactile sensors** rather than just one GelSight sensor, enabling better generalization across sensor types. Third, we annotate **more tactile properties** for comprehensive physical reasoning tasks.\n\nTechnically, **we propose a three-stage training paradigm with optical flow-guided masking for tactile videos, while Octopi uses a basic approach without temporal processing.** Our ablation studies (Table 2) show clear benefits: the full VTV encoder achieves 59.0% average performance versus VideoMAE's 41.2%. Our three-stage interaction design (pressing for pressure, rotation for shear, sliding for friction) captures essential temporal information that single frames cannot provide. This produces significant gains: VTV-LLM achieves 60.4% average performance, substantially outperforming the best baseline's 29.5%.\nTo validate temporal modeling's importance in tactile perception, we used Octopi's data reading method (selecting 5 frames) and trained with the same dataset and pipeline. **Results in the table below confirm the importance of tactile video understanding, especially for dynamic attributes.**\n|                    | Hardness | Protrusion | Elasticity | Friction | Combined | SFD  | SOI  | OSC  | TSA  | Average |\n|--------------------|----------|------------|------------|----------|----------|------|------|------|------|---------|\n| Selecting 5 frames | 62.6     | 60.7       | 41.3       | 31.7     | 10.8     | 58.1 | 47.5 | 27.1 | 51.0 | 43.4    |\n| Ours               | **73.9**     | **75.0**       | **67.3**       | **56.5**     | **35.6**     | **71.3** | **57.6** | **43.2** | **64.0** | **60.4**    |\n\nMoreover, regarding direct comparison with TVL, we need to clarify why fair comparison is not feasible. TVL processes single-frame static tactile images, while our approach handles continuous video sequences of dynamic interactions. **The datasets differ significantly**: TVL contains 43,741 static image-touch pairs with 254 tactile adjectives for classification, whereas our VTV150K features video sequences with a structured four-attribute three-level annotation system. **The evaluation tasks also differ**: TVL focuses on classification accuracy while we assess complex multi-modal reasoning like scenario analysis. These fundamental differences make direct comparison both unfair and meaningless, as they address different sub-problems in tactile understanding. We will include this discussion in the final version.\n\n***\n**Q2: ...Masking 90% of the area may leave insufficient effective information in the remaining regions…**\n\n**A2**: While high masking ratios may seem problematic, our approach has strong theoretical and experimental support.\nVideoMAE [1] research shows that videos contain significant temporal redundancy, with consecutive frames changing slowly and being highly correlated. This allows excellent reconstruction even at 0.9-0.95 masking ratios, as models can infer masked content from limited visible tokens.\n**Tactile videos have even stronger structural advantages than natural videos. Deformations are localized yet constitute important tactile features, and tactile interactions follow predictable physical laws.** Our optical flow-guided masking strategy (Figure 3) preserves key deformation information consistently, while multi-frame sequences provide rich spatiotemporal context even when single frames are limited.\n\nWe conducted dedicated masking ratio ablation experiments specifically for tactile videos. \nResults can be found in the following table, where lower performance at 0.7 masking ratio, optimal performance at 0.9. This proves that **high masking ratio provides sufficient challenge for effective representation learning without causing information insufficiency.**\nFigure 3 demonstrates accurate reconstruction of key tactile features, with additional examples planned for the final version.\n|                    | Hardness | Protrusion | Elasticity | Friction | Combined | SFD  | SOI  | OSC  | TSA  | Average |\n|--------------------|----------|------------|------------|----------|----------|------|------|------|------|---------|\n| masking ratio=0.7  | 68.4     | 61.9       | 58.2       | 45.3     | 14.8     | 59.7 | 43.7 | 38.6 | 49.0 | 48.8    |\n| masking ratio=0.8  | 73.7     | **77.3**       | 66.1       | 54.3     | 28.8     | 70.5 | 52.2 | 40.3 | 59.0 | 58.0    |\n| masking ratio=0.9  | **73.9**     | 75.0       | **67.3**       | **56.5**     | **35.6**     | 71.3 | **57.6** | **43.2** | **64.0** | **60.4**    |\n| masking ratio=0.95 | 72.4     | 76.8       | 62.4       | 48.9     | 30.4     | **72.1** | 51.4 | 40.1 | 60.0 | 57.1    |\n\nFinally, we do not directly transfer VideoMAE parameters but specifically optimize the model for tactile videos through a three-stage training paradigm (Figure 2(b)), with the first stage specifically designing optical flow-guided masking strategy suitable for tactile video characteristics.\n***\n**Q3: ...the authors should include experimental results that demonstrate the model’s performance in zero-shot scenarios.**\n\n**A3**: In Section A.4.4 of the supplementary materials, we tested cross-sensor robustness by evaluating generalization capability. We used original Tac3D videos with signal patterns significantly different from our training data, and results show **our method maintains strong performance across these heterogeneous modalities.**\n\nWe also conducted zero-shot experiments on the XENSE sensor (a variant of the GelSlim sensor [2]). Using the same data collection protocol, we gathered data from 20 objects and generated 100 templates per task with the XENSE sensor. The trained VTV-LLM-7B model was then applied for zero-shot inference. The results in the following table demonstrate that **despite training on only three sensor types, our method generalizes well to other sensors.**\n|                                     | Hardness | Protrusion | Elasticity | Friction | Combined | SFD  | SOI  | OSC  | TSA  | Average |\n|-------------------------------------|----------|------------|------------|----------|----------|------|------|------|------|---------|\n| VTV-LLM-7B (for original 3 sensors) | 73.9     | **75.0**       | **67.3**       | **56.5**     | **35.6**     | 71.3 | **57.6** | **43.2** | 64.0 | **60.4**    |\n| VTV-LLM-7B (for XENSE sensor)       | **77.0**     | 71.0       | 55.0       | 52.0     | 26.0     | **78.0** | 55.0 | 39.0 | **69.0** | 58.0    |\n***\n**Q4: …the dataset’s limited scale...may restrict its applicability and generalization potential in real-world embodied interaction settings.**\n\n**A4**: Our dataset contains 100 objects with 150,000 visuo-tactile video frames collected systematically using three sensors per object, covering pressing, rotation, and sliding interactions. **This represents the first large-scale dataset specifically designed for visuo-tactile video understanding with comprehensive tactile attribute annotations.**\n\n**Results demonstrate the dataset's effectiveness.** VTV-LLM significantly outperforms all baselines including GPT-4o and excels on untrained tactile analysis tasks. Zero-shot performance drops minimally (60.4% to 58.0% on XENSE sensor) and robotic experiments (Figure 8) confirm strong generalization and practical applicability.\n\n***\n**Q5: …What is the size after interpolation?...If so, might this approach fail to fully leverage the fine-grained capabilities of Tac3D data in perceiving force?**\n\n**A5**: As described in lines 123-125, we employ cubic spline interpolation to address the resolution disparity between Tac3D (20×20) and other high-resolution tactile sensors, interpolating Tac3D force field data to **320×320 pixels** for training consistency.\n\nFigure 1 shows the interpolated Tac3D data used for model training, while Figure 7 demonstrates zero-shot testing on original 20×20 force field data. This design proves that despite training only on interpolated image-format data, our model successfully generalizes to original force field format and maintains fine-grained force perception capabilities, validating that our approach preserves Tac3D's essential force-sensing characteristics.\n**Converting force data to image format enables unified processing across heterogeneous sensors and provides denser spatial representations that better capture force distributions for our video-based learning paradigm, rather than diminishing force information.**\n\n***\n**Q6: ...\"maintains strong performance across these heterogeneous input modalities\" is not sufficiently...**\n\n**A6**: We have added more zero-shot tests in **A3**.\n\n***\n**Q7: Is it suitable for real-world dexterous manipulation tasks that demand real-time performance, or is it limited to tactile reasoning applications?**\n\n**A7**: While VTV-LLM focuses on tactile reasoning, our robotic experiments (Section A.4.5) demonstrate its real-world potential by successfully performing object selection and manipulation tasks. Although current computational requirements limit real-time performance, our framework provides essential building blocks for future dexterous manipulation systems.\n***\n[1] Tong, Zhan, et al. \"Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.\" Advances in neural information processing systems 35 (2022): 10078-10093.\n\n[2] Taylor, Ian H., et al. \"Gelslim 3.0: High-resolution measurement of shape, force and slip in a compact tactile-sensing finger.\" 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022."}, {"review_id": "nIBawmUCZY", "reviewer": "Reviewer_9p99", "summary": "The paper introduces a large-scale dataset of visuo-tactile video frames captured from 100 objects across three tactile sensors with tactile attributes annotations. It also presents a multi-modal large language model, VTV-LLM, for understanding visuo-tactile videos. The model was trained in three carefully designed stages, to effectively bridge the gap between tactile perception and natural language understanding. The proposed framework enables physical attributes reasoning that cannot be achieved through visual inspection alone. Experimental results show that VTV-LLM achieves superior performance in tactile reasoning tasks over seven state-of-the-art methods on the newly introduced dataset.", "questions": "- During the annotation process, what is level of variability of classification levels among three independent annotators? In other words, is it highly subjective and dependent on annotator bias? \n- In the question-answer pairs, the success rate of a random guess varies for different tasks, e.g., 50% for SFD and 33% for SOI. Is there any explanation on why some of the open-source models perform much worse than random guesses in Tab. 1?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- It contributes a comprehensive large-scale dataset of video-question-answer pairs across diverse visuo-tactile sensors.\n- The paper provides clear descriptions of dataset construction and model training framework. \n- Extensive experiments were conducted for performance comparison and ablation studies.\n\nWeaknesses:\n- The classification levels for tactile attributes are coarse-grained, lacking fine-grained precision.\n- The manual annotations might be subjective.", "was_revised": true, "final_justification": "This paper contributes a comprehensive large-scale dataset of video-question-answer pairs across diverse visuo-tactile sensors. During the rebuttal stage, the concerns on classification levels of tactile attributes and subjectivity of manual annotations were clarified. Reasonable explanations were provided for the significantly below-random performance of some open-source models. Therefore, I recommend acceptance of the paper.", "rebuttal": "We thank reviewer for the constructive comments. We provide our feedbacks as follows.\n\n**Q1: The classification levels for tactile attributes are coarse-grained, lacking fine-grained precision.**\n\n**A1**: Thanks for this comment. We understand the reviewer's expectation for finer-grained classification, but our three-level classification design is a reasonable choice based on sufficient theoretical considerations and practical validation. Research in psychology and neuroscience shows that human perception of tactile attributes naturally exhibits discrete and hierarchical characteristics [1][2]. **Our three-level classification (0-1-2) design aligns with the natural boundaries of human tactile cognition, rather than artificially imposing overly fine divisions.**\n\n\nSupplementary Material A.3.1 details our annotation process: *\"The annotation process involved three independent annotators evaluating tactile videos of the objects, with final classifications determined by majority consensus.\"* **Three-level classification ensures high inter-annotator consistency, while overly fine classifications would increase subjectivity and reduce annotation quality.**\n\nTable 1's experimental results show that even under three-level classification, the existing strongest baseline models perform significantly lower than our method across all attributes. **This indicates that the current challenge lies in establishing effective cross-modal understanding, not in insufficient classification granularity.** The Tactile Scenario Analysis (TSA) task shown in Figure 8 indicates that actual embodied interaction focuses more on qualitative decision support rather than precise numerical measurements. For example, questions like \"Which object is better for scraping ice?\" require categorical judgments rather than continuous values.\n\nMoreover, our three-stage training paradigm (Figure 2 (b)) has good scalability. If finer classification is needed in the future, we can build upon the current framework by adding classification levels or introducing continuous value regression tasks.\n***\n**Q2: The manual annotations might be subjective. During the annotation process, what is level of variability of classification levels among three independent annotators? In other words, is it highly subjective and dependent on annotator bias?**\n\n**A2**: We provide clear definitions based on objective physical phenomena for each tactile attribute. For elasticity, level 0 represents no obvious rebound, level 1 denotes slow partial recovery, and level 2 denotes rapid full recovery. **These definitions are based on observable physical phenomena, significantly reducing the space for subjective judgment.**\nMoreover, Supplementary Material A.3.1 reports a detailed inter-annotator agreement analysis. We use Fleiss' Kappa coefficient to quantify annotation consistency (Overall: 0.78). According to the standard, 0.61-0.80 indicates \"substantial agreement,\" and 0.81-1.00 indicates \"almost perfect agreement,\" showing that our annotation quality is excellent.\nDuring the annotation process, we conducted quality checks every 10 completed objects. **Throughout the annotation process, Kappa coefficients remained between 0.75 and 0.82, showing good stability.** We will further clarify related information in the final version.\n\n***\n**Q3: In the question-answer pairs, the success rate of a random guess varies for different tasks, e.g., 50% for SFD and 33% for SOI. Is there any explanation on why some of the open-source models perform much worse than random guesses in Tab. 1?**\n\n**A3**: Thank you for this insightful question. The significantly below-random performance of some open-source models in Table 1 can be explained by several key factors:\n\nFirstly, the open-source video-language models (e.g., InternVL2.5-VL-8B achieving only 17.1% average) were primarily trained on natural video datasets containing everyday scenes, objects, and activities. As stated in Section 3.1, **visuo-tactile videos exhibit fundamentally different characteristics from natural images, with unique properties like pressure distributions, surface deformations, and contact dynamics that are absent in conventional video datasets.**\n\nSecondly, **rather than making random errors, these models appear to systematically misinterpret tactile sensor outputs.** For instance, they might consistently confuse high-contrast regions in tactile images (which often indicate surface deformations) with visual features like shadows or textures. This leads to consistently wrong answers rather than random guesses, explaining the below-random performance.\n\nMoreover, as mentioned in Section 3.1, our dataset includes three different sensors (GelSight Mini, DIGIT, Tac3D) with distinct data formats, varying resolutions, and physical property encodings. **Models without tactile-specific training struggle with this multi-sensor variability, leading to consistent misclassification patterns.**\n\nThis systematic failure actually validates our core contribution - **the necessity of specialized tactile training paradigms rather than simply applying existing video-language models to tactile domains.**\nWe will add more discussion in the final version.\n***\n[1] Liu, Huaping, et al. \"Embodied tactile perception and learning.\" Brain Science Advances 6.2 (2020): 132-158.\n\n[2] Packheiser, Julian, et al. \"A systematic review and multivariate meta-analysis of the physical and mental health benefits of touch interventions.\" Nature human behaviour 8.6 (2024): 1088-1107."}]

Author rebuttal:

Meta-review: This work introduces the first multimodal large language model that supports universal visuo-tactile video understanding. The authors collect a large-scale tactile video dataset with annotated attributes and propose a three-stage training paradigm that enables the model to achieve superior performance on tactile reasoning tasks.

Most reviewers consider the proposed dataset valuable for future research in the field. They also appreciate the approach’s design and the demonstrated empirical improvements. One concern was raised regarding the limited fundamental attributes and granularities, but this was well explained during the rebuttal. Other minor concerns from reviewers, except for Reviewer P8wL, who did not participate in the rebuttal or discussion, were also addressed by the authors.

Given the above, the AC recommends acceptance. This decision is made based on down-weighting the comments from Reviewer P8wL due to their lack of participation and discussion.
Decision: Accept (poster)