Title: Universal Visuo-Tactile Video Understanding for Embodied Interaction
Venue: NeurIPS 2025
Accepted: yes
Keywords: Embodied Interaction

Tactile Perception

Video Understanding
Abstract: Tactile perception is essential for embodied agents to understand the physical attributes of objects that cannot be determined through visual inspection alone. While existing methods have made progress in visual and language modalities for physical understanding, they fail to effectively incorporate tactile information that provides crucial haptic feedback for real-world interaction. In this paper, we present VTV-LLM, the first multi-modal large language model that enables universal Visuo-Tactile Video (VTV) understanding, bridging the gap between tactile perception and natural language. To address the challenges of cross-sensor and cross-modal integration, we contribute VTV150K, a comprehensive dataset comprising 150,000 video frames from 100 diverse objects captured across three different tactile sensors (GelSight Mini, DIGIT, and Tac3D), annotated with four fundamental tactile attributes (hardness, protrusion, elasticity, and friction). We develop a novel three-stage training paradigm that includes VTV enhancement for robust visuo-tactile representation, VTV-text alignment for cross-modal correspondence, and text prompt finetuning for natural language generation. Our framework enables sophisticated tactile reasoning capabilities including feature assessment, comparative analysis, and scenario-based decision-making. Extensive experimental evaluations demonstrate that VTV-LLM achieves superior performance in tactile reasoning tasks, establishing a foundation for more intuitive human-machine interaction in tactile domains.


Peer reviews:
comments: SUMMARY:
The paper introduces a large-scale dataset of visuo-tactile video frames captured from 100 objects across three tactile sensors with tactile attributes annotations. It also presents a multi-modal large language model, VTV-LLM, for understanding visuo-tactile videos. The model was trained in three carefully designed stages, to effectively bridge the gap between tactile perception and natural language understanding. The proposed framework enables physical attributes reasoning that cannot be achieved through visual inspection alone. Experimental results show that VTV-LLM achieves superior performance in tactile reasoning tasks over seven state-of-the-art methods on the newly introduced dataset.

STRENGTHS AND WEAKNESSES:
Strengths:
- It contributes a comprehensive large-scale dataset of video-question-answer pairs across diverse visuo-tactile sensors.
- The paper provides clear descriptions of dataset construction and model training framework. 
- Extensive experiments were conducted for performance comparison and ablation studies.

Weaknesses:
- The classification levels for tactile attributes are coarse-grained, lacking fine-grained precision.
- The manual annotations might be subjective.

QUESTIONS:
- During the annotation process, what is level of variability of classification levels among three independent annotators? In other words, is it highly subjective and dependent on annotator bias? 
- In the question-answer pairs, the success rate of a random guess varies for different tasks, e.g., 50% for SFD and 33% for SOI. Is there any explanation on why some of the open-source models perform much worse than random guesses in Tab. 1?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes VTV-LLM, the first multi-modal large language model that supports universal visuo-tactile video understanding. The author constructed VTV150K dataset, covering 150k frames of videos from 100 common objects collected by three sensors, annotated with four tactile attributes: hardness, convexity, elasticity, and friction. To achieve efficient multi-modal fusion, the paper designed a three-stage training paradigm: visual tactile representation enhancement, video and text alignment, and text prompt fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n\n1. This work represents the first systematic attempt to develop a dedicated masking strategy specifically for tactile video data.\n2. The dataset proposed in this work represents good contribution to the field.\n\n\n**Weeknesses:**\n\n1. As one of the similar works to Octopi [1]—the first tactile video understanding model—this paper should provide a detailed comparative analysis highlighting both its differences from and advantages over Octopi. Furthermore, compared to models like TVL [2] that only accept single-frame tactile image inputs, how much benefit does video-based tactile understanding actually provide? A rigorous quantitative comparison would better demonstrate the value of temporal modeling in tactile perception.\n2. This work proposes an optical flow-guided video reconstruction strategy for tactile videos, which specifically masks deformation regions. However, unlike natural videos, tactile videos exhibit highly localized and fine-grained deformations. Masking 90% of the area may leave insufficient effective information in the remaining regions, potentially causing the mask modeling to overfit. The authors should provide examples of masked images and reconstructed results, and conduct an ablation study on masking ratios to validate the approach. Directly transferring parameters from natural video models may not be optimal for tactile video reconstruction.\n3. A key potential of combining tactile sensing with large language models is the ability to achieve zero-shot tactile perception and understanding. To highlight the generalization capability and practical value of their approach, the authors should include experimental results that demonstrate the model’s performance in zero-shot scenarios.\n4. I commend the authors for their efforts in collecting the dataset and conducting detailed annotations. However, the dataset’s limited scale—covering only 100 objects with relatively simple interaction patterns—may restrict its applicability and generalization potential in real-world embodied interaction settings.\n5. The proposed dataset incorporates Tac3D sensor data with a 20×20 resolution force field alongside high-resolution tactile images.  However, the paper does not clearly explain how this data is combined with high-resolution tactile images. Figures 1 and 7 display two different input forms of Tac3D data (interpolated images and force fields), but it remains unclear which one is actually used—or whether these represent different task settings. What is the size after interpolation? Additionally, is the Tac3D data processed into image-form inputs rather than the original point-cloud-like format? If so, might this approach fail to fully leverage the fine-grained capabilities of Tac3D data in perceiving force?\n6. The claim that the method \"maintains strong performance across these heterogeneous input modalities\" is not sufficiently convincing when relying solely on Figure 7. More comprehensive experimental results should be provided to substantiate this conclusion.\n\n\n[1] Yu S, Lin K, Xiao A, et al. Octopi: Object property reasoning with large tactile-language models[J]. arXiv preprint arXiv:2405.02794, 2024.\n[2] Fu, Letian, et al. \"A Touch, Vision, and Language Dataset for Multimodal Alignment.\" *International Conference on Machine Learning*. PMLR, 2024.\n\nQUESTIONS:\nIs it suitable for real-world dexterous manipulation tasks that demand real-time performance, or is it limited to tactile reasoning applications?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a universal Visuo-Tactile Video Large Language Model(VTV-LLM) to address the gap in integrating tactile perception into LLMs. The authors collect a large-scale tactile video dataset, which contains 150K video frames with 4 fundamental tactile attributes. A three-stage training paradigm is proposed to train the VTV-LLM, which includes a VTV enhancement for robust visuo-tactile representation, a VTV-text alignment for cross-modal correspondence, and a text prompt finetuning for natural language generation. After training, the VTV-LLM can execute various tactile reasoning tasks.\n\nSTRENGTHS AND WEAKNESSES:\n【Strengths】\n1. The first work to integrate visual, tactile, and language modalities for universal tactile understanding.\n2. A valuable and comprehensive dataset encompasses a diverse range of objects, sensors, and tactile attributes, which can be used for training tactile-related models.\n3. The three-stage paradigm guarantees effective cross-sensor and cross-modal integration, enhancing tactile reasoning accuracy. \n4. The trained model performs well on the tactile reasoning task and lays the groundwork for the embodied domain.\n\n【Weaknesses】\nLimitation of the dataset: a small number of tactile sensors (GelSight, DIGIT, Tac3D) and limited fundamental attributes.\nReal-world feasibility concerns: The computational overhead of the LLM restricted its application in real-time embodied interaction scenarios.\n\nQUESTIONS:\nThe annotation focuses on four fundamental tactile attributes. Are there plans to expand the dataset to more complex tactile features, such as texture and temperature?\nIs the VTV-LLM possible for edge-device deployment?\nHow to mitigate the adverse effects of the LLMs' hallucination issue?\n\nLIMITATIONS:\nThe dominating limitation is the real-world feasibility of the LLM-based tactile reasoning model in real-time scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper introduces the Visuo-Tactile Video (VTV) dataset, which comprises 150K frames of tactile videos annotated with attributes. Additionally, the author proposes the VTV-LLM framework, which facilitates reasoning in embodied interactions. To train the VTV-LLM framework, the author utilizes the VTV dataset and introduces a three-stage training process for the model.

STRENGTHS AND WEAKNESSES:
**Strength**
1. Introductin of VTV dataset that include attribute annotation and tactile Q&A 
2. Demonstrates the effectiveness of the proposed VTV-LLM and three-step training strategy through experiments on tactile understanding tasks.

**Weakness**
1. I fail to comprehend the advantage of video over tactile. What is the advantage of tactile video compared to tactile images? I believe tactile video doesn’t offer significant differences over time, potentially resulting in redundant frames.

2. Additionally, since the network processes video as input, the number of videos becomes more crucial than the number of frames in the dataset. Furthermore, it’s unclear whether annotations are applied to videos or frames. For instance, what does 150,000 video frames represent? I think including frames per second (FPS) would help understand the dataset size or the number of videos. 150,000 frames at 60 FPS could equate to just 40 minutes of video, which is not a substantial amount.

3. For VTV-LLM, it’s unclear why tactile video is necessary for tactile understanding. A single image would suffice for the task proposed by the task author. Video input only increases the model’s size. To demonstrate the necessity of video, a task that shows temporal tactile information should be introduced.

QUESTIONS:
Please refer to weakness section.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 1
ORIGINALITY: 2
DATE: 2025-07-07