AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors

Authors: Ruoxuan Feng, Jiangyu Hu, Wenke Xia, TianciGao, Ao Shen, Yuhao Sun, Bin Fang, Di Hu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Tactile Representation Learning, Visuo-tactile Sensors, Cross-sensor Transferring

Abstract:
Visuo-tactile sensors aim to emulate human tactile perception, enabling robots to precisely understand and manipulate objects. Over time, numerous meticulously designed visuo-tactile sensors have been integrated into robotic systems, aiding in completing various tasks. However, the distinct data characteristics of these low-standardized visuo-tactile sensors hinder the establishment of a powerful tactile perception system. We consider that the key to addressing this issue lies in learning unified multi-sensor representations, thereby integrating the sensors and promoting  tactile knowledge transfer between them. To achieve unified representation of this nature, we introduce TacQuad, an aligned multi-modal multi-sensor tactile dataset from four different visuo-tactile sensors, which enables the explicit integration of various sensors. Recognizing that humans perceive the physical environment by acquiring diverse tactile information such as texture and pressure changes, we further propose to learn unified multi-sensor representations from both static and dynamic perspectives. By integrating tactile images and videos, we present AnyTouch, a unified static-dynamic multi-sensor representation learning framework with a multi-level structure, aimed at both enhancing comprehensive perceptual abilities and enabling effective cross-sensor transfer. This multi-level architecture captures pixel-level details from tactile data via masked modeling and enhances perception and transferability by learning semantic-level sensor-agnostic features through multi-modal alignment and cross-sensor matching. We provide a comprehensive analysis of multi-sensor transferability, and validate our method on various offline datasets and in the real-world pouring task. Experimental results show that our method outperforms existing methods, exhibits outstanding static and dynamic perception capabilities across various sensors. The code, TacQuad dataset and AnyTouch model are fully available at gewu-lab.github.io/AnyTouch/.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730415630743

thread_type:
official_review

<thread_object>
creator: reviewer_ow9a

<content>
summary:
This paper present a novel framework called UltraTouch and a large dataset TacQuad collected using 4 different tactile sensors. The UltraTouch framework has a multi-stage training in pixel level and semantic level. In stage 1, the authors use MAE to train on static and dynamic tactile images/videos. In stage 2, the authors do contrastive learning on multi-modal aligning and cross-sensor matching to understand semantic-level sensor-agnostic tactile properties. The authors also answer research questions by testing on downstream tasks like material classification, grasp prediction, and pouring.

strengths:
Strengths of this paper includes:

- They collect and open-source a large tactile dataset called TacQuad, which makes good contribution to the multi modal robot learning community since the tactile data are far from enough compared with vision data. 
\
\
- The framework is novel. The idea of learning a unified tactile representation is very interesting and the authors study how tactile aligns with other modalities like vision and text, and how different sensors match with each other. 
\
\
- The experiments are comprehensive. The authors conduct extensive experiments on multiple downstream tasks like material classification, grasp prediction, and pouring, which demonstrate Ultratouch, learning unified representations, has advantages over other models.

weaknesses:
Weaknesses of this paper include:

- Though vision based tactile sensors are very popular in robot learning community, it might not be the ultimate solution that people agree on. Tactile perception are more than images. Humans feel temperature, texture, shape, vibrations, etc. when interacting with objects. This is still an open question and is out of scope of this paper. But it will be interesting to see if how other tactile sensors align or match with the vision-based tactile sensors.
\
\
- I think there can be some study on the effect from the stage 1, the pixel level learning, on the downstream tasks. How does learning the pixel-level details help the downstream tasks? I think material classification and grasp prediction might not need that level of detail to achieve good performance.

questions:
- In data collection process, how do you localize the contact position in world coordinate? How to ensure the contact position is the same for different sensors?

- How many corrections do you have to make when use GPT to generate descriptions? How well does it understand the data?

- Are you using the same sensor throughout the entire data collection process? I feel like most tactile sensors now are not very robust, especially the gel pad, which will degrade over time when it touches objects for too many times. Have you replaced gels during the collection process? Will the data be different after replacement? 

- Even the tactile sensors are of the same type. The difference between each individual sensor is huge. See the pixel-wise difference of DIGIT in Fig 7. in this paper [1]. Combined with my previous question, if you have use multiple sensors of the same type during collection and training, I'm curious how this will affect the performance.

- Missing space in line 797 between TAG and includes

[1] Bhirangi, Raunaq, et al. "AnySkin: Plug-and-play Skin Sensing for Robotic Touch." arXiv preprint arXiv:2409.08276 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730611377944

thread_type:
official_review

<thread_object>
creator: reviewer_deu8

<content>
summary:
This paper presents a novel and comprehensive representation learning framework for tactile data that addresses (1) V-T-L modalities alignment, (2) capturing dynamic information in tactile modality, and (3) generalization across different sensors together. It constructs a multi-sensor dataset called TacQuad and a 2-stage multi-task representation learning framework, equipped with pixel-level MAE, semantic-level multi-modal alignment, and cross-sensor matching techniques. With extensive experiments, the authors show this representation helps in some downstream tasks under various settings (seen and unseen sensors, different training dataset combinations, and real robot experiments).

strengths:
1. Well-motivated and well-written. Authors observed the difference between tactile sensing and vision, and creatively designed corresponding modules to enhance the representation. 

2. Clear visualization. Fig 2 works as a nice abstract of the entire framework.

weaknesses:
1. In Tables 2 and 3, the average performance of the TAG+Feel+YCB-slide+OF 2.0 model is better than the full model with the TacQuad dataset. This observation
  - Questions the necessity of your TacQuad dataset, and 
  - Poses a new question: how do you choose the size & scope of the dataset? Contrastive learning has requirements on the dataset size, while too much data will, in turn, make it "lose focus". 

Please add more discussion and experiments on this issue, and figure out either the detailed reason (i.e. which dataset is bad for some reason) or how to find a good combination of datasets. 

2. Given you claimed the model has the ability to extract sensor-agnostic features from unseen sensors, it would be more convincing to show some downstream task performance on the new sensor, aside from qualitatively showing t-SNE visualization.

3. Some other results are not convincing enough as well. In Fig 4 / Sec 5.3, DuraGel samples are better clustered than baselines, but some samples from GelSight and DIGIT, which are previously seen, are far from each other and mixed with other samples. Is it due to some difficulties in tuning the corresponding weight for loss?

questions:
1. Regarding Fig 7 and 8, is there a significant gap between GPT-4o outputs for OF-Real and TVL/SSVTP, given you didn't use tactile data for the latter? If so, will this affect your training? If not, do you think GPT-4o is making use of tactile input, given it may not be capable of analyzing images from tactile sensors? Can you show some raw output examples?

2. I noticed your "full model" removed Feel and OF 2.0 from the dataset. Is it on purpose? If the full model sees the Feel dataset, will its performance in Table 2 be lower than 80.53, between 80.53 and 82.3, or higher than 82.3? Similarly for OF 2.0. The claim that more is not always better itself is interesting, but I just wonder can it be simply due to the lack of some specific dataset? Your current experiment setting cannot rule out this possibility. 

3. Sec 5.4 explained the performance drop on material classification, but what about grasp? 

4. The tables are a bit hard to read. Is the first row in Table 1 from CLIP as explicitly said in Tables 2 and 3? In Table 2, the colorful detailed dataset configuration makes it hard to find what to compare before carefully reading Sec 5.4. 

Minor:
1. Citation format. Parenthesis missing in lots of scenarios.
2. Line 797 missing space.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685402114

thread_type:
official_review

<thread_object>
creator: reviewer_tlvc

<content>
summary:
This paper proposes UltraTouch, a model that learns unified static-dynamic representations for multiple tactile sensors with different architectures. To train UltraTouch, a dataset called TacQuad is introduced by collecting multimodal data from four different tactile sensors. UltraTouch captures both pixel-level and semantic-level features, which are shown to be transferable and effective through experiments on multiple downstream tasks.

strengths:
1. The collected dataset TacQuad, especially the fine-grained spatio-temporal aligned data can be a significant contribution to the community of tactile sensing. None of the prior works has collected tactile videos with shared speed and depth using multiple sensors. Although not fully explored in this work, I think this dataset and its data collection program will potentially benefit future research if being available to public.

2. The proposed UltraTouch framework, to the best of my knowledge, is the first model that learns the dynamic tactile representations through pretraining. This is a considerable improvement upon the prior works that learn unified tactile representations, which are all limited to static images. Future works that leverage touch for fine-grained manipulation tasks can potentially benefit from this framework.

3. The performance of UltraTouch model on the existing material classification/ grasp stability prediction seems promising, which is outperforming the previous pretraining frameworks significantly in most experiments. 

4. The proposed real-world pouring task demonstrates the effectiveness of learning dynamic representations (Table 4).

weaknesses:
1. As is shown in A.4, only 3 frames are used for each tactile video clip. This seems to be a surprisingly small number of frames, and my concern is that this might not be sufficient for extracting some of the fine-grained tactile information (e.g., hardness). One possible ablation experiment would be training the model with different video length and evaluate the performance. See Questions section for more details.

2. The paper proposes a novel calibration platform for collecting fine-grained spatio-temporal aligned data, and they claim that this can be used for fine-grained tasks such as cross-sensor generation. However, it seems that they don't explore much on the potential of this data, i.e., most of the downstream tasks are performed on static perception, and the only pouring task is not closely related to this part of data. It would be interesting to see if any more impressive tasks can be performed using the proposed dataset.

questions:
1. Referring to my first concern in Weaknesses section, I'm curious about the computational cost of training with tactile videos. It's common that the frame rate of the video being limited due to data loading/ computation limits, but only using 3 frames still seems too few to me. My guess is that using more frames may lead to better dynamic perception ability, and I'm wondering if an ablation experiment on this parameter can be possible?

2. Besides TacQuad, some of the other datasets also contain tactile videos, such as TAG and OF Real. I wonder if they are already treated as videos during the training of UltraTouch, if not, maybe it can further improve the model performance if they are used as videos.

3. I'm curious about the scalability of the calibration platform used for data collection. Is it easy for other researchers to build a similar platform to scale up the current dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708355074

thread_type:
official_review

<thread_object>
creator: reviewer_duex

<content>
summary:
The author introduces an aligned multi-sensor tactile dataset to unify tactile data. The data across four optical-based sensors. Also, the author proposes a unified static-dynamic multi-sensor representation learning. The author also implements real world experiments to test their framework.

strengths:
(1) The idea of aligned multi-sensor dataset, improve generalization across different sensors.
(2) Capturing both static and dynamic tactile features. This is novel to make the system have comprehensive tactile information understanding.
(3) The dataset labeling process involves language model. This could improve the efficiency of the dataset collection.
(4) Using t-SNE visualization is clear to show the impact of components on representation space.
(5) I recognize the authors' effort to collect large volume dataset of tactile

weaknesses:
Major Weaknesses:
1. Although the author's idea of the aligned multi-sensor dataset, the author focuses to optical-based tactile sensors only, which simplifies the problem with similar tactile modalities(tactile image).
2. In sec 5.2, the conclusion of the material classification task seems contradict to the motivation to build the aligned multi-sensor dataset.
3. The experiment results are not convincing in Section 5.5.
4. Some components of methods are not proved effective in the experiment.
4. No limitations and future work are described in the main paper.

Minor Weaknesses:

1. The presentation needs to be improved. Readers may get confused for some point before reading the supplementary material

2. The author use too many colors to represent different sensor types in the main text, which may not be good for reading.

3. Some conclusions according to experiment results are not reasonable.

questions:
1. For those four types of sensors, how many sensors you chose for each type? This is important because the tactile image can be very different even the same types of sensor. The dataset could be more diverse if more sensors for each type are used.

2. [line 189]. The author mentions that collecting fine-grained aligned tactile data is very costly. Please specify the reasons.

3. [line 431]. Incorporating data from more sensors leads to a performance drop. This may potentially reflect the robustness of the mode is not good enough. Using a single sensor for each type may hard to extract effective features of this type of sensor and only focus on some specific details of a single sensor.

4.[line 488-498] The performance of UniTouch and UltraTouch is similar, it is hard to claim the advantages of the current method.

5. Please mention real-world tasks in more detailed in the main paper. The reader will be confused about the task and the meaning of "mean error" before reading the appendix.

6. It would be better if the author have experiments to prove the effectiveness of the text in this dataset. 

7. [line 450-453] Please explain more details for this.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732468704604

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of responses

comment:
We would like to express our sincere appreciation to all reviewers for their insightful and comprehensive feedback. We sincerely appreciate the comments that the framework integrating static and dynamic perception is novel and interesting (Reviewer DUex, dEU8, oW9a and TLvC), the dataset is valuable to the community (Reviewer DUex, oW9a and TLvC), the experiments are comprehensive (Reviewer oW9a and TLvC) and the paper is well-writen and well-motivated (Reviewer dEU8).

For convenience, we have summarized some key concerns and our corresponding responses:

[Performance decline on seen datasets when more data is introduced] Due to the scarcity of tactile datasets, existing methods sometimes validate on downstream datasets that are already included in the pre-training data. In this setting, when the downstream task's dataset has a larger proportion in the pre-training data, it is naturally more likely to perform better, which is aligned with experimental findings in the CLIP paper [1]. Therefore, integrating more data reduces the proportion of the downstream data in pre-training, leading to a performance decline for the seen datasets. What we want to emphasize that when we introduce more multi-sensor data, our method shows an overall performance improvement on unseen datasets. This aligns with the core goal of our dataset and our method: to obtain a unified tactile multi-sensor representation that is applicable to a variety of tasks and sensors.

[Extension of the TacQuad dataset] Due to cost and manpower limitations, the scale of the multi-sensor paired data we collected is somewhat limited compared to the full dataset. Fortunately, as our data collection method is highly scalable, enabling collaborative data collection across multiple researchers and laboratories, the diversity of objects and sensors and the scale of the dataset can be enhanced in future work.

[New downstream task] To more comprehensively demonstrate the value of the dataset we proposed and the effectiveness of UltraTouch, we have conducted cross-sensor generation experiments on the fine-grained spatio-temporal aligned data. The results show that our method outperforms the previous method. This supports our motivation to obtain a unified tactile multi-sensor representation that is applicable to a variety of tasks and sensors.


[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PMLR, 2021.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces TacQuad, a multi-modal tactile dataset, and UltraTouch, a unified static-dynamic multi-sensor representation learning framework, to enhance tactile perception and enable effective cross-sensor transfer, achieving state-of-the-art performance in both offline datasets and real-world tasks.

All reviewers acknowledge the contributions of this work, emphasizing its (1) novelty, (2) technical advancements, including the representation learning framework and the introduced dataset, and (3) clear presentation.

The authors effectively addressed the reviewers’ comments during the Author-Reviewer Discussion phase, resulting in improved scores.

All reviewers are in unanimous agreement to accept this paper. However, the AC recommends that the authors carefully revisit both the original and post-rebuttal reviewer comments to ensure all concerns are adequately addressed in a revised version of the paper.