VTDexManip: A Dataset and Benchmark for Visual-tactile Pretraining and Dexterous Manipulation with Reinforcement Learning
Authors: Qingtao Liu
Yu Cui
Zhengnan Sun
Gaofeng Li
Jiming Chen
Qi Ye
Venue: ICLR 2025 Poster (2025)
Keywords: Robotics; Manipulation; Vision and tactile; Multi-modal pretraining; Reinforcement learning
Primary area: datasets and benchmarks

Abstract: Vision and touch are the most commonly used senses in human manipulation. While leveraging human manipulation videos for robotic task pretraining has shown promise in prior works, it is limited to image and language modalities and deployment to simple parallel grippers. In this paper, aiming to address the limitations, we collect a vision-tactile dataset by humans manipulating 10 daily tasks and 182 objects. In contrast with the existing datasets, our dataset is the first visual-tactile dataset for complex robotic manipulation skill learning. Also, we introduce a novel benchmark, featuring six complex dexterous manipulation tasks and a reinforcement learning-based vision-tactile skill learning framework. 18 non-pretraining and pretraining methods within the framework are designed and compared to investigate the effectiveness of different modalities and pertaining strategies. Key findings based on our benchmark results and analyses experiments include: 1) Despite the tactile modality used in our experiments being binary and sparse, including it directly in the policy training boosts the success rate by about 20\% and joint pretraining it with vision gains a further 20\%. 2) Joint pretraining visual-tactile modalities exhibits strong adaptability in unknown tasks and achieves robust performance among all tasks. 3) Using binary tactile signals with vision is robust to viewpoint setting, tactile noise, and the binarization threshold, which facilitates to the visual-tactile policy to be deployed in reality. The dataset and benchmark are available at \url{https://github.com/LQTS/VTDexManip}.

Published as a conference paper at ICLR 2025
VTDexManip: A DATASET AND BENCHMARK FOR VISUAL-TACTILE PRETRAINING AND DEXTEROUS MA- NIPULATION WITH REINFORCEMENT LEARNING
Qingtao Liu, Yu Cui, Zhengnan Sun, Gaofeng Li, Jiming Chen, Qi Ye"
Department of Control Science and Engineering
Zhejiang University
Hangzhou, Zhejiang, China
(1_qingtao, cculyu, an.sun, gaofeng. ejm, zi.ye]@zju.edu. edu. edu
ABSTRACT
Vision and touch are the most commonly used senses in human manipulation. While leveraging human manipulation videos for robotic task pretraining has shown promise in prior works, is limited to image and language modalities and deployment to simple parallel grippers. In this paper, aiming to address the limitations, collect vision-tactile dataset by humans manipulating 10 daily tasks and 182 objects. contrast with the existing datasets, our dataset is the first Aisual-tactike dataset for complex robotic manipulation skill learning. Also, we introduce novel benchmark, featuring six complex dexterous manipulation tasks and reinforcement learning-based vision-tactile skill learning framework. 18 non non-pretraining and pretraining methods within the framework are designed and compared to investigate the effectiveness of different modalities and pertaining strategies. Key findings based on our benchmark results and analyses experiments include: 1) Despite the tactile modality used in our experiments being binary and sparse, including it directly in the policy training boosts the success rate by about 20% and joint pretraining it with vision gains a further 20%. 2) Joint pretraining visual-tactile modalities exhibits strong adaptability in unknown tasks and achieves robust performance among all tasks. 3) Using binary tactile signals with vision is robust to viewpoint setting, tactile noise, and the binarization threshold, which facilitates to the visual-tactile policy to be deployed in reality. The dataset and benchmark are available hnttp:::s.... http:::///tttb
INTRODUCTION
Achieving human-like dexterous manipulation is one of the most challenging tasks in robotics and embodied artificial intelligence. When humans interact with the world, vision and touch are the common senses used. Touch offers physical information beyond vision and compensates for local perception when vision is obstructed Billard & Kragic (2019). Therefore, extensive studies Lee al. (2020); Chen al. (2022): y Qi al. (2023); Zhang & Demiris (2023); George et al. (2024); Calandra et al. (2018): ; Jin et al. 2024) attach tactile sensors to grippers and leverage both visual and tactile signals in reinforcement learning framework for skill learning. However, the visual- tactile representation is trained jointly with the manipulation policy for specific task in limited environments, struggling to generalize to other tasks and environments.
On the other hand, representation learning large datasets has demonstrated generalization abilities many domains Radford al. (202T): Zitkovich et al. (2023); DDiees et al. (2023); Achiam et al. (2023). To address the generalization issue robotic manipulation, some works have collected extensive robot operation data to train large robotic models Brohan et al. y; Dasari et al. (2019); Bharadhwaj et al. (2023) Fang et al. (2023); y Khazatsky et al. (2024). However, the data collection cost is high, and it is challenging to gather complex manipulation skills on a large scale, resulting in these datasets focusing tasks with simple parallel grippers. In contrast, humans perform
CCoessponing Author

---

Published as a conference paper at ICLR 2025
complex dexterous manipulation tasks, such as in-hand rotation and it is easier acquire large-scale human manipulation videos. Therefore, recent studies have explored the representations learned from large datasets of human manipulation for robotic tasks RRadoavvvvic al. (2023); ; Nair et al. (2022); Karamcheti et al. (2023); Ma et al. (2023b); /Zhan et al. (2023); ; Ma et al. (2023a). Nevertheless, they primarily emphasize the visual and natural language modalities of human manipulation data, neglecting the potential benefits of tactile data.
To bridge the gap, we create a dataset of 2032 vision-tactile sequences covering 10 daily tasks and 182 objects to study the tactile modality and multi-modal pretraining in complex manipulations. Recently, attention has been paid to the fusion of vision and tactile modalities and several visual- tactile datasets are collected to perform tasks like material classification, tactile localization, object property prediction, garment classification Yang et al. (002)); Kerr et al. (2023); Dou et al. (2024); Yu et al. (202bb) as shown in Tab For our manipulation data collection, humans need to wear tactile sensors to collect data for long time and the sensors must be flexible and robust while these optical-based tactile sensors are typically bulky and can not be easily integrated into wearable gloves or attached to human hands at low cost. On the other hand, though these high-precision tactile signals help sophisticated control at the low level, the combination and the tempo of touch status of different hand parts may provide abundant information on how to manipulate a higher planning level. As a glove as
compromise, we adopt solution of low-cost piezoresistive pressure sensors attached shown in aa). Also, in contrast with the existing visual-tactioe datasets, our dataset is the first visual-tactile dataset with multi-fingerte hand for complex robotic manipulation skill learning.
Table I: Comparison of visual-tactile datasets.
Dataset
Whramee/Ohix/SSegg/sisss
Vision
Tactile
Application
Touch and Yang et al. 2022
Tactile driven image sty stylization, multimoaal video prediction
13.9K/3971//-
RGB
GelSight
SSVTP
Anomaly detection, edge filloowigg tactile localization classification
4500/10---
RGB
DIGIT
TaRF-
Tactile localization.
Dou al.
2024
19.3K/M--
RGB
DIGIT
material classification
PHYSICLES EAA
Physical property prediction. scenario reasoning
Yu et al.
2024b
45. 8k/76/0/8
RGB-D
GelSight
Ours
165k/18222032/10
RGB
Pressure sensor
Complex multi-fingered manipulation tasks
To evaluate the role of human vision-taction information in robotic manipulation, we introduce a novel benchmark comprising a vision-tactile dexterous manipulation platform based on Isaac Gym Makoviychuk et al.] (2021) and a reinforcement learning-based skill learning framework, with the hope of providing insights for future research relating to dexterous multi-fingered robotic manipulation, human priors for robotic manipulation, multi-modal learning, tactile sensor design for multi-fingeree hands, etc. Specifically, six complex manipulation tasks are built up in the simulation that require sophisticated coordination of joint movements: Bottlecap Turning, Faucet Screwing, Lever Sliding, Table Reorientation, In-hand Reorientation and Bimanual Hand-over. Though there exists work on some similar tasks like Reorientation Chen et al. (2023), they only focus on one task. Our multi- fingered manipulation platform is the first one consisting of very different types of complex skills, which facilitates the study of generalizable human-lite like complex manipulation skill learning.
For the benchmark, benchmark method fusing vision-tactile via joint pretraining is first constructed to learn manipulation representations from human dision-tactile data inspired by MAE He et al. (2022); Raaosavovicc et al. (2023); Liu et al. (2024). Further, five popular vision pretraining meth- ods Radosavovic et al. (2023): Nair et al. (2022); ; Radford et al. (2021): He et al.] (2016); Karamcheti let al. (2023) used computer vision and roboties and their combination with the tactile modality are benchmarked. In addition to the methods, experiments and analyses regarding the viewpoint setting for visual-tactile fusion, the tactile noise, and thresholds to binarize the tactile signals in pretraining and RL are conducted to provide guidance for deploying the tactile modality to real-world experiment setting. Our main contributions are summarized as follows:
We collect a human visual-tactile manipulation dataset consisting of 565k frames, covering 10 daily tasks and 182 objects for multi-fingered robotic hand manipulation.
We propose tactile benchmark for dexterous manipulation, which includes a manipulation simulation platform with six multi- finteiffieeedd manipulation tasks and a manipulation skill learning framework based on pretraining and reinforcement learning.
2

---

Published as a conference paper at ICLR 2025
We benchmark more than 18 non non-pretraining and pretraining methods to investigate the effectiveness of different modalities and pretraining strategies for dexterous manipulation.
Extensive experiments and analyses regarding viewpoint setting, the noise and binary thresholds of the tactile signals are conducted to provide guidance for the deployment of the tactile modality.
2 RELATED WORKS
Visual-Tactile datasets and benchmarks. In recent years, the integration of visual and tactile sensing has gained increasing attention in robotics, particularly for tasks that involve intricate surface interactions. Optical tactile sensors Lambeta et al. (2020): Yuan et al. (0177). in particular, have enabled the collection of rich visual-tactile data that captures detailed texture information from the contact area. With these sensors, prior works collected datasets that have been primarily used for tasks such texture recognition, classification, localization, and detection Yang al. (2022); Kerr et al.] (2023): Dou et al. 2 Yu et al. (2024b). While visual-tactile information plays a crucial role in dexterous manipulation Qi al. (2023) ) Guzey al. (2023), there currently no dataset available specifically tailored for such tasks. The use of human tactile information for robot manipulation has gained increasing attention. MimicTTouch Yu et al. (2024a) leverages human tactile demonstrations for gripper- based tasks like grasping and insertion. Inspired Liu et al. (2024), our work fills this gap by providing a visual-tactile dataset for learning diverse complex dexterous manipulation skills with multi-fingered hands.
Pretraining for robot manipulation with out-of-domain dataset. Due to the challenges of collect- ing robotic data and generalizing in-domain data, some previous works Radosavovic et al. (2023):
Nair et al. (2022); ; Karamcheti et al. (2023); Ma et al. (2023b); Zhang et (2023): Ma et al. (2023a)
have focused on leveraging out-of-domain datasets for pretraining. These works have shown that representation models pretrained on human data can significantly aid in learning different robotic tasks. These works are mainly divided into two categories: MVP Radosavvvvooiicc et al. (2023), R3M Nair et al. 2022). ) Voltron Karamcheti et al. (2023), and SGR Zhang et al. (2023) pretrain models to extract representations from inputs, while VIP Ma et al. (203b) and LIV Ma et al. (2023a) pretrain represen- tation and reward function models. All these works utilize human manipulation datasets Grauman al. (2022); Damen et al. (2018); Goyal al. (2017), focusing on visual, language, or point cloud modalities. Tactile sensing, despite its crucial role in both human and robotic manipulation, has not been explored extensively due to the absence of human tactile data in existing datasets.
Visual-tactile manipulation. Many studies have attached tactile sensors to parallel grippers and combined the tactile signals with vision to realize manipulation tasks like cable plugging George
et al. [2024) dressing Zhang & Demiris (2023). efficient grasp adjustment Calandra et al. (2018). and insertion Jin et al. (Sferaaa et al. (2023). For multi-finger manipulation tasks, Guzeyet al. Guzey et al. (l.((20) (2023) places uSkin on the Allegro Hand to learn six dexterous manipulation skills. Rotatelt Qi et al. (2023) equip robotic fingertips with four omnidirectional vision-based touch sensors to learn the in- hand rotation skill. These studies focus online learning of visual and tactile modalities, achieving promising results for specific tasks. However, the learned representation models not transferable to other tasks, requiring task-specific data and representation learning for each case, which is inefficient for multi-skill learning. Instead, we mainly study and analyze the efficacy of visual and tactile pretraining for skill learning on various complex manipulation tasks by human manipulation data collection and pretraining benchmarking similar to Liu et al. (2024).
3 VISUAL-TACTIL. MANIPULATION DATASET
Data Collection Procedure. Following Liu et al. 2024)) we set up our collection system as shown in [a). We calibrate each tactile unit on the glove using an F/T sensor to ensure that each tactile unit can be activated by the same amount of force. Due to the simple principle and design, the glove can produced in a more compact form in industry, suitable for large-scale data collection. We utilize the glove and 2ololens2 to synchronously collect visual and tactile data pairs of human manipulation from an ego-centric view. See A.
A.I for more about the glove.
Dataset Statistics and Analysis. The dataset is collected by 5 subjects and includes 2,032 manipula- tion sequences, 10 daily tasks, 182 objects, which ends up with 565k iisual-tactile pairs of frame.
https:// httpps///www.. wwt.w.ii.roooo...coo wwt/
3

---

Published as a conference paper at ICLR 2025
Num dm pars (k)
Nay
Nim of imjocionios
Teat D1
Nan
20
lte
10
(a)
(b)
(e)
Tak
Tal-4
P-Wec Pourina
Bi-Thowing
Nun toaaipiitii
Tek
Tak
S-h-- Sari Reonavalize
Tal
4-0d Uiotadang
Selnin Turiiy
60 o (d)
A-tkicias Objects
Curaat Eoguence colar bus
S-TaMe-TTep Muripuattoon
[e]
Figure 1: Visualization of our dataset. (a) Our collection system. (b) The number of trajectories and objects. (c): The number of total frames (On: frames w/ contact; Off: frames w/o contact ). (d) The
distribution of the number of frames. (e) I-SNE of the tactile signals.
We show the number of trajectories and objects for each task Fig the number of total frames for each task in Fc) and the distribution of the number of frames in each task trajectory in dd.. The tactile signals earry pressure values during manipulation, but to reduce the real-to-sim and sim-to-real gap, only binary signals acquired by thresholding pressure values are used in the following sections. See A.2 for more details.
To further investigate the characteristics of the tactile signals for different manipulation tasks, visualize the tactile signals by t-SNE Van der Maaten Hinton (2008) as shown in Fig [e)): for each sequence, calculate the contact frequency of a tactile sensor and the frequency of all tactile sensors is concatenated as the representation of the sequence for visualization (See (A.3) for more details about t-SNE). In the figure, there exist some clusters while cluster does not correspond to only a task: 1) the clusters may consist of data points from different tasks, e.g. data points for In-hand Reorientation and BottleCap Turning form relatively compact cluster; 2) data points for task may span many clusters, e.g. data points for Bowl Unstacking seatter across different clusters.
The complex distribution however may inspire research in many aspects. The tactile clusters exhibited in our dataset may facilitate research in the dynamic manipulation taxonomy to understand human behaviors using tactile signals, not just the hand poses used Feix al. (2016) for grasp taxonomy study. Second, the hand movement is continuous while the touch is not: the subtle changes of hands in the moment contact are hard be reflected in the images and spacial displacement while the discrete nature of the tactile signal can capture distinct information of finger gait and the synergy during hand-object interaction like opening and closing scissors represented in Fig [ee, which can serve prior for complex manipulation skill learning. Also, the visual modality between different tasks often has large difference, but the similarity presented in touch (c.g.Bow) Unstacking and Peg-in- Peglen-hole depicted in Fig ())) can provide basis for the generalization of task-level strategies in downstream tasks.
VISUAL-TACTILL DEXTEROUS MANIPULATION BENCHMARK
4.1 VISUAL-TACTILE MANIPULATION PLATFORM AND TASKS
In this section, we introduce our disual-tactile dexterous manipulation platform based on Isaac Gym environment Makoviychuk al. (202I). which encompasses six complex tasks with Shadow Hand Sharma et al. (201)) See B.I for more details.
BottleCap Turning. Agents required to rotate the bottle cap counterclockwise by full circle. 10 objects and 5 unseen objects from WWang et al. (2023) are prepared for skill learning and for evaluation. The task is seen in human hand manipulation data.
4

---

Published as a conference paper at ICLR 2025
(a) Manipulatio Platform dexterous manipulationn Insis)
Manipolation Policy Learning PPO
Propriocuption
PP
BosteCap Turning
Fanze Senowing
Lever Sikding
Cyiee
Representation Module Mati
Table
Manipulation Policy
(b) Representazion Module (18 benchmarking models)
Pretrained Models
Non- pretrained Models
VTlliinPPadrrin
CLIPT V. Molnort CLIP
Muleen
T-Pettiin
Roazi-T
MVV-VT
N_MVP
Basc
(Trined on our datnact)
(Exising precained model and Deii variamn))
Actionai
Figure 2: Overview of benchmark. (a) shows the six tasks of manipulation platform (Sec (ecc4.1) (b) lists the 18 pretrained and non-pretrained models in our benchmark 4.2); (c) is the policy learning framework combines proprioceptive inputs and perception representations guide actions within an MDP, with skills learned via PPO (.ec4.3)
Faucet Screwing, Similar to the BottleCap Turing, agents are required to rotate the tap handle one full circle clockwise. The task is unseen in human hand manipulation data. Five faucet models with rotational axis perpendicular to the ground from the SAPIEN dataset Xiang et al. (2220) for manipulation policy learning and generate 10 objects by scaling the taps using two different scaling factors for evaluation.
Lever Sliding. Agents focus on actions to separate a long hole and an axe, which requires the fingers to press or pinch the object and use wrist motion to slide the axis out. We create 5 CAD models based everyday objects and generate URDF (Unified Robot Description Format) files for them. We obtain 10 test objects by scaling the CAD models to two different sizes.
Table Reorientation. This task enables the dexterous hand to learn how to rotate an object on the table without toppling it. Unlike the previous rotation task, the object this task is not fixed. We use 10 objects for training and 5 objects for testing from the YCB dataset Calli et al. (015)).
In-hand Reorientation. Agents are designed to rotate an object within the fingers while keeping the palm facing upward, aiming to rotate the object anticlockwise over half a circle without deviating beyond specified threshold. 10 objects from the YCB dataset Calli er al. (2015) are used for training, and objects are used for evaluation.
Bimanual Hand-over. This a bimanual manipulation task in which one hand throws an object while the other catches it, ensuring the object does not drop. This task primarily trains the dexterous hand in bimanual coordination. We use 5 objects for training and 3 objects for testing from the YCB dataset Calli al. (2015).
Acquisition of Visual and Tactile Sensing. Isaac Gym Makoviychuk et al. (2021) provides users with a rich set of APIS for visual and force sensors. We position ego-centric cameras to capture shown in the manipulation platform of 2 Similar to the collection of human data, we arrange 20 force sensors on the dexterous hand corresponding to the tactile glove shown in the right column of Fig
RGB images of dexterous manipulation in the simulation
4.2 VISUAL-TACTHIL PRETRAINED MODELS
We aim study whether human visual-tactile priors can enhance robotic dexterous manipulation skill learning. We benchmark various pretrained and non-pretrained methods for visual and/or tactile representation. More details for these methods in B.2 of the supplementary material.
Visual-tactile Fusion by Joint Pretraining, Inspired by the masking reconstruction pretraining He et al. (2022); Radosavovic et al. (202)): ) Liu et al. (2024), we construct a benchmark method VT- JointPretrain fuse the visual and tactile modalities, which consists of fusion encoder and reconstruction decoder. The fusion encoder E, integrates image- -actile pairs (V, C). The RGB image V is divided into patches and transformed into embeddings ū and tactile data C is sliced into
5

---

Published as a conference paper at ICLR 2025
patches and converted into embeddings č. We randomly mask patches in certain proportion and use a transformer encoder to fuse the visible patch embeddings along with learnable CLS token, producing fused representation {h,, be: hcr.s). J. The CLS token aggregates the latent features of the visual-tactile pair and serves as the input to the downstream task network. For the reconstruction decoder, (h,, he) and the masked tokens are input into the transformer decoder. We then use Re, to project i to the reconstructed image V and Re, to map each reconstructed tactile patch é; to their reconstructed unit C₁ The network is trained with the discrepancy of the reconstructed images and tactile signals with the original ones. After convergence, the encoder Eg is leveraged for downstream tasks and the token hCLS is input to policy network. Based on the method, we create two single-modality pretrained structures by removing either modality, resulting in V-Pretrain and T-Pretrain. Both methods are trained using our dataset. Also, we construct another baseline V-Pretrain+ T-Pretrain by concatenating the features of V-Pretrain and T-Prefrain.
Other Pretrained Models. We collect 5 pretrained models: CLIP Radford et al. (202T). R3M Nair al. (2022). MVFRadosavovic et al. (2023). VoltronKaramcchtti et al. (2023). ResNet &He al. (2016). We employ their open-source visual encoders to the RL framework, named as V.CIIP, R3M, V MVV. Voltron and V Resnet. We further extend the five methods by adding tactile modality feature extraction with an MLP, concatenating the visual and tactile features to form the visual-tactile representation. We name their variants as V_CLIP+T, R3M+T, V M. V_Voltro++T and V V_Resnet+T.
Non-pretrained Methods. We use the network structure of ResNet18 to extract image features and an MLP to extract tactile features, forming three non- con-preta baseline models. T uses only tactile input, uses only image input, and V+T uses both image and tactile information, concatenating their features to form the visual-tactile representation. Additionally, we prepare Base baseline model that uses only the proprioceptive information of the dexterous hand as input.
4.3 MANIPULATTOON POLICY LEARNING
We model the dexterous manipulation task Markov Decision Process (MDP), defined by tuple: (S, A. T, R, 7) We use the PPO algorithm Schulman et al. (2017) to make the agent learn manipulation skills. The architecture is shown in the right column of Fig We define the state as S= < Ma()). (P) and the action and policy as a = (S) Ma is representation module, which can be the pretrained encoders above or non- pretrriae ones. For the pretrained encoders, Ma is frozen during policy training. MM takes the ego-centric RGB image Vsim or the binarized tactile signals Cssm as input and generates the perceptual representation h. Vsim captured by the ego-centric camera in the simulator and Csim is the binarized result of the signal obtained from the force sensors, with a tactile threshold set to 0.01 across all tasks. P represents the proprioceptive information of the dexterous hand, which includes the joint angles and joint velocities of the hand. See B.3 and B.4 for more details about RL training for each task.
5 BENCHMARKING STUDY
Table 2: Success rate (%) for our method and baselines with different modalities in all tasks.
Tasks
Split
Base
T-Pretrain
V-Pretrain
VT-JointPretrain
Seen
7.2
83.7=
BottleCap Turning
Unseen
36.8* 9.4
68.6t 5.6
58.5=14.2
81.3+ 0.5
Seen
49.0+12.0
60.0+12.3
$7.9= 7.0
80.1 1.8
Faucet Screwing
Unseen
43.9=10.5
51.9+12.1
51.8= 6,5
73.6xt 2.1
Seen
5.8 + 4.4
53, 1323.,
27.9=14.9
89.3+ 3.6
Lever Sliding
Unscea
2.2 # 1.9
48.3+20.7
20.5=10.9
79.62t
Seen
51.84 6.3
68.8+ 1.8
74.2+ 9.4
85.0# 1.4
Table Reorientation
Unseen
46.7
69.8t
69. (9.2=10.0
84.6t
42,14
$5.74
In-hand Reoricntation
Unseen
33.74 1.6
35.84:
$3.5e 1.7
55.1 2.7
Seen
35.0210.2
37.7=10.9
45.52
Bimanual Hand-over
Unseen
3.3 2 1.4
20.74: 6.0
23.14 7.0
26.62 1.9
Soen
34.8t 5.8
55.74 8.8
$4.0z 8.5
72.22 2.4
Task Mean
Unseen
27.84 5.0
49.24 9.4
46.1a 8.1
66.82 2.7

---

Published as a conference paper at ICLR 2025
DotteCap Tuming
Fancrt Screwing
Lever Sliding
80
50
40
500
1030
1500
2000
502
1003
1940
2000
500
1930
1760
200
Table Recrientatio
Ia aa-Band Reorientaton
Eimanual Haan over
Stociss
20
lteratiin
VTT-Jointrrettiin
V-Pretrain
T-Pretrain
Base
Figure 3: The training process of different modalities in all tasks.
In this section, first investigate the effect of the tactile information and compare different pretrained and non prettrrrad methods. Then, we study the robustness of methods under different viewpoints, analyze the impact of the tactile threshold of the pretraining and RI., and investigate the impact of the tactile noise on the policy learning and deployment, which are important aspects for exploiting the tactile modality. In all experiments, the metric for all tasks is Success Rate (%). If the dexterous hand achieves the goals in one episode, it is considered successful. All experiments use random seeds, and the results are tested 100 times for seen and unseen object. All baselines are described in See 4.2
5.1 EFFECTIVENESS OF TACTILE INFORMATION.
We conduct five groups of experiments all six tasks. The training process is shown in Fig 3igBia and the evaluation results are listed in Tab With the tactile informaiton, T-Pretrain demonstrates significant improvements over the Base method. Furthermore, compared to the V-Pretrain methods that utilize only vision, the VTTJoinntretraal methods that incorporate extra tactile modality features show considerably better performance. The results show that incorporating tactile information significantly improves the learning of manipulation skills, even across different types of tasks.
5.2 BENCHMARKING PRETRAINED AND NON-PRETRAINEL METHODS.
Table 3: Success rate (%) for benchmarking different methods.
Method
Modality
Pretrain
Joint peetrain
Unseon
50.8:
47.0s
V
X
24.0= 3.0
22.2s 2.9
V-T
Vet
23.6+ 2.6
19.3+ 2.9
V.MVP
35.2s 2.7
29.4s 2.4
40.0=
31.75
V-R3M
37.0= 0.7
26.2= 2.1
V-CLIP
61.32
49.42 1.8
VeResNet
54.1a
46.82 0.6
V-MVP+T
vet
38.5+ 2.5
35.3+ 2.3
V-Volton=T
39
34.72
V.R3M+T
VSI
X
38.9 2.1
31.0s 1.5
V-CLIP+T
65.4t
55.9=
V-ResNet+T
V4I
55.42
44.15 1.8
V.PretrainwT-Pretrain
62,6t 6.3
53.3#
VI-Jointreetrai
V+t
74.3t
0.6
65.72
0.7
We set up three groups of methods for benchmarking in all six tasks, shown in Tab The first non-pretrained models, taking only tactile (T), only vision (V), or vision and tactile (V+T) as input. The second group is the vision-pretraing
models used in computer vision and robotics, which MVP (V (VM))P, Voltron(V_ (V_Voltron) R3M(V R3MM, CLIP(V -CLIPP, and ResNet(V (essNeet__ReNett. Their features are concatenated with tactile features extracted by the method T to generate visual- tactile pretrained methods (V MVP+T, V Voltron+T, V R3M+T, V CLIP+T, V ResNet+T). The
7

---

Published as a conference paper at ICLR 2025
last group is the visual-tactile concatenation method (V-Pretrein-T-PPeetrain and fusion method (VT-JointPretrain)) pretrained with our dataset.
It can be observed that, regardless of whether pretrained or non-pretrained methods are used, most approaches relying solely on visual inputs show improved performance when augmented with tactile information. However, directly concatenating tactile features is less effective than joint training with VT-JointPretrain. Similarly, combining separately pretrained visual and tactile models (V- Pretrain+ P-Prtrriin) also yields suboptimal results. In contrast, VT-JointPretrain effectively fuses visual and tactile signals by masking input modalities and recovering the original signals, leading to significant performance gains, Additionally, joint pretraining substantially reduces result variance across different tasks and training random seeds. indicating improved consistency and robustness.
5.3 ANALYSIS OF VISUAL AND TACTILE MODALITIES
In this section, we further study the characteristics of the binary tactile signals of our dataset using the tactlle fusion method VT-JointPretrain in the BottleCap Turning task.
1). Viewpoint adaptability for RL. We study the robustness to changes in perspective by further testing the camera mounted the robot (4-arm) and third-person view (*-3rd). *-ego refers to the perspective used in previous experiments. Thbosshows the average success rates for the success rates under each perspective for seen and unseen objects. Adding tactile information helps to improve the average success rate and also reduces the variations due to perspectives.
2). Robustness tactile thresholds for RL. We study the robustness different tactile thresholds. We set tactile thresholds to 0.5N (+-50) and IN (*-100). 0.01N(*-1) is the previous threshold setting, Similarly, Tab shows the average success rates for the success rates of different thresholds. The performance of both tactile-only and visual-tactile methods declines as the threshold increases.
Table 4: Success rate (%) of different visual and tactile modalities.
Vision viewpoints
Tactile thresbolds
Method
Seen
Unseen
V-ogo (V-Pretrain)
ego-centric arm third view arm
70.8 7.2
18.5
V-arm
18.2116.9
58.5217.0
V-3rd
16.2
37.4218.8
VT.amm
DOIIN
78.0+ 4.9
73.3+ 7.0
VT-3rrd
third view
0.0IN
82.4# 2.3
79.4t 4.2
T-1 (T-Pretrain)
0.01N
75.42 2.9
68.62 5.6
T:50
0.5N
60.82 9.1
48.8a14.4
1:100
LON
64.3# 5.9
9.7
VT-50
ego-centric
0.5N
82.9+ 1.2
80.6+ 0.2
VT-100
ego-oentric
I.ON
74.4# 4.8
65.0# 8.8
VT-JoinPretrrin
ego-centric
0.0IN
83.7+ 0.9
81.3t 0.5
3). Impact of tactile threshold on pretraining. addition to the threshold of 0.2V discussed in the paper, we conduct two additional experiment settings: pretrain VT-JointPretrain using threshold voltages of 0.55V and 0.75V for real tactile signals to study the sensitivity of the joint pretraining method to the tactile threshold. According the calibration data from the tactile glove, described in I of the supplementary materials, these thresholds correspond to forces of 0.05 N, 0.5 N. and N. respectively. We use these pretrained models to conduct RL training with three different simulation force thresholds (0.01 N, 0.5 N. and 1.0 presents the results of these experimentsk. The success rate of the policy only declines significantly when the threshold difference between real data pretraining and simulation RL training is as large as 20-fold (0.05 N vs. 1.0 N). These results demonstrate that using binary tactile signal is robust to the thresholds used for real data, and mismatches between the thresholds in real data and simulation.
4). Impact of tactile noise for policy learning and deployment. To assess the impact of tactile noise on downstream tasks, we add Gaussian noise with standard deviations of 0.01N, O. and 1.ON to tactile signals, keeping the binarization threshold at 0.OIN. We train the policy without noise and test it under tactile noise (v1) and hysteresis noise (v2) with thresholds ranging from 0.0IN to 0.5N. Additionally, we train a policy with tactile signals augmented by Gaussian noise (a = 0.1) and test it with varying noise levels (v3). The results (Tablo) show that models trained without noise sensitive to tactile noise during testing, while applying a hysteresis threshold significantly reduces this effect, suggesting that pretrained models using binary tactile signals can be effectively deployed
8

---

Published as a conference paper at ICLR 2025
Table 5; Tactile threshold setting for pertaining
Table 6: Tactile noise setting for RL
threshold (N) triin
-00N0
o=
a=IN
0.0.N
0.5N
LON
Seen
Unseen
Seen
Unsesn
Seen
Unseen
Unscon
Sczn
Uneoea
Soca
62.7s
370t
33.7=
34.4 10.2
33.5e
0.29 (005K)
0.75V (LAN))
0.55V (0.58)
53 0.9
80.52 6.7
$1.64 3.5
1.13
6.3
0.5
82.0a 1.2
8.13551
82
80
94.2330
M448
82.52..3
80.52
65.0=
81.62 4.2
77.88
v3 84.0 4.6 81,7
8384 14
81.21 0.8
79.11
4.2
84.62 7.8
79.9
41.24 10.1
86.82 5.5
83.8s 8.1
10.72 4.8
Table 7: Success rate (%) for deploying single modality after joint pretraining RL
Method
V-Pretriin
VT-JonnPPeerrii-MMaTTT
T-Pretriin
VT-JoindPretrainMaare
VI-JointPretrrin
Scen
70.8= 7.2
73.3+ 2.9
75.4 2.9
72.6 2.7
83.7# 0.9
Unseen
58.5214.2
68.62 5.6
81.32
in real-world scenarios by adjusting the binarization schedule. Furthermore, introducing noise during RL training not only improves performance but also enhances robustness to noise. Notice that the average force is about ION to I5N for different tasks manipulation. The setting of thresholds and stand deviations from 0.01N to IN spans large range.
5). Deploy single modality to RL after joint pretraining. In this part, we investigate whether only using one modality after joint pretraining can benefit from the modality fusion. Therefore, we mask the input visual and tactile modality of the joint pretrained model separately, denoted as "T.JoinnPretraan--MsskY and "T-JointPretrain-Maak
In Tab comparing VT-JointPretrain- MaskT with V-Pretrain, the fusion algorithm demonstrates superior performance in downstream tasks when only visual input is provided. indicating the tactile information contributes to the visual feature extraction during joint pretraining. However, similar distillation is not observed when masking visual modality. D.-JoinPPretrainMaaat
even sees a little drop in the success rate.
5.4 REAL- EXPERIMENTS
The real-world experiment employs a Shadow Hand (Sharma et al. (2014) and an Azure Kinect c To collect tactile data, 20 piczoresistive sensors are attached across the Shadow Hand. The hardware and control algorithm are integrated via ROS, enabling the collection of environmental states and observations. To address the Sim2Real gap, a teacher policy is first trained using domain randomization TTbbin al.) (2017) on joint angles, velocities, and actions, with Gaussian noise (standard deviation 0.1) added to tactile forces before binarization during RL training. This policy is then distilled into a student policy using Dagger Ross et al. (201 1), incorporating augmented visual inputs. The tactile sensors are evaluated on the Shadow Hand, with 0.2V set as the binarization threshold. Further details and examples are provided in of the supplementary material.
(a)
(b)
(c)
(d)
(c)
Figure 4: Real- world experiments. a) Hardware: 1-Shadow Hand, 2-Azure Kinect camera, 3- 3-Tactile
collection board. 4-Bottle. Deployment to real world: (b-f) BottleCap Turning, Faucet Screwing, Lever Sliding, Table Reorientation, In-hand Reorientation.
5.5 DISCUSSION, LIMITATION AND FUTURE WORK
Downstream Task Adaptation. Among the six tasks, only the BottleCap Turing task is in the dataset and the others are unseen tasks. The hand over task is distinct with two hands. Despite the difference, the model pre-trained with our visual-tactile data VT-JointPretrain performs exceptionally well on all tasks. This indicates that the visual-tactile pretraining approach not only improves performance on known tasks but also exhibits adaptability in unknown tasks, particularly in complex manipulation requiring tactile feedback. As shown in Tab for the first four tasks where visual information of the

---

Published as a conference paper at ICLR 2025
objects is prone to blocked by the robotic hands, tactile information significantly enhances model performance. However, in the last two tasks, where visual information is relatively complete, the advantages of visual imput become more pronounced. The results demonstrates the complementary nature of visual and tactile information and the joint pretraining via MAE exploits the nature.
Tactile Modality. In the tactile research community, various tactile sensors have been developed and many of them can capture very high dimensional and dense information like images, e.g. GelSight Yuan et al. (017). Pretraining these dense tactile patches and aligning them with images has been demonstrated effective for downstream tasks like physical reasoning Yu et al. (2024b). However, they have not been evaluated for dexterous manipulation with complex dynamics and coordination between fingers. Though not able to capture highly precise force information, tactile signals carry important information about the coordination between fingers during complex interactions with objects. We attribute the effectiveness of tactile modality demonstrated in the results to the embedded coordination information and also complementing images containing occlusions.
Tactile Pretraining. In pretraining, as discussed in 5) of Sec vision information fails to transfer effectively to the tactile modality after joint learning. Unlike the continuity of images and poses, tactile signals are inherently discrete, reflecting either contact or no contact. Effectively leveraging this discrete nature remains an open challenge. Our initial exploration focuses on binary tactile signals; future work must address signals with pressure values, shear forces, sensor sensitivity variations, and the simulation- to-rality gap, aiming for policies that generalize across tasks. Fully exploiting tactile signals also requires high-speed, high-fidelity simulations of complex object-sensor interactions-2 major challenge for the field. Additionally, for analysis, the visual-tactile pretraining method VT-JointPretrain is trained from scratch on our dataset. Integrating large vision-language models with tactile data presents a promising avenue for enhancing performance.
Cross-Modal Regularization for Dexterous Manipulation. Integrating high-precision tactile modalities into robotic systems often increases cost and complexity. However, as shown in Tab. 7 our visual-tactile fusion approach demonstrates that even sparse tactile data can enhance visual representations by serving as a regularization mechanism. This underscores the potential of tactile signals to boost efficiency and robustness in multimodal learning. Future research could explore distilling tactile into visual modalities, achieving similar performance with reduced reliance on high-precisse tactile hardware-lowering deployment costs without compromising accuracy.
Adaptation to Various Tactile Sensors. Learning manipulation policy with different tactile sensors is a major challenge in robotics due to the diverse physical principles involved, such as capacitive, piezoresistive, optical sensing. Though the pretraining methods evaluated in the work cannot deal with the original tactile signals, in principle they can work with the binarized tactile signals proper preprocessing thresholding step is applied to the original ones. With the thresholding, the differences of tactile sensors become a black box to these pretraining methods, and working with binarized signals mitigates issues like noise, scaling, and sim2real transfer.
Robustness to Tactile Noise. Tactile sensors can produce varying levels of noise when interacting with objects. MAE-bbaed pretraining methods adopted in this work can mitigate this issue by masking the tactile signals during pretraining. The models are forced to learn how to infer missing information, thereby enhancing its robustness to noise and data loss. The denoising effect of MAE is also verified in Yuda Zou (2024), where MAF is used to denoising the key-point labeling noise.
6 CONCLUSION
In this study, we present the first visual-tactile dataset for complex manipulation skill learning and introduce benchmark with six challenging tasks and a reinforcement maaming-baser framework. We evaluate 18 pretraining and non- non-pretraining methods to explore the effectiveness of different modalities and pretraining strategies for dexterous manipulation, conducting extensive experiments on various viewpoints, tactile thresholds, and noise levels. The results reveal key insights: 1) Incorporating binary, sparse tactile information significantly enhances complex dexterous skill learning, especially when combined with vision pretraining. 2) Joint visual-tactile pretraining shows strong adaptability to unseen tasks and achieves robust performance across all tasks. 3) Binary tactile signals exhibit high robustness to threshold variations and tactile noise, helping mitigate the sim-to-real gap. Our work provides the support for advancing dexterous manipulation through visual and tactile modalities. We will release the benchmark and hope it can help find better solutions.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENT
This work was supported by the National Natural Science Foundation of China (Grant No. 62088101, 62233013) and the Key Research and Development Program of Zhejiang Province (Grant No. 2025C01072). We sincerely appreciate Peisen Xu and Zhou Lu for their assistance in data collection, as well as Siyun Wang and Jiaying Chen for their support in conducting the physical experiments.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint FrXXiv23033.8774: 2023.
Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Ku- mar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. arXiv preprint arXiv:2309.01918 2023.
Aude Billard and Danica Kragic. Trends and challenges in robot manipulation. Science, 364(6446); eaat8414, 2019.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068// 2022.
Roberto Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, Edward H. Adelson, and Sergey Levine. More than a feeling: Learning grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters, 3(4)33300-3307, 2018. doi: 10.1 1..11099/RA.200 201.2827777... 2852779.
Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 International Conference on Advanced Robotics (ICAR), pp. 510-517, 2015. doi: 10. 109//AR/ 2015.7251504
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint woXiv:/51..03012. 2015.
Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes. Science Robotics, 8(84): cadc9244, 2023. doi: 10. 1 22tb:sccrobboiics.dd:9444 URL https::/ /wwww science..rog..o
doi Boi/abs/Dns /00 ott../1.e//eerrrrric.o...o..
Yizhou Chen, Mark Van der Merwe, Andrea Sipos, and Nima Fazeli. Visuo-tactile transform- ers for manipulation. In 6th Annual Conference on Robot Learning, 2022. URL https
//openreview.coo new.:nn.e/eeeee www.nee/foruur wow.ooiiddduguiiiii..eo. JQQSTgdQ85F
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 151 2015.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference computer vision (ECCV), pp. 720-736, 2018
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. arXiv preprint arXiv: 1910.1 1215, 2019.
Yiming Dou, Fengyu Yang, Yi Liu, Antonio Loquercio, and Andrew Owens. Tactile-augee augmented radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattrnn Recognition (CVPR), pp. 26529-26539, June 2024.

---

Published as a conference paper at ICLR 2025
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzzan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. In International Conference Machine Learning, pp. 8469-8468999 8488. PMLR, 2023.
Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Chenxi Wang, Junbo Wang. Haoyi Zhu, and Cewu Lu. Rh20t: A comprehensive robotic dataset for learnnng diverse skills in one-shot. In Towards Generalist Robots: Learning Puradigms for Scalable Skill Acquisition@ CoRL2023, 2023.
Thomas Feix, Javier Romero, Seinz--Bodo Schmiedmayer, Aaron M. Dollar, and Danica Kragic. The grasp taxonomy of human grasp types. IEEE Transactions on Human-Machine Systems, 46(1): 66-77, 2016. doi: 10.1 10.1109/TMMS.. 2015.247677.
Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel. Multimodal masked autoencoders learn transferable representations, arXiv preprint arXiv:2205 2022.
Abraham George, Selam Gano, Pranav Katragadda, and Amir Barati Farimani. Visuo-tactile pretrain- ing for cable plugging. arXiv preprint arXiv:2403. 1898, 2024.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Irmak Guzey, Yinlong Dai, Ben Evans, Soumith Chintala, and Lerrel Pinto. See to touch: Learning tactile dexterity through visual incentives, 2023.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR 066), volume 2. pp. 1735-1742. IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Piaopiao Jin, Bidan Huang, Wang Wei Lec, Tiefeng Li, and Wei Yang. Visual-force-tactiee fusion for gentle intricate insertion tasks. IEEE Robotics and Automation Letters, 483):483044877,08 2024. doi: 109/
Siddharth Karamcheti, Suraj Nair, Annie S Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. Robotics: Science and Systems (RSS), 2023.
Justin Kerr, Huang Huang, Albert Wilcox, Ryan Hoque, Jeffrey Ichnowski, Roberto Calandra, and Ken Goldberg. Self-supervised visuo-tactile pretraining to locate and follow garment features. Robotics: Science and Systems (RSS), 2023.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna. Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama. Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403. /2945, 2024.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
12

---

Published as a conference paper at ICLR 2025
Mike Lambeta, Po- Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, Dinesh Jayaraman, and Roberto Calandra. Digit: A novel design for low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3):3838-3845 2020. doi: 10. 1090009 202.....22022.77255.7
Michelle A Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Learning multimodal representations for contact-rich tasks. IEEE Transactions on Robotics, 36(3):582-55) 2020.
Qingtao Liu, Qi Ye, Zhengnan Sun, Yu Cui, Gaofeng Li, and Jiming Chen. Masked visual-tactile pre-training for robot manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp.
2024. doi: 10.1 1..109//CAA551472224..0000
Yecheng Jason Ma. William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. International Conference on Machine Learning (ICML), 2023a.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards universal visual reward and representation via value-implicit pre-training. In The Fleventh International Conference on Learning Representations, 2023b. URI. https
//ppenreview net/ www.mmiiddoX7o2222200. iepers/sieee.ntt//roorrrao2oo>d2
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo. Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2 108. 10470, 2021.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. In 6th Annual Conference on Robot Learning, 2022.
URL / net/
http://eeee///orreeeeeeeeeeee forum? 1d=t
Haozhi Qi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik. General in-hand object rotation with vision and touch. In Conference on Robot Learning, pp. 2549-2564. PMLR, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 0748-8763. PMLR, 2021.
Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. Conference on Robot Learning, pp- 416-426 PMLR, 2023.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR JMLR Workshop and Conference Proceedings, 2011.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastin networks: Self-supervised learning from video. In 2018 IEEE international conference robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.
Carmelo Sferrazza, Younggyo Seo, Hao Liu, Youngwoon Lee, and Pieter Abbeel. The power of the senses: Generalizable manipulation from vision and touch through masked multimodal learning, 2023.
13

---

Published as a conference paper at ICLR 2025
Dhruv Sharma, Kirti Tokas, Aviral Puri, and Krishna Sharda. Shadow hand. Journal of Advance Research in Applied Science (ISSN: 2208-2352), 1(1):04 07, Jan. 2014. doi: 10.53555/nnas.w 692. URLhttps: https: ////u/npubbb.//// org/ http//ndexx... php/ /tp/
pttp:/ppaatpccceee//pppp artaaaattiieeeee wvtp//ieiw///eeeeeeee 692
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23 23-30, 2017. doi: 00.1109/ROSS.221..22000
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Rongkai Wang. Yiyang Jing, Chaojie Gu, Shibo He, and Jiming Chen. End-to-end multi-target flexible job shop scheduling with deep reinforcement learning. IEEE Internet of Things Journal, 2024.
Ruicheng Wang, Jialiang Zhang, Jiayi Chen, Yinzhen Xu, Puhao Li, Tengyu Liu, and He Wang. Dexgraspnet: large-scale robotic dexterous grasp dataset for general objects based on simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11359-11366 IEEE, 2023.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
11097-11107, 2020.
Fengyu Yang, Chenyang Ma. Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, and Andrew Owens. Touch and go: Learning from human-collected vision and touch. Advances in Neural Information Processing Systems, 35:8081 15:8081-8105, 8103, 2022.
Kelin Yu, Yunhai Han, Qixian Wang, Vaibhav Saxena, Danfei Xu, and Ye Zhao. Mimictouch: Leveraging multi-modal human tactile demonstrations for contact-rich manipulation. In 8th Annual Conference on Robot Learning, 2024a. URL btetps:///penrevlew.rr htwpppppwwweeeeeeeeeeeeeeeeeeeee
1d=7yMZAUKXa4]
Samson Yu, Kelvin Lin, Anxing Xiao, Jiafei Duan, and Harold Soh. Octopi: Object property
reasoning with large tactile-language models. Robotics: Science and Systems (RSS), 2024b.
Wenzhen Yuan, Siyuan Dong, and Edward H Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017.
Peilin Zhou Zhichao Sun Bo Du Yongchao Xu Yuda Zou, Xin Xiao. Shifted autoencoders for point annotation restoration in object counting. In Proceedings of the European conference on computer vision (ECCV), 2024.
Fan Zhang and Yiannis Demiris. Visual-tactile learning of garment unfolding for robot-assisted dressing. IEEE Robotics and Automation Letters, 8(9):5512 55199 2023. doi: 10.1109/LRA.2023. 3296371.
Tong Zhang, Yingdong Hu, Hanchen Cui, Hang Zhao, and Yang Gao. A universal semantic-ccometric representation for robotic manipulation. arXiv preprint arXiv:2306. 10474, 2023.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pp. 2165-2183. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2025
BottleCap Tuming
Faucet Screwing
Lever Sliding
Table Reorientation
In-hand Reorientation
Bimanual Hand-over
Figure Visualizaiton of manipulation policies on different tasks and objects in the simulation
BottleCap Turning
Faucet Screwing
Lever Sliding
Table Reorientation
In-hand Reorientation
Bimanual Hand-over
Figure 6: Seen and unseen objects in all tasks. The unseen objects of Faucet Screwing and Lever Sliding are generated by scaling the seen objects with two different scaling factors (0.9 and 1.1)
15

---

Published as a conference paper at ICLR 2025
A MORE DETAILS ABOUT THE DATASET
A.1 TACTILE GLOVE
Our glove system in 8 incorporates 20 commercial piezoresistive pressure sensors as lactile sensing units. For enhanced sensitivity, we utilize an 18.3 mm diameter sensor on the thumb tip and palm, while the remaining areas are equipped with 10 mm sensors. Each sensor's s resistance is converted into voltage sig- nals through a conversion module, which are then processed by an STM32 microcontroller. The system transmits these signals at rate of 200 Hz to central computer via serial port for data storage. For calibration, we conducted 15 force tests per sen- using& force/torque (F/T) sensor, applying forces ranging from 0.5 to 7.5 N in 0.5 increments as shown in Fig 7 Notably, the sliding resistors in the conversion module are fine-tuned to ensure consis- tent voltage readings across all sensors for the same force. And this calibration is applied to both the left and right-hand gloves. The calibrated force-voltage function is given by:
2.0
Tactile data
Fitted curve
Voltage (
1.0
0.5
Pressure (N)
Figure 7: Force and voltage testing results
U 0.7216 F 0.5025 + 0.0398
(1)
mm
mm
SIMEOPMERETS
M Micoornbrrllce
Thin Film Pressure Senser
Thin Film Pressure Sensor
AIDMEG Module
Speciflcatonn short tall
eSpectfccttoon Shortt Range: 20g 6kg Dimeesiios: 32mm (Heighh)) 18mm
RIXSDDD
Range: 20g 2kg Pitch: 2.54mm
Resistinnes
vollage Commussinn
Dimensions: 24m (Heightt * 10mm
Modhiee
*Pi Length: 5mm
PPn L.ngth Smm
Figure Hardware specification of data collection and glove force and voltage testing results
A.2 COLLECTION TASKS
Tab shows the task descriptions we collect in our dataset and some demos.
A.3 T-SNE VISUALIZATION
We set 0.2 (the same as the experiment settings in S.c to binarize the tactile voltage signals, converting them into 0-1 contact signals. We perform t-SNE visualization on the binary tactile signals of each trajectory to further investigate the relationship between tactile combination sequences of different operational tasks. Due to the varying length each operation trajectory and the fact that we rarely care about the relationship between the hand and object before the first contact, we only retain the trajectory between the first and last contact frames for each operation trajectory. According to the length statisties in dd), insufficient length, we Toop the tactile state of the trajectory to reach a length of 500 frames. For any trajectories with more than 500 frames, we directly delete the extra frames.
set the length of each trajectory to 500 frames. For trajectories with
16

---

Published as a conference paper at ICLR 2025
Table 8: Tasks descriptions and demos in the dataset
Collected Tasks
Description
# Objects # Trajs
Examples
PickUp
pick up an object and put it down
50
250
BottleCap Turning
open or close the bottle cap
30
300
Rotate in-hand object with the palm facing up.
In-hand Reorientation
20
205
Bowl Unstacking
unstack a bowl from another
28
269
Manipulate the object around the pivot of the hinge.
Articulated Manipulation
10
168
Insert the plug into the socket Remove the plug from the socket
Peg-in-holc
153
Water Pouring
pour the water into the bowl
10
196
manipulate the objects on the table, such as pushing. rotating, flipping. rolling
Table-top Manipulation
10
196
manipulate tools made using the
Scissors Manipulation
principle of levers, such as scissors, pliers, and clamps.
6
106
Pressing button to perform
Pressing
certain functions, such as
10
189
using remote control
B DETAILS OF THE BENCHMARK
B. TASK SPECIFICATIONS
In this section, introduce visual-tactile dexterous manipulation platform. Due to Isaac Makoviychh et al. (2021) (BSD 3-Clause License) supporting GPU acceleration and rendering, construct all the tasks in the Isaac Gym environment.
B.1.1 BOTTLECAP TURNING
Description. This task provides a platform for dexterous hand learn the skill of rotating bottle caps. Unlike grippers, a dexterous hand can utilize the coordination among its fingers, leveraging lateral friction between the fingers and the cap to induce rotation. In this task, we fix all the bottles on the table, preserving only the rotational freedom of the cap around its own Z-axis. Since rotating bottle cap with one hand is also challenging for humans, we are more focused on whether the manipulation policy can learn the coordination between the fingers rotate the cap. The objects used in this task are sourced from the ShapeNet Chang et al. (2013) dataset. Building upon the URDF files of the objects provided in the work Wang al. (2023) (CC BY-NC 4.0 License), with rotational range of to 6.28 radians to the cap. We prepare 10 objects for skill learning and unseen objects for testing. Using various objects for training allows the learned policy to generalize to unseen bottle caps. This task aims to rotate the bottle cap counterclockwise by one full circle. Achieving this goal requires the dexterous hand to act twisting the cap with fingers in one episode repeatedly.
add a joint
Environment settings. We place the bottle at the center of the table at coordinates (0, 0). with its height just touching the table surface. We determine the bottle's height by measuring the furthest.
17

---

Published as a conference paper at ICLR 2025
distance of the bottle mesh along the Z-axis. The robotic arm is positioned at (-0.38, -..02 0 h+00055) where h is the combined height of the table and the bottle. For this task, the camera parameters are set as eye [-0.25, 0.35, 0.5] and lookat [0.02, -0.02, 0.1]. The damping of the bottle cap joint is set to 0, and all other simulation parameters use default values.
B.1.2 FAUCET SCREWING.
Description. Faucet Screwing is similar to BottleCap Turning. However, when constructing the task, we introduce distinct designs to ensure the diversity of the tasks. We add taps with handles to the task so that the hand needs to push the handle with fingers. Additionally, we define the goal of the task as rotating the tap handle one full circle clockwise, necessitating different finger gaits compared with the goal of BottleCap Turning. We choose five faucet models with a rotational axis perpendicular to the ground from the SAPIEN Xiang et al. (2020) (MIT license) dataset for manipulation policy learning and generate 10 test objects by scaling the taps using two different scaling factors (1.1 and 0.9) to evaluate the generalization of the learned policy. We limit the rotation range of the faucet to 0-6.28 radians.
Environment settings. To place all the faucets on the table. we scale, rotate and ad-
Table Hand position for different objects
just the height of different objects. Then, we
Obj ID
Scale
Height
Eulerizyx)
Hand position
rotate the robotic arm using Euler angles (3.14, 0, 1.57) in the ZYX order and place the objects
0.3
0.1
0.144
(0.32, 0.12, 0.72)
1386
0.3
0.07
(0. 0, 3,144)
(0.38, 0.12, 0.74) (0.36, -0.01, 0.75)
2017
0,25
(0, 0,3.14
in different positions for each faucet. The spe- cific settings for these adjustments are detailed Tab We set the damping for all faucet joints to 0.1 and the friction to 0.5. The mass of each faucet is set to 0.15 kg, For this task, the camera parameters are set as eye = [0.4, -0.4, 0.5] and lookat = [0, 0, 0.05].
2095
0,15
0.02
(0, 1.57.
10.37.-0.015. 0.73)
2113
0.3
0.2
(0. 0. 1.57)
(0. 1.57, D)
B.1.3 LEVER SLIDING.
Description. The lever sliding task focuses on actions to separate the long hole and the axe. Without the assistance of arm movement, this task requires the fingers to press or pinch the object and use wrist motion to slide the axis out. Unlike the previous two tasks, which involve complex coordination between fingers, this task is simple coordinated movement where all fingers perform the same motion pattern. However, it places greater demands on the coordination between wrist joint and finger movements. In this task, we select five common long-hole and axis combinations in daily life, including (1) ClampingHanger, (2) DeadboltLock, (3) GasSpring, (4) Padlock, and (5) TelescopingSlide. We create the CAD models and manually generate the corresponding URDF files. Due to the varying lengths of the holes in each model, we uniformly define the completion goal for all models in this task as sliding the axis out by 15 em. We train with the five objects and, like the Faucet Screwing task, create 10 test objects by scaling in two sets (1.1 and 0.9).
Environment settings. In this task, we rotate the robotic arm using Euler angles (3.14, O, 1.57) in the ZYX order and the scale, positions and rotations of the ob- jects and the hand position are shown in To We set the force and velocity limits for the sliding joints to 1, the joint stiffness to 1, damping to 0.5, and joint friction to 1.5. We set the surface friction coefficient of the objects to and their mass to kg. For this task, the camera parameters are set as eye [0.3, -0.25, 0.3] and lookat [0, 0]
Table 10: Object settings and hand positions in the Lever Sliding task.
Obj ID
Scale
Position
Fuler(zys)
Hand position
0.25
(0, U, 0.63)
(1.57, 0,
(0.4, U.1, 0.71) (0.4, 0.08, 0.71)
0.2
(0, D, 0.67)
(-1.57.0, (1.57)
0.12
(0, -0.12, 0.65)
(-1.57.0,0g 0)
(0.4, 0.13, 0,71)
0.15
(0, -0122, 0.65)
[0, 1.57,
(0.38, 0.08, 0.71)
0.15
(0, 0.66)
(1.57,-1.57,0) 0)
0.08, 0.72)
1.4 TABLE REORIENTATION
Description. Sometimes, when we want to pick up an object from the table, we aim to position it appropriately before grasping it. In this task, the dexterous hand attempts to rotate the object on the table while ensuring that it does not topple over. This task rigorously tests the coordination between
18

---

Published as a conference paper at ICLR 2025
the thumb and the other four fingers, the object is prone to tipping due uneven forces. The goal of this task is to achieve half-circle rotation of the object while keeping it stable. To ensure that the skills learned by the dexterous hand generalize to different objects, selected 10 objects from the YCB dataset for training and chose 5 objects for testing.
Environment settings. In this environment, the dexterous hand rotates object on the table. We set the table height at 0.6 meters and positioned the robotic hand at (-0.03, 0.4, 0.72), rotated 180 degrees around the Y-axis. The object is initialized at (0, 0.02, 0.67) and freely falls onto the table after the round begins. All objects have a friction coefficient of 0.8, and their RGB color is set to (204, 204, 0).
1.1.5 IN-HAND REORIENTATION.
Description The in-hand reorientation task involves rotating object with the fingers while keeping the palm facing upward. The goal is to rotate the object anticlockwise over a large angle without deviating beyond specified threshold. This task adds complexity compared to previous ones, as maintaining an upward-facing palm during rotation increases the risk of the object falling if not properly stabilized. Rotations over half circle without exceeding the deviation threshold along the Z-axis from the target position are considered successful. Due to the frequent changes in the object's Z-axis position, the fall condition is relaxed, allowing for a small amount of Z-axis bias. 10 objects from the YCB dataset are used during training, and 5 are used for testing.
Environment settings. Since the table not involved in this task, it has been removed from the simulation environment. The robot arm is initially positioned at coordinates (0, O, 0.5), and the object placed at (0, -0.39, 0.56), which aligns with the center of the robot's palm. The surface friction coefficient between the objects and the environment is set to 0.8 to simulate realistic contact interactions. The masses the objects are randomly assigned within the range of 0.5kg to 1.5kg to introduce variability in weight. For this task, the camera parameters are configured with the eye position at [0.15, 0.15, 0.165] and the look-at at point at [0.1, 0, 0.05).
1.1.6 BIMANUAL HAND-OVER
Description. Unlike previous tasks, this is a bimanual manipulation task designed to teach the dexterous hands the skill of coordination. In this task, one hand throws the object while the other hand needs to catch it and ensure it does not drop. This requires the dexterous hand to coordinate finger joints and wrist joints, allowing the object be thrown in the correct direction and at the appropriate speed. The catching hand must use its various joints to counteract this speed and stabilize the object in its grasp. This task includes objects for training and 5 objects for testing.
Environment settings. The task environment consists of two Shadow Hands and an object. The throwing hand is positioned at (0, O. 0.5), while the catching hand is placed at (0, -1, 0.5) and rotated (0, O, 3. 3.15) radians in the ZYX order. All objects are initialized at (0, -0.39, 0.54), directly above the throwing hand. Each object's mass is set to 0.1 kg. with RGB color set (204, 204, 0). The target position for the object is (0, -0.64, 0.54), and the task is considered successful when the catching hand brings the object close this target position.
B.2 DETAILS REPRESENTATION LEARNING METHODS
B.2.1 VISUAL-TACTILE FUSION MODEL
Inspired by MAE (masked autoencoder) He et al. (022). we construct a benchmark method to fuse visual and tactile modalities, which consists of two parts: the fusion encoder and the reconstruction decoder. Fig 9 shows the framework of our fusion algorithm.
Fusion Encoder The fusion encoder Eg processes image- tactile data pairs (V,C) to output visual-tactile fused representation h, which encompasses three stages: extraction 2), masking (Eq 3. and fusion 4Fq
19

---

Published as a conference paper at ICLR 2025
hcss
Visual Reconstructor Rd
Trans Pansformee Encoder
Transformer
Decoder
Vois
h
1.0...00 000000
000.
Tactile Reconstructor
c 100...
RØg
masked image patch
masked tactile patch
lactile pulch w/o contact
image mask token
visible image token visible tactile token
lactile patch w/ contact
tactile mask token
Figure 9: The network structure of VT-JointPretrain
For extraction, the input RGB image V E #xxx3 divided and flattened to E (P where N, is the number of image patches and (P, P) is the resolution of each patch. And apply a linear projection (-)-) to image patches y and add 2D sine-cosine positional encoding woos to generate image patches embeddings
R NNN xden, The input tactile C {0, 1). is sliced into N. (-) and add ID sine-cosine positional encodings cpos to produce tactile patch embeddings E R N.N.dde, den is the dimensionality of the encoder output. The extraction is described as:
tactile patches, each of which is projected with an MLP layer sp
Φ0(()) DPOs, , (p(c) pos
(2)
For masking, we introduce a modality seedafity-specific masking function denoted as M( 7), where represents the masking ratio for the respective input modality. Utilizing this function, we apply masking to the image (or tactile) patch embeddings with a given mask ratio (or Ye). This results in the output of visible patch embeddings Evis E 1g (or Eois E N, ). We specify the masking process as follows:
Tvis ( (o), Eois M(2 7c)
(3)
For fusion, the main architecture is transformer encoder TransE(()). We concatenate a learnable CLS token with visiable patched embeddings (Tvis: Evis). The visual-tactile fusion representation Rxxd [hoLS, h he) can be defined as:
h Trans ECCLS, Upis: Cwis)
where s = - 7 No (1 Ye) Ne) + and is the dimensionality of the fusion representation. hCLs, hr and he represent latent features of each input.
Reconstruction Decoder Following MAE, the reconstruction decoder Do is responsible for re- constructing masked patches. Firstly, he h and mask tokens m x Nottie are fed into transformer decoder Tr)nsD()). represented as follow:
{0, c] = TransD he, m)
(5)
where i Nexae (or è R Nexeda is the reconstructed image (or tactile) patch embeddings. dae is the dimensionality of the reconstructed patch embeddings.
We use an MIP as the visual reconstructor Ro.. which projects à into V E Hxx The tactile reconstructor Ro, an ensemble of Multi-Layer Perceptrons (MLPs), where each MLP utilized to individually map reconstructed tactile patch é e è {čo, êi,..., C.. to their reconstructed representations C2 E IR. Thus, the reconstructors formulated follows:
V Ro, (0), Ô Ro. (2)
(6)
20

---

Published as a conference paper at ICLR 2025
Table 12: Visual pretrained models used in our benchmark
Pretrained models
Method
Architsctures
Radford 2021
Contrustive learning Hadsell al. 2006
VIT-R/16
R3M Nair et al 2022)
Time contrastive leaming Sermanet ot
2018
Resect-18
MVP Kadosavove
2025
Masked autoencoder ét
ViT-Base
Voltron Krannchee m. A726
Mullimodal masked antoenerdeer
Geng et al.
2022
VTT Smull
ResNet
Supervised learning
Resnet-18
Loss Function Our visual-tactid representation learning algorithm leverages the reconstruction of invisible patches from visible ones, enabling the integration and interaction of information between modalities. This process facilitates the learning of semantic associations both within and between modalities. To evaluate the quality of the reconstruction, we define the following loss function:
L( X MSE(V, V) + Ae- MSE(C, C)
where X and d are the loss weight for vision and tactile modality.
Dataset preprocessing. We first trim the first and last 10% of each operation sequence. retaining the data pairs that include activated tactile signals. Then, we split the dataset into training and test sets with an 8/2 ratio. The images in the dataset have a resolu- tion of 420x240. Before feeding the images into the network, we perform center crop and resize a resolution of 224x224. The tactile data is binarized by threshold (0.2). We feed the processed visual-tactile data pairs into the network for training and testing.
Table 11: The hyperparameters of our visual-tactile fusion model
Training details Tab 11 lists the hyperparameters used dur- ing our training. For parameters not listed, we followed the MAE settings. We trained on our data for 250 epochs in parallel on two NVIDIA GTX 3090 GPUs, which took ap- proximately 25 hours. We selected the model from the 210th epoch for downstream visual- -tatill representation extraction. The other two single-modality pretrained methods using our data (V-Pretrain and T-Pretrain) also used these parameters but selected models from the 170th and 310th epochs, respectively.
Hyperparameters
Models
Image resolution (H. W)
(224, 224)
Num image patches N.
196
Num tactile patches N
20
Patch resolution (P, P)
(16. 16)
Encoder input den
384
Image mask ratio Yo
0.75
Tactile
"e
0.5
Fusion dim
384
Decoder output dim dae
192
Image loss weight
I
Tactile loss weight Ae
10
Learning
1.5e4
B.2.2 OTHER PRETRAINED METHODS
Batch size
64
We utilize the with the corresponding model names being "resnet18", "vitb-mae-egosse" eegnoup" "ViT-B/16", and "v- cond+vit-small+-th- wtt.sssmlllssssss-eephhhhhhhe sth+epoch-400 The models we use are listed in Tab
models from these works directly,
CLIP Radford et al. (2021). The core of the CLIP model is to train model that understands the relationship between image content and textual descriptions through a contrastive learning approach. It is pre-trained on a large-scale dataset of image and text pairs, learning to map visual and text information into a common feature space. We import the CLIP model into our RL framework by calling c clip.load ( ("ViT-B/16", device=device). which is provided on the Github?)
R3M Nair et al. (202)) R3M employs two contrastive learning objectives: time contrastive learning and image-langus temporal alignment. It leverages the temporal structure of videos, aiming to maximize similarity between adjacent frames while contrasting frames that are further apart. Additionally, R3M utilizes language supervision, combining language descriptions with dual-frame contexts to capture task progression. We import the R3M model into our RL framework by calling r3m
load_r3m ( ("resnet18) ), which is provided on the Githut
MVP Radosavovic et al. 2023). MVP employs the concept of masked autoencoders He et al.
2022), where parts of the input image are masked, and then a transformer is used to integrate and
httess//qithbbb.cm/oonn//////
21

---

Published as a conference paper at ICLR 2025
reconstruct the image. We import the MVP model into our RL framework by calling model mvp. load ("vitb-mme--gosopp which is provided on the Github
Voltron Karamcheti et al. (023). Voltron is a framework for language-driven representation learning from human videos and associated captions. It balances between canguage--onnitioon visual reconstruction to help in learning low-level visual patterns, and iisually-grounder language generation, which encodes high-level semantics. We import the MVP model into RI frame- work by calling vol coltoon... cad ("v-coná", devi ce= ceu"cuda" cuda", freeze=True), which provided on the Githut
ResNet18 He al. (016). ResNet18 is trained with ImageNet- 1k Krizhevsky et al. (2012) by supervised learning. We import the ResNet18 model into our RI. framework by calling corchvision models. resnet18 (pretrained-True) When using the model, we
removed the final linear layer.
Feature extraction in RL. For all pretrained models, we retain only the visual feature extractor and freeze all parameters. When using models for the visual modality alone (such as V-CLIP), the extracted visual features are combined with proprioceptive information and feed into the policy network. For models using visual-tactile modalities (such V .CIII+T), an additional tactile feature extracted by an MLP is also input into the network.
B.2.3 NON-PRETRINED METHODS
We the network structure of ResNet18 extract image features and MLP tract tactile features, forming three non-pretrained baseline models: T. V and V+T. We use torchvision www.soom.oool...mmmmmmmmmo.ooo.. res.eelss.seee11.. load the model.
B.3
RL FRAMEWORK
B.3.1 RL MODELING
We model the dexterous manipulation task as a Markov Decision Process (MDP), defined by a tuple: AS, T, , R, 7) S and A represent the state and action space. The policy अe : A maps the state space S to the action space A. T : S x A + S is the transition dynamic. R S x A + R is the the discount factor. Our goal is to maximize the expected discounted reward J(T)= = Ex 1=== 7'r (st. az)] to train policy network To. We use the PPO Schulman et al. (2017) algorithm to make the agent learn manipulation skills.
reward function and y E (0.
State Space. In all tasks, we define the state as S (h + Mo((), Mo(+) take the RGB image Vsim or the binarized tactile signals Csim as input and generate the perceptual representation h. Vsim captured by the ego-centric camera in the simulator and Csim is the binarized result of the signal obtained from the force sensors, with tactile threshold set to 0.01 N across all tasks. P represents the proprioceptive information of the dexterous hand, which includes the joint angles and joint velocities of the hand. We do not incorporate the extra information of the dexterous hand intentionally, primarily to encourage the manipulation strategy to focus more on visual-tactile perceptual signals. Moreover, the joint angles and joint velocities of the hand are readily accessible in real-world environments.
Action Space. In all tasks, utilize the Shadow Hand as the operator, which possesses 24 degrees of freedom. However, four of these joints are actuated through tendons. We immobilize the arm the Shadow Hand, allowing it to complete each task solely with the motion of its fingers. Consequently, in each task, the action E 20
Reward Design of BottleCap Turning.
We define the reward function as:
r == XIrp + Aero + Agra + Aar.
(8)
httos:://ittuub..m///43333/3
http:://giithbb.com/////o/
22

---

Published as a conference paper at ICLR 2025
where the position reward rp min((0oint: 7.0), the velocity reward r. clamp(U)oind: -10, 10), the distance reward ra exp 10d and the success reward T, 5. d represents the sum of the distances from each fingertip to 2 centimeters below the bottle cap. We set t = 0.3, 2 = 1, A = 0.5 and A = 1. Especially, if r. but the tactile sensors are not activated, then X will be set to 0 when calculating the reward, aiming to encourage the dexterous hand to use the part of the tactile sensor that makes contact with the bottle cap to generate positive rotation.
Reward Design of Faucet Screwing. The reward function for the faucet is similar to that for the bottle cap, with the difference being that the contact condition is not considered when calculating the velocity reward. Additionally, we set a target height for each faucet that we aim for the fingers to reach. In the Z-axis direction, these target heights are offset from the faucet positions by [-0.02, -0.01, 0.08, -0.02, -0.01]. d is the sum of the distances from each fingertip to these positions.
Reward Design of Lever Sliding.
We define the reward function as:
r r= X1 + X2n + AgTa
Pjoint and the distance reward rd exp -10d + exp 110d2, We set the target position 8 em away from the center of the object in the direction of the axis coming out. The dj is the sum of the distances between the fingertips and the target position. The d2 is the distance between the palm and the target position. We set X 1, A2 = 2, λ3 = 1.
where the position reward rp 0g joint, the velocity reward
Reward Design of Table Reorientation.
We define the reward function of this task as:
y= A1ra + XeTrot + AgTa + Aare + rb
(10)
where ra = exp 10d3 d2 contains the distance dj between the fingertips and the object along the z-axis and the Euclidean distance d2 to the target in the xy-plane. Trot represents the difference between the current pose and the target pose the object. ra = ||a| la is the action reward and To clamp(w,, 10, 10) is the velocity reward for the z-axis rotation. T 250 is the bonus when
0.25
the agent achieves the goal. We set the A
A2
A3 0.0000 and A4 = 1.
-10
Reward Design of In-hand Reorientation.
The reward function is defined as:
r ArTa - AuTrot + AgTa + Apr
(11)
where the distance reward ra exp 10d, dz). the absolute rotation Frot the action reward r. and the velocity reward r, = clamp(wz, 10. 10). d1 represents the distance between the fingertips and the object along the z-axis and d2 the Fuclidean distance the target in the xy-plane. We set
0.25
A
d2 1, 0.0000 and 04 1.
- 10
Reward Design of Bimanual Hand-over. sented as:
The reward function of Bimanual Hand-over is repre-
-= + Tb
(12)
where ra erp (55+da)) di is the Euclidean distance between the current position and the target position and de is the distance between the current pose and the target pose of the object. Tb 250 is the bonus reward.
B.4 TRAINING DETAILS
For all tasks, set up 200 environments in Isaac Gym Makoviychuk et al. (222l) to collect trajectories in parallel. The maximum episode length 500 for BottleCap Turning and Faucet Screwing. 250 for Lever Sliding and 600 for other three tasks. Upon each reset, the fixed position of the robotic arm is perturbed by cm, each joint angle is reset to random value within -0.05 to 0.05 rad, and the joint velocities are reset to random value within -0.1 to 0.1 rad/s.
In each step, the camera captures a 224x224 image, or the pressure values from 20 tactile sensors binarized using threshold of 0.01 N. This sensory information is fed into the representation
23

---

Published as a conference paper at ICLR 2025
extraction model to obtain the representation information. A linear layer maps this representation to 128-dimensional vector, which is concatenated with another 128-dimensional vector representing proprioceptive information, also mapped through linear layer. This combined information is then fed into the policy network Tg-
Our policy network has hidden layer sizes of [1024, 1024, 512] and uses ELU et al. (2015) as the activation function. The policy network is optimized using the PPO [Schulman et al. (2017) algorithm. Tab 13 shows the hyperparameterss of the PPO algorithm. Although the handover task involves two hands, we do not use aglti- reinforcement learning Wang
Table 13: The hyperparameters of
PPO
Hyperparmmeters
All Tasks
Num mini batches
Nur ryx-epochs
10
Rollout stop
et al. 2024) to train them. Instead, the policy network directly outputs the actions for both hands.
Hidden
1024,512)
Activation
ELU
Clip range
Max grad
We run all experiments for 2000 or 3000 iterations on a device equipped with Intel Xeon Gold 6326 processor and an NVIDIA 3090 GPU. For methods using both vision and tactile input, training the manipulation policy takes approximately 14 or 21 hours. For methods using only vision input, it takes about 10 or 15 hours to train the visual manipulation policy. For methods using only tactile input, training the manipulation policy takes about or 6 hours. However, for methods without pretraining that include vision, training for 2000 iterations can take up to week.
Leaminng rale
3.e.4
Discount
0.96
GAE lambda (ry)
0.95
Init
Desired kl
0,016
Eut-coef
B.5 VISUALIZAITON OF MANIPULATION POLICES
Fig. S visualizes the results of RL policy training for different tasks and objects on our manipulation platform.
B.6 DETAILS OF VIEWPOINT ADAPTABILITY.
Tab 14 represents the 3 different view settings.
Table 14: Viewpoint settings across three tasks
Viewpoint
Fye
ego centric
(-0.25, 0.35, 0.5)
(0.02, -0.02, 0.1)
BottleCap Turning
the third view
(-0.25, 0.0, 0.5)
(0.02, 0.11
(0.25, 0,35,0
(-0.02, -0.02. 0.1)
C REAL WORLD EXPERIEMNTS
C.I SYSTEM SETUP
In order to evaluate the effectiveness of the trained policy, we established system comprising Shadow Hand Sharma et al. (2014) and an Azure Kinect camera collect tactile data, we attach 20 piezoresistive tactile sensors to each part of the Shadow Hand. We calibrate the tactile sensors and collect the tactile signals using the same system, described in [ We set the tactile threshold to 0.2V for the dexterous manipulation platform. In all tasks, the position of the arm remains fixed.
C.2 SIM2REAL PARAMETERS FOR DIFFERENT MODALITIES
We train robust policy in simulation by first applying domain randomization to proprioceptive data and introducing tactile noise. Next, distill the trained policy into student policy handle the image sim2real transfer.
24

---

Published as a conference paper at ICLR 2025
Bottlel
Bottle2
Bottle3
Figure 10: More BottleCap Turning real experiments with different objects.
For proprioceptive information and actions, there are 24 dimensions of joint angles and 24 dimensions of joint velocities. We apply domain randomization Tobin et al. (2017) to both the proprioceptive data and the actions. The parameters are shown in Tab 15
Table 15: Real-world experiment settings
Joint angles Joint vel (48dim)
Actions (20dim)
Noise (mean, std)
[0, 0.002]
[0, 0.05]
Correlated noise (mean,
[0, 0.001
[0, 0.015]
Operation
Additive
Additive
Distribution
Gaussian
Gaussian
Schedule
Linear
Linear
Schodule_stops
40000
40000
For RGB images, we employ various image augmentation techniques and the color of the manipula- tive objects real RGB images to address the sim2real problem:
Pixel randomization. We add uniform noise in the range of [-5, to each pixel indepen- dently and pixels of values outside the range [0, 255] are clipped
Random perspective transformation. We uniformly randomize the offset of the four corners of the source image with the range of [0. 30).
Contrast and lightness randomization. The contrast parameter is randomly generated in the range of [-20, 20) and the lightness parameter is randomly generated in the range of [-30, 30).
Manipulated object color settings. We set different colors for various objects.
For tactile, we introduce Gaussian noise with standard deviations of O. to the tactile signals, keeping the binarization threshold 0.01N v3 in 4) of See 5.3 dose.
C.3 DEPLOYMENT THE ROBUST POLICY IN REAL WORLD
We ROS to obtain the joint angles and velocities of the Shadow Hand, Kinect capture the visual information the scene, and tactile sensors attached to the hand to collect tactile signals. A Kinect camera is manually calibrated match the position and orientation of the simulated camera
25

---

Published as a conference paper at ICLR 2025
and we crop the RGB images with size of 224x224x3, which matches the pixel observation size in the simulation. The tactile sensors capture voltage signals, which are binarized using a 0.2V threshold. All real-world observations are fed into the trained policy, which outputs actions to control the movement of the rohot hand.
Due to hardware limitations,
only conducted single-hand tasks in the real-world environment. and more real experiments about BottleCap Turning can be seen in Fig 10. For demos, see the supplementary video in the zip file.
The qualitative results are shown in Fig g
Tab 16 provides quantitative results of VT-JointPretrain for each task, with two objects per task. A trial considered successful if the goal is achieved within 15 seconds. Trials exceeding the time limit or resulting in object drops are counted as failures, Each object is tested 10 times.
Table 16: Quantitative results of real-world experiments. We conducted 20 trials for each task.
Tasks
BottleCap Tuming
Fancer Screwing
Lever Sliding
Table Reorientation
In-hand Reoricatation
Suce Times
15
13
26


Reviews:
[{"review_id": "5CtsWOmNF6", "reviewer": "Reviewer_xGyB", "summary": "The author introduces a unique dataset for visual-tactile with 10 daily tasks.  A new benchmark with six complex dexterous manipulation tasks is proposed, using a reinforcement learning-based vision-tactile skill learning framework.\nThe authors have key findings that incorporating tactile data boosts task success rates by up to 40%.", "questions": "1. [line 382-387] Please make clear the definition of robustness to the viewpoint. Usually, when we refer to robustness to the viewpoint, we are talking about training policy in one viewpoint and test on another viewpoint.\n\n2. Please specify the meaning and describe a little bit for each subfigure of Figure 1 and 2, instead of just putting on sentence in the caption.\n\n3. [Line 16]. Only 6 tasks have been found in the paper, but the authors mention they have 10 tasks.\n\n4. The real-world experiments use teacher-student to train the policy. However, this is not mentioned in the method section, which may cause confusion.\n\n5. In section 5 benchmarking study, please specify how many seen objects are used and how many unseen objects are used. It would be better for the author to show all the objects that are used for each task.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. I appreciate the authors' efforts to collect a larger visuo-tactile dataset.\n2. Sufficient experiments as well as real experiments.\n3. The visualization for t-SNE is good to show how tactile differ in different tasks.", "weaknesses": "Major Weaknesses:\n1. The overall presentation and low readability.\n\n2. The contribution 2 is questionable. Half of the tasks have been already widely used in previous benchmark tasks[1], like bottlecap turning, faucet screwing, reorientation, and bimanual hand-over. Even though previous work is state-based not using visual-tactile, the authors cannot mention their contribution to creating tasks. \n\n3. The authors do real-world experiments, but there is no table of experiments to show the number, which cannot make any convincing conclusion.\n\n\n[1] Chen, Y., Wu, T., Wang, S., Feng, X., Jiang, J., Lu, Z., ... & Yang, Y. (2022). Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Systems, 35, 5150-5163.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6phBD7nEiZ", "reviewer": "Reviewer_WkCE", "summary": "This paper introduces a new visual-tactile dataset for training and benchmarking dexterous manipulation skills. It includes 10 tasks with 182 objects collected by human operators and provides a comprehensive benchmark of 6 challenging dexterous manipulation tasks. Additionally, the paper presents an extensive evaluation of over 17 methods on the benchmark, showcasing the efficacy of the dataset. Overall, this research is a valuable contribution, offering a well-structured dataset and benchmark for advancing visual-tactile manipulation.", "questions": "1- How well do the learned skills transfer to real-world dexterous manipulation scenarios? Have you evaluated the performance on a real robotic setup, and if not, are there plans for such an evaluation?\n\n2- Could you elaborate on the rationale for using binary force information in the tactile data? Do you have plans to incorporate higher-resolution force data in future versions of the dataset to improve its applicability for tasks requiring fine-grained force control?\n\n3- Have you considered conducting an ablation study with varying mask rates or with only image/tactile modalities masked? If so, could you share preliminary results or insights on how these variations impact the performance of the VT-JointPretrain framework?\n\n4- Given the variety of tasks in the benchmark, how well does the proposed framework generalize across different types of manipulation tasks? Are there specific tasks or object types where it struggles, and if so, what might be the underlying reasons? (further discussion on failures could improve the paper)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a highly diverse visual-tactile dataset for complex dexterous manipulation tasks, enhancing the realism and diversity of training data.\n\n- Detailed analysis and evaluation of the proposed dataset underscore its robustness and potential impact on advancing dexterous manipulation.\n\n- The proposed VT-JointPretrain method is promising, effectively integrating visual and tactile inputs for manipulation tasks.\n\n- The extensive experimental analysis provides insights into the impact of pre-training on dexterous manipulation, with comparisons between different pre-trained and non-pre-trained methods on the benchmark.", "weaknesses": "- The tactile data in the proposed dataset lacks high-resolution force measurements, providing only binary force information per sensor. This limitation may impact tasks requiring precise force control, such as in-hand rotation or object flipping, where the exact contact force is critical.\n\n- An ablation study of the visual-tactile representation learning framework is necessary to understand its efficacy further. Specifically:\n(i) Examining performance variations with different masking rates of image and tactile data would clarify the framework’s robustness.\n(ii) Analyzing performance with only image or only tactile data masked could highlight the independent contributions of each modality.\n\n- Although the authors demonstrate successful deployment of the trained policy on real robots, the evaluation of its real-world performance is limited and lacks detailed analysis. A more rigorous evaluation of real robotic setups would strengthen the claims of real-world applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JCJtM7OK4G", "reviewer": "Reviewer_bKve", "summary": "This paper proposes a vision-tactile dataset and a novel benchmark for evaluating the vision-tactile pre-training methods. The experiments show that the tactile signals can provide guidance for the deployment of the tactile modality.", "questions": "Seen above in weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The experiments demonstrate both simulation and real-world settings.", "weaknesses": "- This MAE-based method is not novel. There is also another paper try to use the MAE to fuse the vision and tactile information: The power of the senses: Generalizable manipulation from vision and touch through masked multimodal learning. So the author at least should compare with it. \n\n- The comparison is not fair. The pre-trained model-based methods only use an MLP to concatenate with tactile information. This operation cannot illustrate that these models cannot leverage the tactile information better.\n\n- Could you give me some intuitions about why VT-Joint training can improve the robustness of viewpoint changing? I think the reason might be these tasks heavily reply on tactile rather than vision.\n\n- Why Voltron performs so bad?\n\n- Where is the real-world experiment results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JlWhDosdEo", "reviewer": "Reviewer_Z9Mf", "summary": "This paper introduces a dataset and benchmark to address the limitations in current robotic manipulation learning frameworks by incorporating both vision and tactile modalities. It collects human manipulation data with 182 objects and 10 daily tasks and uses low-cost tactile gloves to create a large-scale dataset. The dataset is used to study multi-modal pretraining and dexterous robotic manipulation using reinforcement learning. The authors benchmark 17 pretraining and non-pretraining methods to evaluate the effectiveness of visual and tactile inputs in dexterous manipulation tasks.", "questions": "1. This paper only compare the VT-JointPretrain with V-Pretrain, V-Pretrain+T, T-Pretrain, and V+T. How about using V-Pretrain+T-Pretrain separately?\n\nMore questions have been shown in Weakness. I will consider increasing my score if the contribution can be explained more clearly.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper create the first large-scale visual-tactile dataset for dexterous manipulation tasks, addressing a gap in the existing robotic learning literature. The dataset covers a wide variety of tasks and objects, which makes it valuable for generalizing robotic manipulation skills.\n\n2. The use of both vision and tactile information for joint pretraining boosts performance in complex dexterous tasks.\n\n3. The benchmark evaluates various pretraining and non-pretraining methods, which provides a comprehensive understanding of how different modalities affect task performance.", "weaknesses": "1. The contribution to the community is not super clear to me. Currently, what kind of tactile sensor should be used on the Dexterous Hand is still an open question. Although this paper discuss the Gelsight in the section 5.5, there are still a lot of different sensors. Optical tactile sensor [1, 6] has been used for dexterous manipulation, where \"they have not been evaluated for dexterous manipulation with complex dynamics and coordination between fingers\" shown in the paper might be incorrect. Also, magnetic tactile sensor [2, 3], force tactile sensor [4] are used in dexterous manipulation tasks. It's not clear why this type of tactile data is chosen for pre-training. Why does this benchmark could be widely used since it's still an open question?\n\n2. There are some paper uses visual-tactile representations for reinforcement learning [5, 6]. I think compare using the pre-trained with those SOTA online-learning model is helpful (not only using simple ResNet and MLP).\n\n3. Section 5.4 shows that this framework can do Sim2Real and show some impressive demos. However, no quantitative results have been shown in the paper.\n\n4. A new CoRL paper [7] present a method to learn tactile-guided control policy from human hand demonstrations by embedding tactile sensor on the human finger tip, where the idea is similar to collect human-tactile data with tactile glove shown in this paper. What's the difference between these two approaches? It's helpful if analyzing it in related work.\n\n5. In section 5.2, the paper didn't show which task is it.\n\n[1]. Lambeta et al., DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation, 2020\n\n[2]. Bhirangi et al., ReSkin: a versatile, replaceable, low-cost skin for AI research on tactile perception, 2021\n\n[3]. Bhirangi et al., AnySkin: Plug-and-play Skin Sensing for Robotic Touch, 2024\n\n[4]. Yin et al., Rotating without Seeing: Towards In-hand Dexterity through Touch, 2023\n\n[5]. Sferrazza et al., The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning, 2024\n\n[6]. Qi et al., General In-Hand Object Rotation with Vision and Touch, 2023\n\n[7]. Yu et al., MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)