3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing
Authors: Binghao Huang
Yixuan Wang
Xinyi Yang
Yiyue Luo
Yunzhu Li
Venue: CoRL 2024 (2024)
Keywords: Contact-Rich Manipulation
Multi-Modal Perception
Tactile Sensing
Imitation Learning

Abstract: Tactile and visual perception are both crucial for humans to perform fine-grained interactions with their environment. Developing similar multi-modal sensing capabilities for robots can significantly enhance and expand their manipulation skills. This paper introduces **3D-ViTac**, a multi-modal sensing and learning system designed for dexterous bimanual manipulation. Our system features tactile sensors equipped with dense sensing units, each covering an area of 3$mm^2$. These sensors are low-cost and flexible, providing detailed and extensive coverage of physical contacts, effectively complementing visual information. To integrate tactile and visual data, we fuse them into a unified 3D representation space that preserves their 3D structures and spatial relationships. The multi-modal representation can then be coupled with diffusion policies for imitation learning. Through concrete hardware experiments, we demonstrate that even low-cost robots can perform precise manipulations and significantly outperform vision-only policies, particularly in safe interactions with fragile items and executing long-horizon tasks involving in-hand manipulation. Our project page is available at https://binghao-huang.github.io/3D-ViTac/.

3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing
Binghao Huang!
Yixuan Wang!
Xinyi Yang²
Viyue Luo
Yunzhu Lit
Columbia Univerisity!
University of Illinois Urbana- Champaigmm
University of Washington 3
Abstract: Tactile and visual perception are both crucial for humans to perform fine-grained interactions with their environment, Developing similar multi-modal sensing capabilities for robots can significantly enhance and expand their ma- nipulation skills. This paper introduces 3D- 3D-ViTac, multi-modee sensing and learning system designed for dexterous bimanual manipulation. Our system fea- tures tactile sensors equipped with dense sensing units, each covering an area of 3mm²². These sensors are low-cost and flexible, providing detailed and extensive coverage of physical contacts, effectively complementing visual information. To integrate tactile and visual data, we fuse them into a unified 3D representation space that preserves their 3D structures and spatial relationships. The multi-modae representation then be coupled with diffusion policies for imitation learn- ing. Through concrete hardware experiments, we demonstrate that even low-cost robots can perform precise manipulations and significantly outperform vision-only policies, particularly in safe interactions with fragile items and executing long- horizon tasks involving in-hand manipulation. Our project page available at github. It//D-ViTac//
Keywords: Contact-Riee Manipulation, Multi-Modal Perception, Tactile Sensing,
Imitation Learning
(i) Fragile Objects Grusping
(ii Grasping under Occlusion
3D Tactile Points
hand)
(Right
Tactile Signals
Tactile Signals
In-Hand Adjustment
Rimanual Cooperation
Figure I: We propose 3D- ViTac a multi-modal sensing and lcarning system for dexterous bimanual manipula- tion. This system features flexible, scalable, low-cost tactile sensors, each finger equipped with 16 x 16 sensor array. To demonstrate the capabilities of our system in performing precise manipulations, we showcase four tasks that utilize the force-related and in-hand position information provided by the tactile sensors.
Introduction
Humans heavily rely on both visual and tactile sensing to perform everyday manipulation tasks. Consider grasping egg grape: we start by visually locating the object, and then extract more information from tactile interactions with the object to determine the appropriate amount of force to apply. When eating with spoon, our eyes estimate the global position and geometric information of the spoon, while our sense of touch provides detailed contact information during interactions with the
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

food. Vision and touch complement each other. enhancing our interaction with the environment and significantly increasing our flexibility and robustness, especially in tasks involving large occlusions and in-hand manipulation.
However, building such a multi-modal platform for robots brings two major challenges: (i) Tactile Hardware and System. Due to requirements for minimal range and viewing directions, many existing optical tactile sensors can be either overly bulky or overly rigid for fine- mee-grained manipulation or tasks that require compliant interactions [1-3]. Many commercial tactile sensors can also be expensive due to customized manufacturing and electronic components [4, 5], while some low-cost tactile sensors are usually too sparse to convey effective information [2. 6-8]. (ii) Distinct Nature of Tactile and Visual Modalities. Tactile signals are typically local and physical, while visual data are more global and semantic. This disparity poses significant challenges for models to process and interpret the data effectively. Successfully fusing these distinct modalities requires careful design of the tactile sensor and the learning algorithm.
To address these challenges, we present 3D-ViTac, a novel multi-modas sensing and learning system for contact-rich manipulation tasks. (i) For the tactile sensing hardware system, rather than using existing optical tactile sensors [2. 5, 9, 10], we propose an alternative dense, flexible tactile sensor array that covers larger area of the robot end-effector. Our tactile sensors, inspired by the STAG glove [11]. are cost-effective, flexible, and capable of producing stable continuous signals during manipulation. As illustrated in Fig. 1, the sensor array has a resolution of 16x 16 on each soft finger, totaling 1024 tactile sensing units across our bimanual tactile sensing system. This dense, continuous tactile sensor array provides effective feedback, including the presence of contact, the amount of applied normal force, and local contact patterns. (ii) On the algorithm side, given the multi-modal sensory information, instead of separately inputting visual and tactile data into the policy [12), we propose unified 3D visuo-tactile representation that fuses these two modalities for imitation learning. This representation integrates 3D visual points and 3D tactile points (calculated using robot kinematics) into unified 3D space, which explicitly accounts for the 3D structure and spatial relationship between vision and touch. This approach enables effective imitation learning through diffusion policy [13], allowing the system to react to nuanced force changes and overcome significant visual occlusions.
We conducted comprehensive evaluations on four challenging real-world tasks (e.g., manipulating fragile objects like eggs and fruit, and in-hand manipulation of tools and utensils, as shown in Fig. 1). The results demonstrate that our 3D visuo-tactile representation significantly enhances the performance of contact-rich manipulations by providing more detailed contact states and local geometry or position information. We observed that tactile information is especially critical when there is heavy visual occlusion. We also conducted detailed ablation studies regarding the sensing characteristics, comparing performance across different tactile resolutions, and showed the importance of continuous tactile reading. Additionally, the inclusion of tactile feedback during the data collection process enables the operator to gather higher-quality data, making the final policy more robust.
Related Work
Bimanual Manipulation. Dual-arm robotic setups present a wealth of opportunities for broad applications [13-17]. Traditionally, approaches to bimanual manipulation were based on a classical model-based control perspective using known environmental dynamics [18-25]. However, these methods depend on ground truth environment models that are not only time-consuming construct but also typically require full-state estimation, which is often difficult to obtain, particularly for objects with complex physical properties, such as deformable items. In recent years, many researchers in methods, such as
the robotics community have increasingly shifted their focus to learning- reinforcement learning [15, 26-32], and imitation learning [ [16, 17, 33-41.. However, most bimanual manipulation methods still primarily rely on visual inputs [13, 16, 42-46], limiting the robot's 's ability to achieve human-level flexibility and dexterity due to the sensing gap between humans and robots. To overcome these limitations, recent works [47, 48] employ RGB images from optical tactile sensors. However, due to the minimal range of the cameras in these sensors, the robot fingers are typically very bulky and overly rigid, limiting their effecttivnnes in more complicated dexterous tasks. In

---

Tuctile Readins Luito)
Pulyimide Adhesive 1a.3 cani
Forvs-Sensitvee
Film (5.)
Conductive Thread
Tacille
Sensor Unit (Jtun")
Fuess(N)
(li)
Consistency 116K
16 by 16 Sensor Matrix
Conseeting FC (32H2)
Adaptive Fin-shaped Gripper
Acalisi Board
(n) Tactile Sensing Platform
(b) Tactile Integrated Soft Gripper Design
(c) Physical Churacteristic
Our Tactile Sensing Platform. Part (a) shows our bimanual tactile integrated system setup. We deploy four tactile sensor pads (two for each hand) on the soft grippers. The tactile readings are displayed on the back screen. Part (b) describes the design of tactile-integrated soft gripper. Each sensor comprises 256 sensing units, with their locations on the gripper shown (ii). have also designed a readout board to collect tactile signals and forward them to the host computer. Part (c) shows the physical characteristics and sensing consistency of our tactile sensors (details in Sec. 5).
Figure 2:
contrast, the tactile sensor introduced in this work is based on piezoresistive materials, allowing for larger-scale, compliant coverage of flexible thin fingers.
Cactoleactlleaao Manipulation. Tactile information plays crucial biological role [49], and the integra- tion of vision and touch is fundamental for humans to successfully interact with their environment [50]. Vision provides a broad perspective of the environment but often lacks detailed contact information and suffers from visual occlusion, which can be effectively complemented by tactile sensing [51-62]. Integrating visual and tactile information is also crucial for robotic manipulation. Lin et al. [14] propose a visuo-tactile policy that leverages human demonstrations within a bimanual system. How- ever, their tactile sensor low-resolution, and their approach lacks an explicit account of the spatial relationship between vision and touch. To address this limitation, Yuan al. [27] proposed Robot Synesthesia, which combines visual and tactile data as single input to the policy network. However, this approach only considers lowrresolution binary tactile signals, which are limited in visuo-tactile sensing capacity. In contrast, our method employs dense continuous tactile sensing system that delivers comprehensive information about the contact area. We also provide explicit accounting of the scene structure by integrating both modalities into a unified 3D representation space, leading to more effective policy learning process.
Visuo-Tactile Manipulation System
3.1 Sensor and Gripper Design
Flexible Tactile Sensors. Our sensor pads consist of resistive sensing matrices that convert mechan- ical pressure into electrical signals. Designed with a total thickness of less than 1mm, the tactile sensor pads can be easily integrated onto various robotic manipulators, including the surfaces of robot arms. In this paper, we install the tactile sensors on a soft and adaptable fin-shaped gripper, as shown in Fig. 2(b). These flexible sensor pads bend with the soft gripper and continue to provide effective signal transmission, making the system versatile across a wide range of robotic applications.
As illustrated in Fig. 2(b), each finger of the manipulator is equipped with sensor pad containing 256 sensing units (a 16 by 16 sensor array). The size, density, and spatial sensing resolution of the sensor pads can be customized; in our current design, the resolution set at 3mm? per sensor point. Similar to [1]], the tactile sensing pads leverage a triple-layer design, where a piezoresistive layer (Velostat) sandwiched between two sets of orthogonally aligned conductive yarns serving as electrodes. These layers are then encapsulated between two shaped Polyimide films using a high-strength adhesive (3M 468MP), ensuring robust electrical contact between the electrodes and the piezoresistive film, which is crucial for reliable signal acquisition. The sensor characteristics are repeatable across multiple

---

devices and reliable over multiple cycles. More detailed information on tactile sensor manufacturing available in the Appendix.
The resistance of the piezoresistive layer changes in response to applied pressure, enabling each sensor point to convert mechanical pressure into an electrical signal. These analog signals captured by an Arduino Nano and transmitted to a computer via serial communication. We use customized electrical readout circuit to acquire data at a frame rate of up to approximately 32.2 FPS. The total cost of one sensor pad and the reading board (without Arduino) is about $20. We are committed to releasing comprehensive tutorials for hardware manufacturing.
Integration of Flexible Tactile Sensor and Soft Gripper. We install our tactile sensors on the surface of fully 3D- DD-printed soft gripper made from TPU material (Fig. 2(b)). The tactile sensor pads integrate well with the flexible soft gripper. Our new gripper design offerr several advantages. First, the soft nature of the gripper significantly increases the contact area between the sensors and the target objects. This not only helps stabilize the manipulation process but also ensures consistent reflection of the contacting patterns and geometry of the objects. Second, while our visuo-tactile policy provides certain levels of action compliance, the softness of the gripper adds mechanical compliance [42]. enabling us to handle fragile objects more effectively.
3.2 Multi-Modal Sensing and Teleoperation Setup
We employ a bimanual telcoperation system using two master robots to control two puppet robots [16]. During the data collection process, we gather synchronized multimodal information at a consistent frequency of 10 Hz from various sensors, including tactile sensors, multi-view RGBD cameras (Realsense), and data on robot target actions and current joint states. Synchronization among the sensing information is critical for maintaining the temporal consistency of the multimodal dataset, allowing for accurate alignment between tactile feedback and visual data. We also implement real- time tactile information feedback by visualizing it on the screen, as shown in Fig. 2(a). This enables the human operators to assess the adequacy of contact for secure grasping, which enhances the quality of the collected data, as will be detailed Sec. 5.
robots
4
Learning Visuo-Tactile Dexterity
4.1 Problem Formulation
In our study, we address the challenge of learning contact-rich robot skill trajectories through imitation learning. Fig. 3 provides an overview of the integration of visuo-tactile data and the subsequent action generation processes. Specifically,
introduce a visuo-tactile policy, denoted as π O - A. This policy maps combined visual and tactile observations O to actions A. Our method consists of two critical parts: (1) Dense TactoTectile Representation: Fig. 3(b) shows the integration of visual and tactile data within unified coordinate system, which includes: (i) 3D Visual Point Cloud (visualized by e): Captured by the camera, formatted as Priiuul C IR Na. x-1 including an additional empty channel to match the shape of the tactile data. (ii) 3D Tactile Point Cloud (visualized by +): This tactile point cloud includes all the points of the tactile sensing units and uses the sensing value feature channel, formatted piactile E N (2) Policy Learning: Fig. 3(c) indicates the imitation learning process. Conditioned on 3D dense visuo-tactile representation, leverage the diffusion policy [13] to generate actions as a sequence of robot joint states.
4.2 Dense Visuo-Tactile Representation
In our approach, instead of separately processing tactile and visual modalities for feature extrac- tion [14], we integrate tactile and visual data by projecting them into the same 3D space. As illustrated in Fig. 3(b), the top row demonstrates the processing of visual observations, while the bottom row depicts the processing of dense 3D tactile points using tactile signals and robot proprioception.
3D Visual Point Cloud. We implement a series of data preprocessing procedures on the point cloud captured by the camera, denoted as prisual Na The process involves four steps: (i) Merge: We combine point clouds from multi-view depth observations to ensure comprehensive coverage of the observed environment. (ii) Crop: The point cloud is cropped to the designated work region using

---

Fast Point Sampping
Point
Visuel
Visuo Tactile
Representation
Tactle Signals
PointNe
difuconn Policy
Action
Sequence
Robot Propcloception
Tactile
(a) Real World Environment
(b) 3D Data Processing
(c) Decision Making Module
Figure 3: Visuo-Tactile Policy. Part (a) shows the real-world setup and the manipulated objects. Part (b) illustrates the processing of visual data (upper block) and tactile data (bottom block), followed by their integration within the same 3D coordinates. From the visualization of the tactile signals, depending the relative movements
same gripper can differ even when grasping a symmetric part of the tool. Such nuanced information is particularly important for in-hand object manipulation. Part (c) outlines our decision- decision-making process, where our network takes the integrated 3D visuo-tactile representations input and outputs the predicted action sequenec.
two grippers,
force patterns
two fingers
manually defined bounding box. (iii) Down-Sample To enhance the efficiency of our visual data processing, we down-sample the point cloud using Farthest Point Sampling (FPS) [63] to ensure more uniform coverage of the 3D space (compared to uniform sampling). Here, we set Nvis = 512 to maintain a balance between geometric detail and computational efficiency. (iv) Transform: The point cloud transformed into the robot's base frame.
3D Tactile Point Cloud. The setile-based point cloud, Ptactile RR N represents the positions and the continuous tactile readings from the tactile units in 3D space. To determine the position of each sensor, we calculate the real-time grippers' positions using forward kinematics based on the robot's 's joint states. We set N = 256 x Ningor for the tactile point cloud, where Nünger represents the number of robot fingers equipped with the tactile sensor pads. Each sensor pad consists of 256 tactile points. We set Ninger for single arm task and Ninger = 4 for the bimanual task.
plactile U Prisual, into the same spatial coordinates, visualized in Fig. 3(c). Each point is also assigned a one-hot encoding to indicate whether it is visual point or tactile point. This unified 3D visuo-tactile representation provides the policy network with a detailed and explicit accounting of the spatial relationships between tactile and visual data. This integration introduces an inductive bias that enhances the effectiveness of manipulation contact-rich tasks, particularly those requiring a comprehensive consideration of both modalities.
3D Visuo-Tacteo Points. We then integrate the two types of point clouds,
4.3 Training Procedure
As shown in Fig. 3(c), the decision module in our method formulated conditional denoising diffusion model [64]. It uses the PointNet++ [65] architecture the backbone and is conditioned on the 3D visuo-tactile representation to denoise random Gaussian noise into the actions
5
Experiments
5.1 Tactile Sensor Hardware Characterization
We evaluated the physical characteristics of our tactile sensors to better understand their performance and signal range. As depicted Fig. 2(c), the top figure illustrates how the reading signals of our tactile sensors vary with the applied foree. We selected 10 sensor units from one sensor pad, and the data from each of them were used to fit individual curves. The bottom figure in Fig. 2(c) shows the
5

---

Soarnin??
Open tig Iny
Crmpp Pas
Init Sowe (Prit Prepertion)
(6) and Pooz Mate
(b) Poatic
Grtep Gooe
Maxia Tripe
Init State (Hlet Key Collection)
(o) Right Hand Grasp
(b) Left Hand Grasp
(o) Auuss and Rotaze
(d) Insettion
Serving)
Grap Serving Spocn
(c) Egs
(d) Replace Eigs
Figure 4: Policy Rollout. We evaluate visuo-tactile policy across four nonghooizoon, precisc manipulation tasks. Detailed descriptions and metrics for these tasks can be found Sec. 5.2. The first two rows emphasize tasks that require ine-grained force information, while the last two rows focus on tasks that require in in-hand object state information. Please check videos on our website for more details.
consistency of our tactile sensor pad under different loads. We divided our sensor into an 8x8 grid and summed the readings from each unit. The consistency of the 8x8 grid under different loads is shown using box plot. To demonstrate the capabilities of our dense, continuous tactile sensors, also conducted additional experiments for 6 DoF pose estimation using only 3D tactile information and models of the manipulated objects. More details can be found in the Appendix.
5.2 Experiment Setup for Imitation Learing
We evaluate our multi-modal sensing and learning system on four challenging real-world robotic tasks, each divided into four steps for a more fine-grained assessment of performance (Fig. 14). Tasks are categorized based on how they benefit from the incorporation of tactile signals. Below are the basic descriptions and evaluation metrics for all tasks:
(1) Tasks Requiring FinieGrained Force Information:
Egg Steaming. The robot uses its right hand to open the egg tray first. Then the robot must grasp and place an egg into an egg cooker. Subsequently, the left hand used to relocate and secure the cooker's cover over the egg. Evaluation Metrics: The task is considered successful the egg placed without damage and the cooker's cover is properly positioned over the egg. handling failure occurs if the egg falls due insufficient grip force cracks under excessive force.
Fruit Preparation. The robot uses its left hand to grasp the plate and place it on the table. Subse- quently, both robot arms collaborate to open the plastic bag. Then, the right arm grasps a grape or several grapes and places them on the plate. Evaluation Metrics: The task is considered successful the grapes are placed on plate without sustaining any damage.
(2) Tasks Requiring In Hn-Hand State Information:
Hex Key Collection: The right hand is required to grasp the Hex Key, and then the left hand needs to grasp the Hex key followed by an in-hand adjustment of the Hex Key's position using its left hand. Subsequently, the robot required to accurately insert the Allen wrench into the hole in the box. Evaluation Metrics: A successful insertion of the Allen wrench into the hole is considered successful operation. Additionally, any failure to properly adjust the Hex Key's position may result in the inability when inserting it into the hole.

---

Tasks Requiring Fine-Graied Force Information
Egg Steaming (30 demos)
Fruit Preparation (30 demos)
Modalities
Open Egg Tray
Grasp
Place
Whele Task
Grasp and
Open Plastic Bag
Grusp
Whole
Egg
Egg
Place Plate
Grapes
Task
RGB Only
0.95
0.75
0.60
0.50
0.95
0.75
0.45
0.45
RGB w/ "Tactile Imuge
0.90
0.80
0.70
0.90
0,70
PC. Only
1.00
0.75
0.65
0.55
1.00
0.85
0.50
0.45
PC. Tactile Image
0.85
0.70
0.70
0.95
0.90
0.75
PC, w/ Tactile Points (Ours)
1,00
0.90
0.90
0.85
0.95
0.85
0.80
0.80
Tasks Requiring Hand State Information
Hex
Key Collection (30 demos)
Sandwich Serving (50 demos)
Modalities
Right Hand
Left Hand
In-Hand
Whole
Grasp
Tilt
Get
Whole
Grasp
Grasp
Adjustment
Task
Serving Spoon
Pul
Fried Egg
Task
RGB Only
0,30
0.85
0..55
0.45
1,00
0.95
0.70
0.50
RGB w/ Tactile Image
0.95
0.90
0.70
0.60
1.00
1.00
0.85
0.75
PC. Only
1.00
0.90
0.65
0.65
1.00
1.00
0.75
0.65
PC. w/ Tactile Image
0.95
0.85
0.60
0.50
1.00
0.90
0.75
0.65
PC. w/ Tactile Points (Ours)
1.00
0.95
0.95
0.90
1.00
1.00
0.90
0.85
Table 1: Comparison with Baselines. We evaluate our policy over 20 episodes and the best performance for
each task is holded. Please check our website for more comparison videos.
Sandwich Serving: First, the right hand is required to grasp the serving spoon. Second, the left hand needs to hold the pot handle and then tilt the pot. The right hand should then retrieve the fried egg
from the pot and serve it on the bread. Evaluation Metrics: The robot must successfully retrieve the fried egg from the pot and serve it on the bread.
In the experiments, we primarily compare our methods with the following baselines. We trained all policies for 2,000 epochs. All methods, including ours and the baselines, use three camera views. (I) RGB Only. This method uses multi- view RGB from cameras as input for the image-basel diffusion policy. We use the same implementation [13].
(2) RGB w/ Tactile Image. This method processes multi-view RGB images and tactile images through different branches for the diffusion policy. We use CNN as the feature extractor for tactile images. (3) PC. Only. This method uses only multi-view visual point clouds as the sensing modality for the diffusion policy. We use PointNet++ as the feature extractor.
(4) PC, w/ Tactile Image. This method fuses multi-view visual point clouds and tactile images through different branches for the diffusion policy. We use a CNN as the feature extractor for tactile images and PointNet++ as the feature extractor for point clouds.
5.3 Qualitative Analysis
Our policy integrates three modalities: vision, tactile sensing, and robot proprioception, and has proven to be effective through four challenging, long-horizon tasks. We observe three key benefits of integrating touch. (1) Tactile sensors provide critical feedback on the presence of contact and the appropriate amount of force to apply. For example, a common issue observed with the baseline during the egg- ste-mitnning task is the application of insufficient force, which often results in the egg falling or not being successfully grasped from the tray. Similarly, during the wrape-handling task, a typical error occurs when the gripper attempts to grasp multiple grapes at once, applying excessive force that damages the fruit. (2) Our policy leverages detailed contact patterns provided by touch to address visual occlusions effectively. A common failure in the bascline during the Hex Key collection task is the inability to adjust the hex key's position in hand, leading to failures in subsequent insertion attempts. In the sandwich serving task, tactile feedback is crucial for understanding the in-hand states and the spatial relationship between the contact area and all tools, especially when tools may passively rotate as the spoon interacts with the pot-a frequent cause of failures in baseline approaches. (3) Tactile feedback provides the confidence needed to transition berween task stages. This is especially important in scenarios where the visual information is noisy or highly occluded. During our experiments on grasping grapes in bag, the visual-only policy frequently failed, often getting stuck in the bag and unable to progress to the next stage. More details and videos about failure cases can be found in the Appendix.
A Alltion Study on Varying Levels of Visual Information. Visual occlusion presents a significant challenge in manipulation tasks, especially in bimanual tasks. To determine whether tactile sensing can compensate for visual occlusion, varied the levels of occlusion by changing the number of cameras used both in the training set and during the policy rollout. As detailed in Table 2, Cam refers
7

---

Egg Cooking
(30 demos)
Hex Key Collection (30 demos)
Settings
Open Egg Tray
Grasp
Place
Whole Task
Right Hand
Left Hund
In-Hand Adjustment
Whole
Egg
Egg
Grasp
Grasp
Task
Tactile Points Only
0.70
0.55
0.10
0.10
0.50
0.30
0.30
0.15
Single Cam PC. Only
0.90
0.65
0.00
0.50
0.80
0.60
0.45
0.30
Multi Cam PC. Only
1,00
0.75
0.65
0.55
0.90
0.65
0.65
Single Cam PC. Tactile Points
0.95
0.90
0.85
0.80
0.85
0.80
0.75
0.70
Multi Cam PC, w/ Tactile Points
1,00
0.90
0.90
0.85
1.00
0.95
0.95
0.90
Table 2: How Touch Complement Varying Levels of Visual Information? We manipulate visual occlusion
by varying the number of cameras, thereby testing how tactile feedback compensates for reduced visual input.
Steaming
Hex Collection
Egg Steaming
MAS
CAS LG
Continous 4-4 Tae
100
Copar
wroo
Fopert
6.25
Binury 16x16 Tae
Tac
16-16 Tac
Oyen Egg Try
Grapp Egg
Placs Egg.
Whole Tisk
Hex Key Collection
10
DAS
Thal
Right Hun Grap Left Hand Grap In-Hard Adjusment Whole Task Brooy 16-16
Cone 16-16
Figure 5: Tactile Feedback Improves the Demonstration Data Quality. Users new the system perform better with both visual and tactile feedback than with vision alone.
Figure 6: Comparison of Varying Types of Tactile Information. found that denser and
Across all tested tactile modalities. continuous signals consistently outperform all other comparison groups. Tac refers to the use of tactile signals.
to the use of visual point clouds, while Tactile Points denotes the use of tactile point clouds in 3D space, aligned with the camera coordinates. Results show that even as visual occlusion increases with fewer cameras, the policy continues to perform well with the aid of tactile information.
Ablation Study Varying Levels of Tactile Information. Contact data can vary in form and resolution. Dense and continuous tactile data not only indicate the presence of contact but also measure the amount of applied force and the detailed local contact pattern-capablice that sparse binary tactile signals lack [26]. To validate this, we varied the tactile resolution from 16x 16 to 8 and 4x4 and added an additional baseline that only considers binary signals. All baselines and ours in Fig. 6 incorporate the multi-view point clouds. We found that continuous signals significantly enhance performance, especially in actions requiring precise in-hand orientation of the grasped objects. Dense tactile patterss resolution of 16x 16 slightly outperform the 8 8 resolution and noticeably surpass the 4x4 and binary comparison groups
Demonstration Data Quality with Tactile Feedback. As shown in Fig. 2(a), real-time tactile signals are visualized and displayed on the operator's screen during data collection. This feature not only enhances the efficiency of data collection but also improves the quality of the data. To validate the effectiveness of tactile feedback, enlisted 10 new users complete tasks. Five of these users operated the teleoperation system with tactile feedback, while the other five did not. The duration to complete the tasks is shown in Fig. 5.
Conclusion and Limitation
Conclusion. In this paper, develop a multi-modal sensing and learning system for contact-rich robotic manipulation. We introduce a dense, flexible tactile sensor array that covers larger area of thin, soft robot grippers. We also propose unified 3D visuo-tactile representation, which explicitly accounts for the 3D structure and spatial relationship between vision and touch. We demonstrate the effectiveness of these innovations in a range of challenging manipulation tasks. We are committing to release our code and hardware setup.
Limitations. We acknowledge that collecting real data with a molti- perception system is expensive, due to the data collection itself and the addition of tactile sensing modality. One missing piece of our study is the simulation of our tactile sensor, limiting our ability introduce randomization or scale up the data. In future work, we aim to enhance the system's 's capabilities to better exploit advances in physical simulation, further improving the generalizability and robustness of the policy.
8

---

7 Acknowledgement
The Toyota Research Institute (TRI) partially supported this work. This article solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity.
References
[1] N. Kuppuswamy, A. Alspach, A. Uttanchandani, S. Creasey, Ikeda, and R. Tedrake. Soft- bubble grippers for robust and perceptive manipulation. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9917 99244. IEEE, 2020.
[2] J. Xu, S. Song, and M. Ciocarlie. Tandem: Learning joint exploration and decision making with tactile sensors. IEEE Robotics and Automation Letters, 7(4):10391-1039,, 2022.
[3] I. H. Taylor, S. Dong, and A. Rodriguez. Gelslim 3.0: High-resolution measurement of shape, force and slip in compact suctile-sensing finger. In 2022 International Conference on Roboties and Automation (ICRA), pages 10781-10 177877 IEEE, 2022.
[4] T. P. Tomo, M. Regoli, A. Schmitz, L. Natale, H. Kristanto, S. Somlor, L. Jamone, G. Metta, and S. Sugano. A new silicone structure for uskin a soft, distributed, digital 3-axis skin sensor and its integration on the humanoid robot icub. IEEE Robotics and Automation Letters, 3(3): 2584 2591, 2018.
[5] J. Zhao and E. H. Adelson. Gelsight svelte: A human finger-shaped single-camera tactile robot finger with large sensing coverage and proprioceptive sensing. In 2023 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8979-8984. IEEE, 2023.
[6] R. Bhirangi, T. Hellebrekers, C. Majidi, and A. Gupta. Reskin: versatile, replaceable, lasting tactile skins. In 5th Annual Conference Robot Learning, 2021.
[7] R. Bhirangi, DeFranco, J. Adkins, Majidi, A. Gupta, Hellebrekers, and V. Kumar. All the feels: A dexterous hand with large area sensing. arXiv preprint arXiv:22 15658, 2022.
[8] G. Büscher, M. Meier, G. Walck, R. Haschke, and H. Ritter. Augmenting curved robot surfaces with soft tactile skin. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1514-1519, 2015. doi: 100./..O........ 200.0S..015..33. 20157733566.
[9] A. Padmanabha, F. Ebert, S. Tian, R. Calandra, C. Finn, and S. Levine. Omnitact: A multi- directional higl-resolution touch sensor. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 618-624. IEEE, 2020.
[10] M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon, V. R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, et al. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3):3838-388) 2020.
[11] S. M. Sundaram, P. Kellnhofer, Y. Li, J.-Y. Zhu, A. Torralba, and W. Matusik. Learning the signatures of the human grasp using a scalable tactile glove. Nature, 569:698 702, 2019. URL https://api.
semanticscholar. org Crrp/orDD 169033286.
[12] R. Calandra, A. Owens, M. Upadhyaya, W. Yuan, J. Lin, E. H. Adelson, and S. Levine. The feeling of success: Does touch sensing help predict grasp outcomes? arXiv preprint
17Xiv:1710.05512.220 2017.
[13] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[14] T. Lin, Y. Zhang, Q. Li, H. Qi, B. Yi. S. Levine, and J. Malik. Learning visuotactile skills with two multifingered hands. arXiv preprint arXiv:2404.1688 2024.

---

[15] B. Huang, Y. Chen, T. Wang. Y. Qin, Y. Yang, N. Atanasov, and X. Wang. Dynamic handover: Throw and catch with bimanual hands. arXiv preprint arXiv:2309.05655, 2023.
[16] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 137.033334. 2023.
[17] Z. Fu, T. Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body telcoperation. In arXiv, 2024.
[18] S. S. M. Salchian, N. Figueroa, and A. Billard. A unified framework for coordinated multi-arm motion planning. The International Journal of Robotics Research, 37(11)) 2018. doi:10.1 (0011177707733649117766767
URL https://///ee
org/ 10.. /tt.://.........../.7777//7/0100 10.1177/0777866998866522
[19] C. Smith, Y. Karayiannidis. Nalpantidis, X. Gratal, Qi. D. V. Dimarogonas, and D. Kragic, Dual arm manipulation survey. Robotics and Autonomous Systems, ((10)):340-13533 2012. ISSN 0921-8890. /ttp://ccccaacearaaaaaa/////////
URL https: https://w//
[20] M. Oller, D. Berenson, and N. Fazeli. Tactilevad: Geometric aliasing-aware dynamies for high-resolution tactile control. In J. Tan, M. Toussaint, and K. Darvish, editors, Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 3083-3099. PMIR, 06-09 Nov 2023. URL https: btstps://proceedingg mlr. (ress/2229/oler222
html.
[21] C. Yang, Y. Jiang. J. Na, Z. Li. L. Cheng, and C.-Y. Su. Finite-time convergence adaptive fuzzy control for dual-arm robot with unknown kinematics and dynamics. IEEE Transactions on Fuzzy Systems, 27(3):574 2019. 110::00.10//FUU222288828886
[22] N. Vahrenkamp, M. Przybylski, T. Asfour, and R. Dillmann. Bimanual grasp planning. pages 493-499, 10 2011. 1. 10i: 200///ummmoodd221111666060000
[23] T. Pang, J. T. Suh, L. Yang, and R. Tedrake. Global planning for contact-rich manipulation via local smoothing of quasi-dynamic contact models, 2022. URL httpt: org/ /bs/ 2206. 10787.
[24] H. T. Suh, N. Kuppuswamy, T. Pang, P. Mitiguy, A. Alspach, and R. Tedrake. Seed: Series clastic end effectors in 6d for visuotactile tool use. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4684-4691. IEEE, 2022.
[25] R. Kolamuri, Z. Si, Y. Zhang, A. Agarwal, and W. Yuan. Improving grasp stability with rotation measurement from tactile sensing. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6809-681 16. IEEE, 2021.
[26] Z.-H. Yin, B. Huang, Y. Qin, Q Chen, and X. Wang. Rotating without seeing: Towards in-hand dexterity through touch. arXiv preprint arXi::2303.10880 2023.
[27] Y. Yuan, H. Che, Y. Qin, B. Huang, Z.-H. Yin, K.-W. Lee, Y. Wu, S.-C. Lim, and X. Wang. Robot synesthesia: In-hand manipulation with visuotactile sensing, arXiv preprint arXiv:2312.01853. 2023.
[28] Y. Qin, B. Huang, Z.-H. Yin, H. Su, and X. Wang. Dexpoint: Generalizable point cloud rein- forcement learning for sim-to-real dexterous manipulation. In Conference on Robot Learning, pages 605. PMLR, 2023.
[29] Y. Lin, A. Church, M. Yang, H. J. Lloyd, D. Zhang, and N. F. Lepora. Bi-touch: Biman- ual tactile manipulation with sim-to-real deep reinforcement learning. IEEE Robotics and Automation Letters, 2023.
[30] T. Lin, Z.H. Yin, H. Qi, P. Abbeel, and J. Malik. Twisting lids off with two hands. arX1v:2403.02338, 2024.
10

---

[31] T. Bi, C. Sferrazza, and R. D' A'ddre. Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulation. IEEE Robotics and Automation Letters, 6(3):5761-5768, 2021.
[32] A. Church, J. Lloyd, N. F. Lepora, al. Tactile sim-to-real. policy transfer via real-to- image translation. In Conference Robot Learning, pages 1645-1654, PMLR, 2022.
[33] J. Wang, Y. Qin, K. Kuang, Y. Korkmaz, A. Gurumoorthy, H. Su, and X. Wang. CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation. arXiv preprint arXiv: 2312.09237, 2024.
[34] I. Guzey, B. Evans, S. Chintala, and L. Pinto. Dexterity from touch: Self-supervised pre-training of tactile representations with robotic play, 2023.
[35] Grannen, Y. Wu, B. Vu, and D. Sadigh. Stabilize act: Learning coordinate for bimanual manipulation. In Conference on Robotic Learning (CoRL), 2023.
[36] H. Kim, Y. Ohmura, and Y. Kuniyoshi. Robot peels banana with goal-conditioned dual-action deep imitation learning. arXiv preprint arXiv:2203.097 2022.
[37] R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprint 1xXxv:1806.08766 2018.
[38] Y. Qin, W. Yang, B. Huang, K. Van Wyk, H. Su, X. Wang, Y.W. Chao, and D. Fox. Anytelcop: A general vision-based dexterous robot arm-hand telcoperation system. arXiv preprint arXiv:2307.04577, 2023.
[39] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. Conference on Robot Learning. pages 158-168. PMLR, 2022.
[40] S. P. Arunachalam, S. Silwal, B. Evans, and L. Pinto. Dexterous imitation made easy: A learning-basee framework for efficient dexterous manipulation. In 2023 ieee international conference robotics and automation (icra), pages 5954-5961. IEEE, 2023.
[41] Y. Chebotar, O. Kroemer, and J. Peters. Learning robot tactile sensing for object manipulation. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3368-3375, 2014. doi:10. 1001110//RRSS.00 2014,993,
[42] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and Song. Universal manipulation interface: In-the-win wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402. 10329, 2024.
[43] C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.12422 2023.
[44] C. Wang, H. Shi, W. Wang, R. Zhang, L.. Fei-Fei, and C. K. Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint 47X78.2403.07788 2024.
[45] T. Chen, M. Tippur, S. Wu, V. Kumar, E. Adelson, and P. Agrawal. Visual dexterity: In-hand reorientation novel and complex object shapes. Science Robotics, 8(84)ceadc9244, 2023. scirobotics. adc9244.
URL https: //www/ wwwence.ecccc.............eee..e wwt.aaann.......oooaaeeeeeeee//e 1126/
[46] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. In Conference robot learning, pages 357-368. PMLR, 2017.

---

[47] A. George, S. Gano, P. Katragadda, and A. B. Farimani. Visuo-tactile pretraining for cable plugging. arXiv preprint arXiv:2403. 1898, 2024.
[48] E. Su, C. Jia, Y. Qin, W. Zhou, A. Macaluso, Huang, and X. Wang. Sim2real manipulation on unknown objects with tactile-based reinforcement learning, arXiv preprint arXiv:2403.12170 2024.
[49] R. S. Johansson and G. Westling. Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects. Experimental Brain Research, 56:550-564, 2004. URL https:/ /api. semanticscholas otg/CorpusDD 16631166.
[50] P. Jenmalm and R. S. Johansson. Visual and somatosensory information about object shape control manipulative fingertip forces. Journal of Neuroscience, 0270-6474. doi: 10. 100.1/2//NEUUOSSC.7111------ 1997. URL wrg/content/17/111
1997. ISSN
[51] S. Suresh, H. Qi, T. Wu, T. Fan, L. Pineda, M. Lambeta, Malik, M. Kalakrishnan, R. Ca- landra, M. Kaess, et al. Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation. arXiv preprint arXiv:23 2...::23/2.134000 /3469, 2023.
[52] Z. Ding. G. Chen, Z. Wang, and L. Sun. Adaptive risual-tactil fusion recognition for robotic operation of multi-material system. Frontiers Neurorobotics, 17, 2023. ISSN 1662-5218. doi: 10.3389/fmbot.2223.1 181383. URL httpht::://eee http///wwww.ronttrrininnnn fnbot. 2023. 1181383.
[53] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg. Making sense vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. 2019 International Conference on Robotics and Automation (ICRA), pages 8943-8950. IEFE, 2019.
[54] V. Dave, F. Lygerakis, and E. Rueckert. Multimodal visual-tactile representation learning through self-supervised contrastive pre-training. arXiv preprint (tt::2401.12224, 2024.
[55] A. Bicchi and V. R. Kumar. Robotic grasping and contact: a review. Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cal. No.00CH33065), 1:348-353 vol.1, 2000. URL https: /tttps://ai....
semanticscholar orgo wwrgg/CorpusDD::e. 2650168.
[56] N. Sunil, S. Wang, Y. She, E. Adelson, and A. R. Garcia. Visuotactile affordances for cloth manipulation with local control. In Conference on Robot Learning, pages 1596-1606. PMLR, 2023.
[57] W. Yuan, Y. Mo, S. Wang, and E. H. Adelson. Active clothing material perception using tactile sensing and deep learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 4842 4849, IEEE, 2018.
[58] R. Calandra, A. Owens, D. Jayaraman, Lin, W. Yuan, J. Malik, E. H. Adelson, and S. Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letlers, 3(4):3300-3307, 2018.
[59] H. van Hoof, N. Chen, M. Karl, P. van der Smagt, and J. Peters. Stable reinforcement learning with autoencoders for tactile and visual data. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3928-3934, 2016. doi: N0i:10.1109/RRO..00011 7759578.
[60] Y. Chen, M. Van der Merwe, A. Sipos, and N. Fazeli. Visuo-tactile transformers for manipulation. In 6th Annual Conference on Robot Learning, 2022.
12

---

[61] P. Falco, S. Lu, A. Cirillo, C. Natale, S. Pirozzi, and D. Lee. Cross-modal visuo-tactile object recognition using robotic active exploration. In 2017 IEEF International Conference on Robotics and Automation (ICRA), pages 5273-5280, 2017. 10.1109//CRR22221.7779888
[62] J. Hansen, F. Hogan, D. Rivkin, D. Meger, M. Jenkin, and G. Dudek. Visuotactile-rl: Learning multimodal manipulation policies with deep reinforcement learning, In 2022 In- ternational Conference on Robotics and Automation (ICRA), pages 8298-8304, 2022. doi: 200.:4466666222.882228888
[63] C. Moenning and N. A. Dodgson. Fast marching farthest point sampling. Technical report, University of Cambridge, Computer Laboratory, 2003.
[64] Ho, Jain, and Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[65] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets metric space, Advances neural information processing systems, 30, 2017.
[66] P. M. Djuric, J. H. Kotecha, J. Zhang, Y. Huang, T. Ghirmai, M. F. Bugallo, and J. Miguez. Particle filtering. IEEE signal processing magazine, 2015):19-38,003:
13

---

Supplementary Materials
Contents
Introduction
2
Related Work
3
Visuo-Tactile Manipulation System
3.1
Sensor and Gripper Design
3,2
Multi-Modal Sensing and Telcoperation Setup
4
Learning Visuo-Tactile Dexterity
4.1
Problem Formulation
4.2
Dense Visuo-Tactile Representation
4.3
Training Procedure
S
Experiments
5.1
Tactile Sensor Hardware Characterization
5.2
Experiment Setup for Imitation Learing
5.3
Qualitative Analysis
6
Conclusion and Limitation
Acknowledgement
Tactile Sensor Hardware
14
A.1
Tactile Sensor Manufactory
14
A.1.1
Tactile Sensor Pad Design
14
A.1.2
Reading Board Design
15
A.2
Tactile Hardware Evaluation Experiment
15
A.2.1
Physical Characteristics
A.2.2
6-DoF Object Pose Estimation
16
Experiment details Imitation Learning
17
B.1
System Overview
17
B.2
Experiment Setup Details
18
B.2.1
Details for the Egg Steaming task
18
B.2.2
Details for the Fruit Preparation Task
19
B.2.3
Details for the Hex Key Collection Task
20
B.2.4
Details for the Sandwich Serving Task
22
B.3
Single Arm Experiments.
22
B.4
Learning Details
22
A
Tactile Sensor Hardware
A.1
Tactile Sensor Manufactory
A.1.1 Tactile Sensor Pad Design
The tactile sensing pads leverage triple-layer design, where a piezoresistive layer (Velostat) is sandwiched between two of orthogonally aligned conductive yarns serving electrodes. During the tactile sensor manufacturing, the Velostat layer and then use high-strength adhesive (3M 468MP) to ensure robust electrical contact between the electrodes and the Velostat layer. Additionally, we use adhesive to secure the conductive thread connections to the connector. The connector links all the threads to a flexible flat cable, allowing the signal to be transmitted to the PCB board. This design makes the wires of our tactile sensor highly flexible, facilitating easier installation in various locations, such as the robot end-effector, which requires constant movement during manipulation. To ensure the tactile sensor's long-term robustness, attach polyimide layer top of the adhesive. Polyimides known for their thermal stability, good chemical resistance, excellent mechanical properties, and characteristic
first align 16 Stainless Thin Conductive Threads top of
14

---

(FFC)
Antiioo Niro
66.4
Figure 8: Tactile Reading Board Design.
Force Gauge
15
Right Hlger
15
Lett Fuger
Figure 7: Tactile Physical Characteristics Evaluation Experiment
Figure 9: Tactile Sensor Order Visualization.
orange/yellow color. After completing these steps, we finish aligning the threads for the rows. Then, flip the sensor and align the 16 threads for the columns.
After obtaining the tactile sensor pad, we attach the sensors to the robot fingers. The order of each tactile sensor unit is visualized in Fig. 9. We clearly define the tactile order to ensure that each sensor's position can be accurately calculated, and the tactile signals can correctly correspond to our real setting and dataset.
A.1.2 Reading Board Design
To ensure easy installation of the tactile reading board in the robot, have designed it to be as compact possible, shown in Fig. 8. The tactile reading board measures 45.5 mm X 48.4 .4. mm and includes an Arduino. The small size further enhances the scalability of our tactile sensors. We use two 8-bit shift registers and one 16-channel analog switch to process the tactile signals, which are then input to the Arduino. The ADC the Arduino converts the analog signals from the tactile sensor into digital signals and forwards them to the host via serial communication. We will release a comprehensive reading board scheme so that the community can directly order from PCB supplier easily replicate our tactile sensor.
A.2 Tactile Hardware Evaluation Experiment
A.2.1 Physical Characteristics
To investigate the physical characteristics of our tactile sensors, designed two experiments. As illustrated in Fig. 7, we use a force gauge to apply specific force on the tactile sensor surface. The first experiment tests how individual tactile sensor units react to applied force. The second experiment aims to test the consistency of the entire sensor pad, showcasing the variance between different regions on the tactile sensor pad.
Individual Sensor Performance. We began by randomly selecting 10 sensors from total of 256 sensors in one sensor pad. For each selected sensor, applied normal force incrementally, ranging
15

---

Textile Reading vs Furte (10 Sensor Unitsl
Hestmay Tacill Bealing under Narmal Focte IN
Hermap Tattile Reading under Nurnal Foroe
Consitincy Expedneot (16c16 Sensor Amyy
Teetln Theding
100
253
250
253
750
150
Toxto: Ro0.-Rnonngggg Sgaa
50
Forre
Tactll Reading Forna (10 Sensor Lisitss
Hostmap Tuctile Reading under Nomai Force
Hestrop Tactie Reading Nomoil
IIN
150
Theelin Taaleteee Soga
300
200
293
301
253
100
130
3X
Punne (N)
Semsors
(2) Hestrap for Tactio Reading under Difforont Loads
Plot Toctlle Pod under Different Loods
Figure 10: Results of Physical Characteristics Experiments. Part (a) shows the results of individual sensors' performance according to the force applied to their surface. Part (b) demonstrates the tactile sensor pad's consistency under different normal forces. Each heatmap displays the tactile sensor pad's readings in an x 8 grid. where each number represents the sum of four sensor units. Part (c) presents the results from part (b) in single figure, illustrating the mean and standard deviation.
from 0 to 12 N. and recorded the stable tactile reading accordingly. Each sensor generated average of 24 data points. This method allowed us to observe the individual sensor's s response to varying force levels and identify its saturation thresholds. As shown in Fig. 10 (a), we plot tactile reading versus normal force and identified that the saturation zone begins when the normal force exceeds 9 N. The fitting curve for the 10 sensors is depicted in the black line. Additionally, we applied a logarithmic scale to the x-axis (normal force), resulting in an approximately linear region for normal forces from N N. The region highlighted with blue background, as illustrated in Fig. 10 (a).
Tactile Sensor Pad Consistency. In the second part of the experiment, we used 16 x 16 tactile sensor pad and divided it into 8 x blocks, with each block comprising 4 sensors (2 x 2 matrix area). Uniform loading was applied across each block using the force gauge with a circular contacting area of 176.7 mm? For each x block, we collected the sum of the four tactile readings from individual sensors, enabling us to generate a heat map that visualizes the sensor response under specific loading conditions across the entire pad. Four different loading conditions (1 N. 3 N, N, and 11 N) were applied to comprehensively assess the overall performance, providing detailed representation of the resolution under varying forces. For each force condition, we measured once for each block, resulting in a total of 64 data points per condition. We then generate a heatmap for each force condition as shown in Fig. 10 (b). We calculated the mean and standard deviation for these data points and removed outliers. Finally, as illustrated in Fig. 10 (c), we generated a box plot from 4 sets of 64 tactile readings, demonstrating its consistency across the entire sensor and the stable functionality of the tactile sensors.
A.2.2 6-DoF Object Pose Estimation
In the main paper, we demonstrated the effectiveness of dense, continuous tactile information for fine- grained manipulation tasks. To gain a more comprehensive understanding of the information captured by our proposed sensors, conducted additional experiments on 6-DoF object pose estimation. These experiments revealed that the sensors embed information about object geometry and local contact patterns, which is crucial for manipulation tasks requiring robust and adaptive grasping as well as precise in-hand reorientation behavior.
Specifically, we define the task as estimating the 6-DoF pose of an object using only tactile obser- vations, without any visual input. We assume that the object geometry is known and denote its 3D point cloud pobj Nx3 The tactile observation, obtained by filtering the tactile-based point cloud according to the activation value, is denoted as ptactile e IR Al
Our objective is to track the
16

---

Pose Estimation. In this experiment, estimate the object pose without vision information. We can see that our pose estimation becomes more accurate as we have more complete tactile signals. can also track the object's 's pose as it rotates. Through this estimation, we demonstrate that our hardware can be potentially used for hand pose estimation and other visuotactile tasks.
Figure 11:
pose the object in the 3D space, T E SE(3), where,
0" E (E(3)),
in which the Euclidean group (3(3) = (R, t R E SO t
We solve the pose-tracking problem using particle filtering [66]. We first define our observation
function pobs = f(T) and then the weighting functions w g(g p as follows:
S(T) R.Pobj + t,
gppbs plactile
Σ
min PoE
Ipp: - 131
E
The observation function transforms the object model point cloud using T, while the weighting function calculates the distance from the contact points to the observation points. In practice, we scale the weights using an exponential function to facilitate convergence. Given the observation and weighting functions, we employ a standard particle filter to determine the object's pose.
Some example results are shown Figure 11. object, we can only rely the tactile signals from the left-side robot. Therefore, there lot of plausible solutions. Although our estimated pose is one of the plausible solutions given the one-side tactile signal, the estimation is still inaccurate. When the right-side robot contacts the object, our estimated pose aligns well with visual observation. Also, when the object is rotating in the hand, the object pose is tracked accurately.
Before the right-side robot makes contact with the
B Experiment details for Imitation Learning
B.1 System Overview
As shown in Fig. 12, We employ a bimanual teleoperation system with three Realsense cameras and four tactile sensor pads on four robot fingers. Our tactile signal communication is facilitated by a multi-threaded ROS (Robot Operating System) node. This node captures tactile signals and publishes them at a frequency of Hz. All data, including that from cameras and tactile sensors, is collected through multi-threadd multi-threading. Each data frame received timestamped, and after an episode completed, align all data with these timestamps. This synchronization is crucial for maintaining the consistency of the multimodal dataset, enabling accurate temporal alignment between tactile feedback and visual data. To manage the heavy load of processing frames from three cameras,
17

---

Screen for Tactile Feedback
Figure 12: System Overview. We attach four tactile sensors to four robot fingers and install three Realsense cameras cover the workspace. All the objects used for the task shown in the workspace. Additionally, install background screen to display the tactile feedback.
collect data 10 Hz to ensure consistency. We top camera (Realsense 455) cover the entire workspace and positioned two other cameras (Realsense 435) close the workspace to capture more detailed information. When using point cloud data from multiple cameras, we incorporate data from all cameras. For the baseline method using a single camera, we use only the top camera.
We also implement real-time tactile information feedback, shown Fig. 12 (a). During data collection, tactile signals are visually displayed on the operator's screen, enabling them assess the adequacy of contact for secure grasping. Additionally, during the policy rollout, this visualization helps see in real-time how tactile information relates to robot motion.
B.2 Experiment Setup Details
In this section, we discuss the detailed information of the four tasks described in the main paper. Each task consists of four steps, as illustrated in Fig 14. We will discuss the motions and evaluation metrics for each step, and highlight how these steps demonstrate the capabilities of tactile sensors. The typical failure cases shown in Fig. 13 and will be discussed in the following sections.
B.2.1 Details for the Egg Steaming task
Step 1: Open Egg Tray. The robot uses its right hand to open the egg tray, which mirrors the common scenario where the egg is often occluded by the tray. This realistic setup is maintained to reflect daily life, avoiding task simplification. Evaluation Metrics: The robot must open the tray sufficiently to allow its fingers grasp the egg. Failure open the tray adequately will result in the subsequent task failing. The initial position of the egg tray will be randomized within area of 7-10 em during both data collection and policy rollout.
18

---

Step 2: Grasp Egg. The robot uses its right hand to grasp the egg in the tray. This motion is complex, requiring the robot to slowly increase the force and carefully grasp the egg despite heavy occlusion. The robot with a visuo-tactile policy will retry if there is no stable tactile signal in hand, while a vision-only policy may proceed to the next goal due to heavy occlusion as shown in Fig. 13 (a). Evaluation Metries: The robot can reattempt to grasp the egg, but the step fails if moves to the next stage without the egg or if the egg falls during the transition from the tray to the steaming machine. Additionally, prolonged time spent in the egg tray will also be considered failure.
Step 3: Place Egg. The robot needs to safely place the egg in the steaming machine, which already contains two eggs. It must avoid causing the other eggs to fall while placing the egg in-hand. This step highlights our flexible thin sensor's 's capability to perform fine-grate fine-grained tasks in narrow spaces. As the robot hand exits the steaming machine, tactile information ensures there is no contact between the egg and the gripper, signaling the robot to proceed the next stage. In contrast, a vision-only policy may cause confusion about whether the robot can move out safely, potentially prolonging its stay in the steamer and increasing the risk of dislodging the other eggs. Evaluation Metrics: The robot can place the egg anywhere inside the steaming machine, but the step fails if the robot does not place the egg in the steaming machine or if it causes the other eggs to fall to the ground
Step 4: Cover the Steaming Machine. The robot needs use its left hand to grasp the cover of the steaming machine and place it safely inside. This task is challenging due to the unique shape of the steaming machine's 's handle, as shown in Fig. 13 (a). The robot must apply precise amount of force to the handle; sufficient to lift it but not so much that the cover flips and falls to the ground. The robot must apply a precise amount of force to the handle: sufficient to lift it but not so much that the cover flips and falls to the ground. This step showcases how our tactile sensor enables the robot to perform fine-grain grasping manipulations, similar grasp objects. Evaluation Metrics: The robot is allowed multiple attempts grasp the cover. The task is considered successful if the cover is securely placed the steaming machine. It is considered a failure if the cover flips or falls during the process.
human's 's ability to apply suitable and stable force to
B.2.2 Details for the Fruit Preparation Task
Step 1: Grasp and Place the Plate. The robot needs to use its left hand to grasp and place the plate on the table. This step introduces additional randomization and variance due to the varying positions of the plates, increasing the task's complexity. Evaluation Metrics: The task is considered successful if the robot grasps the plate and places it on the table.
Step 2: Open Plastic Bag. The robot needs to use its two hands to cooperate together to open the bag. The plastic bag is transparent and usually adds additional noise to the point cloud. Evaluation Metrics: The task is considered successful if the robot opens the bag wide enough for the gripper to get in.
Step 3: Grasp the fruit. The robot needs to use its right hand to get inside the plastic bag and grasp the fruit. This step is the most important and difficult in this task. First, as shown in Fig. 14(Task 2: Fruit Preparation), the robot and manipulated objects are highly occluded in the bag, making impossible for visual information to observe critical details. Our visuo-tactile policy will grasp multiple times until there is stable tactile information to secure the grapes, while a vision-only policy typically attempts the motion once regardless of the presence of grapes, making the grasping success a random event. Second, the grapes are usually clustered together, requiring the robot to apply suitable force to avoid damaging the fruit. Our visuo-tactile policy can successfully grasp single or multiple grapes from the bag, while a vision-only policy may break the grapes when the robot grasps multiple grapes (as shown in Fig. 13 (b)). since it aligns the gripper joint states instead of using force-related information. Third, this task also showcases our sensors' human-like dexterous manipulation; our mctile-integrann gripper is thin enough to get into the gaps between grapes, making it easier to grasp the grapes cluster. Evaluation Metrics: The task is considered successful when the robot successfully grasps the fruit out of the plastic bag. The task is considered failure if the robot breaks the grapes or moves to the next stage without the grapes. The policy also fails if the robot stays in
19

---

Typical Failure Cases of Baselines
(We visualize tactile siguals the screen no matter if tactile modality in the observation) (a) Egg Steaming
(b) Froit Preparing
Pyrxeail
Oschaled
Egg Falling Down
M avving W ithout Egi
Apply Too Large Force
Get Stuck Bay
Moving Vilouu Grape
Break Grapes
[c) Hox Key Collection
(d) Sandwich Serving
Bad Placing Modia
Large Slippery
Fail A djustment
Gel Stock
Fall to Get Egg
Egg Falling Down
Fail to Place Egi
Figure 13: Failure Cases. We present typical failure cases of the baseline method for all four tasks and analyze ze the reasons for these failures to highlight the complexity of the tasks and the importance of tactile feedback during these steps.
the bag for long time without moving, which usually happens with the vision-only policy that is confused about the states of the objects and the robot end-effector under high occlusion.
Step 4: Place grape. The robot needs to place the grapes on the plate. The task may fail if the robot uses too much force to grasp the grapes, causing them to stick in the gripper and resulting in failure. Evaluation Metrics: The task is considered successful if the robot successfully places the grapes on the plate and returns to the initial position.
B.2.3 Details for the Hex Key Collection Task
Step 1: Right Hand Grasp. The robot needs to use its right hand to grasp the tail of the hex key and lift it stably to the middle of the air. The initial position of the hex key is tricky, but it reflects a common daily life scenario where only the tail of the hex key accessible, requiring additional adjustments to insert the hex key properly. typical failure case of baselines, shown in Fig. 13 (c), occurs when the robot does not secure a stable grasp, resulting in significant slippage during the lifting process. Even if the hex key remains in-hand, this slippage can cause subsequent task failures. One observation during the experiment is the consistent small slippage during the first grasp, leading to variations in the hex key's 's in-hand pose, which adds complexity to the following steps. Evaluation Metrics: The robot successfully grasps the hex key without significant slippage.
Step 2: Left Hand Grasp. The robot needs to use its left hand to grasp the head of the hex key to ensure the following adjustment step. Evaluation Metrics: The robot left hand successfully grasp the hex key.
Step 3: In-hand Adjusiment. The robot's left and right hands need to cooperate to adjust the hex key's position so that it is in a ready pose for the following insertion. Our goal to adjust the hex key to be perpendicular to the robot's fingers, making the subsequent insertion task easier. The vision-only policy usually fails to adjust the position correctly, shown Fig 13(c), making the following insertion impossible. Evaluation Metrics: The robot's arms must cooperate to adjust the hex key's pose. The final pose should have sufficiently long tail, and the hex key should be almost perpendicular to the robot's 's fingers.
Step 4: Insertion. This step is complex because the pose of the hex key in hand varies, even the robot successfully adjusts the hex key's 's position in the last step. successful policy can implicitly reference the hex key's position in hand and make the necessary adjustments for insertion. Evaluation Metrics: The robot successfully inserts the hex key into the hole rather than placing on the table or getting stuck during the insertion process.
20

---

Task 1: Egg Steaming
Ogen Rgg Truy
Grapp Fay
Fog
Sauring Machine
Lxft Hond
Right Hand
Left Rand
Right Hand
Lait Hand
Right Hond
Eeft Hand
Right Hund
Left Hand
Righ Hand
Task Fruit Preparation
Trit Saae
Gi) and Place Plate
ni Plastio Rag
Griep Grape
Piaor Plasr
Left Hand
Hand
Left Hond
Left Hand
Right
Left Hand
Right Hand
Lett
Hand
Task 3: Hex Key Collection
Right Grepp
Hand Grop
djjstand
Essartios
Lett Hand
Right
Left Iaand
Hand
Lett Hand
Hight Hand
Lett Hand
Right Hand
Left Hand
Right Hand
Task 4: Sandwich Serving
Tnnl State (Sanbxich Serving)
(al Graep Servig Speon
db Tilt Pot
t6 Get Fried Egu
66) Reploce Prisd Foy
Left
Hland
Right
Right Hand
Hend
Figure 14: Quantative Result of Tactile Representation. Here we showcase total of four tasks. For each
task, the first row presents the real image. In the second row, visualize visuo-tactile points in unified 3D space demonstrate how tactile points infer spatial relationships hetween objects and contact The third row provides a image to clearly visualize the tactile signals.
21

---

B.2.4 Details for the Sandwich Serving Task
Step 1:
Grasp Serving Spoon. The robot needs to use its right hand to grasp the spoon and lift it into the air. Evaluation Metrics: The spoon is successfully lifted into the air with minimal slippage.
Step 2:
Tilt Pot. In order to successfully obtain the egg in the next step, the robot's left hand needs to grasp the pot's handle and tilt the pot. The gripper should not exert excessive force to ensure that the handle does not rotate within the robot's ' hand. Evaluation Metrics: The robot's left hand must successfully grasp the handle and then tilt it to a certain angle.
Step 3: Get Fried Egg. The robot's two hands need to cooperate retrieve the fried egg. The right hand will use the spoon to reach the bottom of the pot and maneuver beneath the egg. During this process, the spoon will passively rotate in the hand. Our visuo-tactile policy can explicitly track the states of the spoon, while the baseline policy often fails due to the spoon's 's rotation in the hand. Evaluation Metrics: The robot successfully retrieves the fried egg with the spoon.
Step 4: Replace Fried Egg. The robot needs to move the spoon to the top of the bread and tilt it to place the egg on the bread. A typical failure occurs when the robot does not perform successful tilt motion due to changes in the spoon's position within the hand. Our visuo-tactile policy can account for these changes and adjust the motion accordingly. Evaluation Metrics: The robot successfully places the fried egg on top of the bread.
B.3 Single Arm Experiments
Light Bulb Retrieval: (i) Rotate. The robot locates the light bulb, rotates it, and retrieves it from the base. (ii) Relocation. The robot places the retrieved light bulb carefully the table.
Peg Insertion: (i) Grasp. The robot grasps a peg from the table. (ii) Insertion. The robot inserts the peg into hole.
Single-Arm Tasks
Light Bulb Retrieval (30 demos)
Peg Insertion (30 demos)
Modalities
Rotation
Relocation
Grasp
Insertion
RGB Only
0.85
0.85
1.00
0,30
RGB w/ Tactile Image
0.95
0.95
1.00
0.65
PC, Only
0.85
0.85
1.00
0.65
PC, w/ Tactile Points (Ours)
1.00
1.00
1.00
0.85
Ours (Using New Set of Sensors)
1.00
1.00
Ours (Changing Sensor Location Slightly)
0.95
0.90
Table 3: Comparison with Baselines. We evaluate our single-arm policy over 20 episodes and the best
performance for each task is bolded. Our sensors are repeatable across devices after normalization. Even though the reading may have slight variances among different sensors, the force pattern and distribution are almost. identical after normalization.
B.4 Learning Details
We use Pointnet++ as the learning backbone. we employ hierarchical feature extraction and processing for point cloud data. We use three set abstraction layers: the first set abstraction layer processes 64 points with 0.04 radius and 16 samples using multi-layer perceptron (MLP) with layers [64, 64, 128]; the second set abstraction layer processes 16 points with 0.08 radius and 32 samples using an MI.P with layers [128, 128, 256]: the third set abstraction layer serves as global abstraction layer with an MLP of [256, 512, 1024]. For further feature processing, we use fully connected layers: the first fully connected layer transforms 1024 features to 512, and the second fully connected layer reduces 512 features to 256. We disable batch normalization layers.
22


Reviews:
[{"review_id": "jvV9qrs5Re", "reviewer": "Reviewer_nMTy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a multi-modal sensing and learning system designed for bimanual manipulation. This system is equipped with new tactile sensors and the proposed system is used to perform multi-modal imitation learning on several manipulation tasks.", "summary_of_recommendation": "I think the paper is a systems paper as the algorithmic novelty is minimal. So, I am not sure if the paper is suitable for CoRL.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "Please answer the following questions:\n1. What is the motivation behind the use of the proposed sensors? To make the proposal of the proposed sensors more convincing, the authors should compare the performance against the vision-based tactile sensors like gelsight. In my opinion, the proposed approach would work with other vision-based tactile sensors too. This limits the novelty of the proposed work.\n2. What is the input to your diffusion training network? Is it the sequence of the visuo-tactile point cloud? or there are additional variables like grasping force?\n3. It is interesting to see that some of the failure modes of the baseline (vision-only) policy is failure to grasp or grasping too tight. Im wondering how can the tactile-based policy react to such failures? Unless you allow the grasping force to the adjusted as part of the policy, it may not be possible. Can you please elaborate on this?\n4. A follow up question : you show that the diffusion policy can react to failed grasp during execution. However, the vision-base policy is unreactive? like it doesn't react to the cases it fails to grasp the egg (as the arm moves, one can see the egg). please elaborate on this failure mode too.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6XD1E6f4OE", "reviewer": "Reviewer_qtTt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work introduces a 16x16 piezoresistive flat tactile array, which they install on a pair of grippers for bimanual tactile tasks, and apply visuotactile imitation learning via human teleoperation for a range of practical tasks related to food preparation.", "summary_of_recommendation": "good, well-polished paper of the level for CoRL", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "See above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0keCXZuT3C", "reviewer": "Reviewer_AvTY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper utilizes tactile sensors with resistive sensing mechanism in a bimanual gripper setting to build a 3D continuous visuo-tactile point cloud by aligning both modalities' inputs in the same space, and employs imitation learning to learn behaviors from this unified input. The novel contributions of this paper include the design of a new tactile sensor, which differs from optical and magnetic alternatives, and the use of continuous tactile values to build a 3D visuo-tactile point cloud for downstream imitation learning. This approach differs from that of Yuan et al. [27], who used binary tactile readings to build a similar unified point cloud.  The tasks evaluated by the authors are impressive, showcasing the importance of tactile feedback and the unified point cloud representation in these tasks.", "summary_of_recommendation": "I recommend accepting this paper, as I find the overall framework valuable to share with the community. Even though there is not significant originality in the method, a unified continuous point cloud with vision and tactile inputs has been a missing input modality for learning so far. The tactile sensor seems to be robust, the overall execution is well done, and the resulting tasks are impressive.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Here is the list of additional experiments and recommendations that I believe could improve the paper:\n\n**Tactile Sensor Related Improvements / Recommendations:**\n\n- Could you add an experiment to showcase the transferability of the tactile readings? This could be done by repeating one of the task's results with a new set of sensors: 1) Collect demos with one set of sensors and 2) Evaluate the model trained from those demos using another set of sensors.\n- Could you mention if the tactile sensors wear out over time? If so, how many cycles does it take for the sensors to wear out?\n- Could you add more explanation on the calibration of the sensors, if needed? If calibration is not needed, that should also be mentioned in the paper because not requiring calibration would be valuable in itself.\n\n**Point Cloud Generation Related Improvements / Recommendations:**\n\n- The authors mention that they transfer the point cloud into the robot’s base frame. However, they don’t explain how they do that exactly. Do they use ArUco markers to calibrate the cameras to the environment, or do they know where the cameras are located with respect to the robot’s base? A small explanation there would be helpful.\n- I am curious to see how sensitive the system is to the robot design and the sensors’ location with respect to the base. What happens if we move the location of the sensors a little bit during evaluation, for example?\n\n**Experiments Related Improvements / Recommendations:**\n\n- Could you include other policy rollouts of the other baselines on the website? I am especially curious how the RGB + Tactile Image baseline differs from the point cloud structure. How does the RGB + Tactile Image baseline fail compared to the PC + Tactile Points (3D-Vitac)?\n- The authors have impressive single-arm tasks showcased on the website (such as turning a light bulb). I wonder if they could briefly mention them in the paper and maybe evaluate a few of the baselines on those tasks as well.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers all agree that the paper provides a solid contribution and would be a good addition to the CoRL conference. The authors also did a good job in addressing clarity issues during the rebuttal and providing extensive experiments..
Decision: Accept