MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation
Authors: Kelin Yu
Yunhai Han
Qixian Wang
Vaibhav Saxena
Danfei Xu
Ye Zhao
Venue: CoRL 2024 (2024)
Keywords: Tactile Sensing
Learning from Human
Data Collection
Imitation Learning
Reinforcement Learning
TL;DR: MimicTouch is a multi-modal imitation learning framework that efficiently collects human tactile demonstrations, learns human-like tactile-guided control strategies from them, and zero-shot generalizes to different task settings.

Abstract: Tactile sensing is critical to fine-grained, contact-rich manipulation tasks, such as insertion and assembly. Prior research has shown the possibility of learning tactile-guided policy from teleoperated demonstration data. However, to provide the demonstration, human users often rely on visual feedback to control the robot. This creates a gap between the sensing modality used for controlling the robot (visual) and the modality of interest (tactile). To bridge this gap, we introduce "MimicTouch'', a novel framework for learning policies directly from demonstrations provided by human users with their hands. The key innovations are i) a human tactile data collection system which collects multi-modal tactile dataset for learning human's tactile-guided control strategy, ii) an imitation learning-based framework for learning human's tactile-guided control strategy through such data, and iii) an online residual RL framework to bridge the embodiment gap between the human hand and the robot gripper. Through comprehensive experiments, we highlight the efficacy of utilizing human's tactile-guided control strategy to resolve contact-rich manipulation tasks. The project website is at https://sites.google.com/view/MimicTouch.

Mimic Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation
Kelin Yu",,, Yunhai Han" Qixian Wang Vaibhav Saxena', Danfei Xu Ye Zhao
"qqual Contribution
Georgia Institute of Technology
Zhejiang Technology
kyu85, yhan389, qxian, vsaxena33, danfei, yezhao@gatech.co edu
Time
Humon Tactale Demonstrations
Tactile-kused Robot Executioo
Zero- Zero-sbot shot Generalization
Shifting
Tittng
Two-stage Packing
Multi- mattr-material Packing
Assembly
Figure 1: The first row shows the human tactile demonstrations, including the tactile and proprio- ception data. The second row shows the robot execution with tactile feedback. The third row below dashed line describes the policy's s zero-shot generalization capability in five different domains, including variations in hole positions, angles, inner shapes, materials, and different assembly task.
Abstract: Tactile sensing is critical to fine-grained, contact-rich manipulation tasks, such as insertion and assembly. Prior research has shown the possibility of learning geetiee-guieed policy from teleoperated demonstration data. However, provide the demonstration, human demonstrators often rely visual feedback control the robot. This creates gap between the sensing modality used for controlling the robot (visual) and the modality of interest (tactile). To bridge this gap, we introduce "MimicTouch", novel framework for learning policies di- rectly from demonstrations provided by human users with their hands. The key innovations are i) a human tactile data collection system which collects multi- modal tactile dataset for learning human's tactile-guided control strategy, ii) an imitation learning- based framework for learning human's 's tactile-guide control strategy through such data, and iii) online residual RL framework to bridge the embodiment gap between the human hand and the robot gripper. Through compre- hensive experiments, we highlight the efficacy of utilizing human's tactile -uided control strategy to resolve contact-rich manipulation tasks. The project website is at https::/ //sites google. com/ve http//ieee///////// coouch.
Keywords: Tactile Sensing, Learning from Human, Imitation Learning
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

6D Delta Action Space
Neighbor
Retarget
Imittaiio Learning
Nearest Action
Multi- Modal
Offiine
Proprioception
Obseration
Real-Time
Policy
Library
Oiservation
TOOL
Tactile Images
Funion
Final Policy
Residual Poliey
Akgemmnne
*
RI-Aligo
Audio Signals
(s) Collection
Representation Learning
(d) Ouline RL Fine-tuning
Figure 2: Illustration of the MimicTouuch Framework. In part (a), collect the multi-modal human tactile demonstrations. In part (b), we learn compact low-dimensional tactile representations. In part (c), derive offinn policy through non-parametric imitation learning method. In part (d), refine the offline policy through online residual reinforcement learning on a physical robot.
1 Introduction
Enabling robots to perform contact-rich tasks such as insertion remains formidable challenge in robotics. The primary reason is the complex dynamic interaction between the robot and the object which is influenced by various factors including intricate material properties and low tolerance for error This necessitates an adaptive, data-centric insertion mechanism that utilizes real-time sensor feedback. Recent methods have heavily explored vision-based solutions to tackle this problem [I- 4]. Notably, NVIDIA's sim-to-real transfer approach [1] achieves success rates of up 99.2% in transferring assembly tasks using their customized "Factory" simulator [2]. However, vision- based approaches fall short when visual feedback compromised by cluttered occlusions bad lighting conditions.
Humans exhibit fine-grained manipulation skills through tactile sensing, which allows for successful insertions by solely using tactile feedback to generate complex, continuous, and precise motions [5]. Motivated by this, recent studies collect demonstrations that combine various sensory inputs for pol- icy learning [6-8]. However, these methods assume limited action space, e-g., only 3D translations, to compensate for the complexity of collecting dynamic demonstrations. They also heavily rely robot teleoperation systems [9-11], which inevitably creates a gap between the visual sensing used for data collection and the recorded tactile sensing for policy learning. As a result, these methods are unable to emulate human's 's tactile-guided control strategy for contact-rich tasks.
To tackle these challenges, we present "MimicTouchi" (shown in Fig. 2), novel framework that enables robots to learn human's 's tactile-guided control strategy. Specifically, MimicTouch first intro- duces human tactile-guide data collection system to gather multi-modal tactile feedback (tactile audio) directly from human demonstrators. Next, it incorporates a representation learning model to capture task-specific sensor input features. These compact representations enhance the performance of subsequent imitation learning by abstracting essential sensory information. Then, it employs a non-parane non-parametric imitation learning method [12] derive an offline policy from the collected human tactile demonstrations, Finally, it leverages online residual reinforcement learning (RL) to fine-tur the offline policy the physical robot, aiming to bridge the embodiment gap between the human hand and the robot gripper and enrich contact reasoning.
We conduct comprehensive experiments on contact-rich insertion tasks to evaluate the offline pol- icy derived from demonstrations and the final policy fine-tuned via RL. We find that MimicTouch can collect tactile demonstrations more efficiently than teleoperation. More importantly, the Mim- TcTuu policy can be effectively learned from such demonstrations and significantly outperforms the policy learned from teleoperation data. Additionally, set up generalization tasks in five different domains and show the final policy exhibits superior zero-shot generalization capability.

---

2 Related Works
Multi-modal tactile sensing. Vision-based tactile sensing is integral to robotic manipulation as excels estimating local contact geometry and frictional properties [13-16]. It further enables imitation learning- based methods through policy observations [6, 7, 17] reinforcement learning methods through reward signals [18-20] for various contact-rich manipulation tasks. Additionally, it is also widely used in shape reconstruction [21-23] and grasping [24-27]. On the other hand, audio- based tactile sensors, such as contact microphones have also been demonstrated effective in robotics applications such as manipulation [28], classification [29, 30], and dynamics modelling [31, 32]. These sensors can emulate nerve endings within human skin better detect vibrations during tactile interactions. Therefore, incorporating both sensor modalities can yield tactile feedback more akin to human sensations, enabling the robot to learn human-like tactile- tactile-guided control strategy.
Learning from human demonstrations. Learning from human demonstrations is a long-standing research topic. One group of methods learns the robot behaviors from human videos [33-35]. How- ever, these methods adopt only visual sensing, which can easily collected by low-cost cameras. While visual sensing can provide high-level scene understanding, contact-rich tasks require tactile feedback for precise execution. To incorporate the tactile data into the demonstrations, recent works robot teleoperation systems [6-8, 8, 36-38), where human uses teleoperation system to control sensor-equipped robot during data collection. However, human operators must guide the robot us- ing visual feedback, thereby creating a gap between the visual sensing used for data collection and the tactile sensing recorded for policy learning. Furthermore, collecting 6D dynamic motions for contact-rich manipulations via telcoperation is challenging. Therefore, in this work, we propose to collect human tactile demonstrations, in which the sensing gap is addressed and the demonstration motions are more versatile.
Imitation learning, Offline imitation learning (IL) is an effective strategy to learn robot policies in the real world. We consider two classes of IL methods: parametric methods [39-41] and non- parametric methods [12, 42, 43]. Parametric methods typically train neural networks to map ob- servations to expert actions. While general principle, they are prone to covariant shift and com- pounding errors [44]. Our method instead adopts a non-parametric imitation learning method. These methods constrain robot behaviors the demonstrated data via techniques such nearest neighbr lookup [12]. While they may be less general, they offer safer alternative to their parametric coun- terparts, which is crucial for the real-world contact-rich manipulation tasks considered in this work.
3 Mimic Touch Framework
We aim to enable the robot to resolve contact-rich manipulation tasks by learning tactile-guided control strategy from human demonstrations. To achieve this, we propose a novel learning frame- work named "MimicTouch".. It first introduces human tactile demonstrations (Sec. 3.1) collected by human hands. Then, to emulate the human's tactile-guided control strategy for successful robot execution, MimicTouch has three learning phases. Firstly, learns lower dimensional tactile rep- resentations from the human tactile demonstrations in self-supervised manner (Sec. 3.2). Next, it derives an offline policy with the learned representations using non-parametric imitation learning method [12] (Sec. 3.3). Lastly, it refines the offline policy through online residual RI. on the real rSeo 3.4). The overall MimicTouch framework is shown in Fig. 2.
3.1 Collecting Human Tactile Demonstrations
To collect tactile demonstrations, current teleoperation systems have three key limitations: i). limited scalability due to the need for robot to collect demonstrations [10], ii). long training time and expertise to become proficient with the system for fine- fine-grained manipulation, and iii). sensing gap between the visual sensing used for collection and recorded tactile sensing. To address these, our key innovation is a system that enables humans to provide tactile demonstrations with their hands. The system is elaborated in Fig. 6 (Appendix. A), and it collects the pose of human fingertips, tactile images, and audio signals when human demonstrators perform contact-rich insertion tasks.
The data collection system consists of the following components. We use the RealSense camera with Aruco Marker [45] for human fingertip pose tracking. The tracked fingertip poses are then
3

---

treated as the robot end-effector's poses after calibration and filtering (Appendix. C.2). We also use the GelSight Mini [46], a compact vision-based tactile sensor that is conveniently mounted onto human fingertips using a custom fixture, to estimate the contacts between the object and the fingertip. Notably, we only use one tactile sensor in our experiment setup instead of two. The Audio data, which helpful for manipulation tasks due to its sensitivity to contact vibration signals [28, 47], is captured using the HOYUJI TD-11 piezo-electric contact microphone. Considering the potential discrepancies in the mechanical vibrations between the human and the robot, the microphone is placed at the base of the insertion hole to ensure signal consistency.
3.2 Learning Tactile Representation
The policy learning on high-dimensional sensor inputs struggles with real-world deployment due to computational burden and sensor noise. Additionally, in this work, variations may arise be- tween sensor inputs from human tactile demonstrations and real-time robot feedback due to different finger-object contact force. Inspired by recent works that learn lower-dimensional embeddings for imitation learning [6, 12, 48], we learn the compact representation for both tactile and audio data ing self- self-super learning methods (part (b) in Fig. 2). Intuitively, it identifies low-dimensional embedding space for different augmented data (tactile images or audio spectrum) and projects them to similar embedding. As a result, these embeddings are more computationally efficient and more robust to task-irrelevant sensor noise. This learning phase consists of the following two parts:
Data collection. We collect task-specific tactile-audio data from the human demonstrator, The dataset encompasses successful, failed, and sub-optimal demonstrations. For each, we segment the audio data at 2Hz. In total, collect 100 demonstration trajectories, including 7657 tactile images and 1,000 audio segments. More details are shown in Appendix. B.
Self-supervised learning. We employ the Bootstrap Your Own Latent (BYOL) [49] for tactile im- ages and BYL for audio (BYOL-A) for audio segments [50], since they have demonstrated desired performance in computer vision [49], audio learning [50], and robotics [7. 28, 36]. For BYOL and BYOL-A, the inputs are tactile images and audio spectrums separately, and the embedding spaces are both configured to be 2048-dimensional. Learning details are included in the Appendix. B.
3.3 Learning Offline Policy from Human Tactile Demonstrations
The next step learning robot policy from the human tactile demonstrations. Here, one unique challenge is that the human hand moves much faster than the robot, resulting sparse temporal observation-action samples (i.e., large action values per observation). Also, the embodiment gap (e.g., different motion capabilities and contact forces) can lead to out-of-domain demonstrations, which will make parametric policies prone to covariant shift [44] (see Sec. 4.2.1 for experimental validation). As a result, we use a non-parametric imitation learning method [12] to ensure the execution efficacy of the policy learned from human tactile demonstrations (part (c) in Fig. 2). In addition to the learning algorithm, data pre- pre-processing is necessary for synchronization and the details are included Appendix. C.
Non-parametra imitation learning. We build algorithm on the NN-based framework [12], extending it to include tactile-audio representation without visual signals. At the i-th time step, the observations and actions are denoted as (of of FE, where o" is the tactile representation, A is the audio representation, EE is the robot end-effector pose, and the action a defined by the 6D delta pose of the robot end-effector. Then, we extract tactile and audio features (vf y from (of , o²²) using the pre-trained representation encoders, respectively. These tactile embeddings and the robot end-effector pose (y²,.y' y. , E are formulated as the key features of the demonstra- tion library. Given the varying scales of these inputs, normalize them such that the maximum distance for each input is unity in the library. In robot execution, for given real-time observation (o) 6, (.E) we first obtain the query feature (V7, ..A oFF), and then search the demonstration library for a nearest-neighhor-bbsee action prediction.
3.4 Learning Residual Policy through Online Reinforcement Learning
The offline policy learned from human tactile demonstrations might not guarantee task success when deployed on the physical robot. This could be due to: i). morphological differences between the

---

human hand and the robot gripper, ii). inaccurate fingertip tracking caused by fast movements, and iii). underexplored contact effects. Therefore, motivated by recent works using pure RI. [17-20, 51] to learn tactile policies, we further leverage online reinforcement learning that allows in-domain robot interactions (part (d) of Fig. 2). generate quasi-static motions and utilize a limited action space [17-19, 51] because they learn from scratch without effective motion priors. On the contrary, intend to leverage the best of both advantages by RL fine-tuning the offline policy learned from human tactile demonstrations.
noteworthy that the previous pure RL methods often
Since it is infeasible to directly fine-tune the non-parametric policy, we instead learn a residual policy. The input to the residual policy गy consists of the current observation (53" S o the last policy output, which is the sum of the offline policy output and the residual policy output + Considering use 6D continuous action space and sparse observation-action pairs (around 70 actions per trajectory),
opt for SAC [52] to handle the continuous action space with entropy regularization and to generate replay buffer to increase the size of training data. Finally, the robot action is the su, of the offline policy output and the residual policy output. To ensure that the residual policy only makes slight adjustments, we set specific action limits for the residual policy.
Another critical component of residual RL is the reward design, which must balance exploitation and exploration. To address this, we combine an expert-aligned reward, which is measured by the KI. di- vergence between the human expert trajectory and the robot executed trajectory, with task-specific reward. The expert-aligned reward encourages policy distribution that mimics the demonstrations, whereas the task-specific reward drives exploration to optimize the in in-domain robot policy. More details about pseudocode, action and policy design, and rewards are included in Appendix. D.
Experiments
In this section, we first describe the experiment setting and the data collection throughput to high- light that MinicTouch can efficiently collect useful demonstrations (Sec. 4.1). Then we introduce the Offline Policy Evaluation to validate the efficacy of the offline policy and highlight the benefits of using human tactile demonstrations (Sec. 4.2), and the Online Policy Improvement and General- ization Evaluation demonstrate the efficiency of learning the residual policy through online RL and the superior zero-shot generalization capability (Sec. 4.3).
Hardware setting, All experiments are conducted on a Franka Emika Panda Arm. For each task, the learned policy generates the 6-DoF pose command and then maps it to 7-DoF joint torque actions using an inverse kinematics solver and low-level built-in controller.
Teleoperation setting. We compare our human tactile-guided data collection system (Sec. 3.1) with Spacemouse-basse teleoperation, a commonly used interface for manipulation tasks [9, 11, 37], and a Hand-guided teleoperation similar to Kinesthetic teaching (See Appendix. E for details). For both, to collect a similar number of observation-actin pairs for each trajectory, we collect one robot state, one tactile image, and 0.5s audio segment at 5Hz. Since Spacemouse based teleoperation requires considerable expertise, we allocate approximately hours for practice with this system.
Tasks. We focus on insertion tasks that exemplify the challenge of many contact-rich manipulation tasks. We 3D-print a cylinder and an insertion hole base and set up the same task environments for both data collection settings. An example has been shown in Fig. 1, and the dimensions of each piece, including those used in the generalization tasks (Sec. 4.3), are detailed in Appendix. F.
4.1 Human Tactile Demonstration Collection System
In this section, we demonstrate that Human Tactile Demonstrations can greatly improve data col- lection throughput for contact-rich manipulation tasks. We begin by determining the usability of a demonstration trajectory based on the following two metrics: i) the object is successfully inserted into the hole without any slipping or falling, and ii) the task is completed within 100 actions. Using these criteria, we record the time length of collecting 20 usable demonstration trajectories by using our customized system, the Spacemouse-based teleoperation and the Hand-guided teleoperation. Then, evaluate the data collection throughput (see Table. 1) in two metrics: i) the number of us- able demonstrations collected per hour, and ii) the success rate for collecting usable demonstrations.
5

---

The results in Table 1 support our insights: i) human tactile demonstrations can be collected significantly more efficiently than teleoperation systems for contact- rich tasks, and ii) human tactile demonstrations seamlessly integrate human's tactile feedback and motion capability, whereas teleoperation systems struggle to capture such dynamic tactile-guided motions. These factors together result in much lower data collection efficiency and success rates of the teleoperation systems.
Methods
Frequency
Success Rate
Spacemouse Teleoperation
19 traj/hr
38.5% (20/52)
Hand Teleoperation
traj/hr
58.8% (20/34)
Human Tactile Demonstrations
104 traj/hr
83.3% (20/24)
Table 1:
Data collection throughput for Human Tactile Demon- strations and Telcoperation systems.
4.2 Offfine Policy Evaluation
4.2.1
MimicTTuch effectively learns from human tactile demonstrations
In this subsection, we evaluate offline policies learned from human tactile demonstrations using both nearest-neighbor and parametric imitation learning method. Our goal is to test if these policies can perform within the desired error tolerance in real-world settings, which is crucial for physical robot execution. To evaluate it, we use our data collection system (Sec. 3.1) to collect 20 noise-fres free data sequences as the datasets to learn both offline policies. For testing, we gather another 5 data sequences with random noise to emulate the real-world environments (details are in Appendix. G), and compute the mean square error (MSE, defined in Appendix. H) on the testing set. Also, we conduct the real-world experiments and compare the task success rate over 25 policy rollouts.
For the baseline parametric imitation learning method, we select the MULSA [6], which has been previously demonstrated effective in multisensory robot learning for insertion tasks. In our setting, we use the same sensor input (yT .ya o FF as in the NN-based method to generate the continuous 6D delta action a,. The same MSE loss is used for policy training and validation.
For both offline policies, we calculate the MSE losses on the testing set. We observed that the MSE loss from the NN-based policy is 0.21, which is significantly lower than that of MULSA policy (1.53). Similarly, the NN-based policy achieves a 40 % (10/25) task success rate, while the MULSA policy achieves only 16% (4/25). We further provide three failure rollouts of the MULSA policy in Appendix I, to illustrate the reasons for its poorer performance. These results together suggest that the NN-based policy can generate more reliable 6D continuous actions, indicating more suitable for subsequent online real-worid RL fine-tuning.
Additionally, we conduct ablation study different sensor inputs in Appendix. J. The results sug- gest that multi-modal tactile feedback is crucial for the success of contact-rich insertion tasks.
4.2.2 Human tactile demonstrations trains better policies than teleoperated demonstrations In this subsection, we compare the performance of the offline policies trained from human tactile demonstrations and teleoperation demonstrations. We collect 20 trajectories for each, and then we the same NN-based method to learn the offline policies for each set of demonstrations. We evaluate the policies in two manners: i) the task success rate over 25 policy rollouts, and iī) the action serial numbers (right part of Fig. 3), i.e., the indexed numbers of the selected actions in the corresponding trajectory of the demonstration library, for each action of the rollout trajectories.
The task success rate for offline rollouts from human tactile demonstration is 40% (10/25), whereas the success rate from Spacemouse- teleoperation rollouts is only 12% (3/25), and it from Hand-guided teleoperation is 28% (7/25). Then, in the bottom part of Fig. we show the mean and variance of the action serial numbers of three successful rollout trajectories per policy. Human tactile demonstrations show a linear trend with minimal variance, while teleoperation policies dis- play more nonlinearity and higher variance, especially during the insertion phase. This performance diserepancy arises because the majority of contacts occur during this phase, and the teleoperation systems which lack of human tactile feedback is not well-suited for capturing these contact-rich events. A similar conclusion can be drawn from the sereenshots (top part of Fig. 3) for one suc- cessful rollout of all policies. Notably, the Human Tactile Demonstrations policy exhibits dynamic tilting for object insertion, which emulates the human's 's tactile-guided control strategy (see Fig. 1).

---

Spaceroous Teleoperation
Hand Teleoperation
Human Tactile Demonstration
Reach Insertion
Hand eozeration Patty Rellut
Harrnn Bett Demoretration Fottry Soller
Suicted Actinn Nambel
Scridted Actien Number
Selected
Peach
Insottio
Reach
Insertion
Boatime Action Numbo
Resetime Action Narrber
Reaiime Artion Number
Figure 3: Top: Qualitative results for Spacemouse-basee teleoperation, Hand-guided teleoperation, and Human Tactile Demonstration policies. Bottom: Visualization of the action serial numbers for three successful rollout trajectories generated by each policy. Solid red lines indicate mean trends and shaded areas show standard deviations. The left side of the dashed orange line the Reach phase, and the right side is the Insertion phase.
Human Tactile Demonstrations Policy Rollout
Success Rates Over Training Time
SpacaMoru
Training Samples
HondGuide
Seccu 1%)
60
95
20
121
100
150
Traning Tima (1866)
Qualitative Result
Quantitative Result
Figure 4: Left: Demonstrations of the online RL fine-tuning process, which further improves the task performance. Right: Quantitative task evaluations for offline policies learned from teleoper- ation demonstrations (SpaceMouse and HandGuided) and human tactile demonstrations (Human Tactile) during online RL fine-tuning show that Human Tactile significantly outperforms others in terms of task success rate and RL training efficiency.
Therefore, combined with the results in Sec. 4.1, MimicTouch not only efficiently collects human tactile demonstrations, but also enables effective policy learning using these demonstrations.
4.3 Online Policy Improvement and Generalization to New Settings
In this section, we evaluate the final policy trained through online RL. ensure the robustness of the online policy, we perform domain randomization on the robot's starting state so that the initial object-hole contact is located differently. For the policy update, each iteration, collected trajectories along with another five randomly selected trajectories from the replay buffer.
five newly
Online RL fine-tuning significantly and efficiently improves task performance. We evaluate the trained policy every 20 minutes, approximately after every 13 RL epochs. After each hour, we com- pute the task success rates over 25 policy rollouts, alignment with the offline policy evaluation. For other time instances, we only compute the task success rates over policy rollouts to minimize sensor wear. The evaluation results are shown in Fig. 4, and we can observe that the policy learned with human tactile demonstrations it can reach 88% (22/25) task success rate after 2-hour RL fine-tuning, which is significantly more
reach 96% (24/25) task success rate in 3 hours. Besides,

---

1
Shifting 0.8 cm
TTltiin degree
Tilting 20 degree
Twu-stage Packing
material Packing (Soft)
Mulli-material Packing (Rigid))
Assembly
Figure 5: Setup of zero-shot generalization tasks.
training efficient than the policies learned from both teleoperation demonstrations, which could only achieve 32% (8/25) and 60% (15/25) task success rates at that time, respectively. This result sup- ports the effectiveness of online RL fine-tuning, it allows the robot to further interact with the task environment. Moreover, it once again highlights the importance of using human tactile demonstra- tions since the offline policy learned from teleoperation demonstration exhibit significant training inefficiency. Finally, we compare the action outputs of the offline and online components of the MimicTouch final policy in Appendix. K to measure each one's contribution.
MimicToocch policy exhibits superior zero-shot generalization capability. We evaluate the zero- shot generalizability of the MimicTTouch final policy. We consider the following generalization set- tings: i). Shifting Positions (Shift): an insertion task with the hole shifting for 0.8cm in either tæ or ty directions, ii). Tilting Angles (10 and 20%); ): an insertion task with the hole angle tilted for or 20,, iii). Two-stage Dense Pucking (Two-stage); a two-stage dense insertion task requires the robot perform two consecutive alignments, iv). Multi-material Dense Packing (Rigid and Soft): an insertion task where the box contains multiple objects, such as pen, tissues, or sponge, and v) Furniture Assembly (Assem(I)) and Assem) [53, 54]: an insertion task requires the robot to insert and thread the leg into small hole of table. Each setting is depicted in Fig. 5. To demonstrate the complexity of these tasks, we introduce a baseline: Openloop Policy, where we collect five success- ful insertion trajectories from the initial setting and execute each of these trajectories five times for each the tasks. See Appendix. for the details of each task and the policy evaluation process.
Policy
Shift
10
20
Two-stage
Rigid
Soft
Assem (I)
Assem
Openloop
24/40
14/25
10/25
13/25
13/25
9/25
8/25
3/25
MimicTouch
37/40
23/25
20/25
22/25
20/25
16/25
19/25
13/25
Table 2: Task success rate for each generalization task.
We report the task success rate of both policies Table. Based on these results, we can observe that the MinicTouch policy can significantly outperform all the openloop policies in all those gen- eralization tasks. Also, since the Openloop policy has lower success rates in all the generalization tasks, it indicates the robustness of the Mimic Touch policy in different challenging generalization domains. We also include the qualitative results and detailed analysis in Appendix. M.
5 Limitation and Future Work
MimicTouch pioneers the pathway to learning human tactile-guiie guided control strategies from human tactile demonstrations. However, it still has several limitations for future improvements. Firstly, MimicToucch still requires several hours to refine the policy for addressing the embodiment gap. We will explore better representation learning method to reduce the gap between humans and robots. Secondly, MimicTouch task-specific and can not directly generalize human's metile--uided con- trol strategy to other tasks. One potential solution is to learn generalizable tactile-based dynamic model for different tasks. Thirdly, the method of learning to perform contact-rich manipulation tasks from human tactile demonstrations could be extended to other robot tasks, such as dexterous manipulation, bimanual manipulation, and soft object manipulation.
6 Conclusion
We presented MimicTouch, a multi-modal imitation learning framework that: i) enables humans to perform tactile demonstrations with their hands without a robot in the loop. ii) learns from such demonstrations and safely transfers robot with non-parametric imitation learning, and iii) im- proves the policy performance with residual-based online RL bridge the human-robot embodi- ment gap. We show that MimicTouch enables tigh-throouhhut data collection and achieves high success rate and generalization across wide range of two-piece insertion and assembly tasks.
8

---

References
[1] B. Tang, M. A. Lin, I. Akinola, A. Handa, G. S. Sukhatme, F. Ramos, D. Fox, and Y. Narang. Industreal: Transferring contact-rich assembly tasks from simulation to reality. In Proceedings of Robotics:
Science and Systems (RSS), 2023.
[2] Y. Narang, K. Storey, I. Akinola, M. Macklin, P. Reist, L. Wawrzyniak, Y. Guo, A. Moravan- szky, G. State, M. Lu, A. Handa, and D. Fox. Factory: Fast contact for robotic assembly. In Proceedings of Robotics: Science and Systems (RSS), 2022.
[3] G. Schoettler, A. Nair, J. Luo, S. Bahl, Aparicio Ojea, E. Solowjow, and S. Levine. Deep reinforcement learning for industrial insertion tasks with visual inputs and natural rewards. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 5548 5555, 2020. doi: 10. 10/11009///0004445220000......
[4] G. Schoettler, A. Nair, A. Ojea, S. Levine, and E. Solowjow. Meta-reinnorrcement learning for robotic industrial insertion tasks. In 2020 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 9728 9735, 2020. 0000:0..1000000444452000000000.0
[5] G. Lengyel, G. Zalalyte, A. Pantelides, J. N. Ingram, J. Fiser, M. Lengyel, and D. M. Wolpert. Unimodal statistical learning produces multimodal object-like representations. eLife, 8:e43942, may 2019, ISSN 2050-084X. doi:10.75544eeife-4494. URL https://doi.onrr 10. 75.7554/eLLff eLife...eeeee 43942.
[6] H. Li, Y. Zhang, Zhu, S. Wang, M. A. Lee, H. Xu, E. Adelson, L. Fei-Fei, R. Gao, and J. Wu. See, hear, and feel: Smart sensory fusion for robotic manipulation. In CoRL, 2022.
[7] Guzey, B. Evans, S. Chintala, and L. Pinto. Dexterity from touch: Self-supervised pre- training of tactile representations with robotic play. In 7th Annual Conference Robot Learn- ing. 2023. URL https: http::::////peeeeeeeeeee net peet/forumtt www.fooooooeooioor x=x.XoxxxX33......
[8] A. Mandlekar, Y. Zhu, A. Garg, Booher, M. Spero, Tung, Gao, Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In A. Billard, A. Dragan, J. Peters, and J. Morimoto, editors, Proceedings of The 2nd Conference Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 879-893. PMLR, 29-31 Oct 2018. URL https://proceeding. mlr. mlr.press www.press/vvee v87 http//88/mmmmnl/// mand/ ekar 1aaar18a 18a.html. html.
[9] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[10] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. In Proceedings of Robotics: Science and Systems (RSS), 2024.
[11] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. 6th Annual Conference on Robot Learning (CoRI.), 2022.
[12] Pari, N. M. Shafiullah, S. P. Arunachalam, and Pinto. The surprising effectiveness of representation learning for visual imitation. In Proceedings Robotics: Science and Systems (RSS), 2022.
[13] W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for esti- mating geometry and force. Sensors. 17(12), 2017. ISSN 1424-8220, doi:10.3330/s1111222 URL https: htop::://www..e www.:..pp...com http:///2222222/1/1022 221001//1///2222222

---

[14] E. Donlon, S. Dong, M. Liu, J. Li, E. Adelson, and A. Rodriguez. Gelslim: A high- resolution, compact, robust, and calibrated tactile-sensing finger. In 2018 IEE//RS Inter- national Conference on Intelligent Robots and Systems (IROS), pages 1927-1934, 2018. doi: 10.1109/RROS.0 2018.8593661.
[15] D. Ma, E. Donlon, S. Dong, and A. Rodriguez. Dense tactile force estimation using gelslim and inverse fem. In 2019 International Conference Robotics and Automation (ICRA), pages 5418-5424, 2019. doi: 10. 10i:10.1119/ICC LO.9//CAA.2199.8799911111
[16] S. Dong and A. Rodriguez. Tactile- based insertion for dense box-packing, In 2019 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 7953 2019. 10i:10.1109//ROSS 0et.110009/RS00899..000 8.682244
[17] M. A. Lee, Y. Zhu, K. Srinivasan, Shah, Savarese, L. Fei-Fei, A. Garg, and J. Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 IEEE International Conference Robotics and Automation (ICRA), 2019. URL ptcttp:///rxiv...t
/1///1110...0 10191.
[18] S. Dong, D. K. Jha, D. Romeres, S. Kim, D. Nikovski, and A. Rodriguez. Tactile-rl for inser- tion: Generalization to objects of unknown geometry. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6437-6443, 2021. doi:10. 10011000//CAA4455022222222 9561646.
[19] Kim and A. Rodriguez. Active extrinsic contact sensing: Application to general peg-in-hole insertion. In ICRA, 2022.
[20] Z.-H. Yin, B. Huang, Y. Qin, Q Chen, and X. Wang. Rotating without seeing: Towards in- dexterity through touch. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[21] S. Suresh, Z. Si, J. G. Mangelson, W. Yuan, and M. Kaess. Shapemap 3-d: Efficient shape mapping through dense touch and vision. In 2022 International Conference on Robotics and Automation (ICRA), pages 7073-7080, 2022. 10. 119//CR 200.CCRRAA46692222...99 2022.981204)
[22] E. J. Smith, R. Calandra, A. Romero, G. Gkioxari, D. Meger, J. Malik, and M. Drozdzal. 3d shape reconstruction from vision and touch. arXiv preprint arXiv:2007.0377: 2020.
[23] E. Smith, D. Meger, Pineda, R. Calandra, J. Malik, A. Romero Soriano, and M. Drozdzal. Active 3d shape reconstruction from vision and touch. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 16064-16078. Curran Asso- ciates, Inc., 2021. URL https://proceeddingg neurips. wttp:/eer.....pppprr.ffieeeeeee. ppaper_files/ppperr .df.
[24] R. Calandra, A. Owens, D. Jayaraman, Lin, W. Yuan, Malik, F. Adelson, and Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters, 3(4):3300-3307, oct 2018. doi: 1019/ra..018 URL https: 2852779.
//doi.
[25] I.. Pinto, D. Gandhi, Y. Han, Y.-.. Park, and A. Gupta. The curious robot: Learning visual representations via physical interactions. In European Conference on Computer Vision, pages 3-18. Springer, 2016.
[26] R. Calandra, A. Owens, M. Upadhyaya, W. Yuan, J. Lin, E. H. Adelson, and S. Levine. The feeling of success: Does touch sensing help predict grasp outcomes? In S. Levine, V. Van- houcke, and K. Goldberg, editors, Proceedings of the 1st Annual Conference on Robot Learn- ing, volume 78 of Proceedings of Machine Learning Research, pages 314-323. PMLR, 13-15 www.mmww..pp......mmmmmmmmm
Nov 2017. URL https: http:///eooeeeee..ooeeeeee mlr wwwps/eeeeeeeee v78/ calanddal17 html.
10

---

[27] Y. Han, K. Yu, R. Batra, N. Boyd, C. Mehta, T. Zhao, Y. She, S. Hutchinson, and Y. Zhao. Learning generalizable vision-tactile robotic grasping strategy for deformable objects via trans- former. arXiv preprint 12.06374 2023.
[28] A. Thankaraj and L. Pinto. That sounds right: Auditory self-supervision for dynamic robot ma- nipulation. 7th Annual Conference on Robot Learning. 2023. URL WtRLhttts://operevieee net/forum? ? Le.kkekkke....
[29] J. Sinapov, M. Wiemer, and A. Stoytchev. Interactive learning of the acoustic properties of household objects. 2009 IEEE International Conference on Robotics and Automation, pages 2518-2524, 2009. doi: 10. 109/00000. 201...220000050
[30] J. Christie and N. Kottege. Acoustics based terrain classification for legged robots. 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 3596-3603, 2016. doi: 20.110/ 6.7777553.
[31] C. Matl, Y. Narang, D. Fox, R. Bajesy, and F. Ramos. Stressd: Sim-to-real from sound for stochastic dynamics. In J. Kober, F. Ramos, and C. Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 935-958. PMLR, 16-18 Nov 2021. URL https://pproceeddngg mlr press/v155/
mat12ia. html.
[32] D. Gandhi, A. Gupta, and L. Pinto. Swoosh! rattle! thump! actions that sound. In Proceed- ings of Robotics: Science and Systems (RSS), 2020.
[33] P. Sharma, D. Pathak, and A. Gupta. Third-person visual imitation learning via decoupled hierarchical controller. Advances in Neural Information Processing Systems, 32, 2019.
[34] H. Bharadhwaj, A. Gupta, S. Tulsiani, and V. Kumar. Zero-shot robot manipulation from passive human videos. arXiv preprint arXiv:2302.0201 2023.
[35] H. Xiong, Q. Li, Y-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg. Learning by watching: Physical imitation of manipulation skills from human videos. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7827-7834. IEEE, 2021.
[36] S. P. Arunachalam, I. Güzey, S. Chintala, and L. Pinto. Holo-dex: Teaching dexterity with immersive mixed reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5962-5969, 2023.
[37] C. Wang. L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. In 7th Annual Conference on Robot Learning, 2023. URL
met/forummmte 2it..
[38] M. Xu, Z. Xu, C. Chi, M. Veloso, and S. Song. Xskill: Cross embodiment skill discovery. In Conference on Robot Learning, 2023.
[39] J. Ho and S. Ermon. Generative adversarial imitation learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 29, Curran Associates, Inc., 2016. URL BtLp http:///rpoooeiinn....eippppppp cc/ faper_files/pie oaeeee//eeeeeeeeooooc/c/ccc/r/aa
pdf.
[40] M. Xie, A. Handa, S. Tyree, D. Fox, H. Ravichandar, N. D. Ratliff, and K. V. Wyk. Neural ge- ometrie fabrics: Efficiently learning high-dimensional policies from demonstration. In Liu, D. Kulie, and Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1355-1367, PMLR, 14-18 Dec 2023.
[41] Y. Han, M. Xie, Y. Zhao, and H. Ravichandar. On the utility of koopman operator theory in learning dexterous manipulation skills. In 7th Annual Conference Robot Learning, 2023. URL htttps://operreee ootprrrrreee.tttttooori///-----e OTIYra..
11

---

[42] Y. Huang, L. Rozo, J. Silvério, and D. G. Caldwell. Non- Non-parametri imitation learning of robot motor skills. In 2019 International Conference on Robotics and Automation (ICRA), pages 5266-5272, 2019. doi:10. 1109/CCA.2019.8899222.
[43] M. Vaandrager, R. Babuška, L. Buşoniu, and G. A. Lopes. Imitation learning with non- parametric regression. In Proceedings of 2012 IEEE International Conference on Automation, Quality and Testing, Robotics, pages 91-96, 2012. doi: 10. 1100/AQTR..012.222321
[44] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 3: 297-330, 2020.
[45] S. Garrido-Jurado, R. Muñoz-Salinas, F. J. Madrid-Cuevas, and M. J. Marin-Jiménez. Auto- matic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognit, 47:2280-2292, 2014. URL https://appp www.ssscscmaanclhlaa.a.... Tng/CorpusID: 12519167.
[46] Gelsight mini https///www.gggg.g........gaaaa.
[47] M. Du, O. Y. Lee, S. Nair, and C. Finn. Play it by ear: Learning skills amidst occlusion through
audio-visual imitation learning. In Proceedings of Robotics: Science and Systems (RSS), 2022.
[48] S. Haldar, J. Pari, A. Rai, and L. Pinto. Teach robot to fish: Versatile imitation from one minute of demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[49] J.-B. Grill, Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, Doer- sch, B. Avila Pires, Guo, M. Gheshlaghi Azar, B. Piot, k. kavukeuoglu, Munos, and M. Valko. Bootstrap your own latent ing. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21271-21284. Curran As- sociates, Inc., 2020. URL https://proceeddnngg neur courrpp.com 2020/
approach to self-supervised learn-
wwwurips.ccpppaaaee wwtppcpppppppppe/ee feaper_fllesppperr. Paper. Paper.ddf..
[50] D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada. and K. Kashino. Byol for audio: Self- supervised learning for general-purpose audio representation. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2021. doi: 10019000/2NN5223..000111 9534474.
[51] J. Ding, C. Wang, and C. Lu. Transferable force-torque dynamics model for peg-in-hole task, 2019.
[52] Haarnoja, A. Zhou, Abbeel, and S. Levine. Soft actor-critic: Off Off-policy maximum en- tropy deep reinforcement learning with stochastic actor. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Pro- ceedings of Machine Learning Research, pages 1861-1870. PMLR, 10-15 Jul 2018. URL https://procee:
mlr taoes//v80/haarnjjaabbbb html.
[53] M. Heo, Y. Lee, D. Lee, and J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation. In Robotics: Science and Systems, 2023.
[54] H. Lin, R. Corcodel, and D. Zhao. Generalize by touching: Tactile ensemble skill transfer for robotic furniture assembly, 2024.
[55] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 1b0.:00.1109//CPP.20000..
12

---

Appendices
A Data Collection
In this section, show the novel data collection system in Fig. 6.
Proprioception
Tactile Images
Audio Signals
Figure The human tactile demonstrations collection system.
B Representation Learning
Data Collection For Representation Learning, we collect a large task-specific dataset that contains 7657 tactile images and 1000 audio sequences. We collect 100 trajectories, each of them approxi- mately five seconds long and containing around 70 tactile images and 10 audio sequences. These trajectories contain various data qualities, which include successful cases, failure cases, and sub- optimal cases. In detail, successful cases refer the cases that human finishes the task with smooth trajectories; failure cases mean that human did not insert the object successfully or used more than five seconds to finish this task; sub-optimal cases mean that human used unnecessary motions to finish the insertion task.
BYOL BYOL [49] generates two augmented views, t(x) and v' A t'(x), from given x by applying image augmentations T and t' T' respectively, where T and T' represent distinct augmentation distributions. The architecture of BYOL comprises primary encoder fo and target encoder fę. where the latter being an exponential moving average of the former. Given the aug- mented views v and v', they are processed to yield representations y and y'. These representations are subsequently transformed by projectors gø and g: to produce dibber-iimensional vectors z and v' The primary encoder and associated projector are designed to predict the output from the tar- get projector, resulting qo (zz) and sg (f(z)) The model's output consists of lg-normalized versions of these predictions, which are trained using similarity loss function. Post-training the encoder fo utilized for feature extraction from observations.
13

---

To utilize BYOL in tactile images, we scale the tactile image up to 256x256 to work with standard image encoders. We use the ResNet [55] architecture, also starting with pre-trained weights. Un- like SSL (self-supervised learning) techniques used in visual images, we only apply the Gaussian blur and small center-resized crop augmentations, since other augmentations such as color jitter and grayscale would violate the assumption that augmentations do not change the tactile signal signifi- cantly. For each imput, the trained model will generate 1 2048 representation vector.
Audio Representation Learning BYOL-A [50] is an extended version of BYOL to audio rep- resentation learning. processing log-scaled mel-spectrograms through specialized augmentation module. To utilize BYOL-A in our audio data, we down-sampled signals from 44.1kHZ to 16kHz, with window size of 64 ms, a hop size of 10 ms, and mel-spaced frequency bins F= 64 in the range 60-7,800 Hz. Then, the Pre-Normalization step stabilizes the input audio for subsequent augmenta- tions. Once normalized, the Mixup step introduces contrasts in the audio's background, defined by the log-mixup-exp formula:
x log((1 (1 - X) exp(x:) + A exxp(xx)
where X& is mixing counterpart and À is ratio from uniform distribution. The next one is the RRC block, an augmentation technique, that captures content details and emulates pitch shifts and time stretches, For each input, the trained model will generate a 1 x 2048 representation vector.
C Data Pre-processing
C.1 Sensor Data Alignment
Each sensor operates different frequency: i) RealSense operates at 60 Hz with a resolution of 320x240 pixels, ii) GelSight Mini streams tactile images at 15 Hz with 400x300 pixel resolution, and iii) HОYUJI TD-11 piezo-electric contact microphone has a 44.1 kHz sampling rate, and the audio data is segmented at 2Hz, meaning each audio input to our framework is a 0.5s sequence containing 22,050 audio signals.
Therefore, ensure synchronization across our sensors, we first address the disparate sampling rates of the fingertip poses, tactile images, and audio sequences, which are 60Hz, 15Hz, and 2Hz, respectively. Specifically, we downsample the fingertip poses to 15 Hz. For the audio data, instead of collecting entirely new 0.5-second segments, we record the extended audio signals at intervals of every 0.07 seconds. As result, each 0.5s segment has a new-collected 0.07s interval and an old overlapped 0.43s segment in the past, which results in an overlap of 0.43 seconds between consecutive audio segments. Therefore, all sensor inputs are sampled 15Hz.
C.2 Calibration and Filtering of Fingertip Poses
The 6D human fingertip poses extracted from the AruCo marker include 3D positions along with rotation vectors. To these fingertip poses the end-effector's poses for robot policy learning, we need to address two problems: i). developing calibration method to align the data collection system with the robot execution system, and ii). implementing filtering method to generate smooth trajectories.
Calibration Given that data collection and robot experiments occur in disparate scenarios, it is crucial to align human tactile data collection system with the physical robot system. Initially, we record the distance between the object (starting point) and the base (ending point) within the data collection system and replicate this setup in the robot environment. Following this, six equidistant positions between the starting and ending points are identified within both systems. The object is gripped at these predetermined positions using both hands and the robot's end-effector so that we can capture the corresponding poses. In this calibration process, the hand poses, denoted as the "Eye" in the calibration function, are referenced to the camera frame, while the end-effector poses, represented the "Hand" in the calibration system, referenced to the robot frame. Conclusively, employ the Hand function from OpenCV, using the six captured poses, to calibrate
14

---

Algorithm Online Residual Reinforcement Learning
1: Input: offline policy To randomly initialized residual policy TT.
2: Input: step size sequences {Be}, number of iterations K, Replay Buffer D
3: Initialize replay buffer D with pre-collected data
4: for k = 1 K do
5: Sample mini-batch D from Replay Buffer D
6: Obtain current trajectory Og by executing To T
7: To collect more data in D: D <- D U Cj
8: Combine D2 and Cj to form batch By for update
9:
for all (s, ".a.rrs'). B3 do
10:
a, - -T((s)
D. Obtain offline action
11:
Or +- (7(8,00)
D. Obtain residual action
4 a, + Ur Q + Qr(s, ao) + t
Combine offline and residual actions
13:
14:
Tr + T4 - a aV x, ((Tr)
> Update residual policy with gradient step
15:
end for
16:
end for
17:
Return: Trained residual policy xy
these two frames (camera frame and robot frame). It should be noted that during robot execution, do not use camera. Therefore, we cannot collect human fingertip trajectories in the camera frame and use inverse kinematics for robot execution, as commonly done in vision-based policies.
Filtering Given the inherent noise and occasional outliers in the poses obtained from the RealSense and AruCo markers, it imperative implement post-processing techniques ensure the quality and smoothness of the trajectories. For each pose sequence, outliers are detected by sorting the values of each delta transformation (i.e., the delta translations and the delta Euler angles). The Interquartile Range (IQR) method is employed to establish the upper and lower bounds, which are then used to identify outliers. The IQR is defined as: IQR = Q3 - Q1 where Q3 and Q1 are the third and first quartiles, respectively. Outliers are replaced using a median filter with a window size of 3. To enhance the temporal consistency of the estimated hand and object pose, digital low-pass filter is applied to eliminate gigh-freueency noise. Specifically, the filter has sampling frequency of 5Hz and cutoff frequency of 2Hz. The low-pass filter can be represented as:H (f) (+) where f is the sampling frequency and fe is the cutoff frequency.
D Details of RL training
Training pipeline The overall pseudo-code for RL Training is given in Alg.
Action limits The residual RL policy is designed to make only slight adjustments to the offline IL policy. To ensure this, during RL fine-tuning, constrain the action ranges of the residual policy to (-0.15em, 0.15cm) for each x, y, translation and (-3 degrees. 3 degrees) for each Euler angle.
RL policy details For the residual policy म, within our framework, we employ the Soft Actor- Critic (SAC) algorithm with an MLP architecture. The training strategy aims to effectively combine reinforcement learning principles with residual corrections, thus enhancing the overall performance of the system. The following formula represents the objective for training the residual policy:
T. = argmax (Bre,a)<D [Q(s, a + x()) - a log n (als))}
Q(s, (s)): The Q-value function, which estimates the value of executing the residual action (Ts) in addition to the base action in the state This represents the total action influenced by both the offline policy and the residual corrections suggested by ny.
(saa--n: The expectation over state-action pairs sampled from the replay buffer D. which contains data from both past experiences and current new explorations.
15

---

a log (a)s): The entropy regularization term for the policy :, which encourages explo- ration by penalizing the certainty of the policy's action selection. This term is crucial in SAC to ensure sufficient exploration and avoid premature convergence to suboptimal poli- cies.
x (als)): The policy network (MLP) outputs the probability distribution over actions given the state 8; from which the action is sampled.
This formula ensures that the residual policy H, learns adjust the actions generated by the offline policy by optimizing the SAC objective. It balances the maximization of expected returns (via Q-values) and the maintenance of behavioral diversity (via entropy regularization), allowing म, to adapt and refine actions based on real-time environmental feedback and historical data from the replay buffer.
RL Reward Design We will give detailed explanation each component in our reward design. Distance Reward:
d = 1 - 1awhaann000 + ||distance|
where the distance is between the current position of the gripper center and the target gripper center.
Orientation Reward:
0 = t7.5 * f
where the forf stands as the quaternion difference between the current gripper orientation and the target gripper orientation. Penalty for blocking
0.2,
successfully complete this action
(1)
0,
cannot complete this action in 0.5s
Penalty for Slippery
0.5,
|
1/2 11
0.5
0,
otherwise
The yt and vt -1 stands for the embeddings of the tactile images in the current step and the last step
i
Overall Rewards
I, (PI() - -
if success otherwise
In Eqn. 3, P is the executed trajectory, Q is the expert trajectory, and Dxz(PllQ) is the KL Diver- gence between the executed trajectory and the expert trajectory, which makes the executed trajecto- ries close to the expert trajectory, ensuring that the robot does not take risky behavior. In addition, d, o, C, $ are defined above separately. The setup each weight: a = 0.5, B = 0.3, 7 = 0.2.
E Hand-guided Teleoperation Setting
As shown in Fig. 7, the human operator can directly control the robot end-effector. This method of- fers more intuitive approach data collection and has proven effective manipulation tasks [10].
F Details of the inserted objects and the holes
In all the experiments, except for the furniture assembly, the object being inserted is a cylinder with radius of 17.5 mm and height of 80 mm. In both the shifting and tilting experiments, the only variable among the holes is the angle between the hole opening and the horizontal plane, which is
16

---

Figure 7: Hand -aidee Teleoperation System.
Ø40
40
41
(b)
0.30 mm
20
R30mmn
25
0.0 mm
50mmn
R22mm
20 mm
040mm
017
(c)
Figure 8: Detailed dimensions of the inserted objects and holes.
set to 0º, 10", and 20", respectively (see Fig. 8, parts (a), (b), and (c)). For all, the hole openings have a uniform radius of 20 mm and a depth of 40 mm.
In the two-stage packing experiment, the hole consists of two sections. The first section flared hole with bottom radius of 22 mm, top radius of 30 mm, and an angle of 25 relative to the horizontal plane. The second section is cylindrical hole with radius of 20 mm. The combined height of the two parts is 50 mm (see Fig. 8, part (d)).
17

---

In the last furniture assembly experiment, the threaded section of the inserted objects is a cylinder with a radius of 8.5 mm and height of 20 mm (see Fig. 8, left of parts (e)). connects to the tabletop through a thread with a diameter of 17 mm and a pitch of 8 mm. To facilitate tightening, the radius of the threaded hole in the tabletop is 10 mm (see right of part (e)).
G Emulate the Physical Environment for Policy Evaluation
3To emulate the physical robot environment, we introduce random noise to those 10 unseen data sequences. The robot state space input undergoes a random position noise within the range [-3cm, for each axis. Gaussian noise, denoted as (O,, or, is added to both the tactile image and audio signal. In this notation, (0, or) signifies a Gaussian distribution with a mean of O and a standard deviation of For tactile images, the noise affects pixel values in the range [0, 253], while for audio data, it impacts signal values in the range [0, 1]. Given their distinct ranges, apply Gaussian noise with standard deviations of o 100 for tactile images and 0.4 for audio data.
H
MSE Loss
For calculating the MSE Loss between two action sequences, we need to normalize the actions' translation vectors and rotation vectors since they have different scales. Specifically, we use min- max normalization both the translation vectors and rotation vectors, where the max vector and min vector selected from the training set. As result, translation vectors and rotation vectors will have the same scale for calculating the MSE Loss. The formula is shown as:
MSE Σ -t)²
Where: y, represents the ground truth normalized action, ý, represents the generated normalized action, and is the number of all action steps.
I Failure Examples of the MULSA Policy
We provide three failure examples to better illustrate why the MULSA policy performs worse. As shown in Fig, 9, when the robot makes contact at the wrong position or fails to make contact, it tends drift further away from the hole. This occurs due to covariant shift and compounding errors [44], leading the policy continue select erratic actions, ultimately resulting in task failures. Based on these observations, believe the MULSA policy is less suitable for real-world RL fine-tuning.
J Ablation Study: Do Multi-Modal Tactile Feedback Improve the Task Performance?
In this section, evaluate the performance of our multi-modal tactile embeddings. We consider the following baselines: i). MinicTouch w/o T & A: MimicToochh without tactile or audio embed- dings, ii). MimicTouch (T): MimicTToch incorporating only tactile embeddings, iii). MimicToue (A): MimicTouch incorporating only audio embeddings., and iv). MimicTouch (T + A, Ours): Mim- icTouch incorporating both tactile and audio embeddings. We evaluate the policy performance in terms of the MSE losses over the test sets (see Sec. 4.2.1) and task success rates over 25 policy rollouts. In addition, we visualize the impact of each sensor modality during policy execution. Specifically, we plot the normalized distance between the query feature and the selected key feature for each sensor input. A larger distance means that the corresponding sensor modality contributes more in selecting the key feature from the demonstration library. From Table. 3, we observe that without using both tactile images and audio signals, the MSE loss (task success rate) 0.62 (4%), which is significantly higher (lower) than the others. By incorporat- ing both tactile and audio feedback, the MSE loss can be low 0.21, and more importantly, the task success rate can reach 40%.
18

---

Example 1
Example 2
Example 3
Figure 9: Three failure rollouts of the MULSA policy. For all of them, the robot drifts away from the hole, resulting complete task failures.
Models
MimicTouch w/o T & A MimicTouch (T)
MinicTouch (A)
MimicTouch (T + A)
MSE Loss
0.62 4% (1/25)
0.38
0.48
0.21
Success Rate
24% (6/25)
16% (4/25)
40% (10/25)
Table 3: MSE Loss over test sets and Task success rates of 25 policy rollouts.
As shown in Fig. 10, tactile and audio inputs start play important components during the Insertion phase, when most contacts occur. We also have qualitative results shown the Fig. 11. According to those results, contact appears, whereas the lack of audio feedback easily leads to an inability to detect external collisions.
find that lack of tactile feedback easily leads incorrect motion when
Therefore, we can conclude that the multi-modal tactile feedback crucial for the success of contact-rich insertion tasks.
K Comparison of Action Magnitudes Between Offline IL Policy and Residual RL Policy
To quantitatively evaluate the contributions of both the IL and RL components of the final policy, we compare the action outputs (i.c., the 6D delta pose) of each across three successful rollout tra- jectories. the left plot of Fig. 12, we show the average distances and the right plot, we show the average rotation angles. From these results, we observe that the IL policy's action outputs account for higher proportion of the final output, indicating that the II. policy is overall more critical. This underscores the importance of collecting Human Tactile Demonstrations and processing them using the NN-based method. However, it is also important to note that the smaller contributions from the RL policy are also necessary for the successful execution of the fine-grained manipulation tasks.
L Generalization Setting
In this section, we describe the setting of each generalization task.
Shifting Positions: In this generalization task, we shift the hole positions for 0.8 em either tor by test the generalization ability for finishing the insertion task with under varied alignment conditions.
19

---

Contribution of Modalities Over Actions
1.0
State
Normalized Contribution
Tactile
Audio
0.6
Insertion
Reach
0.4
0.2
0.0
10
20
30
5C
Actions Serial Number
Figure 10: Visualization of the impact of each sensor modality during policy execution. The left side of the dashed orange line Reach Phase, and the right side Insertion Phase.
No Tactile
No Audio
Figure 11: Policy rollouts some failure examples with only tactile feedback or only audio feed- back.
Tilting Angles: this generalization task, we tilted the hole angle for 10 degrees or 20 degrees to test the generalization ability for finishing the insertion task with different contact positions.
Two-stage Dense Packing:
We introduce the two-stage dense packing task, which requires the robot to perform two consecutive alignment adjustments complete the dense packing process. Each hole will challenge the robot's s ability to adjust its alignment according to tactile feedback efficiently.
In this generalization task, the robot required to insert the cylin- der into the hole which contains multiple objects: pen, tissue, and sponge. This setting has rigid objects (pen) and deformable soft objects (tissue and sponge) with different materials and shapes, which challenge the robot's ability to accomplish the task with different tactile feedback from dif- ferent materials.
Multi-material Dense Packing:
Furniture Assembly: In this generalization task, the robot required to insert the cylinder into a small hole in a table for screwing. This task will test two aspects of our policy: whether the robot can insert the object into a smaller hole, and whether the robot can adjust it to position, that deep enough to be skewed by human-defined simple script (to rotate the end-effector for 120%), based on the tactile feedback from the threads in the hole.
We evaluate the policy performance in those five different generaliza- tion settings for both MonicTouch final policy and the Openloop Policy. For Shifting Positions, we rollout the policies 10 times in each direction of or +y,, resulting total of 40 evaluations. For Tilting Angles, we rollout the policies 25 times for both tilting directions in ir, resulting in a total of 50 evaluations for the 10º tilting and 20 tilting, respectively. For Two-stage Dense Packing,
Policy Evaluation Process:
rollout the policies 25 times. For Multi material Dense Packing, rollout the policies 25 times
20

---

Final Actions Rollout (IL vs RL)
Final Actions Rollout (IL RL)
Action Distance Magnitude (cm)
Mean_IL Jt
Action Angle Magnitude (degrees)
Mean IL
Mean RL
Mean RL
STD
STD_LL
0.6
Reach
Insertion
0.4
Reach
Insertion
10
70
10
70
Realtime Action Number
Realtime Action Number
Figure 12: In the left, we show the average distances and in the right, we show the average rotation angles. We observe that the IL policy's action outputs account for higher proportion of the final output.
both rigid object "pen" (Rigid in the table) and deformable soft objects "tissue and sponge" (Soft in the table). For Furniture Assembly (Assem in the table). we rollout the policies 25 times. This task has two sub-evaluation metrics: insertion (Assem (l) in the table) and adjustment (Assem (A) in the table). Notably, the success rate for Assem (I) is the success rate for the number of attempts (25), and the success rate for Assem (A) is the success rate when the insertion is successful.
M Generalization Results
In this section, summarize the zero-shot generalization results based on the quantitative results shown in Table. 2, and the qualitative results shown in Fig. 13.
MinicTouch policy can zero-shot transferring to insertion tasks with different contact po- sitions, tilted angles, and even different sizes of holes (Aseembly (Insertion)).
For the Two-stage Dense Packing, Mimic'Touch policy displays its robustness to adjust according to multiple stages of contact information according to the quantitative result and the video. This shows that our model can make continuous and correct adjustments based on the continuously varied contact information.
MimicTouch policy also shows its power in the multi-material task domains. Due to dif- ferent materials in the environment, the sensor feedback will be different from the training environment. In this case, policy still has great performance other rigid objects (pen). For the deformable soft object (tissue and sponge), The success rate is bit lower because of two challenging issues: i). it's hard to get audio feedback for contact with soft objects, ii). sponge sometimes is too soft to get tactile feedback. With those issues, our policy still gets 64% success rate. Moreover, the qualitative result from the video shows impressive performance in adjusting continuously according to the deformation of the tissue.
In the assembly task, Mimic'Touh policy not only can insert the object into a small hole but also can adjust the object to correct pose and insert to a deep-enough position according the tactile feedback from the contact between screw threads. This allows us use very simple script (to rotate the end-effector for 120") to solve the assembly task.
21

---

Time
Shifting
20 Tiktng 10- Titting
Two stage
Packing
Soft Tissue
Pen
Assembly
L tasks
Figure 13: Task setup and qualitative results for zero-shot generalization
22


Reviews:
[{"review_id": "ahxcMFT5Z8", "reviewer": "Reviewer_W9Dy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a framework to collect multi-modal human demonstrations and use them for robot policies. The embodiment involves a robotic arm with a gripper, and optical-tactile and audio sensors attached to it. Humans can wear an attachment with ArUco markers and the same sensors attached, allowing them to collect demonstrations using their fingers as a gripper. Calibration between the robot and the human hand is done using a set of poses collected by both the human and the robot hand. After data collection, they train tactile and audio representations using self-supervision and use nearest neighbor-based action retrieval as an offline policy. They fine-tune their offline policy by learning an online residual policy added atop the offline policy. They evaluate their method on a peg insertion task and showcase its generalization capabilities in various settings for that task.  The major novel contribution of the paper includes a framework to collect human audio-tactile demonstrations, including the design of an attachment for use and deploying policies learned on a gripper. There is no significant originality in the policy learning aspect of the framework.", "summary_of_recommendation": "Even though I find this work and overall framework valuable to share with the community, I believe there are significant robot experiments to be made before fully accepting the paper. Post Rebuttal Edit: I believe after the rebuttal, the paper is now suitable for the conference.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Here is the list of additional experiments and recommendations that I believe could improve the paper:\n\n**Calibration Related Improvements / Recommendations:**\n\n- I recommend that the authors calibrate the robot’s base only once with respect to the camera and use that transformation for inverse kinematics. Is there a specific reason why they didn’t do this?\n\n**Residual RL Related Improvements / Recommendations:**\n\n- Could you explain how you set the action space during residual learning? Do you learn the residual policy for all dimensions of the action space, and do you have any action limits for your residual policy?\n- During reward calculation, the authors mention using KL divergence to align the human expert and the robot data. What exactly is the input of this KL divergence? Do they include only the proprioception, or do they include the tactile and the audio as well?\n- A small explanation of the RL learning in the main paper for these questions could be useful.\n\n**Experiments Related Improvements / Recommendations:**\n\n- For Section 4.1 Human Tactile Demonstration Collection System: Could you please reiterate how you calculate the success rate here? Are you only running trajectory replay of the demonstration to see if that succeeds, or are you counting the number of trials during data collection? If it is trajectory replay, could you explain why the teleoperated demonstrations fail? If they succeeded during data collection, why would they fail in the open-loop trajectory?\n- The authors used MSE on the test set to compare their method with the baselines, both for different modalities and different offline policies. I find this evaluation somewhat deficient. For offline policies, NN-based action retrieval compared to a parametric policy would, of course, give lower MSE since it outputs very similar actions to those in the test set. However, a parametric offline policy could perform better in situations that require different actions than those in the demonstration set. I recommend that the authors run real robot experiments for both Section 4.2.1 (offline policy learning) and Table 3 (modality ablation).\n\n**Generalization Related Improvements / Recommendations:**\n\n- Could you explain the generalization experiments in more detail? Are you using the collected 5 demonstrations for the MimicTouch in action retrieval as well, or are you using the old demonstrations for the offline policy and training a residual policy on top?\n- Could you include NN-based action retrieval on the new demos and NN-based action retrieval on the old demos with the residual RL as baselines for Table 2? This could help understand how much the residual RL improves generalization and how much better the NN-based policy performs atop open-loop.\n\n**Syntax / Writing Related Improvements / Recommendations:**\n\n- Instead of using percentages for the success rates of the policy deployments, I recommend using <# of successful runs> / <# of evaluation runs>. It makes it easier to read and comment on.\n- For Figure 7, Contribution of Modalities, the authors mention that as the distance of the modality is higher, it has a higher contribution. I didn’t fully understand this part. Wouldn’t it be that as the distance of the modality is lower to the nearest neighbor, it has a higher contribution? Could the authors explain this, please?\n- The authors refer to their offline policy as VINN. However, VINN uses interpolation between a set of K nearest neighbors as the final action, but from what I understand, MimicTouch only uses the nearest neighbor's action without any interpolation. If that’s the case, I recommend adding this small difference and not referring to their offline policy as VINN, perhaps using NN-based action retrieval or something similar.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kcSN8sITC5", "reviewer": "Reviewer_4Bsg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This study presents learning from human tactile demonstrations for contact-rich manipulation.", "summary_of_recommendation": "Overall, this study is suitable for this conference. I would recommend weak accept. However, several points should be addressed. After rebuttal comment: Thank you very much for the detailed rebuttal. The additional experiments significantly improved the quality.  Although I would like to see direct comparisons between the proposed and pure RL methods in this environment, I understand the difference. I believe this paper is suitable for this conference. I would like to maintain the score.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Please address the above comments, mainly focusing on comment a).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9YenfE6J5w", "reviewer": "Reviewer_VHmX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a novel multi-modal manipulation learning pipeline from human demonstrations. In this pipeline, a human operator collects demonstrations with pose, tactile, and acoustic signals, then a BYOL-based SSL architecture is used to learn a representation, from which a IL policy is trained. At the end a RL policy learns a residual policy to compensate for the IL policy.", "summary_of_recommendation": "In summary, I don’t think the paper in its current state is suitable for publication at CoRL mainly due to the lack of proof of the effectiveness of the proposed pipeline v.s SOTAs and the lack of more complicated tasks.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Since there is a RL process at the end to learn a residual policy, how important are the representation learning process and the IL process? Can the proposed method actually beat the performance of e2e RL or e2e of a modern IL policy (e.g. diffusion policy or mulsa)?\n2. Why doesn’t the residual policy pi_r also take a_i (the action output from IL) as an input?\n3. Not really a question, but I’d encourage the authors to rearrange the paper a bit and move some architecture/method explanations from the appendix to the main paper.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers concur that the paper should be accepted. Please incorporate reviewer feedback in the camera ready.
Decision: Accept