<figure><img src="image_1.png" /></figure>

HAL
open science

# Toward synthetic data generation for robotic tactile
manipulations

Guillaume Duret, Florence Zara, Jan Peters, Liming Chen

► To cite this version:

Guillaume Duret, Florence Zara, Jan Peters, Liming Chen. Toward synthetic data generation for robotic tactile manipulations. Workshop on "Robot Embodiment through Visuo-Tactile Perception" - 2024 IEEE International Conference on Robotics and Automation (ICRA) Conference Workshop, May 2024, Yokohama, Japan. hal-04566202

HAL Id: hal-04566202

https://hal.science/hal-04566202v1

Submitted on 18 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Toward synthetic data generation for robotic tactile manipulations *

Guillaume Duret¹,³, Florence Zara², Jan Peters³ and Liming Chen¹

## I. INTRODUCTION

Data collection for tactile-based robotic manipulation plays a crucial role in improving existing models and enabling general models to perform effectively across various scenarios. These tactile sensors can infer essential information, such as force measurements, contact area, and marker displacements. These tactile sensors have been successfully employed in diverse applications, including: rich-contact tasks, grasping transparent objects, slipping detection or force-controlled grasping for fragile and deformable items. Most of these tasks rely on small datasets from real-world scenarios. Consequently, a significant limitation persists: the scarcity of data.

In recent years, numerous simulations designed for tactile sensors have emerged. Initial efforts focused on rigid body simulation, using conventional rendering techniques like rasterization [1], [2], [3] and ray tracing [4] to create synthetic tactile images. The deformation of the Gelsight sensor was represented in the image through smoothing [5], and promising results have been achieved with domain adaptation techniques [6], [7]. However, tactile deformation in real-world tactile sensors is essential for shear force or tactile marker displacement. Consequently, research has shifted towards physically accurate simulations for robotic manipulation. Physical deformation has been modeled using the Material Point Method (MPM) to generate more precise tactile images [8], [9]. The Finite Element Method (FEM) has been used to introduce marker-based and shear force data [10], [11], [12]. This work has demonstrated the effectiveness of deformable simulation in producing high-quality tactile images. However, none of these recent works proposed general grasping manipulation for data generation.

To address this limitation, our research aims to develop a general method for generating synthetic data tailored for tactile-oriented research problems. This paper presents the first simulation of the Gelsight Mini sensor, a market-available, user-friendly sensor that requires no expertise or time in 3D printing. We propose an FEM simulation pipeline

<figure><img src="image_3.png" /><figcaption>Fig. 1: Two configurations of the same grasping pose are presented. The left part presents a grasping pose with the object oriented in the direction of gravity. The right part shows the object oriented perpendicularly. The respective tactile images show that the second configuration requires much more force on the object and puts the sensor under high stress and shear force, which is not suitable.</figcaption></figure>

to generate meaningful data for major grasping-based tactile research problems, thereby facilitating automatic data generation for training general on tactile manipulation. Our approach incorporates recent advancements in tactile sensor simulation into the existing FEM-based simulator, Defgraspsim [13]. Our extension enables the recording of deformable tactile data during grasping through a specific pipeline designed to address dataset generation for multiple tactile challenges, as detailed in Section II. Additionally, for the purpose of data generation, the large-scale dataset Acronym [14] is used as data input to offer a variety of more than 17.7 million grasping poses for more than 8872 objects across 262 categories. This enable the generation of extensive tactile annotations such as grasping success, forces, slipping, stress, deformation, and tactile images.

## II. SIMULATION PIPELINE

We introduce a grasping pipeline for tactile data recording extending the capabilities of Defgraspsim [13], which was originally designed for generating data related to grasping deformable objects. Our pipeline includes high-level annotations such as grasping success and slipping, as well as frame-to-frame ground truth information such as stress, deformation, forces data, and marker-based tactile images similarly to [15], [11]. Additionally, one can load any grasping pose from the large-scale dataset Acronym [14]. The simulation serves as an all-in-one pipeline, generating meaningful data for tactile-based manipulation.

a) Gravity aware grasping success: The ability to evaluate grasping poses is essential for control models. However, this evaluation is dependent on gravity compared to pure

*This work was in part supported by the French Research Agency, l'Agence Nationale de Recherche (ANR), through the projects Learn Real (ANR-18-CHR3-0002-01), Chiron (ANR-20-IADJ-0001-01), Aris totle (ANR-21-FAI1-0009-01). It was granted access to the HPC re sources of IDRIS under the allocation 2023-[AD011013894] and 2024- [AD011015271] made by GENCI.

¹Centrale Lyon, CNRS, LIRIS, UMR5205, F-69130 Ecully, France, {guillaume.duret, liming.chen}@ec-lyon.fr

²UCBL, CNRS, LIRIS, UMR5205, F-69622 Villeurbanne, France, florence.zara@liris.cnrs.fr

³Intelligent Autonomous Systems Lab, Technical University of Darmstadt, 64289 Darmstadt, Germany, jan.peters@tu-darmstadt.de
<figure><img src="image_4.png" /><figcaption>Fig. 2: Example of a simulated rectangle grasped with two Gel-Sight mini sensors and a franka panda gripper during the different steps of the simulation.</figcaption></figure>

rigid body grasping [14]. Indeed, tactile sensors are inherently fragile, and their purpose may involve manipulating fragile or deformable objects. As depicted in Fig. 1, gravity plays a critical role in determining the success of a grasping pose and affects the outcome of deformation and tactile images. The grasping process can be divided into two phases. First, the simulation closes the finger until contact (see Fig. 2a) is established, with the force reaching a parameter *N* newtons (refer to Fig. 2b and 2c). The second phase incorporates gravity. This two-phase approach allows us to generate grasping success metrics that account for gravity. During this step, ground truth data was recorded, including the deformation and stress of the sensor for each force applied to the object as seen in Fig. 3.

<figure><img src="image_5.png" /><figcaption>Fig. 3: For the rectangle example: (left) illustration of the deformation; (right) stress applied on it.</figcaption></figure>

b) *Slipping detection:* The final step involves gradually reducing the grasping force until contact between the gripper and the object is lost. This step provides information about the force required for gravity aware stable grasping and can introduce slipping tactile data (see Fig. 2d). Taking advantages of the simulation, slipping movement was detected by a change of contact between the object and the gripper. Using this slipping detection, we can annotate during the pipeline the frames where there was slipping. The data generated can then be used for slipping detection or in controlled slipping manipulation models.

c) *Tactile images and experiments:* Tactile images were computed in a second step using the simulation data. Indeed, the deformed surface was extracted and used to render the tactile image. To take into account the marker and real-world texture, a calibration of the camera sensor was performed. This calibration step offers multiple advantages, such as reducing camera distortion on the tactile image and using this undistorted image as a texture for the synthetic tactile image. Fig. 4 shows three examples of marker-based textures that can be used. To qualify the quality of the tactile data, a comparison of the synthetic tactile image and the real tactile image is illustrated in Fig. 5, where the similarity of the

marker is demonstrated. In general, the tactile data can be used to learn global inference, establishing the global link between grasping tactile images and other annotations (as previously described in paragraph II-.0.a).

<figure><img src="image_6.png" /><figcaption>Fig. 4: Three examples of texture were applied to the tactile sensor: the left one uses a small marker texture; the middle one uses a real size marker texture; and the last one uses a real tactile image as texture. The first row shows the undeformed texture, while the second row shows the deformation induced by grasping the rectangular object.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 5: Visual comparison between synthetic tactile image (left) and a real tactile image (right) for a rectangle object</figcaption></figure>

d) *Limitations:* This work is a preliminary step towards generating comprehensive data for tactile robotic manipulation using Isaac Gym. Despite the high-quality output from the FEM simulation, it is computationally expensive, reducing the effectiveness of parallelization.

### III. CONCLUSION

Our study extends the data recording pipeline of Defgrasp-sim for tactile sensors, proposing a comprehensive pipeline for tactile data generation related to manipulation. The generated data targets multiple research objectives in tactile-oriented studies and paves the way for large-scale tactile data datasets. Consequently could also facilitate the pre-training of general tactile based models for tactile manipulation, thereby significantly advancing tactile manipulation research.
REFERENCES

[1] S. Wang, M. Lambeta, P. wei Chou, and R. Calandra, "Tacto: A simulator for learning control from touch sensing," 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:231646400

[2] D. F. Gomes, P. Paoletti, and S. Luo, "Generation of gelsight tactile images for sim2real learning," *IEEE Robotics and Automation Letters*, vol. 6, pp. 4177-4184, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:231632323

[3] D. Gomes, P. Paoletti, and S. Luo, "Beyond flat gelsight sensors: Simulation of optical tactile sensors of complex morphologies for sim2real learning," in *Robotics: Science and Systems*, 2023.

[4] A. Agarwal, T. Man, and W. Yuan, "Simulation of vision-based tactile sensors using physics based rendering," in *2021 IEEE International Conference on Robotics and Automation (ICRA)*. IEEE, 2021, pp. 1-7.

[5] Z. Si and W. Yuan, "Taxim: An example-based simulation model for gelsight tactile sensors," *IEEE Robotics and Automation Letters*, vol. 7, no. 2, pp. 2361-2368, 2022.

[6] Y. Lin, J. Lloyd, A. Church, and N. F. Lepora, "Tactile gym 2.0: Sim-to-real deep reinforcement learning for comparing low-cost high-resolution robot touch," *IEEE Robotics and Automation Letters*, vol. 7, no. 4, pp. 10754-10761, 2022.

[7] W. Chen, Y. Xu, Z. Chen, P. Zeng, R. Dang, R. Chen, and J. Xu, "Bidirectional sim-to-real transfer for gelsight tactile sensors with cyclegan," *IEEE Robotics and Automation Letters*, vol. 7, no. 3, pp. 6187-6194, 2022.

[8] Z. Chen, S. Zhang, S. Luo, F. Sun, and B. Fang, "Tacchi: A pluggable and low computational cost elastomer deformation simulator for optical tactile sensors," *IEEE Robotics and Automation Letters*, vol. 8, no. 3, pp. 1239-1246, 2023.

[9] Z. Si, G. Zhang, Q. Ben, B. Romero, Z. Xian, C. Liu, and C. Gan, "DIFFTACTILE: A physics-based differentiable tactile simulator for contact-rich robotic manipulation," in *The Twelfth International Conference on Learning Representations*, 2024. [Online]. Available: https://openreview.net/forum?id=eJHnSg783t

[10] W. Du, W. Xu, J. Ren, Z. Yu, and C. Lu, "Tacipc: Intersection- and inversion-free fem-based elastomer simulation for optical tactile sensors," *IEEE Robotics and Automation Letters*, 2024.

[11] S. Cui, Y. Wang, S. Wang, Q. Li, R. Wang, and C. Zhang, "Tactile imprint simulation of gelstereo visuotactile sensors," in *2023 IEEE International Conference on Mechatronics and Automation (ICMA)*. IEEE, 2023, pp. 650-656.

[12] W. Chen, J. Xu, F. Xiang, X. Yuan, H. Su, and R. Chen, "General-purpose sim2real protocol for learning contact-rich manipulation with marker-based visuotactile sensors," *IEEE Transactions on Robotics*, 2024.

[13] I. Huang, Y. Narang, C. Eppner, B. Sundaralingam, M. Macklin, R. Bajcsy, T. Hermans, and D. Fox, "Defgraspsim: Physics-based simulation of grasp outcomes for 3d deformable objects," *IEEE Robotics and Automation Letters*, vol. 7, no. 3, pp. 6274-6281, 2022.

[14] C. Eppner, A. Mousavian, and D. Fox, "Acronym: A large-scale grasp dataset based on simulation," in *2021 IEEE International Conference on Robotics and Automation (ICRA)*. IEEE, 2021, pp. 6222-6227.

[15] Y. Narang, B. Sundaralingam, M. Macklin, A. Mousavian, and D. Fox, "Sim-to-real for robotic tactile sensing via physics-based simulation and learned latent projections," in *2021 IEEE International Conference on Robotics and Automation (ICRA)*. IEEE, 2021, pp. 6444-6451.