<figure><img src="image_1.png" /></figure>

HAL
open science

# Haptic interaction with virtual deformable objects

Emanuele de Santis, Youenn Le Jeune, Maud Marchal, Claudio Pacchierotti, Marilena Vendittelli

► To cite this version:

Emanuele de Santis, Youenn Le Jeune, Maud Marchal, Claudio Pacchierotti, Marilena Vendittelli. Haptic interaction with virtual deformable objects. I-RIM 3D 2024 -Conference on Italian Institute of Robotics and Intelligent Machines, Oct 2024, Rome, Italy. pp.1-3. hal-04733238

HAL Id: hal-04733238

https://inria.hal.science/hal-04733238v1

Submitted on 11 Oct 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Haptic interaction with virtual deformable objects

E. De Santis*, Y. Le Jeune**, M. Marchal**, C. Pacchierotti†, M. Vendittelli*

*DIAG Department Sapienza University of Rome, Italy, {vendittelli, edesantis}@diag.uniroma1.it

**University of Rennes, INSA, IRISA, France, {youenn.le-jeune, maud.marchal}@insa-rennes.fr

†CNRS, University of Rennes, Inria, IRISA, France, {claudio.pacchierotti@irisa.fr}

***Abstract***—High-fidelity virtual reality (VR) simulations are becoming increasingly attractive in the medical and surgical training field. In this context, the sensory feedback returned to the surgeon or practitioner is of primary importance to create a significant experience. One of the main challenges in delivering VR systems for surgical training and planning that include the possibility to interact with the virtual scene, and render the associated touch sensation to the user, is the real-time simulation of soft tissues. This paper reports preliminary results in the design and implementation of an integrated system that includes dynamic simulation of the interaction with deformable objects and tissues, a VR environment with finger motion tracking, and haptic feedback provided by a wearable device. In addition, it explores the challenges and advances in integrating such technologies with a focus on creating realistic tactile experiences. It also addresses the complexity of combining hardware and software components, proposing some solutions to overcome integration problems.

***Index Terms***—Simulation, Soft material, Virtual Reality, Haptic rendering, Motion retargeting

## I. INTRODUCTION

In recent years, there has been an increasing demand for integration of virtual environments with the real world, where a sensation generated by events happening in the virtual space is conveyed to the user through different types of devices providing, e.g., visual and haptic stimuli. Research progress in this area will enable VR technology to be used in fields that require a high degree of fidelity of the experience, such as surgical training and planning, as well as remote surgery. Accurate and realistic simulation allows also having augmented interaction with a physical object through its virtual counterpart. For example, in surgical training the physical phantom used by the trainees is usually disposable and very expensive: the possibility of having a digital twin or even a virtual-only phantom can instead enable advanced "virtual" sensors and the possibility to reuse the phantom after the training session, thus drastically reducing the costs.

Simulation of biological tissues deformation and haptic rendering of the interaction with them has been attacked in the literature as separate problems. The simulation of deformable bodies, and in particular of biological tissues, has been traditionally treated via Mass-Spring Model (MSM), like in [1]–[3], and Finite Element Method (FEM) analysis, as in [4]–[6]. These methodologies usually suffer the very-high computation times, making them unfeasible for the real-time

<figure><img src="image_3.png" /><figcaption>Fig. 1. Architecture of the proposed integrated solution.</figcaption></figure>

interaction with these soft bodies, unless having to reduce the number of masses (for MSM) or meshes (for FEM) to make the computation fitting the required real-time constraints. In recent times other approaches based on AI tried to mitigate such issue, as in [7], [8], but computing the only steady-state deformation without its transient.

Few results are available for the interaction with soft bodies. In [9], [10] the interaction is obtained through grounded interfaces, without the possibility of grasping or manipulating the virtual object. As a matter of fact, works on wearable haptic interfaces, allowing retargeting the real-hand motion to a virtual-hand, are mainly focused on the device design, as in [11], [12].

This paper proposes an integration of dynamic simulation of the interaction with soft objects, VR technology, and wearable haptic devices - capable of detecting finger movements and providing feedback such as touch pressure, temperature and texture of the touched object - with a focus on motion retargeting for a customized hand model. In addition, this paper highlights the challenges and advances in the integration of these technologies, particularly in representing the complexities of motion retargeting and deformable objects, which are critical for realistic tactile experiences.

## II. PROPOSED ARCHITECTURE

The system architecture, shown in Fig. 1, includes a VR-ready high-end consumer laptop, equipped with an Intel Core i7 10875H and an Nvidia RTX 2070 Super graphics card, running the MuJoCo physics engine [13], [14], responsible for the simulation of the virtual scene and the deformable objects,

<footnote>This work was partially supported by Rome Technopole Flagship Project 4, CUP B83C22002820006, through the Sapienza project "Phygital Twin Technologies for innovative Surgical Training and Planning"</footnote>
<figure><img src="image_4.png" /><figcaption>Fig. 2. Hand model with cuts along the knuckles grasping a deformable cube.</figcaption></figure>

and a Python application that manages the interconnection between MuJoCo and the other components.

The virtual scene includes a 3D model of a physical anatomic phantom of the abdomen (upper right corner of Fig. 1), which contains a deformable kidney (upper left corner of Fig. 1). For immersive 3D view of the scene, an Oculus Rift S VR headset is connected to the laptop via a customized OpenXR interface [15].

A Weart TouchDIVER [16] wearable haptic device renders touch feeling through pressure on the user's fingers, temperature and texture of the object touched in the virtual scene by the digital hand model. The haptic glove is composed of three thimbles, worn on thumb, index and middle fingers, and a main component to be worn on the hand palm.

The Weart TouchDIVER is connected to the laptop via a Bluetooth connection to a Weart proprietary middleware, which provides information on the palm pose and fingers closure state and takes the commands (e.g. pressure change on the fingertips, temperature change, etc.) via a TCP connection to the Python application that integrates all the components.

### III. HAND MODEL AND MOTION RETARGETING

The hand model used in the virtual environment available from the Weart assets, and developed for Unity, was not ready for integration in MuJoCo due to its skeletal rig model style, which is not supported in MuJoCo. It was then necessary to develop a new hand model. Two approaches were then considered: to cut the geometry along the knuckles as shown in Fig. 2 or to implement the skin over a hand skeleton model available in MuJoCo. The first solution is more portable and can be used also in robotic simulation environments like, e.g., CoppeliaSim, shown in the figure. Simulation of soft bodies in CoppeliaSim is however possible only for a limited set of geometrical shapes like, e.g., cube, sphere, cilinder. The second option was then chosen to allow the simulation of soft bodies of any geometrical shape in MuJoCo.

The use of a hand model different from the one provided by Weart, required the development of a customized motion retargeting method. The information on the hand configuration retrievable from the TouchDIVER middleware consists in the palm pose and in a *closure* parameter, varying between 0 (fully open) and 1 (fully closed). To retarget this information to the virtual hand, each finger is modeled as a planar 3R robot, as sketched in Fig. 3. The manipulator base tracks

<figure><img src="image_5.png" /><figcaption>Fig. 3. Planar 3R robot as simplification of each finger.</figcaption></figure>

the palm pose, while the closure parameter is mapped to the normalized distance of each finger tip to the TouchDIVER main component mounted on the hand palm (Fig. 1, bottom).

The kinematics of the closure task is then defined by

$$ \dot{d} = \frac{1}{d} p^T(q) J(q) \dot{q} $$

with $p \in \mathbb{R}^2$ the position of the 2R manipulator tip, $q \in \mathbb{R}^3$ the vector of joint variables and $J(q)$ the manipulator Jacobian.

Denoted with $d_r$ the value of the closure parameter retrieved from the glove, and using the following definitions (dropping dependencies on the joint variables $q$ for brevity)

$$ J_d = \frac{1}{d} p J \qquad J_t^\# = J_d^T (J_d J_d^T + \mu^2)^{-1}, \quad (1) $$

where a damped least square solution is used to compute the task Jacobian pseudoinverse $J_t^\#$.

The tracking task is then achieved trough the control law

$$ \dot{q} = J_t^\# [K_p(d_r - d)] + (I - J_t^\# J_d)u, \quad (2) $$

with $u$ defined so as to enforce joint limit constraints.

This definition of the closure task makes the control of the virtual hand independent of the hand size of users and allows a simple control of the fingers without resorting to pre-specified trajectories. Moreover, the kinematic redundancy of each manipulator with respect to the closure task allows to specify additional constraints both on the singular finger, like e.g., joint limit enforcement, and on the collective motion to make, for example, the grasp more natural and effective.

### IV. DEFORMABLE KIDNEY MODEL

The right kidney of the virtual phantom has been made deformable using the MuJoCo 3.0 `flexcomp` object type and it is possible to manipulate it with feedback such as touch, temperature, and texture sensations.

Since the 3D model of the phantom abdomen was reconstructed from CT scans of the physical phantom, a number of transformations of the 3D mesh model were necessary. In particular, holes and voids were removed and the planar meshes converted to volumetric ones. Reduction of the number of mesh vertices was also necessary for computational cost issues. In fact, finding the right trade-off between the number of mesh vertices and accuracy of simulation is a critical
<figure><img src="image_6.png" /><figcaption>Fig. 4. Sequence diagram of the execution of the simulation cycle.</figcaption></figure>

aspect. Low-vertex deformable meshes are faster to simulate but are not realistic, while high-vertex meshes can make the simulation much slower than real time and may not fit the computational resources of most hardware equipment.

## V. INTEGRATION OF THE OCULUS VR WITH OPENXR INTERFACE

The integration of Oculus VR with MuJoCo required the use of the OpenXR library. This choice is motivated by the fact that it is currently supported by almost all commercial VR headsets and has an (unofficial, but up-to-date) Python interface. This makes it possible to use Python code (which is portable and does not require compilation) with different VR headsets without having to write hardware-specific code.

Code optimization was necessary to make the VR rendering smooth and prevent that the simulation of the physics engine was slowed down. In particular, a multi-threaded approach to decouple the rendering process from the simulation process have been put in place. This has made possible to render a frame while at the same time computing the simulation for the next frame, thus reducing the overall time spent for simulating physics and rendering the corresponding frame.

Moreover, given the fact that the Oculus Rift S renders images on its screens at a fixed rate of 80Hz, while the physics engine runs simulation at higher frequency (depending on the simulation time set in MuJoCo), a mechanism to synchronize the simulation time and the rendering time has been put in place. To achieve this, the simulation thread (which calls MuJoCo for the physics simulation) runs a number of steps (depending on the frame rate of the VR headset, which information is available from OpenXR) before invoking the rendering thread, thus running the simulation thread at a higher frequency than the rendering thread. The sequence diagram of the simulation and rendering threads is represented in Fig. 4.

## VI. CONCLUSIONS

This paper presented preliminary results of ongoing research on interaction with virtual worlds including soft material, with rendering of the haptic sensation generated by the virtual interaction. The presented experimental system integrates a 3D vision head set, a wearable haptic interface and a physics engine providing simulation tools for deformable materials. The integration of these components required the definition of appropriate models, both digital and mathematical, and control methods. In particular, to retarget the motion of the real

hand to its virtual counterpart, a custom task has been defined allowing effective motion tracking and independence on the real hand size. The experimented limitations to the simulation realism due to the computationally demanding simulation of soft objects motivates the use of alternative methods, possibly including pre-trained models. This is the direction of future research paths.

## REFERENCES

[1] Y. Duan, W. Huang, H. Chang, W. Chen, K. K. Toe, J. Zhou, T. Yang, J. Liu, S. K. Teo, C. W. Lim, Y. Su, C. K. Chui, and S. Chang, *Modeling and Simulation of Soft Tissue Deformation*. Springer Berlin Heidelberg, 2013, p. 221–230. [Online]. Available: http://dx.doi.org/10.1007/978-3-642-41083-3_25

[2] W. Mollemans, F. Schutyser, J. Van Cleynenbreugel, and P. Suetens, *Tetrahedral Mass Spring Model for Fast Soft Tissue Deformation*. Springer Berlin Heidelberg, 2003, p. 145–154. [Online]. Available: http://dx.doi.org/10.1007/3-540-45015-7_14

[3] L. Xu, Y. Lu, and Q. Liu, “Integrating viscoelastic mass spring dampers into position-based dynamics to simulate soft tissue deformation in real time,” *Royal Society Open Science*, vol. 5, no. 2, p. 171587, Feb. 2018. [Online]. Available: http://dx.doi.org/10.1098/rsos.171587

[4] G. R. Joldes, A. Wittek, and K. Miller, “Suite of finite element algorithms for accurate computation of soft tissue deformation for surgical simulation,” *Medical Image Analysis*, vol. 13, no. 6, p. 912–919, Dec. 2009. [Online]. Available: http://dx.doi.org/10.1016/j.media.2008.12.001

[5] T. Chanthasopeephan, J. P. Desai, and A. C. W. Lau, “Modeling soft-tissue deformation prior to cutting for surgical simulation: Finite element analysis and study of cutting parameters,” *IEEE Transactions on Biomedical Engineering*, vol. 54, no. 3, p. 349–359, Mar. 2007. [Online]. Available: http://dx.doi.org/10.1109/TBME.2006.886937

[6] W. F. Larrabee, “A finite element model of skin deformation. i. biomechanics of skin and soft tissue: A review,” *The Laryngoscope*, vol. 96, no. 4, p. 399–405, Apr. 1986. [Online]. Available: http://dx.doi.org/10.1288/00005537-198604000-00012

[7] A. Mendizabal, P. Márquez-Neila, and S. Cotin, “Simulation of hyperelastic materials in real-time using deep learning,” *Medical Image Analysis*, vol. 59, p. 101569, Jan. 2020. [Online]. Available: http://dx.doi.org/10.1016/j.media.2019.101569

[8] A. Odot, R. Haferssas, and S. Cotin, “Deepphysics: A physics aware deep learning framework for real-time simulation,” *International Journal for Numerical Methods in Engineering*, vol. 123, no. 10, p. 2381–2398, Mar. 2022. [Online]. Available: http://dx.doi.org/10.1002/nme.6943

[9] C. Hui, S. Hanqiu, and J. Xiaogang, “Interactive haptic deformation of dynamic soft objects,” in *Proceedings of the 2006 ACM international conference on Virtual reality continuum and its applications*, ser. VRCIA06, vol. 98. ACM, Jun. 2006, p. 255–261. [Online]. Available: http://dx.doi.org/10.1145/1128923.1128965

[10] Y. Tian, Y. Yang, X. Guo, and B. Prabhakaran, “Haptic-enabled interactive rendering of deformable objects based on shape matching,” in *2013 IEEE International Symposium on Haptic Audio Visual Environments and Games (HAVE)*, vol. 3. IEEE, Oct. 2013, p. 75–80. [Online]. Available: http://dx.doi.org/10.1109/HAVE.2013.6679614

[11] A. Frisoli and D. Leonardis, “Wearable haptics for virtual reality and beyond,” *Nature Reviews Electrical Engineering*, Sep. 2024. [Online]. Available: http://dx.doi.org/10.1038/s44287-024-00089-8

[12] G. Frediani, D. Mazzei, D. E. De Rossi, and F. Carpi, “Wearable wireless tactile display for virtual interactions with soft bodies,” *Frontiers in Bioengineering and Biotechnology*, vol. 2, Sep. 2014. [Online]. Available: http://dx.doi.org/10.3389/fbioe.2014.000031

[13] Google Deepmind, “MuJoCo physics engine,” accessed 2024-09-18. [Online]. Available: https://mujoco.org/

[14] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in *2012 IEEE/RSJ International Conference on Intelligent Robots and Systems*. IEEE, Oct. 2012. [Online]. Available: http://dx.doi.org/10.1109/IROS.2012.6386109

[15] The Khronos Group, “OpenXR standard interface,” accessed 2024-09-18. [Online]. Available: https://www.khronos.org/openxr/

[16] Weart, “Weart Touchdiver G1,” accessed 2024-09-18. [Online]. Available: https://weart.it/haptic-vr-products/touchdiver-haptic-glove/