# Deconstructing haptic feedback information in robot-assisted needle insertion in soft tissues

Marco Ferro, Claudio Pacchierotti, Sara Rossi, Marilena Vendittelli

► To cite this version:

Marco Ferro, Claudio Pacchierotti, Sara Rossi, Marilena Vendittelli. Deconstructing haptic feedback information in robot-assisted needle insertion in soft tissues. IEEE Transactions on Haptics (ToH), 2023, 16 (4), pp.536-542. 10.1109/TOH.2023.3271224. hal-04082247

HAL Id: hal-04082247

https://inria.hal.science/hal-04082247

Submitted on 26 Apr 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Deconstructing haptic feedback information in robot-assisted needle insertion in soft tissues

Marco Ferro, Claudio Pacchierotti, *Senior Member, IEEE*, Sara Rossi, Marilena Vendittelli

**Abstract**—This paper evaluates the role and effectiveness of different types of haptic feedback in presenting relevant feedback information during needle insertion in soft tissues through a remotely operated robot. We carried out three experiments with human subjects to analyze the effect of grounded kinesthetic feedback, cutaneous vibrotactile feedback, and cutaneous pressure feedback for rendering the elastic and the viscous force components of a simplified needle-tissue interaction model in a simulated environment. Results showed that providing the two pieces of feedback information through different channels, i.e., kinesthetic and cutaneous, led to the best performance, yielding an improvement in detecting a different tissue layer with respect to providing both information through the same commercial grounded kinesthetic interface. Moreover, results indicate that cutaneous pressure feedback is more suited for rendering the elastic component of the interaction with respect to vibrotactile cutaneous sensations. Finally, results suggest that rendering this elastic component where the user holds the input interface is not so important, confirming that delocalized cutaneous sensations can be an effective solution.

## I. INTRODUCTION

NEEDLE insertion into soft tissue is a minimally-invasive surgical procedure performed in a wide range of applications, such as for targeted drug delivery, biopsies, anesthesia, blood sampling, prostate brachytherapy, ablation. Providing rich feedback information to the clinician is paramount for the success of these procedures. Haptic feedback has been indeed proven useful for conveying guidance/navigation information [1], [2] as well as interaction forces [3], [4] during robot-assisted needle insertion, enhancing the operator awareness without overloading other sensory channels. The main forces acting on the needle during the insertion are developed at the needle tip as well as along its shaft. The needle tip force is due to tissue deformation at the contact (puncturing), rupture and cutting, while the force along the shaft is due to friction between the needle and the tissue [5].

Literature provides evidence [3], [4] that needle insertion procedures benefit from the separation of forces at the needle tip from friction along the needle shaft, that increases during penetration. Removing friction is indeed useful to improve the perception of the events occurring at the needle tip, particularly puncturing and ruptures. Friction force, however, can convey important information about the nature of the tissue that the needle is traversing and rendering this component of the force can make the haptic perception during insertion procedure more natural and even more safe/stable [1]. As it is useful to separate the two main components of the needle tissue interaction force,

it is also interesting to study the problem, not yet addressed in the literature, of how to render these two separate pieces of information during a teleoperated procedure.

This paper presents an evaluation of haptic feedback for teleoperated needle insertion in soft tissues. Using an established simulator [6], we evaluate the importance and role of providing kinesthetic and cutaneous feedback for conveying information related to the elastic and viscous friction forces, the two most relevant components of the interaction forces acting on the needle during the insertion [7]. The research questions addressed in this paper can be summarized as follows:

* Shall the visco-elastic components of the needle-tissue interaction force be all provided through the same single-point kinesthetic interface (as it usually happens today) or using different haptic devices and sensory channels?

* Which combination of kinesthetic and/or cutaneous feedback stimuli is best for providing the considered pieces of information during the insertion?

* Does providing such composite haptic feedback away from where the operator holds the input interface (e.g., the fingertip) negatively affects performance?

## II. EXPERIMENTAL METHODS AND APPARATUS

The setup used to conduct the experiments has the architecture of a teleoperated system (see Fig. 2, left). On the remote side, a simulated scene, built with CoppeliaSim, reproduces the robot-assisted needle insertion system described in Sec. II-A. On the user side, a Geomagic Touch interface is used to teleoperate the simulated robot. According to the condition at hand, haptic feedback about the needle-tissue interaction is provided by combinations of (i) the Geomagic Touch, providing kinesthetic feedback at the fingers; (ii) the hBracelet (see Sec. II-B and Fig. 1a), conveying vibrotactile feedback at the forearm; (iii) the hRing (see Sec. II-C and Fig. 1b), conveying pressure feedback on the fingers; and/or (iv) the TouchDIVER (see Sec. II-D and Fig. 1c), conveying pressure and vibration to the fingertips.

The use of a simulated needle insertion scenario guarantees the repeatability of conditions during the insertion. In particular, the kinematic control of the simulated robot dynamics coupled with a virtual fixture allows to execute the motion commanded by the user through the haptic interface to insert the needle always in the same direction, so as to obtain consistent feedback for all users. Due to the low velocities and the absence of delays in the local communication channel, no stability issues arose during the experiments. Below we describe the different parts of the experimental system more in details.

### A. Needle insertion system

The virtual environment is visible on the screen of the laptop in Fig. 2 (left), with a closeup view on the upper right

M. Ferro, S. Rossi, and M. Vendittelli are with the Department of Computer, Control and Management Engineering, Sapienza University of Rome - Rome, Italy. e-mail: ferro@diag.uniroma1.it.

C. Pacchierotti is with CNRS, Univ Rennes, Inria, IRISA - Rennes, France.

The research leading to this paper was partially supported by the "Visiting Professorship" program of the University of Rome "La Sapienza".
<figure><img src="image_2.png" /><figcaption>Fig. 1. The three cutaneous devices we used: (a) the hBracelet (see Sec. II-B), conveying vibrotactile feedback at the forearm; (b) the hRing (see Sec. II-C), conveying pressure feedback on the fingers; (c) the TouchDIVER (see Sec. II-D), conveying pressure and vibration to the fingertips.</figcaption></figure>

picture in Fig. 2. It is composed of a rigid unbeveled-tip needle, mounted on the end-effector of a KUKA LWR 4+ robot, and an abdominal phantom built from images acquired through 3D CT scan imaging. The phantom is placed on a table below the robot.

During the experiments, the virtual robot is teleoperated by the user through the Geomagic Touch interface, to insert the needle through the multiple tissue layers of the phantom. The motion of the Geomagic Touch interface controls in velocity that of the simulated robot with a scale factor of $s = 0.4$, resulting in a slower motion of the robot with respect to that of the haptic interface. A button on the interface works as a clutch, allowing the user to disconnect from the robot, move the haptic device, and then resume the control of the robot. This approach is commonly used to address issues of limited workspace of the control interface, as in our case. The motion of the robot is constrained so as to move the needle only along the direction of insertion.

Each layer is characterized by a given elasticity and friction (see Sec. III-A). The resulting needle-tissue interaction force, along the insertion direction, is generated by measuring the relative position $z(t)$ and velocity $v(t)$ of the needle with respect to the tissue layers, according to the multi-layer visco-elastic model inspired from [8], [9], [10] and experimentally validated in [6] (see Fig. 2):

$$f(t) = \underbrace{-K_i \Delta z_i(t)}_{f_e(t)} \underbrace{-B_i \Delta z_i(t)v(t)}_{f_f(t)} \underbrace{-\bar{B}_d v(t)}_{f_{cf}(t)} \quad (1)$$

where $\Delta z_i(t) = z(t) - \bar{z}_{i-1}$ is the needle tip relative penetration inside the i-th layer $L_i$, $i = 1, ..., n$ which starts at depth $\bar{z}_{i-1}$ of the first contact with the considered layer, $K_i$ and $B_i$ are the elastic and friction coefficients, $d_i$ is the layer thickness along the insertion axis and $\bar{B}_d = \sum_{j=1}^{i-1} B_j d_j$ is the constant cumulative friction term due to contact of the needle shaft with the previous layers $L_j$, with $j = 1, ..., i - 1$. The horizontal brackets below Eq. (1) highlight the three contributions of the elastic (e), friction (f), and cumulative friction (cf) forces composing the multi-layer visco-elastic interaction model.

Whenever the needle touches a new layer for the first time, the latter deforms and provides an increasing elastic force $f_e(t)$ to the user. Whenever this force exceeds a certain threshold [6], the tissue ruptures, the needle enters the new layer, and the

<figure><img src="image_3.png" /><figcaption>Fig. 2. Experimental setup. The virtual Kuka LWR robot is teleoperated through the Geomagic Touch haptic interface. One or more devices are then used to provide composite haptic feedback about the needle-tissue interaction forces (for example, in the picture, the user receives feedback from the hRing and the Geomagic Touch). During the trial, the inside view of the phantom (top right) is hidden. The needle traverses four layers of tissue, corresponding to fat, muscle, liver, and lesion (bottom right).</figcaption></figure>

elastic force drops to $f_e = 0$. In this experimental study, we only render the visco-elastic components of the force $f_e$ and $f_f$ relative to the tissue in which the needle tip is currently inserted. This choice is motivated by the observation that, during the needle penetration, the friction $f_{cf}$ accumulated along the needle shaft may hinder the detection of subsequent tissue ruptures and transitions, that are informative for a proper detection of the current insertion state. However, when executing insertions on real tissues, $f_{cf}$ can be rather easily identified and subtracted from the perceived force, so as to render to the user only the viscous component related to the tissue being punctured, i.e., $f_e$ and $f_f$ [6]. In the remainder of this Section, we briefly present the devices employed in our study for the rendering of the mentioned force components.

## B. hBracelet vibrotactile cutaneous device (interface B)

The hBracelet, shown in Fig. 1a, conveys distributed vibrotactile sensations around the user's wrist or forearm [4], [11]. It is composed of four vibrotactile motors (Pico-Vibe 304-116) attached to an elastic fabric strap. Each motor has an effective vibration frequency range between 80 and 280 Hz, with a proportionally-related amplitude between 0.2 and 1.1 g. They are positioned evenly around the arm at 90 degrees from each other and are connected to a small control board, which enables to control each motor independently. To render the force contributions $f_e$ or $f_f$ along the insertion direction, each motor is actuated by mapping the force magnitude value to the vibration intensity. This reproduces a homogeneous rendering of the chosen scalar force contribution along the arm's surface in contact with the bracelet.

## C. hRing pressure cutaneous device (interface R)

The hRing, shown in Fig. 1b, is capable of providing pressure and skin stretch stimuli at one finger [12], but in this paper we only use it for pressure feedback. It is composed
of two servo motors (HiTec HS-40) and a fabric belt, that applies the requested stimuli to the skin. A Velcro strap band secures the device to the finger. When the two motors rotate in opposite directions, the belt is pulled up or down, providing a varying force normal to the finger (pressure). The maximum displacement range of the device is 6 mm, leading to a maximum estimated normal force of 3 N [13]. This device is made to be worn at the middle or proximal finger phalanx of the finger, so as to leave the fingertip of the user free to naturally hold the pen-like end-effector of the Geomagic Touch interface. The rendering of $f_e$ or $f_f$ is accomplished by mapping the magnitude of the selected force component to the pressure applied by the belt.

### D. TouchDIVER fingertip cutaneous device (interface D)

The TouchDIVER (WEART, Italy), shown in Fig. 1c, is capable of providing pressure, vibrations, and temperature at the fingertip, but in this work we only use it for pressure and vibrotactile feedback. This enables us to provide the same stimuli of interfaces B and R, but with a different localization. The device is composed of a control unit placed on the user wrist and three independent fingertip devices. A servo motor moves a rigid platform to apply the requested pressure to the fingertip skin. The maximum attainable vertical displacement is 5 mm, leading to a maximum estimated normal force of 2.5 N [13]. Vibrations are rendered through a piezo actuator with an effective vibration frequency range in 50-450 Hz, with amplitude of 0.7-3.5 g. The elastic component $f_e$ is rendered through the contact pressure module, mapping the force magnitude to the pressure level, while the friction component $f_f$ through the vibration module, mapping the force magnitude to the vibration amplitude.

## III. EXPERIMENTAL EVALUATION

### A. Experimental setup and protocol

The 3D model of the abdomen phantom is composed of 4 layers, fat, muscle, liver, lesion (see Sec. II-A and Fig. 2). Each layer $L_i$, $i = 1, ..., 4$ is characterized by a nominal thickness $d_i$={7.3, 13, 56, 11} mm, an elastic coefficient $K_i$={200, 300, 500, 400} N/m and a friction per unit length coefficient $B_i$. We considered two tissue environments having different layer characteristics, one with less friction (LF) and one with more friction (HF), having $B_{iLF}$={700, 800, 1000, 900} Ns/m² and $B_{iHF}=B_{iLF}+150$ Ns/m², respectively. Moreover, when the needle reaches a new layer, a tissue rupture is simulated when the needle-tissue interaction force exceeds a certain threshold $t_i$=0.8, 1.5, 1.2, 1.5 N. Further details regarding this model and its medical validity can be found in [6]. How this model generates haptic feedback has been discussed in Sec. II-A.

During the insertion, participants are provided with haptic feedback about the elastic and friction force, according to the considered feedback condition and device. No visual feedback on the inside of the phantom is provided; the participant is only able to see the robot steering the needle and the needle penetrating into the tissue.

In all experiments, participants were asked to hold the end-effector of the Geomagic Touch interface with their right hand and steer the needle inside the simulated soft tissue. They

wore one or more wearable cutaneous devices according to the experiment at hand. The hBracelet was always worn on the left arm, while the hRing and the TouchDIVER were worn on the index and the middle fingers of the right hand, respectively. The thumb was left free to press the Geomagic button. Participants pressed a predefined key stroke whenever they perceived the needle penetrating into a new layer of the tissue. Inspired from [3], [6], this task is relevant in many medical scenarios, e.g., spinal tap or biopsy for interventional radiology. In fact, the proper detection of the tissue transitions is fundamental to assess the current state of the insertion when also the real-time visual feedback is poor or lacking, which is often the case.

### B. Evaluation metrics

To evaluate whenever the user identified that the needle had penetrated the *i*-th layer of the tissue, we focused on the two main events that happen during a penetration (see Sec. II-A): i) the *puncturing* event, corresponding to the time instant $t_{pi}$ when the needle tip starts to be in contact with the layer, i.e., such that $f_e(t) > 0$ for $t > t_{pi}$; and ii) the *rupture* event, corresponding to the time instant $t_{ri}$ ($> t_{pi}$), when the needle exceeds the elastic resistance of the layer surface and actually penetrates it, i.e., resulting in a force drop such that $f_e(t_{ri}) = 0$. Both events are important during needle insertion, depending on the application at hand, and they are both clearly detectable in real insertion scenarios. This leads to the definition of three performance indicators:

* True positives (TP) happen whenever the user correctly identifies the requested event, puncturing or rupture, of the new tissue layer within 3 s.

* False negatives (FN) happen whenever the user fails to recognize the requested event of the new tissue layer within 3 s, e.g., the needle penetrates a new layer but the user does not recognize it.

* False positives (FP) happen whenever the user mistakenly identifies the requested event of the new tissue layer within 3 s, e.g., the user indicates that the needle as touched (punctured) a new layer but this event did not happen.

The timespan of 3 s was chosen following the results of a set of pilot experiments, showing that recognition happens within 3 s or not at all. We can then calculate:

* the *recall* (or *sensitivity*), i.e., the ratio of correctly perceived penetrations into a new layer to all the penetrations;

* the *precision*, i.e., the ratio of correctly perceived penetrations into a new layer to the total of perceived penetrations;

* the *F1 score*, i.e., the weighted harmonic average of precision and recall.

The above metrics can be calculated for both puncturing and rupture events.

At the end of each experiment, subjects were asked to rank the conditions according to their perceived effectiveness, from the best condition (ranked 1st) to the worst one (ranked either 4th or 5th according to the experiment). We also encouraged participants to comment the tasks out loud throughout the experiment.

### C. Experiment #1: single-point vs. distributed feedback

While most haptic-enabled systems for needle insertion use a grounded kinesthetic interface to convey all the feedback
information, there is evidence that providing elastic and viscous components of the interaction force through different sensory channels might improve the intuitiveness and effectiveness of the operation. In fact, while the elastic component conveys information on the deformation of the tissue in contact with the needle tip and, hence, may enable prediction of a rupture event, the viscous force feedback is related to the interaction during motion of the needle and can enable predicting tissue damage [10].

1) **Hypotheses:** Literature results show that, in remotely operated needle insertion, navigation cues provided by haptic feedback [14] lead to a significant improvement in the targeting accuracy. This feedback can be enriched by also rendering the cutting force at the needle tip through a different haptic channel [4], reaching similar targeting accuracy while getting additional relevant information about the advancement of the procedure. Inspired by these results, we propose two hypotheses:

(H1.1) Providing feedback on elastic and friction interaction force components through different and distributed haptic channels (kinesthetic and vibrotactile) outperforms providing the same information through a single grounded kinesthetic interface;

(H1.2) Vibrotactile cutaneous sensations are effective for providing elastic force feedback.

2) **Feedback conditions:** Elastic (E) and friction (F) force feedback are computed as described in Sec. II-A. In this experiment, we employed the Geomagic Touch kinesthetic interface (G) and the hBracelet vibrotactile device (B). We considered five feedback conditions:

(E<sub>G</sub>F<sub>G</sub>) Both the elastic and friction feedback components are provided by the Geomagic Touch interface, while the hBracelet is disabled;

(E<sub>G</sub>F<sub>B</sub>) Elastic and friction feedback components are provided by the Geomagic Touch interface and the hBracelet device, respectively;

(E<sub>B</sub>F<sub>G</sub>) Elastic and friction feedback components are provided by the hBracelet device and Geomagic Touch interface, respectively;

(E<sub>B</sub>F<sub>B</sub>) Both elastic and friction feedback components are provided by the hBracelet device, while the Geomagic Touch interface provides no feedback;

(E<sub>GB</sub>F<sub>GB</sub>) Elastic and friction feedback components are combined and provided by both the Geomagic Touch interface and hBracelet device.

In all conditions, the Geomagic Touch interface is used to insert the needle inside the soft tissue, as described in Sec. II-A.

3) **Subjects:** 10 participants took part in the experiment, including 3 women and 7 men (age 24–28 years old). One practice trial per feedback condition was allowed. Each participant repeated the insertion 2 times per feedback condition, one with LF and one with HF tissue characteristics (see Sec. III-A), leading to 10 insertions per subject.

4) **Results:** As mentioned in Sec. III-B, we analyzed the F1 scores and the reported subjective effectiveness for each condition. We ran two-way repeated-measures ANOVA tests for analyzing the F1 scores. The five feedback conditions

<figure><img src="image_4.png" /><figcaption>Fig. 3. Experiment #1. Mean and 95% confidence interval of (a) F1 score (puncture events), (b) F1 score (rupture events) and (c) subjective ranking for the five feedback conditions (E<sub>G</sub>F<sub>G</sub>, E<sub>G</sub>F<sub>B</sub>, E<sub>B</sub>F<sub>G</sub>, E<sub>B</sub>F<sub>B</sub>, E<sub>GB</sub>F<sub>GB</sub>) and the two tissue environments (HF, LF). The * symbol indicates statistically significant differences in the post-hoc tests.</figcaption></figure>

(E<sub>G</sub>F<sub>G</sub>, E<sub>G</sub>F<sub>B</sub>, E<sub>B</sub>F<sub>G</sub>, E<sub>B</sub>F<sub>B</sub>, E<sub>GB</sub>F<sub>GB</sub>) and the two tissue environments (HF, LF) were treated as within-subject factors. Data were transformed using a log transformation whenever not normal. A Greenhouse-Geisser correction was used when the assumption of sphericity was violated. Sphericity was assumed for variables with only two levels of repeated measures. Fig. 3a shows the F1 score per each condition for the puncturing event. The two-way repeated-measure ANOVA revealed a statistically significant difference for this metric across feedback condition (F(4, 36) = 7.441, *p* < 0.001), but not across tissue environments. Post hoc analysis with Bonferroni adjustments revealed a statistically significant difference between conditions E<sub>G</sub>F<sub>G</sub> vs. E<sub>B</sub>F<sub>B</sub> (*p* = 0.049), E<sub>G</sub>F<sub>B</sub> vs. E<sub>B</sub>F<sub>B</sub> (*p* = 0.009), and E<sub>B</sub>F<sub>G</sub> vs. E<sub>B</sub>F<sub>B</sub> (*p* = 0.027). Fig. 3b shows the F1 score per each condition for the rupture event. The two-way repeated-measure ANOVA revealed a statistically significant difference for this metric across feedback condition (F(4, 36) = 9.248, *p* < 0.001), but not across tissue environments. Post hoc analysis with Bonferroni adjustments revealed a statistically significant difference between conditions E<sub>G</sub>F<sub>G</sub> vs. E<sub>B</sub>F<sub>B</sub> (*p* < 0.001), E<sub>G</sub>F<sub>B</sub> vs. E<sub>B</sub>F<sub>B</sub> (*p* = 0.003), and E<sub>B</sub>F<sub>G</sub> vs. E<sub>B</sub>F<sub>B</sub> (*p* = 0.017). We ran Friedman tests for analyzing the condition rankings. The five feedback conditions were treated as within-subject factors. Fig. 3c shows the average ranking per each condition. The Friedman test revealed a statistically significant difference for this metric across feedback condition (χ<sup>2</sup>(4) = 18.160, *p* < 0.001). Post hoc analysis with Bonferroni correction revealed a statistically significant difference between conditions E<sub>G</sub>F<sub>B</sub> vs. E<sub>B</sub>F<sub>B</sub> (*p* < 0.001), E<sub>GB</sub>F<sub>GB</sub> vs. E<sub>B</sub>F<sub>B</sub> (*p* = 0.019), and E<sub>G</sub>F<sub>G</sub> vs. E<sub>B</sub>F<sub>B</sub> (*p* = 0.047).

5) **Discussion:** Results are rather mixed, disproving in part both H1.1 ad H1.2. Indeed, results do not show better (or worse) performance when providing elastic and friction
feedback components through different haptic channels. The only clear result is that providing all forces through vibrotactile sensations ($E_B F_B$) perform worse than other solutions. Verbal feedback from the users gives us interesting insights regarding the underlying reasons for this outcome. Indeed, most subjects complained about the effectiveness of the vibrotactile feedback. While many subjects appreciated receiving the two information separately ("It was useful to receive two pieces of information in two different ways"), they all reported that vibrations were not effective in conveying the target information ("It was hard to discern the different levels of vibrations after a while."). Indeed, it is known that providing sustained vibrations becomes quickly uncomfortable [15], so other types of feedback might be preferable. Finally, results did not highlight any relevant difference between the two tissue environments LF and HF, meaning that the viscosity in the current tissue did not affect significantly the layer recognition process. We further emphasize here that this is motivated by the choice of rendering only the friction of the layer where the needle tip is currently located, and not the friction accumulated along the needle shaft, that could rather compromise the detection of the puncturing events [9], [16], [6].

## D. Experiment #2: vibrotactile vs. contact pressure feedback

Following the results of the previous experiment, we decided to introduce an additional haptic interface, the hRing (see Sec. II-C), so as to better represent the nature of the forces acting on the environment during this task. We expect the pressure sensations delivered by the hRing to be more suited than the vibrotactile ones rendered by the bracelet.

1) **Hypotheses:** Cutaneous feedback has been shown to be quite effective in teleoperated needle insertion and other medical procedures, as well as safer than kinesthetic feedback provided by grounded haptic interfaces [17], [18]. To analyse the effectiveness of vibrotactile and contact pressure cutaneous feedback, and considering the results of our previous experiment, we propose two hypotheses:

(H2.1) Providing elastic and friction feedback components through different and distributed haptic channels (kinesthetic and contact pressure) outperforms providing the same information through a single grounded kinesthetic interface;

(H2.2) Contact pressure cutaneous sensations are effective for providing elastic force feedback.

2) **Feedback conditions:** Elastic (E) and friction (F) interaction force feedback are again computed as described in Sec. II-A. In this experiment, we employed the Geomagic Touch kinesthetic interface (G), the hBracelet vibrotactile device (B), and the hRing pressure device (R). We considered four feedback conditions:

(E<sub>G</sub>F<sub>G</sub>) Both the elastic and friction force feedback components are provided by the Geomagic Touch interface, while the hBracelet and hRing devices are disabled;

(E<sub>G</sub>F<sub>B</sub>) Elastic and friction force feedback components are provided by the Geomagic Touch interface and the hBracelet device, respectively;

<figure><img src="image_5.png" /><figcaption>Fig. 4. Experiment #2. Mean and 95% confidence interval of (a) F1 score (puncture events) and (b) subjective ranking for the five feedback conditions (E<sub>G</sub>F<sub>G</sub>, E<sub>G</sub>F<sub>B</sub>, E<sub>R</sub>F<sub>G</sub>, E<sub>R</sub>F<sub>G</sub>!) and the two tissue environments (HF, LF). The * symbol indicates statistically significant differences in the post-hoc tests.</figcaption></figure>

(E<sub>R</sub>F<sub>G</sub>) Elastic and friction force feedback components are provided by the hRing device and Geomagic Touch interface, respectively;

(E<sub>R</sub>F<sub>G</sub>!) elastic and friction force feedback components are provided by the hRing device and Geomagic Touch interface, respectively, while the hBracelet provides a vibration burst when the needle touches the new layer so as to alert the user that a penetration is about to occur.

As before, in all conditions, the Geomagic Touch interface is used to steer the needle inside the soft tissue (see Sec. II-A). We considered again condition E<sub>G</sub>F<sub>G</sub>, as it represents the most common solution for currently-available haptic-enabled needle insertion systems, and E<sub>G</sub>F<sub>B</sub>, as it enables us to compare vibrotactile vs. pressure feedback in this scenario and performed the best among the feedback conditions considered in Experiment #1.

3) **Subjects:** 20 participants took part in the experiment, including 8 women and 12 men (age 24–29 years old). One practice trial per feedback condition was allowed. 10 of them participated in Experiment #1 as well. Each participant repeated the insertion 2 times per feedback condition, one with LF and one with HF tissue characteristics, leading to 8 insertions per subject.

4) **Results:** We analyzed the data as done in Sec. III-C4. Fig. 4a shows the F1 score per each condition for the puncturing event. The two-way repeated-measure ANOVA revealed a statistically significant difference for this metric across feedback condition (F(2.199, 41.782) = 16.265, *p* < 0.001), but not across tissue environments. Post hoc analysis with Bonferroni adjustments revealed a statistically significant difference between conditions E<sub>G</sub>F<sub>G</sub> vs. E<sub>R</sub>F<sub>G</sub> (*p* < 0.001), E<sub>G</sub>F<sub>G</sub> vs. E<sub>R</sub>F<sub>G</sub>! (*p* = 0.001), E<sub>G</sub>F<sub>B</sub> vs. E<sub>R</sub>F<sub>G</sub> (*p* = 0.002), E<sub>G</sub>F<sub>B</sub> vs. E<sub>R</sub>F<sub>G</sub>! (*p* = 0.043), and E<sub>R</sub>F<sub>G</sub> vs. E<sub>R</sub>F<sub>G</sub>! (*p* = 0.017). On the other hand, a two-way repeated-measure ANOVA revealed no statistically significant difference for the F1 score related to the rupture events (*p* > 0.05; not shown in the Figure, all mean F1 scores higher than 0.96).

Fig. 4b shows the average ranking per each condition. A Friedman test revealed a statistically significant difference for this metric across feedback condition (χ<sup>2</sup>(3) = 33.300, *p* < 0.001). Post hoc analysis with Bonferroni correction revealed a statistically significant difference between conditions
$E_R F_G$ vs. $E_G F_G$ ($p < 0.001$), $E_R F_G$ vs. $E_G F_B$ ($p < 0.001$), and $E_R F_G$! vs. $E_G F_B$ ($p = 0.020$).

5) **Discussion:** Results of this second experiment are more encouraging, supporting both H2.1 and H2.2. Indeed, decoupling elastic and friction force feedback components ($E_R F_G$) significantly outperformed providing both pieces of information through the Geomagic Touch ($E_G F_G$). The two conditions providing elastic force through the hRing ($E_R F_G$, $E_R F_G$!) performed the best among the four considered ones. However, a bit surprisingly, providing additional vibration bursts ($E_R F_G$!) did not improve the performance with respect to not providing them ($E_R F_G$); the latter was also preferred by most users. Users reported the pressure feedback from the hRing to be sufficient and that vibrotactile feedback did not add valuable information while even distracting them ("The vibration prevented me from concentrating on the pressure at the fingers.").

## E. Experiment #3: co-located vs. non co-located feedback

Following the results of the previous experiments, we wanted to better study the role of the co-location of the feedback information, taking advantage of the distributed nature of the cutaneous receptors under our skin and their different activation thresholds. For this reason, we introduced another haptic interface, the TouchDIVER, as it is able to provide contact pressure and vibration sensations at the same point.

1) **Hypotheses:** Inspired by the insights on force-feedback substitution and augmentation from [19], [20] and the results above, we propose two hypotheses:

(H3.1) Providing elastic and friction force feedback components through different but co-located haptic channels (kinesthetic, vibrotactile, and/or contact pressure) outperforms providing the same information through a single grounded kinesthetic interface;

(H3.2) Providing elastic and friction force feedback components through co-located diverse cutaneous feedback (vibrotactile and contact pressure) outperforms providing the same information but at different, farther points on the body.

2) **Feedback conditions:** Elastic (E) and friction (F) force feedback components are again computed as described in Sec. II-A. In this experiment, we employed the Geomagic Touch kinesthetic interface (G), the hRing pressure device (R), and the TouchDIVER fingertip cutaneous device (D). We considered five feedback conditions:

(E<sub>G</sub>F<sub>G</sub>) Both the elastic and friction force feedback components are provided by the Geomagic Touch interface, while the TouchDIVER and hRing devices are disabled;

(E<sub>R</sub>F<sub>G</sub>) Elastic and friction force feedback components are provided by the hRing device and Geomagic Touch interface, respectively;

(E<sub>D</sub>F<sub>D</sub>) Elastic and friction force feedback components are provided through the contact pressure and vibrotactile modules of the same TouchDIVER device, respectively;

(E<sub>D</sub>F<sub>G</sub>) Elastic and friction force feedback components are provided by the contact pressure module of the TouchDIVER device and Geomagic Touch interface, respectively;

<figure><img src="image_6.png" /><figcaption>Fig. 5. Experiment #3. Mean and 95% confidence interval of (a) F1 score (puncture events) and (b) subjective ranking for the five feedback conditions (E<sub>G</sub>F<sub>G</sub>, E<sub>R</sub>F<sub>G</sub>, E<sub>D</sub>F<sub>D</sub>, E<sub>D</sub>F<sub>G</sub>, E<sub>G</sub>F<sub>D</sub>) and the two tissue environments (HF, LF). The * symbol indicates statistically significant differences in the post-hoc tests.</figcaption></figure>

(E<sub>G</sub>F<sub>D</sub>) Elastic and friction force feedback components are provided by the Geomagic Touch interface and the vibrotactile module of the TouchDIVER device, respectively.

As before, in all conditions, the Geomagic Touch interface is used to teleoperate the needle insertion inside the soft tissue (see Sec. II-A). We considered again condition E<sub>G</sub>F<sub>G</sub> as it represents the most common solution for currently-available haptic-enabled needle insertion systems, and E<sub>R</sub>F<sub>G</sub> because it enables us to compare co-localized vs. de-localized feedback in this scenario and performed the best among the feedback conditions considered in Experiment #2.

3) **Subjects:** 30 participants took part in the experiment, including 14 women and 16 men (age 24–35 years old). One practice trial per feedback condition was allowed. 20 of them participated in Experiment #2 as well. Each participant repeated the insertion 2 times per feedback condition, one with LF and one with HF tissue characteristics, leading to 10 insertions per subject.

4) **Results:** We analyzed the data as done in Secs. III-C4 and III-D4. Fig. 5a shows the F1 score per each condition for the puncturing event. The two-way repeated-measure ANOVA revealed a statistically significant difference for this metric across feedback condition (F(2.665, 77.275) = 16.731, *p* < 0.001), but not across tissue environments. Post hoc analysis with Bonferroni adjustments revealed a statistically significant difference between conditions E<sub>G</sub>F<sub>G</sub> vs. E<sub>R</sub>F<sub>G</sub> (*p* < 0.001), E<sub>G</sub>F<sub>G</sub> vs. E<sub>D</sub>F<sub>G</sub> (*p* = 0.001), E<sub>R</sub>F<sub>G</sub> vs. E<sub>D</sub>F<sub>D</sub> (*p* = 0.014), E<sub>R</sub>F<sub>G</sub> vs. E<sub>G</sub>F<sub>D</sub> (*p* < 0.001), E<sub>D</sub>F<sub>D</sub> vs. E<sub>G</sub>F<sub>D</sub> (*p* = 0.017), and E<sub>D</sub>F<sub>G</sub> vs. E<sub>G</sub>F<sub>D</sub> (*p* < 0.001). Fig. 5b shows the average ranking per each condition. A Friedman test revealed a statistically significant difference for this metric across feedback condition (χ<sup>2</sup>(4) = 17.680, *p* = 0.001). Post hoc analysis with Bonferroni correction revealed a statistically significant difference between conditions E<sub>R</sub>F<sub>G</sub> vs. E<sub>G</sub>F<sub>D</sub> (*p* = 0.015) and E<sub>R</sub>F<sub>G</sub> vs. E<sub>G</sub>F<sub>G</sub> (*p* = 0.011). The two-way repeated-measure ANOVA also revealed a statistically significant difference for the F1 score for the rupture event across feedback condition (F(1.834, 53.194) = 5.620, *p* = 0.007; not shown in the Figure, all mean F1 scores higher than 0.89). Post hoc analysis with Bonferroni adjustments revealed a statistically significant difference between E<sub>G</sub>F<sub>G</sub> vs. E<sub>D</sub>F<sub>D</sub> (*p* = 0.019), where E<sub>D</sub>F<sub>D</sub> outperformed E<sub>G</sub>F<sub>G</sub>.
5) *Discussion*: Results partially support H3.1, as co-located diverse information ($E_D F_G$) outperforms providing the same information through the kinesthetic channel only ($E_G F_G$). However, this was not true in all cases, as $E_D F_D$ or $E_G F_D$ did not perform significantly better than $E_G F_G$. This result can be justified by recalling that $E_D$ is provided through the contact pressure module of the TouchDIVER while $F_D$ by the vibrotactile module of the same device, confirming that vibrotactile feedback is not effective in providing this type of information (as already seen in Experiment #1). On the other hand, results do not support H3.2, as co-located information (e.g., $E_D F_G$) did not outperform non-colocated feedback ($E_R F_G$). While this might seem surprising, there have been successful attempts to apply the haptic feedback away from the ideal/expected point of application [13]. Results confirm that providing elastic and friction force feedback, respectively, through the cutaneous and kinesthetic channels ($E_R F_G$) is better than providing them through the same one (either kinesthetic, $E_G F_G$, or cutaneous, $E_D F_D$). Results also confirm the good performance of the cutaneous pressure delivered by the hRing for conveying elastic force feedback ($E_R F_G$), which performed the best among all the considered conditions. Finally, results confirm that rupture events are generally easier to detect than puncturing ones.

## IV. CONCLUSIONS

We evaluated the role of haptic feedback during needle insertion. We carried out three experiments, enrolling 10, 20, and 30 human subjects, to evaluate the effect of grounded kinesthetic feedback, cutaneous vibrotactile feedback, and cutaneous pressure feedback for rendering elastic and friction components of the needle-tissue interaction force during simulated needle insertion. Results showed that providing these two pieces of feedback information through different haptic channels outperforms providing both information through the same one, either kinesthetic or cutaneous, showing the importance of multisensory haptic feedback in this context. We also found out that pressure cutaneous sensations are very effective at conveying the elastic component of the interaction, while vibrotactile feedback should be avoided for this type of information. Finally, results showed that delocalized cutaneous sensations, e.g., pressure on the proximal finger phalanx instead of the fingertip, can lead to high performance and even outperform localized feedback in some cases, opening an interesting line of research questions regarding how far this delocalization can go.

To conclude, it is also important to highlight a few limitations of the presented approach. Cutaneous sensations were provided through three different devices, i.e., the hBracelet (vibrations), the hRing (pressure/skin stretch), and the TouchDIVER (pressure/vibration/temperature) interfaces, which provide diverse stimuli at different parts of the body, i.e., the forearm, the proximal phalanx of the finger, and the fingertip. For this reason, it is of course possible that using other rendering parameters, e.g., different stimuli strengths and patterns, or body locations, e.g., on the torso, back of the hand, leads to different results than those presented here. The rendering and location choices evaluated in this work stemmed from previous work on the topic, as discussed throughout the text, as well as the author's

experience in cutaneous haptics; however, future work needs to extend this evaluation towards an even broader range of feedback renderings and parameters. Finally, validating the proposed approach in a real needle insertion in soft tissue will be beneficial towards its final use and translational to the medical field.

## REFERENCES

[1] M. Abayazid, C. Pacchierotti, P. Moreira, R. Alterovitz, D. Prattichizzo, and S. Misra, "Experimental evaluation of co-manipulated ultrasound-guided flexible needle steering," *Int. J. Med. Robot.*, vol. 12, no. 2, pp. 219-230, 2016.

[2] A. Majewicz and A. M. Okamura, "Cartesian and joint space teleoperation for nonholonomic steerable needles," in *Proc. IEEE World Haptics Conf.*, 2013, pp. 395-400.

[3] L. Meli, C. Pacchierotti, and D. Prattichizzo, "Experimental evaluation of magnified haptic feedback for robot-assisted needle insertion and palpation," *Int. J. Med. Robot.*, vol. 13, no. 4, p. e1809, 2017.

[4] M. Aggravi, D. A. Estima, A. Krupa, S. Misra, and C. Pacchierotti, "Haptic teleoperation of flexible needles combining 3d ultrasound guidance and needle tip force feedback," *IEEE Robotics and Automation Letters*, vol. 6, no. 3, pp. 4859-4866, 2021.

[5] M. Khadem, C. Rossa, R. S. Sloboda, N. Usmani, and M. Tavakoli, "Mechanics of tissue cutting during needle insertion in biological tissue," *IEEE Robotics and Automation Letters*, vol. 1, no. 2, pp. 800-807, 2016.

[6] M. Ferro, C. Gaz, M. Anzidei, and M. Vendittelli, "Online needle-tissue interaction model identification for force feedback enhancement in robot-assisted interventional procedures," *IEEE Trans. Medical Robotics and Bionics*, vol. 3, no. 4, pp. 936-947, 2021.

[7] R. J. Roesthuis, Y. R. Van Veen, A. Jahya, and S. Misra, "Mechanics of needle-tissue interaction," in *Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems*, 2011, pp. 2557-2563.

[8] O. Gerovich, P. Marayong, and A. Okamura, "The effect of visual and haptic feedback on computer-assisted needle insertion," *Computer Aided Surgery*, vol. 9, no. 6, pp. 243-249, 2004.

[9] L. Barbé, B. Bayle, M. de Mathelin, and A. Gangi, "In vivo model estimation and haptic characterization of needle insertions," *Int. J. Robotics Research*, vol. 26, no. 11-12, pp. 1283-1301, 2007.

[10] M. Mahvash and P. E. Dupont, "Mechanics of dynamic needle insertion into biological material," *IEEE Trans. Biomedical Engineering*, vol. 57, no. 4, pp. 934-943, 2010.

[11] M. Aggravi, G. Sirignano, P. R. Giordano, and C. Pacchierotti, "Decentralized control of a heterogeneous human-robot team for exploration and patrolling," *IEEE Trans. Automation Science and Engineering*, 2021.

[12] C. Pacchierotti, G. Salvietti, I. Hussain, L. Meli, and D. Prattichizzo, "The hring: A wearable haptic device to avoid occlusions in hand tracking," in *Proc. IEEE Haptics Symposium (HAPTICS)*, 2016, pp. 134-139.

[13] X. De Tinguy, C. Pacchierotti, M. Marchal, and A. Lécuyer, "Enhancing the stiffness perception of tangible objects in mixed reality using wearable haptics," in *Proc. IEEE Conf. Virtual Reality and 3D User Interfaces (VR)*, 2018, pp. 81-90.

[14] C. Pacchierotti, M. Abayazid, S. Misra, and D. Prattichizzo, "Teleoperation of steerable flexible needles by combining kinesthetic and vibratory feedback," *IEEE Trans. Haptics*, vol. 7, no. 4, pp. 551-556, 2014.

[15] L. Devigne, M. Aggravi, M. Bivaud, N. Balix, C. S. Teodorescu, T. Carlson, T. Spreters, C. Pacchierotti, and M. Basel, "Power wheelchair navigation assistance using wearable vibrotactile haptics," *IEEE Trans. Haptics*, vol. 13, no. 1, pp. 52-58, 2020.

[16] D. De Lorenzo, Y. Koseki, E. De Momi, K. Chinzei, and A. M. Okamura, "Coaxial needle insertion assistant with enhanced force feedback," *IEEE Transactions on Biomedical Engineering*, vol. 60, no. 2, pp. 379-389, 2013.

[17] C. Pacchierotti, F. Chinello, and D. Prattichizzo, "Cutaneous device for teleoperated needle insertion," in *Proc. IEEE RAS&EMBS Intl. Conf. Biomedical Robotics and Biomechatronics (BioRob)*, 2012, pp. 32-37.

[18] C. Pacchierotti, D. Prattichizzo, and K. J. Kuchenbecker, "Cutaneous feedback of fingertip deformation and vibration for palpation in robotic surgery," *IEEE Trans. Biomed. Eng.*, vol. 63, no. 2, pp. 278-287, 2015.

[19] C. Pacchierotti, A. Tirmizi, G. Bianchini, and D. Prattichizzo, "Enhancing the performance of passive teleoperation systems via cutaneous feedback," *IEEE Trans. Haptics*, vol. 8, no. 4, pp. 397-409, 2015.

[20] Z. F. Quek, S. B. Schorr, I. Nisky, W. R. Provancher, and A. M. Okamura, "Sensory substitution and augmentation using 3-degree-of-freedom skin deformation feedback," *IEEE Trans. Haptics*, vol. 8, no. 2, pp. 209-221, 2015.