# Wearable multichannel haptic device for encoding proprioception in the upper limb

Patrick Sagastegui Alva, Silvia Muceli, Seyed Farokh Atashzar, Lucie

William, Dario Farina

► To cite this version:

Patrick Sagastegui Alva, Silvia Muceli, Seyed Farokh Atashzar, Lucie William, Dario Farina. Wearable multichannel haptic device for encoding proprioception in the upper limb. Journal of Neural Engineering, 2020, 10.1088/1741-2552/aba6da . hal-02908309

HAL Id: hal-02908309

https://inria.hal.science/hal-02908309

Submitted on 7 Sep 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
ACCEPTED MANUSCRIPT • OPEN ACCESS

# Wearable multichannel haptic device for encoding proprioception in the upper limb

To cite this article before publication: Patrick G. Sagastegui Alva et al 2020 *J. Neural Eng.* in press https://doi.org/10.1088/1741-2552/aba6da

## Manuscript version: Accepted Manuscript

Accepted Manuscript is "the version of the article accepted for publication including all changes made as a result of the peer review process, and which may also include the addition to the article by IOP Publishing of a header, an article ID, a cover sheet and/or an 'Accepted Manuscript' watermark, but excluding any other editing, typesetting or other changes made by IOP Publishing and/or its licensors"

This Accepted Manuscript is © 2020 The Author(s). Published by IOP Publishing Ltd.

As the Version of Record of this article is going to be / has been published on a gold open access basis under a CC BY 3.0 licence, this Accepted Manuscript is available for reuse under a CC BY 3.0 licence immediately.

Everyone is permitted to use all or part of the original content in this article, provided that they adhere to all the terms of the licence https://creativecommons.org/licenses/by/3.0

Although reasonable endeavours have been taken to obtain all necessary permissions from third parties to include their copyrighted content within this article, their full citation and copyright line may not be present in this Accepted Manuscript version. Before using any content from this article, please refer to the Version of Record on IOPscience once published for full citation and copyright details, as permissions may be required. All third party content is fully copyright protected and is not published on a gold open access basis under a CC BY licence, unless that is specifically stated in the figure caption in the Version of Record.

View the article online for updates and enhancements.
# Wearable Multichannel Haptic Device for Encoding Proprioception in the Upper Limb

Patrick G. Sagastegui Alva<fnref n="1" />,<fnref n="7" />, Silvia Muceli<fnref n="2" />,<fnref n="7" />, S. Farokh Atashzar<fnref n="3" />,<fnref n="4" />,<fnref n="5" />,<fnref n="7" />, Lucie William<fnref n="6" />,<fnref n="7" />, Dario Farina<fnref n="1" />,<fnref n="8" />

## Abstract.

*Objective.* We present the design, implementation, and evaluation of a wearable multichannel haptic system. The device is a wireless closed-loop armband driven by surface electromyography and provides sensory feedback encoding proprioception. The study is motivated by restoring proprioception information in upper limb prostheses. *Approach.* The armband comprises eight vibrotactile actuators that generate distributed patterns of mechanical waves around the limb to stimulate perception and to transfer proportional information on the arm motion. An experimental study was conducted to assess: the sensory threshold in 8 locations around the forearm, the user adaptation to the sensation provided by the device, the user performance in discriminating multiple stimulation levels, and the device performance in coding proprioception using four spatial patterns of stimulation. Eight able-bodied individuals performed reaching tasks by controlling a cursor with an EMG interface in a virtual environment. Vibrotactile patterns were tested with and without visual information on the cursor position with the addition of a random rotation of the reference control system to disturb the natural control and proprioception. *Results.* The sensation threshold depended on the actuator position and increased over time. The maximum resolution for stimuli discrimination was four. Using this resolution, four patterns of vibrotactile activation with different spatial and magnitude properties were generated to evaluate their performance in enhancing proprioception. The optimal vibration pattern varied among the participants. When the feedback was used in closed-loop control with the EMG interface, the task success rate, completion time, execution efficiency, and average target-cursor distance improved for the optimal stimulation pattern compared to the condition without visual or haptic information on the cursor position. *Significance.* The results indicate that the vibrotactile device enhanced the participants' perceptual ability, suggesting that the proposed closed-loop system has the potential to code proprioception and enhance user performance in the presence of perceptual perturbation.

<footnote n="1">¹Department of Bioengineering, Imperial College London, London, United Kingdom</footnote>

<footnote n="2">²Division of Signal Processing and Biomedical Engineering, Department of Electrical Engineering, Chalmers University of Technology, Gothenburg, Sweden</footnote>

<footnote n="3">³Department of Electrical and Computer Engineering, New York University (NYU), United States of America</footnote>

<footnote n="4">⁴Department of Mechanical and Aerospace Engineering, NYU</footnote>

<footnote n="5">⁵NYU WIRELESS center</footnote>

<footnote n="6">⁶INRIA, University of Montpellier, CNRS, Montpellier, France</footnote>

<footnote n="7">⁷These authors contributed equally and shared the first authorship</footnote>

<footnote n="8">⁸E-mail: d.farina@imperial.ac.uk</footnote>
**Keywords:** Sensory substitution, Vibrotactile feedback, Haptics, Assistive Devices, Biofeedback, Prosthetic devices, Wearable technologies.

# 1. Introduction

Proprioception is an imperative sensory modality that is required for the proper execution of a wide range of activities of daily living. In the absence of a biological limb, corresponding sensory information is partially lost along with motor abilities.

Myoelectric prostheses, which are controlled by electromyography (EMG) signals from the residual muscles of the stump [1], can partially replace the lost motor functionality using pattern decoding and machine learning. Although the available prostheses yield complex movements [2], they do not provide explicit sensory feedback. This degrades the functionality of the system and results in an unintuitive control [3]–[5]. Lack of sensory feedback results in excessive visual and cognitive loads for the users [3] and has been indicated as one of the main reasons causing a high rejection rate of active prostheses [5], [6].

Proprioception and haptics (sense of touch) are likely the two sensory modalities that are needed for prosthetic users [7]. Proprioception is the most challenging perception to restore because of its multimodal nature. Here, we focus on the restoration of proprioception using a wearable multichannel vibrotactile feedback system that provides a wide variety of spatiotemporal vibrotactile patterns.

The concepts of sensory supplementation and proprioception restoration have been investigated in past studies that relied on invasive or non-invasive approaches. Invasive methods use electrodes to directly stimulate the nerves [8]–[9] or the primary

sensory cortex in the brain [10]. For the non-invasive methods, cutaneous electro-tactile [11] and vibrotactile [12] feedback systems have been tested in the literature, eliciting different sensations by activating different receptors on the skin [13]. To augment the bandwidth and complexity of the feedback, a typical approach is to increase the number of feedback channels [10]. These methods provide a sensation different from natural proprioception (sensory substitution). This information is provided by the modulation of parameters on the stimulation point, such as amplitude or frequency.

In this study, we focus on vibrotactile stimulation. The feasibility of using vibrotactile devices as sources of sensory information has been previously studied [14], [15], mainly for haptic sensation [16], [17]. For example, adding vibrotactile feedback for the touch sensation to myoelectric prostheses has shown improvement for task completion time and the controllability over force generation during grasping [18]–[19]. Most previous studies have focused on grasp parameters, such as force control [18]–[19], variability and characteristics of contact point [20], and the combination of force and one degree of freedom of hand motion [21]. Conversely, in this study, we investigate the encoding of multiple degrees of freedom of proprioception through multichannel closed-loop stimulation. This is achieved with an armband comprising eight vibrators. With this system and a myocontrol interface, we evaluated the ability of users to utilize the provided sensory feedback to complete a reach-
Artificial proprioception

ing task without visual feedback and in the
presence of unpredictable perturbations of
the reference control system.

It should be noted that the proprio-
ceptive feedback provided by the system
to the user was calculated in a closed-loop
fashion based on the predicted intention
of the user for generating the needed mo-
tion. In the experiments, we have also
disturbed (by unpredictable random rota-
tion) the mapping between the natural mo-
tion and how the cursor moves. This was
done to disturb the natural propriocep-
tive awareness of the user and thus to in-
vestigate whether the provided multichan-
nel positional sensory feedback could re-
cover the distracted proprioceptive aware-
ness. The result of this analysis will illus-
trate the potential capability of the pro-
posed system in restoring the lost propri-
oception. Thus, the proposed system pro-
vides information about the spatial motion
of the user's limb to give the user concur-
rent awareness of where the limb position
is in space so that the user can steer the
control consequently. The distributed pat-
tern of vibration provides positional infor-
mation in parallel with the user's natural
proprioception.

## 2. Materials and Methods

2.1. *Design of multichannel vibrotactile*
*interface*

The proposed closed-loop vibrotactile in-
terface consists of a wearable wireless de-
vice (Figure 1) that allows full customiza-
tion of closed-loop stimulation patterns.
This wearable armband embeds eight vi-
bration modules. Each module includes
a motor (310-103 Vibration ERM, Preci-

sion Microdrive's, USA), and the driver
(DR2605L Haptic Driver, Texas Instru-
ments, USA). An embedded Arduino Micro
was used to generate the control signals of
the waveforms for the vibration patterns.

In order to control the motors and pro-
vide the needed voltage, each stimulating
module includes embedded amplifiers. The
motors' driver was a DRV2605L to play
custom-made waveforms directly through
the integrated circuit (I2C). This allowed
us to control all motors by using only four
wires, which would allow for the expansion
of vibration units on the same I2C bus. In
addition, an address translator (LTC4316,
Analog Devices, USA) was utilized to mod-
ify the default address of the motor driver.

The frequency of the vibrotactile mo-
tors was controlled by tuning the voltage
to each motor separately. The voltage was
computed by the microprocessor and sent
by I2C to the corresponding motor driver.
An increase of the voltage determined a si-
multaneous increase of the vibration am-
plitude and frequency, due to the coupled
characteristics of the vibrotactile motors.
The communication between the PC and
the armband occurred via USB and Blue-
tooth v2.0.

## 2.2. Demographic Data

The study included two experimental ses-
sions run in two days. Nine able-bodied
participants were included in the experi-
ments. The experimental procedure was
approved by the Imperial College Research
Ethics Committee (ref number 18IC4685).
All participants signed an informed consent
before the experiments. Experiment 1 (4
males and 4 females, age 27.0 ± 3.1 years)
aimed at determining the sensitivity to vi-
## Artificial proprioception

bration around the forearm delivered by the developed armband and the ability to discriminate different levels of vibrations. Experiment 2 (5 males and 3 females, age $27.0 \pm 3.1$ years) aimed at investigating if proprioception information could be substituted by the vibration. Seven subjects were common to both experiments.

<figure><img src="image_2.png" /><figcaption>Figure 1: A) Assembly diagram of the vibrotactile unit. B) Front and lateral views of an individual vibrotactile unit. C) Armband with all the units. D) Position of the motors around the forearm.</figcaption></figure>

## 2.3. Experimental setup

The experimental setup comprised:

(i) A host PC (Intel® Core(TM) i7-

8750H CPU at 2.20 GHz, 16 GB RAM) running a closed-loop myocontrol framework and an external 15.6” screen to provide visual feedback to the participants

(ii) Sixteen pre-gelled monopolar EMG electrodes (Ambu Neuroline 72000-S/25, Denmark)

(iii) A ground electrode

(iv) An EMG amplifier (OT EMG USB2+, OT Bioelettronica, Italy)

(v) The developed multichannel vibrotactile interface (section 2.1)

The vibrotactile interface (armband) was placed on the proximal third of the forearm, according to Figure 1, D. Using the specific mechanical design, motors around the forearm were positioned so that motor #1 was on top of the ulna, and motor #5 was at the opposite side. The other six motors were placed equidistant, with three motors (motor #2, #3, and #4) on the volar side and the others on the dorsal side (see Figure 1, D). Sixteen EMG electrodes were placed in two rows circumferentially and equidistant around the forearm, distally with respect to the armband to record muscle electrical activity. The center-to-center electrode distance was 2 cm in the forearm longitudinal direction. The center-to-center distance between the motors and the most proximal row of electrodes was 2.5 cm. The ground electrode was placed at the wrist. A tabletop EMG amplifier, OT EMG USB2+, was used to retrieve high-quality signals for reliable control. The amplification gain was set to be 100 V/V, sample rate at 2048 Hz with a band-pass filter at 10 - 500 Hz. The control loop was implemented using a myoelectric interface developed in MATLAB

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
that processed and decoded the EMG signals. The interface displayed a virtual environment that represented the position of the hand (prosthesis).

Two degrees of freedom of the wrist were used for online control (wrist flexion/extension and radial/ulnar deviation). A linear regressor was used as the machine learning core of the implemented software to map the EMG signals to cursor movements. The input for the linear regressor was the RMS of the EMG signals calculated over sliding time windows of 160 ms with 120 ms overlap. Three series of movements were included in the training phase: flexion/extension, radial/ulnar deviation, and a combination of the two along with the 45°, 135°, 225°, and 315° directions. Each movement was repeated 3 times. The trained regressor was used in the controller allowing for simultaneous and proportional steering of the two degrees of freedom. There were no visible artefacts due to the stimulation on the EMG; the cursor control was not affected when the vibration was activated. This can also be concluded from the high performance in the task when the visual feedback was provided (the simplest motor control task), in which the system was able to regress the intended motion of the user.

## 2.4. *Experiment 1: Resolution*

Experiment 1 was designed to determine the individual's sensitivity to the vibration system and the ability to discriminate different stimulation parameters. Participants were comfortably seated in front of a table with the dominant (right) arm in a rest position (palm facing inwards and thumb pointing upwards) over an arm sup-

port. Experiment 1 included an initial assessment of the sensation threshold (Experiment 1.a), a vibration intensity discrimination test (Experiment 1.b), and a final re-assessment of the sensation threshold (Experiment 1.c).

In Experiment 1.a, the vibration frequency was incremented from 0 Hz (no stimulation) in steps of 3% (6.69 Hz) of the motor vibration ($V_{max}$ = 232 Hz) until the participant reported perceiving the stimulus ($V_{l2h_m}$, where *m* is the motor index). Then, the vibration frequency was set to 40% of the maximal vibration of the motors (116.4 Hz) and was decremented until the participant reported feeling no sensation of the vibration ($V_{h2l_m}$). For each motor, we considered as sensation threshold ($V_{th_m-i}$, where *i* stands for initial) the maximum between $V_{l2h_m}$ and $V_{h2l_m}$. The lower limit ($V_{l_m}$) used during Experiment 1.b was $1.1 \cdot V_{th_m-i}$.

In Experiment 1.b, we determined the number of vibrotactile feedback levels that the user could effectively discriminate. We refer to this test as intensity discrimination rather than frequency discrimination because vibration frequency and amplitude are coupled in the motors we used (section 2.1). The provided levels of vibration ranged from $V_{l-m}$ to $V_{max}$. The vibration range was discretized in 4 and 5 equidistant levels. The different numbers of levels were provided in a randomized order across different participants. As in previous studies [22], [23], for these tests, we included learning, reinforced learning, and validation. The learning phase was meant to familiarize the participant with the vibration levels. For each location, a series of different vibration levels was presented five times in ascending order, and the level
## Artificial proprioception

6

delivered was shown on a screen in front of the participant. For the reinforced learning phase, a random stimulus of the 4 or 5 levels was delivered to the participant until the perceived level was reported and selected on the interface. The correct answer was then shown on the screen. All levels were presented randomly 5 times each (20 trials for 4 levels and 25 trials for 5 levels). The validation step had a structure similar to the reinforced learning step, but in this case, the answer was not provided.

Experiment 1.c aimed at investigating the adaptation phenomena, an increase in the sensation threshold after a persisting stimulation by vibration. This phenomenon has been reported in the literature and has a fast recovery process for the vibrotactile sensation (2 or 3 minutes) [24]. Following Experiment 1.b, the sensation threshold for the position corresponding to each motor ($V_{th\_m\_f}$, where *f* stands for final) was determined again following the procedure described for Experiment 1.a.

## 2.5. Experiment 2: Evaluation of proprioception encoding

The reliability and performance to control the position of the hand in closed-loop were evaluated in Experiment 2 through the delivery of different feedback patterns around the forearm to map the direction of the wrist. The task consisted of reaching 20 random targets in a virtual environment (Figure 2).

As in Experiment 1, participants were comfortably seated in front of a table with the dominant arm over a support, the hand palm facing inwards, and thumb pointing upwards. The position of the user's hand was mapped to the motion of a red cursor

in the virtual representation. Wrist extension and flexion movements (when detected correctly by the machine learning algorithm) produced displacements of the cursor to the right and left, respectively, whereas radial and ulnar deviation moved the cursor respectively up and down. During the test phase, the mapping was perturbed by applying a random rotation of +90 or -90 degrees. This rotation was introduced to investigate whether participants could rely more on the added sensory channels (in comparison to their natural proprioception) to reconstruct the perturbed sensory navigation task.

<figure><img src="image_3.png" /></figure>

Figure 2: Virtual reality environment for position control. Flexion/extension movements controlled the horizontal direction of the red cursor, ulnar/radial deviation controlled its vertical direction. The magnitude of the movement is the distance to the rest position (the center of the environment), *M*, and the angular position is given by $\alpha$. The purple circle is the target. The cyan circle, with radius *r*, is the workspace where the target may appear.

Three feedback modes were evaluated to perform the task: Visual, NVNV (no visual no vibration), and Vibrotactile feedback.
back. During the *Visual* feedback modal-
ity, the participant could see the cursor on
the screen. For the other feedback modal-
ities (*Vibrotactile* and *NVNV* mode), the
cursor was hidden so that the participants
could only see when the cursor hit the tar-
get at the end of the task by changing the
color to green if it is hit correctly. In the
case of the *Vibrotactile* feedback, 4 pat-
terns of vibration were tested, as described
below. No vibration and no visual feedback
were applied in the *NVNV* mode. The
task was considered successful if the par-
ticipant was able to position the red cursor
within the purple circle for at least 300 ms
(dwelling time) consecutively, within 20 s
(task failure time) [25]. In this case, the
color of the purple circle changed to green
to notify the user that the task was suc-
cessful.

The regressor processed the 16 surface EMG input signals and estimated the intended intensity and orientation of the motion. The estimated intended motion was used to control the two characteristics of the patterns of vibrotactile feedback (i.e., the intensity and direction of vibration) to close the loop. The sensory feedback was provided as follows: the amplitude of the movement (*M*) was mapped to the number of active motors and to the frequency of vibration of the active motors (*α*) was associated to the location of the group of active motors. The whole range of amplitude of the movements was divided into 4 levels of vibration frequency (see section 3.1).

The patterns used to map the propri-
oception are shown in Figure 3. The green
color represents the active motors, and the
frequency of vibration is represented by
the color intensity with darker colors cor-
responding to a higher frequency. The con-

trol performance obtained with these pat-
terns were compared to the *Visual* and
*NVNV* modes that acted as a benchmark.

<figure><img src="image_4.png" /><figcaption>Figure 3: Different feedback modalities. In the code for the *Vibrotactile* modality, *SpMod* stands for spatial modulation, *FrMod* for frequency modulation, the first number indicates the *number of modulated motors*, and the second the *number of active motors*. *NVNV* stands for no vision, no vibration. The intensity of the colors (light vs dark) corresponds to the intensity of the vibration at the corresponding point (low vs high intensity).</figcaption></figure>

In Figure 3, *VibSpMod* refers to one
possible activation pattern that started
with one motor aligned with the direction
of the movement and then it activated the
surrounding motors at the same frequency
while the number of motors activated in-
creased with the amplitude of the move-
ment (level). *VibAmpMod3-3* activated
three motors, while their frequency in-
creased with the level. For *VibAmpMod1-1*
only one motor was used to map the cursor
direction and the frequency increased with
the level. *VibAmpMod1-3* activated always
## Artificial proprioception

three motors, but only the frequency of the central one increased with the level.

For each feedback pattern, the user had 2 minutes to free play with both the *Visual* and *Vibrotactile* feedback modalities in order to explore the different stimulation levels around the workspace (for the NVNV feedback, as there was no feedback at all, participants could not explore the environment). The mapping rotation was not applied during the familiarisation phase. Then the user had the task to reach the 20 targets based only on the vibrotactile feedback, the visual feedback or no feedback (NVNV mode). The red cursor that indicated the visual online feedback was disabled for all patterns except for the *Visual* feedback.

## 2.6. Data analysis

For Experiment 1.b, the outcome measure was the success rate, expressed as the percentage of the correctly identified vibration levels. Confusion matrices were also generated to evaluate and visualize how much the participants confused different levels and how the confusion varied among different levels. For Experiments 1.a and 1.c, one-way ANOVA and Turkey's honestly significant criterion were used to compare the sensation threshold in different physiological positions (i.e. corresponding to different motors) and the initial ($V_{th\_m\_i}$) and final ($V_{th\_m\_f}$) thresholds. For Experiment 1.b, one-way ANOVA was used to compare the ability of participants to distinguish four or five vibration levels. For Experiment 2, the following performance metrics were evaluated: (a) the percentage of reached targets out of the 20 attempted (success rate), (b) the average distance between

the cursor's position and the target along the traveled path, (c) the median pathway efficiency (PathE), (d) the median duration of the trial (trial time), (e) the median speed, (f) the median completion time of the successful trials (successful trial time).

The pathway efficiency (PathE) was defined as the ratio between the shortest pathway to reach the target and the path traveled by the cursor, expressed as a percentage. The metrics used for quantifying the trial duration (trial time and successful trial time) were complementary. For example, trial time discriminated between a participant who quickly reached a target once and failed all the other trials and a participant who reached all targets in a similar time. Successful trial time discriminated a participant who could accomplish a reach in > 20 s from another one who could not finish irrespective of the time given.

As data did not pass the normality Shapiro test, the non-parametric Wilcoxon signed-rank test was used. The threshold for the statistical significance was set at *p* < 0.05. The results were expressed as *mean* ± *standard deviation* or as a box plot.

# 3. Results

## 3.1. Experiment 1: Resolution

The statistical analysis demonstrated a significant difference of the sensation threshold for all participants and all the motors around the forearm between Experiment 1.a (initial, 87.2±12.6 Hz) and Experiment 1.c (final, 100.1 ± 15.3 Hz, *p* < 0.001, see Figure 4, A).

In addition, a significant difference
was found between the initial (Experiment 1.a) and final (Experiment 1.c) sensation threshold for all locations (motors), except for the ulnar and radial positions (see black starts in Figure 4, B, physiological positions 1 and 5).

Statistical differences were observed between the sensation thresholds of the different physiological positions of stimulation (both Experiment 1.a (green stars) and 1.c (blue stars), see Figure 4, B).

In all cases, the threshold of the forearm region situated on top of the ulna (physiological position 1) was significantly lower than the threshold of the dorsal side of the forearm (physiological positions 6, 7 and 8).

The statistical analysis of the results for Experiment 1.b demonstrated a significant difference in the success rate for all participants and all motors around the forearm for the discrimination of 4 and 5 levels of intensity (86.6 ± 11.4 % and 75.7 ± 11.5 %, respectively *p* < 0.001, see Figure 5, A).

Additionally, on average around the forearm, there were significant statistical differences between the success rate for 4 and 5 different levels of intensity for the positions 2, 5, 6 and 7 (corresponding to the ulnar/volar, the radial, the radial/dorsal and the dorsal parts of the forearm, see 5, B and C).

## 3.2. Experiment 2: Evaluation of proprioception encoding

Experiment 2 tested the ability to perceive the cursor movement with different feedback strategies (*Visual* mode, *NVNV*, *vibrotactile* mode).

Figure 6 shows the performance of each participant (column) for each performance

metrics (rows A to F) and feedback modalities. In each panel, the green area corresponds to the region where the results are better than *NVNV* feedback, and the grey area is the one where the results are worse than *NVNV* feedback.

Figure 7 shows a representative example of differences between the path traveled in case of the *NVNV* feedback and the best feedback (in that case, *VibFrMod3-3* feedback) for one participant.

From the results of Figure 6, it is possible to find for each subject the best vibrotactile pattern which had the most frequent statistical significant difference or higher success rate. For each individual, such a pattern is the one that has the maximum benefit for the user in terms of substituting proprioception when compared with no feedback experiment (*NVNV* mode).

The feedback with the best performance was *VibFrMod1-1* for participants 5 and 8, *VibFrMod1-3* for participants 3 and 6, *VibFrMod3-3* for participants 2 and 7 and *VibSpMod* for participants 1 and 4.

As expected, *Visual* feedback resulted in better performance compared to *NVNV* (Figure 8). In fact, success rate and path efficiency were higher, the average distance from the target and the time to accomplish the trials and successful trials were lower. However, speed was lower in case of *Visual* feedback.

The comparison of the *Visual* and *NVNV* feedback with the *Best* vibrotactile feedback demonstrated an improvement of the performance with the vibrotactile feedback compared to the *NVNV* mode for all metrics but speed, and no statistical differences with respect to the *Visual* feedback for 3 (success rate, path efficiency, successful trial time) out of the 6 considered metrics (Figure 8).
<figure><img src="image_5.png" /><figcaption>Artificial proprioception</figcaption></figure>

Figure 4: (A) Distribution of the sensation threshold for the 8 physiological positions of the motors around the forearm and 8 participants in the initial and final phase of Experiment 1 (Experiment 1.a and Experiment 1.c, see section 2.4 for details on the experimental protocol). (B) Comparison of the distribution of the average initial (green stars), final (blue stars) and initial vs final values (black stars) of the sensation threshold (white and grey box plots, respectively) across the 8 participants for the 8 physiological positions. (C) Representation of the initial (green, Experiment 1.a) and final sensation threshold (blue, Experiment 1.c) around the forearm. The dots correspond to the average threshold for each motor across subjects, and the shaded area to the standard deviation. *: $p < 0.05$, **: $p < 0.01$, ***: $p < 0.001$

## 4. Discussion

In this study, a multichannel wearable vibrotactile device was proposed to encode proprioceptive information. We showed that this could be achieved using eight channels of vibrotactile feedback spatially distributed around the arm and 4 vibration

levels per channel controlled in a closed-loop manner based on measurements of surface EMG.

The custom-made system features a modular design that supports straightforward modification of the number of vibration units and complete control of the vibration characteristics.
<figure><img src="image_6.png" /><figcaption>Artificial proprioception</figcaption></figure>

Figure 5: (A) Distribution of the success of level differentiation for 4 and 5 different levels of intensity for all physiological positions and participants. (B) Comparison of the distribution of the average success rate for 4 and 5 different levels of intensity (white and grey box plots, respectively) across the 8 participants and the 8 physiological positions. (C) Representation of the success rate around the forearm with 4 levels (green) and 5 frequency levels (blue). The dots correspond to the average threshold for each motor, and the shaded area to the standard deviation. (D) and (E) Confusion matrix for 4 and 5 vibration intensity levels, respectively. *: *p* < 0.05, **: *p* < 0.01, ***: *p* < 0.001

4.1. *Experiment 1: Resolution*

Experiment 1 provided a psychometric characterization of the proposed vibration armband. Experiment 1.a and 1.c showed differences between the threshold perceived in different positions around the forearm. The sensation thresholds of the ulnar and volar/ulnar portions of the forearm were lower than those of the dorsal region. Therefore, the observed difference in the vibration sensation threshold in dif-

ferent forearm positions may be due to
anatomy, e.g. differences in the density of
the Pacinian corpuscles (mechanoreceptors
responsible for vibration) in the dermis, or
the dermis thickness where Pacinian cor-
puscles are located [25].

The relatively high threshold fre-
quency is determined by the simultane-
ous modulation of amplitude and frequency
with the voltage of the motor. The am-
plitude of the motor has a steeper slope
with the voltage than the frequency. Con-
<figure><img src="image_7.png" /></figure>

Figure 6: Performance metrics for each participant and feedback modality. A) success rate, B) average distance between the cursor and the target, C) pathway efficiency, D) trial time, E) speed, F) successful trial time. The *Visual* feedback is represented in blue, and the *Best* feedback for each participant in yellow. *: *p* < 0.05, **: *p* < 0.01, ***: *p* < 0.001. Unit for average distance and speed: *r* is the workspace radius as indicated in Figure 2.
<figure><img src="image_8.png" /></figure>

Figure 7: Example of the four best paths for A) the NVNV mode and B) the Best feedback (VibFrMod3-3) for participant 2. Unit: *r* is the workspace radius as indicated in Figure 2. The blue circle indicates the rest position, the red circle the target, and the green one the cursor at the end of the task, i.e. either when the task was successful or at the task failure time.

<figure><img src="image_9.png" /></figure>

Figure 8: Performance metrics for all participants. A) success rate, B) Average distance between the cursor and the target, C) pathway efficiency, D) trial time, E) speed, F) successful trial time. \*: *p* < 0.05, \*: \*: *p* < 0.01, \*: \*: \*: *p* < 0.001. Unit for average distance and speed: *r* is the workspace radius as indicated in Figure 2.

sequently, at low voltage and frequency, the
amplitude is considerably low.

An increase of the sensation threshold
was observed between Experiment 1.a and
Experiment 1.c, which could be associated
with sensory adaptation. For a prolonged
use of the device, the observed adaptation
could be taken into account by gradually

increasing the stimulation intensity.

In Experiment 1.b, participants were
able to discriminate 4 levels of vibration
intensity for each motor better than 5 (Fig-
ure 5, A). Classification accuracy of 86.6%
was obtained in discriminating 4 vibration
levels compared to 75.7% in the case of 5
levels (Figure 5, D and E). Likely, accu-
racy would have further improved in case of three vibration levels only. However, in Experiment 2 we decided to maintain 4 levels for the various vibration patterns to convey high-resolution information about proprioception.

## 4.2. *Experiment 2: Evaluation of proprioception encoding*

In Experiment 2, four patterns of vibration (shown in Fig. 3) with variable location and distribution of stimulation intensity were tested in comparison with two modes where the participants could rely on visual feedback (*Visual mode*) or were deprived of both visual and vibrotactile feedback (*NVNV mode*). The *Visual mode* was provided with the most comprehensive source of feedback. Therefore, all metrics were better compared to the *NVNV mode* where feedback was absent, with the only exception of speed (Figure 8). Participants were faster in the *NVNV mode*. This was likely due to the fact they started moving very fast around the workspace until they accidentally hit the target. This is also reflected in the result that the path taken was quite long and random in the *NVNV mode* (Figure 7, A).

When provided with vibrotactile feedback (*Best pattern*), performance improved with respect to the *NVNV mode*. For each subject, we selected the pattern that provided the greatest improvement compared to the absence of feedback. Each vibrotactile pattern resulted to be the best for 25% of the participants (Figure 6). We believe that the underlying mechanism which causes one pattern to be the best for one particular subject depends on the neurophysiological differences between

the users. For amputees, the effectiveness of a specific stimulation pattern with respect to others will be affected by the type of surgery and potential nerve and muscle damages during the surgery. Therefore, the optimal pattern will need to be determined on a patient-specific basis. Although the test of multiple patterns may be cumbersome in patients, the initial evaluation of the optimal stimulation pattern will need to be done only once at the time of prosthetic fitting and revised occasionally.

The *Best* pattern among the 4 patterns of vibration tested in Experiment 2 determined an improvement of 5 out of the 6 performance metrics compared to the *NVNV mode* (Figure 8). Only speed was worst, likely for the same reason mentioned in *Visual Mode*. It is relevant to note that speed is a measure of performance which should be interpreted together with other measures, such as accuracy or success rate. In our experiments, when users did not have any feedback, they started random fast motions exploring the whole workspace with the hope of hitting the target in the given time. This random motion resulted in a higher speed which was unrelated to accuracy in control. Conversely, when the user tracked the target by using sensory feedback, the increase in speed corresponded to better control, as revealed by the other performance metrics. Thus, high speed can represent opposing scenarios in terms of quality of the control. As a result, speed should not be studied as a stand-alone measure. However, alongside other measures which can decode the ability of the user in interpreting the sensory feedback resulting in high motor performance, higher speed can be considered as an indicator for improved motor ability.
In 3 out of 6 metrics, the vibrotactile feedback reached a similar performance as the *Visual* mode (Figure 8).

Since able-bodied subjects, differently from amputees, have intact proprioception, a sensory perturbation (in the form of random rotation of the control) was applied to ensure that participants actually relied on the sensory feedback provided by the vibration rather than on their own proprioception. This was evident in the path taken, as shown in the representative example in Figure 7, B. When the target appeared, the device wearer started to contract the proper muscles in order to reach the target. However, this resulted in an diverging undesired movement due to the random rotation applied by the system. Therefore, he/she had to change strategy and rely on the cue provided by the vibration to accomplish the reaching task and control successfully. In fact, Figure 7, B shows the trajectory was directed towards the target once the user started relying on the vibrotactile feedback after an initial divergence due to the rotation in control. This initial divergence resulted in a decrease of path efficiency compared to previous studies where a similar myoelectric control paradigm was applied but the control was unperturbed [25]. The performance metrics found when the vibrotactile stimulation was applied have to be considered in relation to the *NVNV* and *Visual* modes rather than in absolute terms due to the paradigm employed (proprioception perturbation) in order to evaluate the performance of the proposed closed-loop system when sensation is impaired. Taken together, the results showed that the participants were able to exploit the proprioceptive information conveyed by the proposed vibrotactile device to overcome the

perturbation imposed by the control system. We can conclude that the proposed device showed significant flexibility in controlling the stimulation parameters and reconstructing the perturbed proprioception.

With the vibrational feedback, the user was able to recover the performance which was comparable with that of visual feedback for some of the outcome measures. One of the major problems with the use of current prosthetic devices is that the user needs to visually track the motions of the limb to be able to successfully implement a meaningful motion. This results in high visual processing load during task execution, which negatively affects the usability of prostheses. This is one of the reasons for rejection of active prosthetic devices by many users. An ideal substitutional proprioceptive feedback could reduce the need for visual inspection of motor tasks. This study provides a step forward in this direction. We have shown that by the particular spatial distribution of vibrotactile feedback and through modulation of the intensity of the feedback, we were able to recover several aspects of motor control. This shows the importance and significance of vibrotactile feedback and the corresponding potential for recovering the lost proprioception. With the use of the proposed armband, the user will also have the option of combining visual and vibrotactile feedback, which may result in a multimodal approach to compensate for the lost proprioception.

In this study, the proposed device was systematically tested in able-bodied users to provide a benchmark for future implementation in amputees. However, the paradigm was motivated and designed in order to simulate the proprioception impairment occurring in individuals with limb
## REFERENCES

16

deficiency. Our future step will be to test the device in amputees to investigate whether axonal regrowth or phantom limb can affect the performance of the users in utilizing substitution sensory channels for recovering proprioception.

## 5. Conclusion

In this paper, we have systematically evaluated the performance of an surface EMG-driven closed-loop sensory feedback system based on a wearable vibrotactile haptic device that provides eight channels of vibrotactile feedback for the encoding of proprioception. The results strongly support the capability of the proposed device of substituting the perturbed proprioception, suggesting a customizable pattern of stimulation with user-specific level of vibration for each participant. The next step will be to investigate the use of the device and test patterns with amputees.

## 6. Funding

This project was funded by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (project NaturalBionicS; grant agreement No 810346). SM was financed by the Life Science Engineering Area of Advance at Chalmers University of Technology.

## References

[1] D. Farina, N. Jiang, H. Rehbaum, A. Holobar, B. Graimann, H. Dietl, and O. C. Aszmann, "The extraction of neural information from the surface emg for the control of upper-limb prostheses: Emerging avenues and challenges," *IEEE Transactions on Neural Systems and Rehabilitation Engineering*, vol. 22, no. 4, pp. 797-809, 2014.

[2] J. T. Belter, J. L. Segil, A. M. Dollar, and R. F. Weir, "Mechanical design and performance specifications of anthropomorphic prosthetic hands: A review.," *Journal of Rehabilitation Research & Development*, vol. 50, no. 5, pp. 599-618, 2013.

[3] C. Pylatiuk, S. Schulz, and L. Döderlein, "Results of an internet survey of myoelectric prosthetic hand users," *Prosthetics and orthotics international*, vol. 31, no. 4, pp. 362-370, 2007.

[4] C. H. Jang, H. S. Yang, H. E. Yang, S. Y. Lee, J. W. Kwon, B. D. Yun, J. Y. Choi, S. N. Kim, and H. W. Jeong, "A survey on activities of daily living and occupations of upper extremity amputees," *Annals of rehabilitation medicine*, vol. 35, no. 6, pp. 907-921, 2011.

[5] F. Cordella, A. L. Ciancio, R. Sacchetti, A. Davalli, A. G. Cutti, E. Guglielmelli, and L. Zollo, "Literature review on needs of upper limb prosthesis users," *Frontiers in neuroscience*, vol. 10, no. 209, 2016.

[6] S. Lewis, M. Russold, Dietl, and E. Kaniusas, "User demands for sensory feedback in upper extremity prostheses," *MeMeA 2012 - 2012 IEEE Symposium on Medical Measurements and Applications, Proceedings*, May 2012.

[7] C. Antfolk, M. D'alonzo, B. Rosén, G. Lundborg, F. Sebelius, and C. Cipriani, "Sensory feedback in upper limb prosthetics," *Expert review of medical devices*, vol. 10, no. 1, pp. 45-54, 2013.

[8] E. D'Anna, G. Valle, A. Mazzoni, I. Strauss, F. Iberite, J. Patton, F. M. Petrini, S. Raspopovic, G. Granata, R. Di Iorio, et al., "A closed-loop hand prosthesis with simultaneous intraneural tactile and position feedback," *Science Robotics*, vol. 4, no. 27, pp. 88-92, 2019.

[9] G. S. Dhillon and K. W. Horch, "Direct neural sensory feedback and control of a prosthetic arm," *IEEE Transactions on Neural Systems and Rehabilitation Engineering*, vol. 13, no. 4, pp. 468-472, Dec. 2005.
[10] S. S. Hsiao, M. Fettiplace, and B. Darbandi, "Sensory feedback for upper limb prostheses," in *Progress in brain research*, vol. 192, Elsevier, 2011, pp. 69–81.

[11] S. Dosen, M. Markovic, M. Strbac, M. Belić, V. Kojić, G. Bijelić, T. Keller, and D. Farina, "Multichannel electrotactile feedback with spatial and mixed coding for closed-loop control of grasping force in hand prostheses," *IEEE Transactions on Neural Systems and Rehabilitation Engineering*, vol. 25, no. 3, pp. 183–195, 2017.

[12] C.-Y. Ko, Y. Chang, S.-B. Kim, S. Kim, G. Kim, J. Ryu, and M. Mun, "Evaluation of physical and emotional responses to vibrotactile stimulation of the forearm in young adults, the elderly, and transradial amputees," *Physiology & behavior*, vol. 138, pp. 87–93, 2015.

[13] K. A. Kaczmarek, J. G. Webster, P. Bachy-Rita, and W. J. Tompkins, "Electrotactile and vibrotactile displays for sensory substitution systems," *IEEE Transactions on Biomedical Engineering*, vol. 38, no. 1, pp. 1–16, Jan. 1991.

[14] I. Oakley, Y. Kim, J. Lee, and J. Ryu, "Determining the feasibility of forearm mounted vibrotactile displays," in *2006 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems*, IEEE, 2006, pp. 27–34.

[15] L. A. Jones and N. B. Sarter, "Tactile displays: Guidance for their design and application," *Human factors*, vol. 50, no. 1, pp. 90–111, 2008.

[16] H. J. Witteveen, E. A. Droog, J. S. Rietman, and P. H. Veltink, "Vibro-and electrotactile user feedback on hand opening for myoelectric forearm prostheses," *IEEE transactions on biomedical engineering*, vol. 59, no. 8, pp. 2219–2226, 2012.

[17] R. Christiansen, J. L. Contreras-Vidal, R. B. Gillespie, P. A. Shewokis, and M. K. O'Malley, "Vibrotactile feedback of pose error enhances myoelectric control of a prosthetic hand," pp. 531–536, Apr. 2013.

[18] A. De Nunzio, S. Dosen, S. Lemling, M. Marković, M. Schweisfurth, N. Ge, B. Graimann, D. Falla, and D. Farina, "Tactile feedback is an effective instrument for

the training of grasping with a prosthesis at low- and medium-force levels," *Experimental brain research*, vol. 235, Aug., 2017.

[19] F. Clemente, M. D'Alonzo, M. Controzzi, B. B. Edin, and C. Cipriani, "Non-invasive, temporally discrete feedback of object contact and release improves grasp control of closed-loop myoelectric transradial prostheses," *IEEE Transactions on Neural Systems and Rehabilitation Engineering*, vol. 24, no. 12, pp. 1314–1322, Dec. 2016.

[20] A. Ninu, S. Dosen, S. Muceli, F. Rattay, H. Dietl, and D. Farina, "Closed-loop control of grasping with a myoelectric hand prosthesis: Which are the relevant feedback variables for force control?" *IEEE Transactions on Neural Systems and Rehabilitation Engineering*, vol. 22, no. 5, pp. 1041–1052, Sep. 2014.

[21] H. J. B. Witteveen, F. Luft, J. S. Rietman, and P. H. Veltink, "Stiffness feedback for myoelectric forearm prostheses using vibrotactile stimulation," *IEEE Transactions on Neural Systems and Rehabilitation Engineering*, vol. 22, no. 1, pp. 53–61, Jan. 2014.

[22] M. Štrbac, M. Belić, M. Isaković, V. Kojić, G. Bijelić, I. Popović, M. Radotić, S. Došen, M. Marković, D. Farina, et al., "Integrated and flexible multichannel interface for electrotactile stimulation," *Journal of neural engineering*, vol. 13, no. 4, p. 046014, 2016.

[23] S. Muceli, W. Poppendieck, K.-P. Hoffmann, S. Dosen, J. Benito-León, F. O. Barroso, J. L. Pons, and D. Farina, "A thin-film multichannel electrode for muscle recording and stimulation in neuroprosthetics applications," *Journal of neural engineering*, vol. 16, no. 2, p. 026035, 2019.

[24] D. E. Barker, "Skin thickness in the human," *Plastic and Reconstructive Surgery*, vol. 7, no. 2, pp. 115–116, 1951.

[25] S. Muceli, N. Jiang, and D. Farina, "Extracting signals robust to electrode number and shift for online simultaneous and proportional myoelectric control by factorization algorithms," *IEEE transactions on neural systems and rehabilitation engineering: a publication of the IEEE Engineering in Medicine and Biology Society*, vol. 22, Oct. 2013.