# Multisensory neurofeedback design for KMI embodiment

Gabriela Herrera Altamira, Nathalie Skiba, Anatole Lécuyer, Laurent Bougrain, Stéphanie Fleck

► To cite this version:

Gabriela Herrera Altamira, Nathalie Skiba, Anatole Lécuyer, Laurent Bougrain, Stéphanie Fleck. Multisensory neurofeedback design for KMI embodiment. IMX 2023 - ACM International Conference on Interactive Media Experiences, Jun 2023, Nantes, France. pp.1-3, 10.1145/3604321.3604352. hal-04228939

HAL Id: hal-04228939

https://hal.science/hal-04228939v1

Submitted on 4 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Multisensory neurofeedback design for KMI embodiment

Gabriela Herrera Altamira*

gabriela.herrera-altamira@loria.fr

Université de Lorraine, CNRS, LORIA,

Nancy

France

Nathalie Skiba

Open Edge-Alchimies Group

France

nathalie@openedge.cc

Anatole Lécuyer

Hybrid Team, Inria, University of

Rennes, Irisa, UMR CNRS 6074,

Rennes

France

Laurent Bougrain

Université de Lorraine, CNRS, LORIA,

Nancy

France

Stéphanie Fleck*

Université de Lorraine, PErSEUs, Metz

France

## ABSTRACT

This study describes some key steps in the design-based research process related to developing a multimodal brain-computer interface aiming to support kinesthetic motor imagery learning and rehabilitation after stroke. We highlight some of the challenges of sensory interface design to afford and embody imagined hand movement successfully. The result is a multi-sensory neurofeedback solution offering three modalities (visual, kinesthetic, and vibrotactile) to meet future users' needs and realities best.

## CCS CONCEPTS

* Human-centered computing → User centered design; Haptic devices.

## KEYWORDS

Post-stroke rehabilitation, Multisensory feedback, Brain-Computer Interaction

### ACM Reference Format:

Gabriela Herrera Altamira, Nathalie Skiba, Anatole Lécuyer, Laurent Bougrain, and Stéphanie Fleck. 2023. Multisensory neurofeedback design for KMI embodiment. In *ACM International Conference on Interactive Media Experiences Workshops (IMXw '23), June 12–15, 2023, Nantes, France*. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3604321.3604352

## 1 INTRODUCTION

In the context of motor rehabilitation after a stroke, kinesthetic motor imagery (KMI) can be used to stimulate patients' brain plasticity. KMI consists of imagining a movement without performing it by reactivating the haptic sensations experienced during a real movement (e.g., tactile, proprioceptive, and kinesthetic). Consequently, KMIs are difficult to perform and learn. Indeed, unlike real movement, people do not have any sensory feedback during KMI.

As a result, they cannot know if they have actually performed KMI, know their performance, or find the most efficient way to perform the task. Some brain-computer interfaces (BCI), which allow a user to control a computer using their brain's electrical signals, use KMI as an interaction modality. Therefore, and in line with [1, 3], we thought coupling a BCI with sensory interactions could overcome the KMI learning barriers and thus provide an alternative to traditional rehabilitation therapies. This paper describes the interrelated phases that led to the design of a BCI solution that provides the user with three sensory neurofeedback (NFB) modalities: visual, kinesthetic, and vibrotactile, in order to support KMI learning and ultimately support brain stimulation. Furthermore, it underlines the importance of the human-centered approach throughout the design process.

## 2 DESIGNING MULTIMODAL SENSORY-NEUROFEEDBACK

To develop solutions to real-world problems, we implemented an interdisciplinary Design-Based Research (DBR) approach, a methodology used in educational research and HCI that emphasizes iterative cycles of design, implementation, and evaluation of the interventions. In addition, all along the design, a participatory approach was implemented to define the problems and orient the design choices as closely as possible to human realities.

### 2.1 Problem-setting: the central rule of all users

First, we chose to involve potential users, therapists, and post-stroke patients from the exploratory phases. Most previous studies rarely involve patients in the early design stages. Instead, practitioners are often solicited as intermediaries of the patient's needs. As a result, the reality of patients is not fully accessible, and the needs of professionals are rarely taken into account. In order to know and understand the realities of both sides and to adjust the design choices, we specifically developed and implemented a participatory approach involving ten caregivers from different specialties (physicians, occupational and physical therapists), 12 patients, as well as researchers and engineers. By combining gamified workshops, card-based focus groups, and pre-tests with explanatory supports, we sought to facilitate cross-views and authentic expression of the various participants. These participatory methods allowed the patients to express themselves as individuals, beyond their pathology, and the caregivers to share their feelings and expectations. Priceless information such as the stress generated by sounds for

<footnote>*Both authors contributed equally to this research.</footnote>

<footnote>Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.</footnote>

<footnote>IMXw '23, June 12–15, 2023, Nantes, France</footnote>

<footnote>© 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.</footnote>

<footnote>ACM ISBN 979-8-4007-0845-9/23/06...$15.00</footnote>

<footnote>https://doi.org/10.1145/3604321.3604352</footnote>
<table><caption>Table 1: Summary results of the exploration phase</caption><thead><tr><th>Dimension</th><th>Needs</th><th>Design choices</th></tr></thead><tbody><tr><td>Therapeutic</td><td>Remain close to classical rehabilitation practices<br>Limit stress and spasticity (abnormal increase in muscle tone)<br>Stimulate cerebral plasticity</td><td>KMI targeted: Grasp, Pinch, Open hand<br>no sound and soft haptic stimuli<br>facilitate KMI-based brain-computer interactions</td></tr><tr><td>Pedagogical</td><td>Limit the mental load<br>Support motivation<br>Support engagement<br>Develop KMI skills</td><td>Uncluttered visual interfaces, without verbal instruction & task affordance<br>Gamification training program & rounded in daily life activities<br>Making the patient an actor & situational affordance<br>Scaffold the information embodiment through multisensorial NFB</td></tr><tr><td>Ergonomic</td><td>Increase caregiver efficiency & reduce patient fatigue<br>Maintain the caregiver-patient relationship</td><td>Easy-to-use and to-implement solutions<br>Training program customizable by the caregiver and the patient</td></tr></tbody></table>

some patients, the vigilance concerning, e.g., the patients' heterogeneity (e.g., age), cognitive overload risk, hemineglect syndrome, or spasticity, but also the patients' desire to make choices in their rehabilitation and to feel capable, and the professionals' desire to take better care of the patients individually, oriented our design choices (see summary in the Table 1) and complete state of the art (see [2].)

## 2.2 Design choices: multimodality and multisensoriality

The ongoing second phase of DBR corresponds to the interfaces design cycles that have resulted in a multimodal KMI-based BCI (see Figure 1) within which three sensory NFB modalities: visual, kinesthetic, and vibrotactile, are offered in a modular way.

- **Visual stimulations** Thanks to the Grasp-IT VR game, the visual stimulation is involved in both input and output interactions. Its design mobilizes three theoretical principles. The first one is based on the low of parsimony (or Occam's razor) to limit the cognitive load. The second one is based on the influence of visual affordances. Thus, the visual scene invites the user to imagine, for

example, that he/she is squeezing a ketchup bottle. The hypothesis is that this simple task, evoked by a familiar object, is easier to construct mentally than without it. The third is gamification, which is known to increase emotional reinforcement and motivation. Three forms of performance feedback are visually given: score, gauge, and hand movements and their consequences on the object (e.g., the ketchup flows proportionally).

- **Kinesthetic stimulation** This second stimulation aims to provide a kinesthetic consequence to an imagined movement. To do this, Sense-IT (see figure 1) simulates the mechanical deformation of an object held by the person to provide a more rewarding and realistic experience. The device is akin to an actuated deformable interface designed to make tangible (i.e., perceptible by senses and carrying meaning) the effect of a specific hand gesture on objects with different mechanical properties and/or shapes. The artifact deformations are created by small balloons placed inside an empty joystick. Pumps are used to inflate and deflate the balloons, inducing the movement of the user's passive fingers.

<figure><img src="image_2.png" /><figcaption>Figure 1: Grasp-IT Xmod BCI consists of 4 elements: An EEG system to record the electrical brain activity; the Grasp-IT VR game: a gamified virtual environment that affords the tasks to execute and gives visual feedback about the KMI performance; Sense-IT: an actuated, tangible interface designed to render a kinesthetic sense of a hand gesture on virtual objects (e.g., squeezing a ketchup bottle); Feel-IT: A vibrotactile device consisting of 3 vibration motors (VM) that stimulate the skin on top of some of the key muscles involved during hand movement.</figcaption></figure>
<figure><img src="image_3.png" /><figcaption>Figure 2: Box plots of the minimum sensory thresholds (a) for each age group and vibration motor, (b) for each vibration motor grouped by age group. (\*p<0.05, \*\*p<0.01, \*\*\*p<0.001), RPM stands for revolutions per minute.</figcaption></figure>

**- Vibrotactile stimulation** The third modality aims to provide the patient with a more embodied KMI experience. Thus, we designed a vibrotactile interface called Feel-IT. It consists of three 10mm ERM vibration motors held in dedicated 3D-printed cases and fixed with adjustable bracelets. The vibration motors do not stimulate the muscles but the Pacinian corpuscles on the skin. Two of them are placed on the main forearm muscles involved in a grasping hand movement so that the stimulation coincides with the natural muscular activation. These muscles are the flexor digitorum superficialis and the extensor digitorum. The third motor was placed on the hand, aligned with the third metacarpal.

## 2.3 The challenge of the vibrotactile feedback design

Each solution is designed to be used separately and coupled to increase the possibilities of embodiment and individualization. However, due to the lack of previous studies on vibrotactile NFB, many challenges had to be overcome, which required a new participatory design step.

**- Study 1: Definition of the vibrotactile sensory threshold.**
To ensure a reliable and positive experience, we determined each motor's minimum sensory threshold (MST) and the uncomfortable threshold (UT). We recruited 40 subjects with no neurological disorders and with hand and arm sensibility, 18 women and 22 men within three age groups (under 40 years of age, n=16; over 60, n=12; and intermediate, n=12). Participants were asked to indicate when they felt the vibration by pressing a button under two conditions (1) increasing intensity, (2) decreasing intensity, counterbalanced (one vibration every 2 to 4 seconds, in 2 to 5% increments). The order of the three motors was randomized. For the UT, participants indicated each time the vibration was uncomfortable or annoying. A significant difference (repeated measures ANOVA) of minimum sensory feeling at the forearm level was noted between the participants younger than 40 and the other age groups. This difference is not observed at the hand level. Furthermore, the older the individuals the more heterogeneity in terms of sensitivity level increase (Fig.2.a). In addition, the forearm and hand sensitivity significantly differed regardless of age (Fig.2.b). The distribution of mechanoreceptors on the hand may explain these results. We did not observe

a significant difference for UT, with most participants reaching the maximum vibration intensity without complaint.

**- Study 2: Vibration patterns coherence.** The perceptual coherence when coupling vibrotactile and visual stimuli was also assessed. Through a user test (N=18), we questioned (1) which activation sequence to favor, whether the motors vibrate synchronously (Simultaneous Condition) or in a sequence inspired by the natural activation of the muscles during grasping (Sequential Condition), and with a 2 or 3-motor configuration; (2) the consistency between tactile and visual feedback (fidelity, reliability, and synchronicity); (3) to what extent the stimulation was perceived as comfortable. Participants reported that regardless of the number of motors and activation patterns, the vibration was comfortable and synchronized with the animation on the screen. Furthermore, its intensity was judged proportional to the visual feedback displayed. 11 participants considered the 3-motor configuration more consistent and better synchronized with the visual animation. Thus, we kept this configuration, as well as the sequential pattern because it was preferred in 3 of the 4 items assessed.

## 3 FUTURE WORK

Currently, we are performing tests with the functional BCI to evaluate the unimodal (visual or vibrotactile) and bimodal NFB modalities. At the moment of publication of this paper, 38 participants (20 men, mean age: 32.5, s.d.: 10.78) have participated in our study. Preliminary results show that 71% of the participants prefer bimodal feedback. Further analysis of UX questionnaires and electroencephalographic signals will allow us to explain this choice better. Moreover, a third cycle of participatory assessment is currently running for the kinesthetic NFB, posing specific design and use problems. Finally, we will assess the relevance of combining the third NFB.

## REFERENCES

[1] Patricia Cornelio, Carlos Velasco, and Marianna Obrist. 2021. Multisensory Integration as per Technological Advances: A Review. *Frontiers in Neuroscience* 15 (2021). https://doi.org/10.3389/fnins.2021.652611

[2] Salomé Le Franc, Gabriela Herrera Altamira, Maud Guillen, Simon Butet, Stéphanie Fleck, Anatole Lécuyer, Laurent Bougrain, and Isabelle Bonan. 2022. Towards an adapted Neurofeedback for post-stroke motor rehabilitation: state of the art and perspectives. *Frontiers in Human Neuroscience* (2022), 448.

[3] Richard E Mayer. 2005. Cognitive theory of multimedia learning. *The Cambridge handbook of multimedia learning* 41 (2005), 31–48.