# Introducing the Use of Thermal Neurofeedback

François Le Jeune, Emile Savalle, Anatole Lécuyer, Marc J.-M. Macé, Pierre Maurel, Léa Pillette

► To cite this version:

François Le Jeune, Emile Savalle, Anatole Lécuyer, Marc J.-M. Macé, Pierre Maurel, et al. Introducing the Use of Thermal Neurofeedback. BCI 2024 - 9th Graz Brain-Computer Interface Conference, Sep 2024, Graz, Austria. pp.1-6. hal-04607599

HAL Id: hal-04607599

https://hal.science/hal-04607599

Submitted on 10 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Introducing the Use of Thermal Neurofeedback

François Le Jeune, Emile Savalle, Anatole Lécuyer, Marc J.-M. Macé, Pierre Maurel, Léa Pillette

<figure><img src="image_1.png" /></figure>

## To cite this version:

François Le Jeune, Emile Savalle, Anatole Lécuyer, Marc J.-M. Macé, Pierre Maurel, et al.. Introducing the Use of Thermal Neurofeedback. BCI 2024 - 9th Graz Brain-Computer Interface Conference, Sep 2024, Graz, Austria. pp.1-6. hal-04607599v3

HAL Id: hal-04607599

https://hal.science/hal-04607599v3

Submitted on 30 Sep 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# INTRODUCING THE USE OF THERMAL NEUROFEEDBACK

F. Le Jeune<fnref n="1" />, *, E. Savalle<fnref n="1" />, *, A. Lécuyer<fnref n="2" />, M. J.-M. Macé<fnref n="1" />, P. Maurel<fnref n="3" />, L. Pillette<fnref n="1" />

E-mail: francois.le-jeune@inria.fr

**ABSTRACT:** Motor imagery-based brain-computer interfaces (MI-BCIs) enable users to control digital devices by performing motor imagery tasks while their brain activity is recorded, typically using electroencephalography. Performing MI is challenging, especially for novices. To tackle this challenge, neurofeedback (NFB) training is frequently used and usually relies on visual feedback to help users learn to modulate the activity of their sensorimotor cortex when performing MI tasks. Improving the feedback provided during these training is essential. This study investigates the feasibility and effectiveness of using thermal feedback for MI-based NFB compared to visual feedback. Thirteen people participated to a NFB training session with visual-only, thermal-only, and combined visuo-thermal feedback. Both visual-only and combined visuo-thermal feedback elicited significantly greater desynchronization over the sensorimotor cortex compared to thermal-only feedback. No significant difference between visual-only and combined visuo-thermal feedback was found, thermal feedback thus not impairing visual feedback. This study outlines the need for further exploration of alternative feedback modalities in BCI research.

## INTRODUCTION

A brain-computer interface (BCI) relies on a neurophysiological acquisition method, often electroencephalography (EEG), to record brain activity that is in turn processed and interpreted as a command to control a digital device. BCIs have been used to control external devices for both non-clinical applications, such as video games [1], and clinical applications, such as motor rehabilitation after stroke [2].

One of the main challenges for BCIs, is for their users to generate a brain activity that is reliably recognizable by the computer. One commonly used mental task that produces consistent brain activity is motor imagery (MI), which has been proven to consistently activate the sensorimotor cortex [3]. However, MI is not an easy task for novice BCI users. They have to learn to modulate their brain activity by exploring different MI strategies, such as imagining different gestures or focusing on different sensations, to reliably activate their sensorimotor cortex.

Consequently, BCIs rely on training users to control their brain activity. To this end, neurofeedback (NFB) is mostly used: it consists in a closed-loop technique providing users with feedback on their own brain activity so they can learn to modulate it. Thus, providing users with feedback that they can understand and interpret intuitively is of utmost importance in the learning process, and improving the feedback is a key factor of improvement for BCI efficiency. A feedback can be defined through three main characteristics: (i) its content, i.e., the information that it conveys (ex: neuromarker on which the feedback is based), (ii) its modality, i.e., the way this information is conveyed (ex: haptic feedback using vibrators), and (iii) its presentation timing, i.e., the moment when it is provided (ex: continuous presentation with a refreshing rate at 0.1Hz) [4]. Among those, the modality of feedback is the most investigated characteristic.

Several modalities of feedback are reported in the literature. A majority of the studies displayed visual feedback, probably because vision is the sense on which daily life perception relies the most [4]. Haptic feedback including vibrotactile, functional electrical stimulation and robotic orthosis was also provided during MI-BCI user training (see [5] for a review on haptic NFB). Such feedback could particularly be interesting for MI-BCI as it activates similar cortical structures as the ones involved in MI. Conversely, the use of haptic feedback could also contribute to overtax the cortical structures associated with both its processing and the performance of the MI tasks. Previous MI-BCI experiments involving vibrotactile feedback did not find any significant negative or positive influence on the resulting electrophysiological activity or BCI performances compared to visual feedback [6–8], despite the fact that participants reported perceiving the haptic feedback as more natural than the visual one [6]. Multimodal feedback, involving both visual and haptic vibrotactile stimuli at once, seemed however to improve MI-BCI performances [9, 10].

To our knowledge, the thermal component of haptic stimulation has never been investigated as a potential NFB modality. Yet, thermal feedback appears promising as thermal stimulation is inexpensive to develop and

<footnote n="1">¹Univ. Rennes, Inria, CNRS, IRISA, Rennes, France</footnote>

<footnote n="2">²Inria, Univ. Rennes, IRISA, CNRS, Rennes, France</footnote>

<footnote n="3">³Univ. Rennes, Inria, CNRS, Inserm, Empenn ERL U1228, Rennes, France</footnote>

<footnote>*Co-first authorship. Both authors contributed equally.</footnote>
# INTRODUCING THE USE OF THERMAL NEUROFEEDBACK

F. Le Jeune<fnref n="1" />, *, E. Savalle<fnref n="1" />, *, A. Lécuyer<fnref n="2" />, M. J.-M. Macé<fnref n="1" />, P. Maurel<fnref n="3" />, L. Pillette<fnref n="1" />

E-mail: francois.le-jeune@inria.fr

**ABSTRACT:** Motor imagery-based brain-computer interfaces (MI-BCIs) enable users to control digital devices by performing motor imagery tasks while their brain activity is recorded, typically using electroencephalography. Performing MI is challenging, especially for novices. To tackle this challenge, neurofeedback (NFB) training is frequently used and usually relies on visual feedback to help users learn to modulate the activity of their sensorimotor cortex when performing MI tasks. Improving the feedback provided during these training is essential. This study investigates the feasibility and effectiveness of using thermal feedback for MI-based NFB compared to visual feedback. Thirteen people participated to a NFB training session with visual-only, thermal-only, and combined visuo-thermal feedback. Both visual-only and combined visuo-thermal feedback elicited significantly greater desynchronization over the sensorimotor cortex compared to thermal-only feedback. No significant difference between visual-only and combined visuo-thermal feedback was found, thermal feedback thus not impairing visual feedback. This study outlines the need for further exploration of alternative feedback modalities in BCI research.

## INTRODUCTION

A brain-computer interface (BCI) relies on a neurophysiological acquisition method, often electroencephalography (EEG), to record brain activity that is in turn processed and interpreted as a command to control a digital device. BCIs have been used to control external devices for both non-clinical applications, such as video games [1], and clinical applications, such as motor rehabilitation after stroke [2].

One of the main challenges for BCIs, is for their users to generate a brain activity that is reliably recognizable by the computer. One commonly used mental task that produces consistent brain activity is motor imagery (MI), which has been proven to consistently activate the sensorimotor cortex [3]. However, MI is not an easy task for novice BCI users. They have to learn to modulate their brain activity by exploring different MI strategies, such as imagining different gestures or focusing on different sensations, to reliably activate their sensorimotor cortex. Consequently, BCIs rely on training users to control their brain activity. To this end, neurofeedback (NFB) is mostly used: it consists in a closed-loop technique providing users with feedback on their own brain activity so they can learn to modulate it. Thus, providing users with feedback that they can understand and interpret intuitively is of utmost importance in the learning process, and improving the feedback is a key factor of improvement for BCI efficiency. A feedback can be defined through three main characteristics: (i) its content, i.e., the information that it conveys (ex: neuromarker on which the feedback is based), (ii) its modality, i.e., the way this information is conveyed (ex: haptic feedback using vibrators), and (iii) its presentation timing, i.e., the moment when it is provided (ex: continuous presentation with a refreshing rate at 0.1Hz) [4]. Among those, the modality of feedback is the most investigated characteristic.

Several modalities of feedback are reported in the literature. A majority of the studies displayed visual feedback, probably because vision is the sense on which daily life perception relies the most [4]. Haptic feedback including vibrotactile, functional electrical stimulation and robotic orthosis was also provided during MI-BCI user training (see [5] for a review on haptic NFB). Such feedback could particularly be interesting for MI-BCI as it activates similar cortical structures as the ones involved in MI. Conversely, the use of haptic feedback could also contribute to overtax the cortical structures associated with both its processing and the performance of the MI tasks. Previous MI-BCI experiments involving vibrotactile feedback did not find any significant negative or positive influence on the resulting electrophysiological activity or BCI performances compared to visual feedback [6–8], despite the fact that participants reported perceiving the haptic feedback as more natural than the visual one [6]. Multimodal feedback, involving both visual and haptic vibrotactile stimuli at once, seemed however to improve MI-BCI performances [9, 10].

To our knowledge, the thermal component of haptic stimulation has never been investigated as a potential NFB modality. Yet, thermal feedback appears promising as thermal stimulation is inexpensive to develop and

<footnote n="1">¹Univ. Rennes, Inria, CNRS, IRISA, Rennes, France</footnote>

<footnote n="2">²Inria, Univ. Rennes, IRISA, CNRS, Rennes, France</footnote>

<footnote n="3">³Univ. Rennes, Inria, CNRS, Inserm, Empenn ERL U1228, Rennes, France</footnote>

<footnote>*Co-first authorship. Both authors contributed equally.</footnote>
fairly easy to use. Besides, thermal stimulation could
be perceived as natural feedback since external thermal
sensation is continuously involved when exploring our
environment with our body. Furthermore, such feed-
back could particularly benefit therapeutic applications as
studies showed that thermal stimulation facilitates sens-
sory and motor recovery in stroke patients [11, 12].
Hence, it appears of interest to study the feasibility of
including the so far disregarded thermal feedback in MI-
NFB training. The goal of the present study was to in-
vestigate the effects of thermal neurofeedback on users'
ability to control their brain activity and on their user ex-
perience, in comparison with a visual feedback. To this
aim, we investigated NFB training with thermal feedback
only, visual feedback only, and with a combination of
both visual and thermal feedback.

MATERIALS AND METHODS

We investigated the influence of three different feedback modalities on participants' ability to modulate their own brain activity and on their user experience. The three modalities were the following ones: **visual-only feedback (V)** in which participants experienced solely visual feedback, **thermal-only feedback (T)** in which participants experienced solely thermal feedback, and a bi-modal **visuo-thermal feedback (VT)** in which participants experienced both thermal and visual feedback simultaneously. All participants experienced the different feedback and their order of presentation was pseud-randomized (so that the order of conditions was counter-balanced across participants).

*Participants:* Twenty-four healthy participants completed the study (8 women, 15 men, 1 non-binary, age $25.8 \pm 3.8$ years). None of them had any history of neurological or psychiatric disorders. All participants provided written informed consent before the experiment in accordance with the Declaration of Helsinki and following amendments. The thirteen initial participants were tested with a thermal feedback slightly different from the others (see Section *Thermal Feedback*). Authors of the study observed that thermal feedback obtained from equation 4 resulted in a wider stimulation range than desired. The equation was updated (5) and eleven additional participants were tested. Results for the second group of participants were similar and conclusions thus identical to the ones of the first group. No significant difference was found between groups. Both datasets were thus merged and analysed as a single one.

*Experimental protocol:* The experiment lasted about
two hours. Participants were seated in a comfortable
armchair, in front of a monitor placed flat on a table
right above their arms (Fig. 1.A). First, participants were
asked to fill in two questionnaires regarding general de-
mographic information and their handedness. Partici-
pants were then equipped with a wearable thermal stimu-
lation system on their right hand. It consisted of a Peltier
cell attached to a heat sink and assembled on a 3D-printed

wearable element (Fig. 1.B). The wearable system was
strapped to the right hand of participants so that the
Peltier cell would be in complete contact with the palm
of their hand. Because thermal stimulation varies from
user to user, we performed a calibration of the thermal
stimulation range to adapt it to each participant (see Sec-
tion *Thermal Feedback*). Subsequently, participants were
equipped with an EEG headset (see Section *EEG Record-
ings & Signal Processing*) and an electrode was placed
on the participants’ skin above the anterior proximal part
of their forearm to assess hand electromyographic activ-
ity (EMG). Afterwards, the participants were given in-
structions regarding the experimental protocol, including
the modalities of feedback and the motor imagery task to
perform. They were asked to repetitively imagine clos-
ing and opening their right hand while focusing on the
sensations related to the movement, such as hand mus-
cle contraction, skin and tendon stretch, and tactile and
thermal sensations on the hand. Then, participants were
asked during one run to perform the MI task while look-
ing at a fixation cross (no feedback was provided at that
point of the experiment). Each run consisted of twenty
successive trials of five seconds of rest followed by ten
seconds of MI, for a total run duration of five minutes.
We calibrated the BCI based on the data from this run by
defining a reference ERD (ERD<sub>ref</sub>) set as the 30th per-
centile of the produced ERDs. Then, all the participants
successively experienced the three feedback modalities
(pseudo-randomized order across participants). For each
modality, we proceeded as follows. First, we asked par-
ticipants to rest while staring at the center of a white cross
displayed on the screen for one minute while we recorded
their brain activity as a baseline for future analyses. Sec-
ond, we asked participants to perform two training runs
(separated by a short break) with the feedback modality
associated with one of the three modalities. Afterwards,
participants were asked to fill in a questionnaire regarding
their user experience of the two NFB runs they just per-

<figure><img src="image_2.png" /><figcaption>Figure 1: Experimental setup. **A-Participant** seated on a chair in front of the computer screen and wearing the EEG headset. The virtual hand is superimposed above the participant's real hand. **B-Thermal stimulation system.** Top: top view of the system; Middle: top view of the wearable system on the user's hand; Bottom: left view of the wearable system on the user's hand.</figcaption></figure>
fairly easy to use. Besides, thermal stimulation could
be perceived as natural feedback since external thermal
sensation is continuously involved when exploring our
environment with our body. Furthermore, such feed-
back could particularly benefit therapeutic applications as
studies showed that thermal stimulation facilitates sens-
sory and motor recovery in stroke patients [11, 12].
Hence, it appears of interest to study the feasibility of
including the so far disregarded thermal feedback in MI-
NFB training. The goal of the present study was to in-
vestigate the effects of thermal neurofeedback on users'
ability to control their brain activity and on their user ex-
perience, in comparison with a visual feedback. To this
aim, we investigated NFB training with thermal feedback
only, visual feedback only, and with a combination of
both visual and thermal feedback.

MATERIALS AND METHODS

We investigated the influence of three different feedback modalities on participants' ability to modulate their own brain activity and on their user experience. The three modalities were the following ones: **visual-only feedback (V)** in which participants experienced solely visual feedback, **thermal-only feedback (T)** in which participants experienced solely thermal feedback, and a bi-modal **visuo-thermal feedback (VT)** in which participants experienced both thermal and visual feedback simultaneously. All participants experienced the different feedback and their order of presentation was pseud-randomized (so that the order of conditions was counter-balanced across participants).

*Participants:* Twenty-four healthy participants completed the study (8 women, 15 men, 1 non-binary, age $25.8 \pm 3.8$ years). None of them had any history of neurological or psychiatric disorders. All participants provided written informed consent before the experiment in accordance with the Declaration of Helsinki and following amendments. The thirteen initial participants were tested with a thermal feedback slightly different from the others (see Section *Thermal Feedback*). Authors of the study observed that thermal feedback obtained from equation 4 resulted in a wider stimulation range than desired. The equation was updated (5) and eleven additional participants were tested. Results for the second group of participants were similar and conclusions thus identical to the ones of the first group. No significant difference was found between groups. Both datasets were thus merged and analysed as a single one.

*Experimental protocol:* The experiment lasted about two hours. Participants were seated in a comfortable armchair, in front of a monitor placed flat on a table right above their arms (Fig. 1.A). First, participants were asked to fill in two questionnaires regarding general demographic information and their handedness. Participants were then equipped with a wearable thermal stimulation system on their right hand. It consisted of a Peltier cell attached to a heat sink and assembled on a 3D-printed

wearable element (Fig. 1.B). The wearable system was
strapped to the right hand of participants so that the
Peltier cell would be in complete contact with the palm
of their hand. Because thermal stimulation varies from
user to user, we performed a calibration of the thermal
stimulation range to adapt it to each participant (see Sec-
tion *Thermal Feedback*). Subsequently, participants were
equipped with an EEG headset (see Section *EEG Record-
ings & Signal Processing*) and an electrode was placed
on the participants’ skin above the anterior proximal part
of their forearm to assess hand electromyographic activ-
ity (EMG). Afterwards, the participants were given in-
structions regarding the experimental protocol, including
the modalities of feedback and the motor imagery task to
perform. They were asked to repetitively imagine clos-
ing and opening their right hand while focusing on the
sensations related to the movement, such as hand mus-
cle contraction, skin and tendon stretch, and tactile and
thermal sensations on the hand. Then, participants were
asked during one run to perform the MI task while look-
ing at a fixation cross (no feedback was provided at that
point of the experiment). Each run consisted of twenty
successive trials of five seconds of rest followed by ten
seconds of MI, for a total run duration of five minutes.
We calibrated the BCI based on the data from this run by
defining a reference ERD (ERD<sub>ref</sub>) set as the 30th per-
centile of the produced ERDs. Then, all the participants
successively experienced the three feedback modalities
(pseudo-randomized order across participants). For each
modality, we proceeded as follows. First, we asked par-
ticipants to rest while staring at the center of a white cross
displayed on the screen for one minute while we recorded
their brain activity as a baseline for future analyses. Sec-
ond, we asked participants to perform two training runs
(separated by a short break) with the feedback modality
associated with one of the three modalities. Afterwards,
participants were asked to fill in a questionnaire regarding
their user experience of the two NFB runs they just per-

<figure><img src="image_3.png" /><figcaption>Figure 1: Experimental setup. **A-Participant** seated on a chair in front of the computer screen and wearing the EEG headset. The virtual hand is superimposed above the participant's real hand. **B-Thermal stimulation system.** Top: top view of the system; Middle and Bottom: top and left view (respectively) of the wearable system on the user's hand.</figcaption></figure>
<figure><img src="image_3.png" /><figcaption>Figure 2: Experimental protocol. Edinburgh and general information questionnaires, setup, and calibration were performed once, at the beginning of the experiment. The post-session questionnaire was filled once at the end of the experiment. The three tested condition (V, T, VT) were performed successively in a pseudo-random order, in a single session.</figcaption></figure>

formed. Finally, during one run (which we called *Replay*, Fig. 2) we replayed the exact stimuli generated during the second user training run while the participants were asked to only pay attention to these stimuli without imagining the movement. A short break was also proposed inbetween modalities. At the end of the three modalities, participants were asked to fill in a final questionnaire regarding their preferences in terms of feedback modality. All the equipment was then removed and we made a short debrief (Fig. 2).

**EEG Recordings & Signal Processing:** The EEG data was recorded with 31 active electrodes, using a g.USBAmp EEG amplifier (g.tec, Austria). The electrodes were placed over the sensorimotor cortex (at locations Fp1, Fp2, FC5, FC6, F3, F4, FCz, T7, T8, C3, C4, Cz, P3, P4, O1, O2, CP1, CP2, CP5, CP6, FC1, FC2, CP4, C5, C6, FC3, FC4, Pz, C1, C2 and CP3 in the 10-20 system). They were referenced to the left earlobe and grounded to AFz. The data was sampled at 512 Hz, and processed online using OpenViBE 3.4.0 [13].

During the user training runs, the online data used to provide feedback was processed as follows. We first selected the signal from electrodes C3, FC1, FC5, CP1 and CP5, which was then filtered between 8 Hz and 20 Hz. The signals were then passed through a Laplacian filter centered on C3 (with electrodes CP1, CP5, FC1 and FC5). During the resting phases of the runs, the output signal was then epoched using a one second window every 0.1 second. The power over the 8-20Hz frequency band was computed, time-averaged, and the data from the epochs of the last two seconds of the resting time (20 epochs) was averaged. This average was used as the *Rest* value to compute the ERDs of the following MI phase. During the MI phases of the runs, the output signal of the Laplacian filter was epoched using a 0.25 s window every 0.25 s, epochs whose signal's power was computed, time-averaged, and for every epoch, the data from the current epoch and last three epochs were averaged. For every epoch, this average was used as the *Task* value to compute the online ERDs (ERD<sub>on</sub>) as follows:

$$ERD_{on} = (Task - Rest)/Rest * 100 \quad (1)$$

We then used the ERD<sub>ref</sub> defined from the calibration run to compute an ERD score (SERD):

$$S_{ERD} = ERD_{on}/ERD_{ref} * 100 \quad (2)$$

The SERD was then used to define the feedback score (S<sub>FB</sub>) according to the following thresholds:

$$ \begin{aligned} S_{ERD} < 30\% & \Rightarrow S_{FB} = 0 \\ S_{ERD} \ge 30\% & \Rightarrow S_{FB} = 1 \\ S_{ERD} \ge 60\% & \Rightarrow S_{FB} = 2 \\ S_{ERD} \ge 100\% & \Rightarrow S_{FB} = 3 \end{aligned} \quad (3) $$

For the offline analysis, the EEG data has been pre-processed using MNE-Python [14]. The signal was filtered using a zero-phase shift notch filter with a 50 Hz cut-off frequency and a finite impulse response (FIR) band-pass filter with cut-off frequencies of 1 and 25 Hz and then average-referenced. We used an independent component analysis (ICA) to limit the impact of muscular artefacts. In average, 3 components were removed from the analysis of each participant. We extracted 14 s window epochs from 4 s before the MI instruction cue to 10 s after (one epoch per trial). Epochs with peak-to-peak amplitude greater than 200 µV were rejected. In total, an average of 1 epochs out of 20 were removed for each run. The data was then filtered using a Laplacian filter centered on C3 (with electrodes CP1, CP5, FC1 and FC5). Then, we re-sampled our data at 256 Hz, computed time-frequency representation using Morlet wavelets between 8 and 20 Hz, and normalized it with baseline correction by taking the logratio of the signal over the average power during the rest period (-4 s to -1 s before cue). This gave us the power relative to rest period. Then, we averaged this power between 1 s and 9 s post-cue and across trials to obtain ERD<sub>off</sub>.

**Visual Feedback:** The visual feedback (developed using Unity 2019.4.18f1) presented to participants during the modalities V and VT consisted of a right virtual hand superimposed over the participants' real right hand. The virtual hand performed wrist rotations to go towards or away from a virtual cup containing a steaming hot beverage placed on a table. The virtual scene would take place in a chalet with a view of a snowy environment. There was a 60° rotation range between the starting position of the hand and the mug. The S<sub>FB</sub> i.e., 0, 1, 2 or 3, corresponded to different rotation speeds of the virtual hand, i.e., -2°/s, 4°/s, 6°/s or 10°/s respectively. The virtual hand could not move further back than the starting position. Participants were informed that the better the MI task was performed, the closer to the mug the hand would move and that independently from their brain activity, the virtual hand would continuously open and close (2s period). During the resting period, a white cross was displayed on the screen. During the calibration and for the runs with thermal-only feedback modality, a white frame was additionally displayed on the border of the screen to inform the participants when they should perform MI tasks.
<figure><img src="image_4.png" /><figcaption>Figure 2: Experimental protocol. The three tested condition (V, T, VT) were performed successively in a pseudo-random order, in a single session.</figcaption></figure>

formed. Finally, during one run (which we called *Replay*, Fig. 2) we replayed the exact stimuli generated during the second user training run while the participants were asked to only pay attention to these stimuli without imaging the movement. A short break was also proposed inbetween modalities. At the end of the three modalities, participants were asked to fill in a final questionnaire regarding their preferences in terms of feedback modality. All the equipment was then removed and we made a short debrief (Fig. 2).

**EEG Recordings & Signal Processing:** The EEG data was recorded with 31 active electrodes, using a g.USBAmp EEG amplifier (g.tec, Austria). The electrodes were placed over the sensorimotor cortex (at locations Fp1, Fp2, FC5, FC6, F3, F4, FCz, T7, T8, C3, C4, Cz, P3, P4, O1, O2, CP1, CP2, CP5, CP6, FC1, FC2, CP4, C5, C6, FC3, FC4, Pz, C1, C2 and CP3 in the 10-20 system). They were referenced to the left earlobe and grounded to AFz. The data was sampled at 512 Hz, and processed online using OpenViBE 3.4.0 [13].

During the user training runs, the online data used to provide feedback was processed as follows. We first selected the signal from electrodes C3, FC1, FC5, CP1 and CP5, which was then filtered between 8 Hz and 20 Hz. The signals were then passed through a Laplacian filter centered on C3 (with electrodes CP1, CP5, FC1 and FC5). During the resting phases of the runs, the output signal was then epoched using a one second window every 0.1 second. The power over the 8-20Hz frequency band was computed, time-averaged, and the data from the epochs of the last two seconds of the resting time (20 epochs) was averaged. This average was used as the *Rest* value to compute the ERDs of the following MI phase. During the MI phases of the runs, the output signal of the Laplacian filter was epoched using a 0.25 s window every 0.25 s, epochs whose signal's power was computed, time-averaged, and for every epoch, the data from the current epoch and last three epochs were averaged. For every epoch, this average was used as the *Task* value to compute the online ERDs (*ERD*<sub>on</sub>) as follows:

$$ERD_{on} = (Task - Rest)/Rest * 100 \quad (1)$$

We then used the *ERD*<sub>ref</sub> defined from the calibration run

to compute an ERD score (*S*<sub>ERD</sub>):

$$S_{ERD} = ERD_{on}/ERD_{ref} * 100 \quad (2)$$

The *S*<sub>ERD</sub> was then used to define the feedback score (*S*<sub>FB</sub>) according to the following thresholds:

$$ 
\begin{aligned}
S_{ERD} < 30\% & \Rightarrow S_{FB} = 0 \\
S_{ERD} \ge 30\% & \Rightarrow S_{FB} = 1 \\
S_{ERD} \ge 60\% & \Rightarrow S_{FB} = 2 \\
S_{ERD} \ge 100\% & \Rightarrow S_{FB} = 3
\end{aligned} 
\quad (3) $$

For the offline analysis, the EEG data has been pre-processed using MNE-Python [14]. The signal was filtered using a zero-phase shift notch filter with a 50 Hz cut-off frequency and a finite impulse response (FIR) band-pass filter with cut-off frequencies of 1 and 25 Hz and then average-referenced. We used an independent component analysis (ICA) to limit the impact of muscular artefacts. In average, 3 components were removed from the analysis of each participant. We extracted 14 s window epochs from 4 s before the MI instruction cue to 10 s after (one epoch per trial). Epochs with peak-to-peak amplitude greater than 200 µV were rejected. In total, an average of 1 epochs out of 20 were removed for each run. The data was then filtered using a Laplacian filter centered on C3 (with electrodes CP1, CP5, FC1 and FC5). Then, we re-sampled our data at 256 Hz, computed time-frequency representation using Morlet wavelets between 8 and 20 Hz, and normalized it with baseline correction by taking the logratio of the signal over the average power during the rest period (-4 s to -1 s before cue). This gave us the power relative to rest period. Then, we averaged this power between 1 s and 9 s post-cue and across trials to obtain *ERD*<sub>off</sub>.

**Visual Feedback:** The visual feedback (developed using Unity 2019.4.18f1) presented to participants during the modalities V and VT consisted of a right virtual hand superimposed over the participants' real right hand. The virtual hand performed wrist rotations to go towards or away from a virtual cup containing a steaming hot beverage placed on a table. The virtual scene would take place in a chalet with a view of a snowy environment. There was a 60° rotation range between the starting position of the hand and the mug. The *S*<sub>FB</sub> i.e., 0, 1, 2 or 3, corresponded to different rotation speeds of the virtual hand, i.e., -2°/s, 4°/s, 6°/s or 10°/s respectively. The virtual hand could not move further back than the starting position. Participants were informed that the better the MI task was performed, the closer to the mug the hand would move and that independently from their brain activity, the virtual hand would continuously open and close (2s period). During the resting period, a white cross was displayed on the screen. During the calibration and for the runs with thermal-only feedback modality, a white frame was additionally displayed on the border of the screen to inform the participants when they should perform MI tasks.

**Thermal Feedback:** To adapt the thermal stimulation range to participants' perception, we calibrated the device
**Thermal Feedback:** To adapt the thermal stimulation range to participants' perception, we calibrated the device at the beginning of the every experiment using the participant's (i) minimum warmth threshold perception and (ii) potential uncomfortable warmth perception. The temperatures were chosen among the following predefined range [21.5°C (room temperature), 38°C]. This range was chosen based on previous experiments and should not induce painful temperatures [15]. On average (mean ± standard deviation), participants defined the thresholds as follows. Lower threshold: 25.9°C ± 1.9°C, upper threshold: 34.8°C ± 3.0°C). The thermal feedback delivered to the participant during the T and VT conditions was then bounded according to their individually defined thresholds, i.e., [minimum warmth threshold perception-1°C, potential uncomfortable warmth perception-1°C]. With the goal of having consistent visual and tactile feedback modalities, the thermal feedback was commanded by the angle between the hand and the mug according to the following functions (see Section Participants):

$$T_{com} = T_{com,max} * \theta / \theta_{max} + (T_{com,min} - 11) \quad (4)$$

$$T_{com} = (T_{com,max} - T_{com,min}) * \theta / \theta_{max} + T_{com,min} \quad (5)$$

$$T_{stim} = T_{amb} + T_{com} * 0.15 \quad (6)$$

where $T_{com}$, $T_{com,min}$ and $T_{com,max}$ are the current, minimum and maximum stimulation commands (respectively) sent to the Peltier system, $\theta$ and $\theta_{max}$ the current and maximum (i.e. 60°) angle between the hand and the mug (respectively), and $T_{stim}$ and $T_{amb}$ (21.5°C) the stimulation temperature and room temperature respectively. Participants were informed that the better the MI task was performed, the warmer the thermal stimulation would be. The Peltier cell temperature was controlled through an Arduino Uno R3 (Arduino.cc) receiving the $\theta$ angle information from the Unity application.

**Variables & Factors:** Our analyses focused on assessing the potential difference induced by the modalities of feedback on the electrophysiological changes resulting from the NFB user training. To that extent, we used three different variables. The previously described $ERD_{on}$ and $ERD_{off}$ (see Section EEG Recordings & Signal Processing), and a NFB performance variable, $P_{FB}$, based on the feedback produced during a run and defined as the sum of all the $S_{FB}$ produced during a run divided by the maximum total score (i.e. all $S_{FB} = 3$):

$$P_{FB} = \frac{\sum_{i=1}^{N} S_{FB,i}}{3 \times N} \times 100 \quad (7)$$

where N is the total number of online epochs thus of $S_{ERD}$ computed during a run and $S_{FB,i}$ the $S_{FB}$ of the $i^{th}$ epoch. The statistical analysis of these values consisted of repeated measures ANOVAs, with associated post-hoc analyses using the false discovery rate (FDR) method to correct for multiple comparisons. Our goal was to study the effects of the *modality* of feedback (i.e., V, T and VT) and of the *run* (i.e., Run1 and Run2) on our variables ($ERD_{on}$, $ERD_{off}$, $P_{FB}$).

<figure><img src="image_4.png" /><figcaption>Figure 3: Box and scatter plot of $ERD_{on}$ for runs Run1 and Run2. Asterisks: p<0.05: * ; p<0.01: ** ; p<0.001: ***.</figcaption></figure>

## RESULTS

We analysed the potential influence of our conditions on the three variables defined earlier: $P_{FB}$, $ERD_{on}$, and $ERD_{off}$. As repeated measures ANOVA results are particularly sensitive to outliers, we removed the results from the participants that were over plus or minus two standard deviations relative to the median of this variable. In total, two participants were removed from $P_{FB}$ analyses, five from $ERD_{off}$, and one participant from $ERD_{on}$ analyses. We tested our data for deviation of normality using a Shapiro-Wilk test. No significant deviation was found. Then, we computed three 2-way repeated measures ANOVAs to assess if there was an effect of or an interaction between “Modality” (V, T, VT) and “Run” (Run1, Run2) on either of the dependent variables ($P_{FB}$, $ERD_{on}$, $ERD_{off}$).

Table 1 reports the results of the three repeated measure ANOVAs. We found a significant main effect of run repetition on $ERD_{on}$, but not on $ERD_{off}$ nor on $P_{FB}$ (Tab. 1). $ERD_{on}$ were found significantly greater during Run2 ($M = -46.9\%; SD = 13.5\%$) than during Run1 ($M = -44.9\%; SD = 13.9\%$) [$t=3.67, df=21, p=0.001$](Fig. 3). We found a significant main effect of feedback modality on $ERD_{off}$, but not on $ERD_{on}$ nor on $P_{FB}$. Post-hoc analyses of the effect of “Modality” revealed significantly greater $ERD_{off}$ for V ($M = -0.14; SD = 0.08$) compared to T ($M = -0.07; SD = 0.06$) [$t=5.2483, df=18, p<0.001$], and significantly greater $ERD_{off}$ for VT ($M = -0.14; SD = 0.08$) compared to T [$t=5.31, df=18, p<0.001$]. No significant difference was found between V and VT [$t=-0.12, df=18, p=0.899$] (Fig. 4). P-values were FDR-corrected for multiple comparisons.

## DISCUSSION

The aim of the present study was to investigate the feasibility of introducing thermal stimulation as a new neurom feedback modality. To that extent, we compared neuropsychological outcomes of MI-based NFB between a
at the beginning of the every experiment using the participant's (i) minimum warmth threshold perception and (ii) potential uncomfortable warmth perception. The temperatures were chosen among the following predefined range [21.5°C (room temperature), 38°C]. This range was chosen based on previous experiments and should not induce painful temperatures [15]. On average (mean ± standard deviation), participants defined the thresholds as follows. Lower threshold: 25.9°C ± 1.9°C, upper threshold: 34.8°C ± 3.0°C). The thermal feedback delivered to the participant during the T and VT conditions was then bounded according to their individually defined thresholds, i.e., [minimum warmth threshold perception-1°C, potential uncomfortable warmth perception-1°C]. With the goal of having consistent visual and tactile feedback modalities, the thermal feedback was commanded by the angle between the hand and the mug according to the following functions (see Section Participants):

$$T_{com} = T_{com,max} * \theta/\theta_{max} + (T_{com,min} - 11) \quad (4)$$

$$T_{com} = (T_{com,max} - T_{com,min}) * \theta/\theta_{max} + T_{com,min} \quad (5)$$

$$T_{stim} = T_{amb} + T_{com} * 0.15 \quad (6)$$

where $T_{com}$, $T_{com,min}$ and $T_{com,max}$ are the current, minimum and maximum stimulation commands (respectively) sent to the Peltier system, $\theta$ and $\theta_{max}$ the current and maximum (i.e. 60°) angle between the hand and the mug (respectively), and $T_{stim}$ and $T_{amb}$ (21.5°C) the stimulation temperature and room temperature respectively. Participants were informed that the better the MI task was performed, the warmer the thermal stimulation would be. The Peltier cell temperature was controlled through an Arduino Uno R3 (Arduino.cc) receiving the $\theta$ angle information from the Unity application.

**Variables & Factors:** Our analyses focused on assessing the potential difference induced by the modalities of feedback on the electrophysiological changes resulting from the NFB user training. To that extent, we used three different variables. The previously described $ERD_{on}$ and $ERD_{off}$ (see Section EEG Recordings & Signal Processing), and a NFB performance variable, $P_{FB}$, based on the feedback produced during a run and defined as the sum of all the $S_{FB}$ produced during a run divided by the maximum total score (i.e. all $S_{FB} = 3$):

$$P_{FB} = \frac{\sum_{i=1}^{N} S_{FB,i}}{3 \times N} \times 100 \quad (7)$$

where N is the total number of online epochs thus of $S_{ERD}$ computed during a run and $S_{FB,i}$ the $S_{FB}$ of the $i^{th}$ epoch. The statistical analysis of these values consisted of repeated measures ANOVAs, with associated post-hoc analyses using the false discovery rate (FDR) method to correct for multiple comparisons. Our goal was to study the effects of the *modality* of feedback (i.e., V, T and VT) and of the *run* (i.e., Run1 and Run2) on our variables ($ERD_{on}$, $ERD_{off}$, $P_{FB}$).

<figure><img src="image_5.png" /><figcaption>Figure 3: Box and scatter plot of $ERD_{off}$ for runs Run1 and Run2. Asterisks: p<0.05: * ; p<0.01: ** ; p<0.001: ***.</figcaption></figure>

## RESULTS

We analysed the potential influence of our conditions on the three variables defined earlier: $P_{FB}$, $ERD_{on}$, and $ERD_{off}$. As repeated measures ANOVA results are particularly sensitive to outliers, we removed the results from the participants that were over plus or minus two standard deviations relative to the median of this variable. In total, two participants were removed from $P_{FB}$ analyses, five from $ERD_{off}$, and one participant from $ERD_{on}$ analyses. We tested our data for deviation of normality using a Shapiro-Wilk test. No significant deviation was found. Then, using JASP [16] we computed three 2-way repeated measures ANOVAs to assess if there was an effect of or an interaction between “Modality” (V, T, VT) and “Run” (Run1, Run2) on either of the dependent variables ($P_{FB}$, $ERD_{on}$, $ERD_{off}$).

We found a significant main effect of *run* repetition on $ERD_{on}$ and $ERD_{off}$, but not on $P_{FB}$ (Tab. 1). $ERD_{on}$ were found significantly greater during Run2 ($M=-46.9\%$; $SD=13.5\%$) than during Run1 ($M=-44.9\%$; $SD=13.9\%$) [$t=3.67, df=21, p=0.001$]. $ERD_{off}$ were also found significantly greater during Run2 ($M=-0.15\text{dB}$; $SD=0.11\text{dB}$) than during Run1 ($M=-0.12\text{dB}$; $SD=0.10\text{dB}$) [$t=3.467, df=17, p=0.003$] (Fig. 3). We found a significant main effect of feedback *modality* on $ERD_{off}$, but not on $ERD_{on}$ nor on $P_{FB}$ (Tab. 1). Post-hoc analyses of the effect of “Modality” revealed significantly greater $ERD_{off}$ for V ($M=-0.18\text{dB}$; $SD=0.12\text{dB}$) compared to T ($M=-0.09\text{dB}$; $SD=0.08\text{dB}$) [$t=-3.993, df=17, p<0.001$], and significantly greater $ERD_{off}$ for VT ($M=-0.14\text{dB}$; $SD=0.10\text{dB}$) compared to T [$t=-2.444, df=17, p=0.040$]. No significant difference was found between V and VT [$t=-1.549, df=17, p=0.131$] (Fig. 4). P-values were Holm-corrected for multiple comparisons.

## DISCUSSION

The aim of the present study was to investigate the feasibility of introducing thermal stimulation as a new neu-
<table><caption>Table 1: Result of the two-way repeated measures ANOVA for *Modality* and *Run*. Significant main effects are indicated in bold.</caption><thead><tr><th></th><th>ERD<sub>on</sub></th><th>ERD<sub>off</sub></th><th>P<sub>FB</sub></th></tr></thead><tbody><tr><td>Modality</td><td>F(2,42)=0.11;p=0.83;η<sup>2</sup>=0.001</td><td>F(2,36)=22.1;p&lt;0.001;η<sup>2</sup>=0.18</td><td>F(2,40)=0.93;p=0.39;η<sup>2</sup>=0.01</td></tr><tr><td>Run</td><td>F(2,21)=13.4;p=0.001;η<sup>2</sup>=0.01</td><td>F(1,18)=2.68;p=0.11;η<sup>2</sup>=0.01</td><td>F(1,20)=3.16;p=0.09;η<sup>2</sup>=0.01</td></tr><tr><td>Modality*Run</td><td>F(2,42)=1.61;p=0.21;η<sup>2</sup>=0.004</td><td>F(2,36)=3.29;p=0.052;η<sup>2</sup>=0.03</td><td>F(2,40)=1.05;p=0.36;η<sup>2</sup>=0.01</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Figure 4: Box and scatter plot of ERD<sub>off</sub> for modalities V, T, and VT. Asterisks: p<0.05: * ; p<0.01: ** ; p<0.001: ***.</figcaption></figure>

thermal-only feedback modality, a visual-only feedback modality, and a visuo-thermal modality (combining both aforementioned feedback modalities). Through repeated measures ANOVAs, we studied the effects of the feedback *modality* and *run* repetition.

We found that *run* repetition had a significant main effect on online NFB performances (i.e., ERD<sub>on</sub>), indicating a learning effect. We also found that *modality* had a significant main effect on offline performances (i.e., ERD<sub>off</sub>) with significantly greater desynchronization over the left sensorimotor cortex for the visual-only and visuo-thermal modalities compared to the thermal-only modality.

The main effect of modality of feedback on brain activity modulation was observed on the offline NFB performance (ERD<sub>off</sub>) but not on the online NFB performance (ERD<sub>on</sub>, P<sub>FB</sub>). The opposite was found for the effect of run repetition that was observed on ERD<sub>on</sub> but not on ERD<sub>off</sub>. Two main differences between the online and offline NFB performances most likely explain such differences. Firstly, the offline performances benefit from the ICA muscle activity artifacts correction, which is not applied online. Secondly, during online processing the positive values of ERD<sub>on</sub>, i.e., event related synchronization (ERS), were automatically set to 0% not to give negative feedback to users, whereas offline processing took into account both ERD and ERS. Thus, participants were trained online to increase their ERD but not to minimize their ERS, which would explain the presence of the run repetition effect only on online performances. Besides, feedback modality shows a significant effect when both ERD and ERS are taken into account in the processing (ERD<sub>off</sub>) but not when only ERD are (ERD<sub>on</sub>). This suggests that feedback modality has mainly an effect on

ERS, with V and VT modalities resulting in less ERS than their thermal counterpart.

Thermal feedback alone thus appears less effective than its visual counterpart for MI-NFB learning. A first hypothesis to explain this result may lie in the difference of intuitiveness between both feedback. Considering the visual feedback, participants could see the starting and goal position of the hand (i.e. the mug) and thus appreciate at all times their performance based on the position of the hand between these two positions. On the contrary, during thermal-only NFB runs, participants could use the room temperature as a starting point, but did not have any reference cue related to the goal temperature to reach. Furthermore, thermal noticeable difference, i.e., the minimum temperature difference between two stimuli that a person is able to perceive, may reach a minimum of 0.75°C at 38°C, but rapidly increases for lower temperature (1.6°C at 32°C) [16]. Thus, considering the temperature range we used and that a prior characterisation of our thermal stimulation system gave a temperature variation speed of around 1°C/s, the perceived feedback variation experienced by participants during a 10 s task was very likely lower for thermal stimulation than for visual stimulation. This is corroborated by comments from two participants: one claimed to have felt a delay between their MI and the thermal feedback, and the other participant mentioned having a hard time figuring out if they were performing the MI task correctly with the thermal-only feedback condition and even feeling lost without visual feedback. Both aforementioned arguments suggest that visual feedback alone contains a richer information than thermal feedback alone. Despite eliciting lower modulation of brain activity than its visual counterpart and its probable lack of intuitiveness, adding thermal sensations to visual feedback did not decrease the efficiency of the visual-only feedback, as we did not find any significant difference in brain activity modulation between V and VT conditions. A stronger desynchronization could have been expected for the visual and thermal condition as previous studies found that multimodal feedback composed of haptic and visual stimuli has a beneficial influence [9, 10]. This could be caused by an inconsistency between our visual and thermal feedback due to an increased delay in thermal stimulation compared to the visual one. Our results suggest that using thermal feedback coupled with a visual feedback could be an interesting solution, especially for NFB application that would benefit from thermal stimulation such as sensory and motor post-stroke rehabilitation as previously mentioned [11, 12]. Further analyses of the user experience will provide us with complementary information.
<table><caption>Table 1: Result of the two-way repeated measures ANOVA for *Modality* and *Run*. Significant main effects are indicated in bold.</caption><thead><tr><th></th><th>ERD<sub>on</sub></th><th>ERD<sub>off</sub></th><th>P<sub>FB</sub></th></tr></thead><tbody><tr><td>Modality</td><td>F(2,42)=0.12; p=0.83; η<sup>2</sup>=0.004</td><td>F(2,34)=8.1; p=0.001; η<sup>2</sup>=0.23</td><td>F(2,40)=0.93; p=0.4; η<sup>2</sup>=0.027</td></tr><tr><td>Run</td><td>F(1,21)=13.4; p=0.001; η<sup>2</sup>=0.05</td><td>F(1,17)=12; p=0.003; η<sup>2</sup>=0.05</td><td>F(1,20)=3.16; p=0.09; η<sup>2</sup>=0.01</td></tr><tr><td>Modality*Run</td><td>F(2,42)=1.61; p=0.21; η<sup>2</sup>=0.016</td><td>F(2,34)=0.6; p=0.55; η<sup>2</sup>=0.005</td><td>F(2,40)=1.05; p=0.36; η<sup>2</sup>=0.02</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>Figure 4: Box and scatter plot of ERD<sub>off</sub> for modalities V, T, and VT. Asterisks: p<0.05: * ; p<0.01: ** ; p<0.001: ***.</figcaption></figure>

rofeedback modality. To that extent, we compared neu-
rophysiological outcomes of MI-based NFB between a
thermal-only feedback modality, a visual-only feedback
modality, and a visuo-thermal modality (combining both
aforementioned feedback modalities). Through repeated
measures ANOVAs, we studied the effects of the feed-
back *modality* and *run* repetition.

We found that *run* repetition had a significant main effect on NFB performances (*ERD*<sub>on</sub> and *ERD*<sub>off</sub>), indicating a learning effect. We also found that *modality* had a significant main effect on offline performances (i.e., *ERD*<sub>off</sub>) with significantly greater desynchronization over the left sensorimotor cortex for the visual-only and visuo-thermal modalities compared to the thermal-only modality.

The main effect of *modality* of feedback on brain activity modulation was observed on the offline NFB performance (*ERD*<sub>off</sub>) but not on the online NFB performance (*ERD*<sub>on</sub>, *P*<sub>FB</sub>). Two main differences between the online and offline NFB performances most likely explain such differences. Firstly, the offline performances benefit from the ICA muscle activity artifacts correction, which is not applied online. Secondly, during online processing the positive values of *ERD*<sub>on</sub>, i.e., event related synchronization (ERS), were automatically set to 0% not to give negative feedback to users, whereas offline processing took into account both ERD and ERS. Thus, feedback *modality* shows a significant effect when both ERD and ERS are taken into account in the processing (*ERD*<sub>off</sub>) but not when only ERD are (*ERD*<sub>on</sub>). This suggests that feedback modality has mainly an effect on ERS, with V and VT modalities resulting in less ERS than their thermal counterpart.

Thermal feedback alone thus appears less effective than

its visual counterpart for MI-NFB learning. A first hypothesis to explain this result may lie in the difference of intuitiveness between both feedback. Considering the visual feedback, participants could see the starting and goal position of the hand (i.e. the mug) and thus appreciate at all times their performance based on the position of the hand between these two positions. On the contrary, during thermal-only NFB runs, participants could use the room temperature as a starting point, but did not have any reference cue related to the goal temperature to reach. Furthermore, thermal noticeable difference, i.e., the minimum temperature difference between two stimuli that a person is able to perceive, may reach a minimum of 0.75°C at 38°C, but rapidly increases for lower temperature (1.6°C at 32°C) [17]. Thus, considering the temperature range we used and that a prior characterisation of our thermal stimulation system gave a temperature variation speed of around 1°C/s, the perceived feedback variation experienced by participants during a 10 s task was very likely lower for thermal stimulation than for visual stimulation. This is corroborated by comments from two participants: one claimed to have felt a delay between their MI and the thermal feedback, and the other participant mentioned having a hard time figuring out if they were performing the MI task correctly with the thermal-only feedback condition and even feeling lost without visual feedback. Both aforementioned arguments suggest that visual feedback alone contains a richer information than thermal feedback alone. Despite eliciting lower modulation of brain activity than its visual counterpart and its probable lack of intuitiveness, adding thermal sensations to visual feedback did not decrease the efficiency of the visual-only feedback, as we did not find any significant difference in brain activity modulation between V and VT conditions. A stronger desynchronization could have been expected for the visual and thermal condition as previous studies found that multimodal feedback composed of haptic and visual stimuli has a beneficial influence [9, 10]. This could be caused by an inconsistency between our visual and thermal feedback due to an increased delay in thermal stimulation compared to the visual one. Our results suggest that using thermal feedback coupled with a visual feedback could be an interesting solution, especially for NFB application that would benefit from thermal stimulation such as sensory and motor post-stroke rehabilitation as previously mentioned [11, 12]. Further analyses of the user experience will provide us with complementary information.

Finally, it must be mentioned that including a repetitive grasping movement of the virtual hand in the visual feedback might have brought about the effect of action ob-
Finally, it must be mentioned that including a repetitive grasping movement of the virtual hand in the visual feedback might have brought about the effect of action observation (AO) from participants and thus enhanced the ERD amplitude in those conditions, as recently reported by Nagai et al. [17]. Indeed, they found that participants produced significantly greater ERDs during MI of fist clenching while synchronously looking at a video of someone else's hand performing the movement compared to pure motor imagery (without AO). Future investigation of the *Replay* runs should enable us to quantify the respective influence of AO and MI on the ERDs amplitude.

## CONCLUSION

In conclusion, this study sheds light on the potential of thermal feedback as a novel modality for motor imagery-based neurofeedback training. We found that it elicits a less pronounced modulation of the left sensorimotor cortex compared to visual feedback alone and to combined visuo-thermal feedback. Nevertheless, adding thermal feedback to other feedback modalities could remain interesting as it does not negatively affect NFB performances when combined with visual feedback. Overall, this study contributes to our understanding of feedback mechanisms in BCI and highlights the importance of considering alternative modalities in pursuit of more intuitive and effective human-computer interaction paradigms. The investigation into thermal feedback opens new avenues for improving neurotechnologies, particularly in clinical applications, such as sensorimotor rehabilitation. By addressing the challenges associated with MI tasks, we can bring further the development of NFB and BCIs and broaden their practical applications. Future research may further refine thermal feedback protocols and explore its potential synergies with other modalities to optimize user experience and BCI performances.

## ACKNOWLEDGEMENT

This work was supported by the GRASP-IT project (ANR-19-CE33-0007, January 2020) and PEPERONI project (CominLabs, 2022-2024). We would also like to thank Justine Saint-Aubert, Jeanne Hecquard and Kyung-Ho Won for their help in the technical development of the setup, as well as all the participants.

## REFERENCES

[1] Kerous B, Skola F, Liarokapis F. Eeg-based bci and video games: A progress report. Virtual Reality. 2018;22:119-135.

[2] Le Franc S et al. Toward an adapted neurofeedback for post-stroke motor rehabilitation: State of the art and perspectives. Frontiers in Human Neuroscience. 2022;16:917909.

[3] Pfurtscheller G, Neuper C. Motor imagery activates primary sensorimotor area in humans. Neuroscience letters. 1997;239(2-3):65-68.

[4] Pillette L. Redefining and adapting feedback for mental-imagery based brain-computer interface user training to the learners' traits and states. Ph.D. dissertation. Bordeaux. 2019.

[5] Fleury M, Lioi G, Barillot C, Lécuyer A. A survey on the use of haptic feedback for brain-computer interfaces and neurofeedback. Frontiers in Neuroscience. 2020;14:524967.

[6] Cincotti F et al. Vibrotactile feedback for brain-computer interface operation. Computational intelligence and neuroscience. 2007;2007.

[7] Gwak K, Leeb R, Millán JdR, Kim DS. Quantification and reduction of visual load during bci operation. In: 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 2014, 2795-2800.

[8] Lukoyanov M et al. The efficiency of the brain-computer interfaces based on motor imagery with tactile and visual feedback. Human Physiology. 2018;44:280-288.

[9] Barsotti M, Leonardis D, Vanello N, Bergamasco M, Frisoli A. Effects of continuous kinaesthetic feedback based on tendon vibration on motor imagery bci performance. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2017;26(1):105-114.

[10] Pillette L, N'kaoua B, Sabau R, Glize B, Lotte F. Multi-session influence of two modalities of feedback and their order of presentation on mi-bci user training. Multimodal Technologies and Interaction. 2021;5(3):12.

[11] Chen JC, Liang CC, Shaw FZ. Facilitation of sensory and motor recovery by thermal intervention for the hemiplegic upper limb in acute stroke patients: A single-blind randomized clinical trial. Stroke. 2005;36(12):2665-2669.

[12] Wu HC, Lin YC, Hsu MJ, Liu SM, Hsieh CL, Lin JH. Effect of thermal stimulation on upper extremity motor recovery 3 months after stroke. Stroke. 2010;41(10):2378-2380.

[13] Renard Y et al. Openvibe: An open-source software platform to design, test, and use brain-computer interfaces in real and virtual environments. Presence. 2010;19(1):35-53.

[14] Gramfort A et al. Meg and eeg data analysis with mne-python. Frontiers in neuroscience. 2013;7:70133.

[15] Darian-Smith I, Johnson KO. Thermal sensibility and thermoreceptors. Journal of Investigative Dermatology. 1977;69(1):146-153.

[16] Bunk SF, Lautenbacher S, Rüsseler J, Müller K, Schultz J, Kunz M. Does eeg activity during painful stimulation mirror more closely the noxious stimulus intensity or the subjective pain sensation? Somatosensory & motor research. 2018;35(3-4):192-198.

[17] Nagai H, Tanaka T. Action observation of own hand movement enhances event-related desynchronization. IEEE transactions on neural systems and rehabilitation engineering. 2019;27(7):1407-1415.
servation (AO) from participants and thus enhanced the ERD amplitude in those conditions, as recently reported by Nagai et al. [18]. Indeed, they found that participants produced significantly greater ERDs during MI of fist clenching while synchronously looking at a video of someone else's hand performing the movement compared to pure motor imagery (without AO). Future investigation of the *Replay* runs should enable us to quantify the respective influence of AO and MI on the ERDs amplitude.

## CONCLUSION

In conclusion, this study sheds light on the potential of thermal feedback as a novel modality for motor imagery-based neurofeedback training. We found that it elicits a less pronounced modulation of the left sensorimotor cortex compared to visual feedback alone and to combined visuo-thermal feedback. Nevertheless, adding thermal feedback to other feedback modalities could remain interesting as it does not negatively affect NFB performances when combined with visual feedback. Overall, this study contributes to our understanding of feedback mechanisms in BCI and highlights the importance of considering alternative modalities in pursuit of more intuitive and effective human-computer interaction paradigms. The investigation into thermal feedback opens new avenues for improving neurotechnologies, particularly in clinical applications, such as sensorimotor rehabilitation. By addressing the challenges associated with MI tasks, we can bring further the development of NFB and BCIs and broaden their practical applications. Future research may further refine thermal feedback protocols and explore its potential synergies with other modalities to optimize user experience and BCI performances.

## ACKNOWLEDGEMENT

This work was supported by the GRASP-IT project (ANR-19-CE33-0007, January 2020) and PEPERONI project (CominLabs, 2022-2024). We would also like to thank Justine Saint-Aubert, Jeanne Hecquard and Kyung-Ho Won for their help in the technical development of the setup, as well as all the participants.

## REFERENCES

[1] Kerous B, Skola F, Liarokapis F. Eeg-based bci and video games: A progress report. Virtual Reality. 2018;22:119-135.

[2] Le Franc S et al. Toward an adapted neurofeedback for post-stroke motor rehabilitation: State of the art and perspectives. Frontiers in Human Neuroscience. 2022;16:917909.

[3] Pfurtscheller G, Neuper C. Motor imagery activates primary sensorimotor area in humans. Neuroscience letters. 1997;239(2-3):65-68.

[4] Pillette L. Redefining and adapting feedback for mental-imagery based brain-computer interface user training to the learners' traits and states. Ph.D. dissertation. Bordeaux. 2019.

[5] Fleury M, Lioi G, Barillot C, Lécuyer A. A survey on the use of haptic feedback for brain-computer interfaces and neurofeedback. Frontiers in Neuroscience. 2020;14:524967.

[6] Cincotti F et al. Vibrotactile feedback for brain-computer interface operation. Computational intelligence and neuroscience. 2007;2007.

[7] Gwak K, Leeb R, Millán JdR, Kim DS. Quantification and reduction of visual load during bci operation. In: 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 2014. 2795-2800.

[8] Lukoyanov M et al. The efficiency of the brain-computer interfaces based on motor imagery with tactile and visual feedback. Human Physiology. 2018;44:280-288.

[9] Barsotti M, Leonardis D, Vanello N, Bergamasco M, Frisoli A. Effects of continuous kinaesthetic feedback based on tendon vibration on motor imagery bci performance. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2017;26(1):105-114.

[10] Pillette L, N'kaoua B, Sabau R, Glize B, Lotte F. Multi-session influence of two modalities of feedback and their order of presentation on mi-bci user training. Multimodal Technologies and Interaction. 2021;5(3):12.

[11] Chen JC, Liang CC, Shaw FZ. Facilitation of sensory and motor recovery by thermal intervention for the hemiplegic upper limb in acute stroke patients: A single-blind randomized clinical trial. Stroke. 2005;36(12):2665-2669.

[12] Wu HC, Lin YC, Hsu MJ, Liu SM, Hsieh CL, Lin JH. Effect of thermal stimulation on upper extremity motor recovery 3 months after stroke. Stroke. 2010;41(10):2378-2380.

[13] Renard Y et al. Openvibe: An open-source software platform to design, test, and use brain-computer interfaces in real and virtual environments. Presence. 2010;19(1):35-53.

[14] Gramfort A et al. Meg and eeg data analysis with mne-python. Frontiers in neuroscience. 2013;7:70133.

[15] Darian-Smith I, Johnson KO. Thermal sensibility and thermoreceptors. Journal of Investigative Dermatology. 1977;69(1):146-153.

[16] JASP Team. JASP (Version 0.18.3)[Computer software]. 2024. [Online]. Available: https://jasp-stats.org/.

[17] Bunk SF, Lautenbacher S, Rüsseler J, Müller K, Schultz J, Kunz M. Does eeg activity during painful stimulation mirror more closely the noxious stimulus intensity or the subjective pain sensation? Somatosensory & motor research. 2018;35(3-4):192-198.

[18] Nagai H, Tanaka T. Action observation of own hand movement enhances event-related desynchronization. IEEE transactions on neural systems and rehabilitation engineering. 2019;27(7):1407-1415.