# Improving portability of bimodal neurofeedback: predicting NF-fMRI scores from EEG signals

Caroline Pinte, Claire Cury, Pierre Maurel

<figure><img src="image_1.png" /></figure>

## To cite this version:

Caroline Pinte, Claire Cury, Pierre Maurel. Improving portability of bimodal neurofeedback: predicting NF-fMRI scores from EEG signals. OHBM 2023 - Organization for Human Brain Mapping, Jul 2023, Montréal, Canada. pp.1-1. hal-04168694

HAL Id: hal-04168694

https://inria.hal.science/hal-04168694

Submitted on 21 Jul 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

# Can neural networks help improve portability of combined EEG – fMRI neurofeedback?

## Improving portability of bimodal neurofeedback: predicting NF-fMRI scores from EEG signals

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

Caroline Pinte

<figure><img src="image_9.png" /></figure>

@Caroline_Pinte

<figure><img src="image_10.png" /></figure>

caroline.pinte@irisa.fr

## Introduction

* Neurofeedback (NF) training is more effective with simultaneous EEG-fMRI acquisitions [1].

* However, the use of MRI is expensive and draining for the subject.

* So, we need to reduce its use.

→ We propose a method based on a one-dimensional convolutional neural network (1D CNN). With this model, we aim to predict fMRI information from EEG recordings.

## Objective

Learning a **neural network** model from simultaneous EEG-fMRI data to predict **NF-fMRI** scores from **EEG** signals alone.

## Methods

1. **Data:** We acquired **simultaneous EEG-fMRI data** on 15 healthy subjects in a motor imagery context, described in [2]. This includes fMRI and EEG data for three NF runs of five minutes per subject, with their respective NF scores.

2. **Preprocessing:** We compute a **supervised learning dataset**, where a certain number of timesteps from a selection of EEG channels become one sample that has one NF score as target.

3. **Training:** We train the **1D CNN model**. Since we have 15 subjects with 3 runs each, the training dataset consists of 14 subjects and 2 runs, the validation dataset of the third run from these 14 subjects, and the test dataset of the 3 runs from the 15th subject.

4. **Analysis:** After the model is trained, **NF-fMRI scores are predicted**, and compared to the ground truths with the RMSE and Pearson correlation metrics.

## Discussion

We are currently working on improving the correlations, which are negatively impacted by:

* Sensitivity to the neural network hyperparameters

* Noisy nature of the EEG modality

* Lack of diversity in data

## Conclusion

* The interest of this method compared to the previous modeling [3] is to train a single **deep learning model** for all subjects, which would eventually allow the use of the model to predict these fMRI scores directly online.

* An important aspect of the issue is to **interpret this model**, in order to understand links between EEG and fMRI.

<figure><img src="image_11.png" /><figcaption>**Fig. 1:** Architecture of the 1D CNN used to predict NF-EEG and NF-fMRI scores from raw EEG signals. The input sample consists of 4400 timesteps from every EEG channels selected. When predicting NF-EEG scores, we select 9 channels corresponding to the laplacian of the C3 channel, and when predicting NF-fMRI scores, we select 25 channels above the motor area.</figcaption></figure>

## Results

<table><caption>### Predicting NF-EEG scores to validate our approach:</caption><thead><tr><td></td><td>Predicting NF-EEG scores</td><td>Predicting NF-fMRI scores</td></tr></thead><tbody><tr><td>Mean RMSE</td><td>0.23</td><td>0.47</td></tr><tr><td>Mean correlation</td><td>0.94</td><td>0.23</td></tr><tr><td>Min-max correlation</td><td>(0.92 → 0.95)</td><td>(0.00 → 0.44)</td></tr></tbody></table>

<figure><img src="image_12.png" /><figcaption>**Fig. 2:** Prediction of NF-EEG scores: train and valid loss curves (top left) and comparison between NF-EEG ground truth and NF-EEG prediction for the 1st run (top right), the 2nd run (bottom left) and the 3rd run (bottom right) from the test subject.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>**Fig. 3:** Prediction of NF-fMRI scores: train and valid loss curves (top left) and comparison between NF-fMRI ground truth and NF-fMRI prediction for the 1st run (top right), the 2nd run (bottom left) and the 3rd run (bottom right) from the test subject.</figcaption></figure>

<figure><img src="image_14.png" /></figure>

1. Perronnet et al. "Unimodal versus bimodal EEG-fMRI neurofeedback of a motor imagery task." (2017)

2. Lioi et al. "Simultaneous EEG-fMRI during a neurofeedback task, a brain imaging dataset for multimodal data integration." (2020)

3. Cury et al. "A sparse EEG-informed fMRI model for hybrid EEG-fMRI neurofeedback prediction." (2020)

<figure><img src="image_15.png" /></figure>

Caroline Pinte¹, Claire Cury¹, Pierre Maurel¹

¹University of Rennes, CNRS, Inria, Inserm, IRISA UMR 6074, EMPENN — ERL U 1228, Rennes, France.

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>