<figure><img src="image_1.png" /></figure>

HAL
open science

# RNN-LSTM neural network for predicting fMRI neurofeedback scores from EEG signals

Caroline Pinte, Claire Cury, Pierre Maurel

<figure><img src="image_2.png" /></figure>

## To cite this version:

Caroline Pinte, Claire Cury, Pierre Maurel. RNN-LSTM neural network for predicting fMRI neurofeedback scores from EEG signals. rtFIN 2022 - Real-time Functional Imaging and Neurofeedback, Oct 2022, New Haven, United States. hal-03798824

HAL Id: hal-03798824

https://hal.science/hal-03798824

Submitted on 5 Oct 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# RNN-LSTM neural network for predicting fMRI neurofeedback scores from EEG signals

Caroline Pinte¹, Claire Cury¹, Pierre Maurel¹

1 - University of Rennes, CNRS, Inria, Inserm, IRISA UMR 6074, Empenn Team ERL U 1228, Rennes, France.

**Introduction** In the context of neurofeedback (NF), simultaneous acquisitions with electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) provide more effective NF training due to their complementarity [1]. However, the use of MRI is expensive and draining for the subject. Therefore, we would like to reduce its use. Following the work of Cury et al. [2], we propose a method based on a recurrent neural network (RNN) that consists in learning a model from simultaneous EEG-fMRI acquisitions to predict NF-fMRI scores with EEG signals alone.

**Materials and Methods** We acquired simultaneous EEG-fMRI data on 15 healthy subjects, in a motor imagery context, following the XP2 protocol described in [3]. This includes fMRI and EEG data for three NF runs of five minutes per subject, with their respective NF scores whose calculation is detailed in the same article. Our method consists in extracting, for each EEG electrode, traditional features such as mean, standard deviation, skewness, kurtosis, α, β, and σ band powers. We then proceed to train a long short-term memory recurrent neural network (RNN-LSTM). The training, validation and test datasets are each composed of 15 runs, one per subject. After the model is trained, NF-fMRI scores can be predicted, to which the true NF-EEG scores are added for comparison with the NF-EEG-fMRI ground truth.

## Results and Conclusions

The predictions of our current best model have an RMSE of 1.35 and a correlation of 0.57 with the ground truths on average over the 15 test runs. However, there are large variations in performance across subjects, ranging from a correlation of 0.79 for the example in Fig. 1 to correlations close to 0.30 for some subjects, which is not yet satisfying. Indeed, there are still many ways to improve the model, which suffers in particular from a sensitivity to the learning parameters, from the noisy nature of the EEG modality, and from the lack of data. To conclude, the interest of this method compared to the previous modeling [2] is to test the contribution of deep learning, mainly to train a single model for all subjects, which would eventually allow the use of the model to predict these fMRI scores directly online.

<figure><img src="image_4.png" /><figcaption>**Fig. 1:** Prediction of NF-fMRI scores for the best subject, to which the true NF-EEG scores are added, and compared with the true combined NF-EEG-fMRI scores.</figcaption></figure>

## References

[1] Perronnet, Lorraine, et al. "Unimodal versus bimodal EEG-fMRI neurofeedback of a motor imagery task." Frontiers in Human Neuroscience 11 (2017): 193.

[2] Cury, Claire, et al. "A sparse EEG-informed fMRI model for hybrid EEG-fMRI neurofeedback prediction." Frontiers in Neuroscience 13 (2020): 1451.

[3] Lioi, Giulia, et al. "Simultaneous EEG-fMRI during a neurofeedback task, a brain imaging dataset for multimodal data integration." Scientific data 7.1 (2020): 1-15.