# What is the best model for decoding neurophysiological
signals? Depends on how you evaluate

Bruno Aristimunha, Thomas Moreau, Sylvain Chevallier, Raphael Y de
Camargo, Marie-Constance Corsi

► To cite this version:

Bruno Aristimunha, Thomas Moreau, Sylvain Chevallier, Raphael Y de Camargo, Marie-Constance Corsi. What is the best model for decoding neurophysiological signals? Depends on how you evaluate. CNS 2024 - 33rd Annual Computational Neuroscience Meeting, Jul 2024, Natal, Brazil. hal-04743845

HAL Id: hal-04743845

https://inria.hal.science/hal-04743845v1

Submitted on 18 Oct 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# What is the best model for decoding neurophysiological signals? Depends on how you evaluate

Bruno Aristimunha¹,², Thomas Moreau³, Sylvain Chevallier¹, Raphael Y. de Camargo², Marie-Constance Corsi⁴

1. Inria TAU, LISN-CNRS, Université Paris-Saclay, 91405, Orsay, France

2. Center for Mathematics, Computing and Cognition, Universidade Federal do ABC, Santo André, Brazil

3. Inria Mind team, Université Paris-Saclay, CEA, Palaiseau, 91120, France

4. Sorbonne Université, Institut du Cerveau – Paris Brain Institute -ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière, Paris, France

5. Inria NERV team, Paris, France

Non-invasive brain-computer interface (BCI) is a framework that establishes direct communication between a computational external device and the brain activity, mostly via electroencephalography (EEG) signals. Despite its clinical applications, EEG-based BCI presents several challenges, such as performance variability across subjects and a low amount of data. To tackle these issues, many approaches have been proposed to better highlight and understand the neural dynamics reflected in the signals, including signal processing tools (e.g. band-pass filters, alignments), alternative features such as functional connectivity (Corsi M.-C. et al. 2022), or more sophisticated classification models (based on manifolds or deep learning). Despite all these efforts, many results fail to provide a consistent answer to which type of model is the best to understand brain dynamics; even when they use the same data, they use different evaluation schemes. In this study, we are interested in the following questions: (i) Does the way the model is evaluated impact the ranking of the best model? (ii) Does the amount of data impact the decoding of the brain signals, and is this reflected in the ranking? (iii) Do the best models also deliver better interpretability? Here, we systematically evaluated different methods, 7 using deep learning, 10 using Riemannian Manifold, and Common Spatial Patterns across six EEG-based BCI datasets during the sensory-motor rhythms tasks. All these methods were benchmarked using the same data split, with the classification task determining which motor imagery task occurred during a trial. The results were consistent with prior reports (Chevallier, S. et al. 2024). For instance, the best deep learning model, Attention Net (Wimpff, M., Gizzi, et al. 2024), outperformed the best Riemannian model, Fucone (Corsi M.-C. et al. 2022), by 16% in five-fold cross-validation at the subject level on the BCI 2014 competition dataset. However, the ranking remained inconsistent when we changed the evaluation method. The amount of data
used as input in the model was decisive for deep learning models, while the manifold models proved to be more invariant to this factor when trained with one model for a subject. Finally, the inherent interpretability of the functional connectivity models was not effective for scenarios with many subjects. These results emphasize the necessity of a systematic comparison of brain decoding models, drawing a parallel with the benchmark approaches that built the foundation in deep learning fields, which now could be adopted in neuroscience.

## References

Corsi, M. C., Chevallier, S., De Vico Fallani, F., & Yger, F. (2022). Functional connectivity ensemble method to enhance BCI performance (FUCONE). *IEEE Transactions on Biomedical Engineering*, 69(9), 2826-2838.

Wimpff, M., Gizzi, L., Zerfowski, J. and Yang, B., 2024. EEG motor imagery decoding: A framework for comparative analysis with channel attention mechanisms. *Journal of Neural Engineering*, 21(3), p.036020.

Chevallier, S., Carrara, I., Aristimunha, B., Guetschel, P., Sedlar, S., Lopes, B., ... & Moreau, T. (2024). The largest EEG-based BCI reproducibility study for open science: the MOABB benchmark. *arXiv preprint arXiv:2404.15319*.

## Acknowledgments

The work of BA was supported by DATAIA Convergence Institute as part of the ``Programme d'Investissement d'Avenir'', (ANR-17-CONV-0003) operated by LISN.

## E-mails

b.aristimunha@gmail.com

thomas.moreau@inria.fr

sylvain.chevallier@universite-paris-saclay.fr

raphael.camargo@ufabc.edu.br

marie-constance.corsi@inria.fr
<figure><img src="image_2.png" /></figure>

# WHAT IS THE BEST MODEL FOR DECODING NEUROPHYSIOLOGICAL SIGNALS?

## DEPENDS ON HOW YOU EVALUATE

Bruno Aristimunha¹,², Thomas Moreau³, Sylvain Chevallier¹, Raphael Y. de Camargo², Marie-Constance Corsi⁴

1. Inria TAU, LISN-CNRS, Université Paris-Saclay, 2. CMCC, UFABC, Santo André, Brazil | 3. Inria Mind, CEA, Université Paris-Saclay, France | 4. Sorbonne Université, ICM, CNRS, Inria NERV, Inserm, France

## What is decoding neurophysiological

### signals?

Here, the neurophysiological time series *x* depends on the stimulus *y*

Motor Imagery
Task

<figure><img src="image_3.png" /></figure>

x

y

Left

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

When you train a machine learning model, you learn how to decode an task.

## What is it benchmark?

Benchmarking is an **emerging science**, and we understand it as the iron rule to tame anything goes. All disputes must be settled by competitive empirical testing:

1) Agree on metric;

2) Agree on benchmark data;

3) Compete (Compute).

## Dataset and models (what we agree)

Here, we selected four motor-imagery datasets:

<table><caption>Table 1: Motor Imagery datasets considered during this study</caption><thead><tr><td>Dataset</td><td>Subjects</td><td>Channels</td><td>Sampling Rate (Hz)</td><td>Sessions</td><td>Tasks</td><td>Trials/Class</td><td>Epoch (s)</td></tr></thead><tbody><tr><td>BNCI2014001</td><td>9</td><td>22</td><td>250</td><td>2</td><td>4</td><td>144</td><td>[2, 6]</td></tr><tr><td>BNCI2014004</td><td>9</td><td>3</td><td>250</td><td>5</td><td>2</td><td>360</td><td>[3, 7.5]</td></tr><tr><td>Weibo2014</td><td>10</td><td>60</td><td>200</td><td>1</td><td>7</td><td>80</td><td>[3, 7]</td></tr><tr><td>Zhou2016</td><td>4</td><td>14</td><td>250</td><td>3</td><td>3</td><td>160</td><td>[0, 5]</td></tr></tbody></table>

And we selected 17 machine learning models:

<table><thead><tr><td>NAME</td><td>YEAR</td><td>CATEGORY</td></tr></thead><tbody><tr><td>BIOT</td><td>2023</td><td>DEEP</td></tr><tr><td>AUG-COV</td><td>2024</td><td>FC</td></tr><tr><td>ATCNet</td><td>2022</td><td>DEEP</td></tr><tr><td>AttentionNet</td><td>2024</td><td>DEEP</td></tr><tr><td>EEGITNet</td><td>2022</td><td>DEEP</td></tr><tr><td>EEGInception</td><td>2020</td><td>DEEP</td></tr><tr><td>EEGNetv4</td><td>2018</td><td>DEEP</td></tr><tr><td>ShallowFBCSPNet</td><td>2016</td><td>DEEP</td></tr><tr><td>TIDNet</td><td>2020</td><td>DEEP</td></tr><tr><td>Cov-CSP-LDA</td><td>2008</td><td>FC</td></tr><tr><td>Cov-CSP-LDA</td><td>2008</td><td>FC</td></tr><tr><td>Cov-FgMDM</td><td>2010</td><td>FC</td></tr><tr><td>Cov-MDM</td><td>2010</td><td>FC</td></tr><tr><td>Cov-Tang-LogReg</td><td>2010</td><td>FC</td></tr><tr><td>Fucone</td><td>2022</td><td>FC</td></tr><tr><td>Cov-Tang-SVM</td><td>2010</td><td>FC</td></tr><tr><td>LogVar-LDA</td><td>2008</td><td>FC</td></tr><tr><td>LogVar-SVM</td><td>2008</td><td>FC</td></tr></tbody></table>

### Deep learning

<figure><img src="image_6.png" /></figure>

### Functional Connectivity

<figure><img src="image_7.png" /></figure>

## How do we evaluate?

### Inter vs Intra models

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

## Experimental results

### 1. We can decode the time series!
One subject for model

multi subject for model

<table><thead><tr><th>solver_name</th><th>BNCI2014_001</th><th>BNCI2014_004</th><th>BNCI2015_001</th><th>Weibo2014</th><th>BNCI2014_001</th><th>BNCI2014_004</th><th>BNCI2015_001</th><th>Weibo2014</th></tr></thead><tbody><tr><td>ATCNet</td><td>38.93±12.80</td><td>63.01±18.96</td><td>55.79±13.37</td><td>27.21±6.41</td><td>35.07±4.11</td><td><strong>74.78±4.46</strong></td><td>53.27±2.90</td><td>18.45±2.62</td></tr><tr><td>AttentionBaseNet</td><td>39.85±13.73</td><td>57.65±16.93</td><td>57.39±16.03</td><td>26.70±6.87</td><td>47.41±6.61</td><td>67.36±7.31</td><td><strong>66.03±5.57</strong></td><td>18.70±1.95</td></tr><tr><td>AUG-COV</td><td>67.47±18.67</td><td>70.22±18.04</td><td>81.43±14.08</td><td>46.53±10.92</td><td>33.70±3.77</td><td>60.95±6.27</td><td>59.12±5.17</td><td>18.46±4.11</td></tr><tr><td>BIOT</td><td>33.33±10.47</td><td>55.75±16.01</td><td>58.00±14.33</td><td>27.94±6.80</td><td>26.65±1.72</td><td>66.44±2.24</td><td>53.28±0.93</td><td>17.43±1.81</td></tr><tr><td>Cov-CSP-LDA_shr</td><td>61.53±17.93</td><td>71.15±17.55</td><td>78.46±15.14</td><td>36.83±11.66</td><td>32.95±5.68</td><td>65.24±2.88</td><td>54.42±2.06</td><td>16.67±3.23</td></tr><tr><td>Cov-CSP-LDA_svd</td><td>62.87±17.41</td><td><strong>72.29±16.20</strong></td><td>77.36±16.65</td><td>38.34±12.05</td><td>35.45±5.15</td><td>63.69±3.25</td><td>59.04±6.00</td><td>17.93±4.45</td></tr><tr><td>Cov-FgMDM</td><td>69.20±16.29</td><td>70.78±18.21</td><td>81.00±13.94</td><td>60.54±11.44</td><td>32.76±3.85</td><td>61.61±3.66</td><td>55.44±1.39</td><td>18.48±3.17</td></tr><tr><td>Cov-MDM</td><td>60.27±15.26</td><td>69.03±17.92</td><td>75.43±17.26</td><td>32.56±11.08</td><td>30.45±5.67</td><td>66.83±3.31</td><td>57.65±5.98</td><td>18.98±2.53</td></tr><tr><td>Cov-Tang-LogReg</td><td>70.19±16.49</td><td>72.10±17.45</td><td>81.96±14.49</td><td>64.36±9.41</td><td>36.35±6.75</td><td>64.56±3.31</td><td>57.00±2.69</td><td>18.30±2.91</td></tr><tr><td>Fucone</td><td><strong>70.27±15.49</strong></td><td>70.64±18.16</td><td>81.50±13.61</td><td><strong>63.25±10.96</strong></td><td>32.38±3.73</td><td>60.45±4.46</td><td>54.59±1.48</td><td>17.80±2.43</td></tr><tr><td>Cov-Tang-SVM</td><td>67.32±15.83</td><td>69.80±17.53</td><td><strong>82.36±13.89</strong></td><td>65.30±9.92</td><td>30.24±4.75</td><td>65.20±3.57</td><td>54.76±3.20</td><td>17.72±2.75</td></tr><tr><td>DUMMY</td><td>15.71±4.24</td><td>40.53±7.81</td><td>41.82±6.27</td><td>9.03±3.04</td><td>25.00±0.00</td><td>50.00±0.00</td><td>50.00±0.00</td><td>14.29±0.00</td></tr><tr><td>EEGInception</td><td>30.77±12.07</td><td>54.41±16.92</td><td>53.89±12.58</td><td>21.66±6.79</td><td>30.21±1.95</td><td>70.41±4.81</td><td>58.20±1.69</td><td>16.39±2.72</td></tr><tr><td>EEGNetv4</td><td>32.91±9.77</td><td>52.13±13.97</td><td>51.89±9.99</td><td>22.98±7.97</td><td>42.73±5.08</td><td>74.74±3.42</td><td>65.01±4.45</td><td>18.13±3.53</td></tr><tr><td>LogVar-LDA</td><td>56.36±14.77</td><td>69.02±17.51</td><td>72.43±16.84</td><td>51.55±8.31</td><td>35.69±5.47</td><td>64.80±5.49</td><td>55.98±3.30</td><td>20.29±1.84</td></tr><tr><td>LogVar-SVM</td><td>42.76±16.46</td><td>66.52±20.04</td><td>63.86±17.39</td><td>41.94±7.51</td><td>33.19±2.29</td><td>63.33±1.38</td><td>58.39±4.41</td><td>18.16±2.71</td></tr><tr><td>ShallowFBCSPNet</td><td>48.24±16.57</td><td>67.69±19.03</td><td>67.07±17.03</td><td>45.47±10.30</td><td><strong>56.09±6.17</strong></td><td>73.26±2.24</td><td>64.67±4.11</td><td><strong>27.07±4.91</strong></td></tr><tr><td>TIDNet</td><td>34.21±10.21</td><td>62.72±20.05</td><td>55.25±13.12</td><td>30.21±7.45</td><td>35.40±2.77</td><td>75.10±4.45</td><td>63.41±1.64</td><td>21.68±3.54</td></tr></tbody></table>

### 2. The model ranks change completely!

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

## Take-Home insights

1) The way the evaluation is the devil in the details!

2) The variance of ranks between datasets is large; the best model for one dataset isn't necessarily the best for all. New models should be evaluated across different datasets, and statistical conclusions should be drawn.

3) More data appears to improve deep learning models. It doesn't make sense to build deep learning models if we train one model per subject, traditional models will be better;

4) We need to optimize the models more, but we have a tradeoff in computational costs, just here in an exploratory study, we trained more than 10500 models.

LinkedIn:

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>