# Ensemble of Riemannian Classifiers for Multimodal Data: FUCONE Approach for M/EEG Data

Marie-Constance Corsi, Sylvain Chevallier, Fabrizio de Vico Fallani, Florian Yger

<figure><img src="image_1.png" /></figure>

## To cite this version:

Marie-Constance Corsi, Sylvain Chevallier, Fabrizio de Vico Fallani, Florian Yger. Ensemble of Riemannian Classifiers for Multimodal Data: FUCONE Approach for M/EEG Data. ISBI 2023 - IEEE International Symposium on Biomedical Imaging, Apr 2023, Cartagena de Indias, Colombia. hal-04140126

HAL Id: hal-04140126

https://inria.hal.science/hal-04140126

Submitted on 24 Jun 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
Ensemble of Riemannian classifiers for multimodal data:
FUCONE approach for M/EEG data

<figure><img src="image_3.png" /></figure>

Marie-Constance Corsi*, Sylvain Chevallier, Fabrizio De Vico Fallani, Florian Yger

*Inria, Paris Brain Institute, France
Non-invasive Brain-Computer Interface - BCI

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /><figcaption>Control</figcaption></figure>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /><figcaption>Communication</figcaption></figure>
## Behind the magic...

<figure><img src="image_14.png" /></figure>

## Non-invasive tools

EEG

<figure><img src="image_15.png" /></figure>

NIRS

<figure><img src="image_16.png" /></figure>

MEG

<figure><img src="image_17.png" /></figure>

fMRI

<figure><img src="image_18.png" /></figure>
Behind the magic...

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>
Behind the magic...

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>
Behind the magic...

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>
Non-invasive Brain-Computer Interface - BCI

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

Adapted from (Ahn & Jun, 2015)

**Problem:** Current BCIs fail to detect the mental intentions in ~30% of users –
**BCI inefficiency** (Thompson, 2018)
# Riemannian Geometry (RG) & Functional connectivity (FC)

## Geometric methods, gold standard in BCI (Yger et al, 2017)

- Covariance matrices, $\Sigma = \frac{1}{N-1}XX^T$, $\Sigma$ captures some invariants of the EEG signal

- Manifold of Symmetric Positive Definite (SPD) matrices

- Robust & effective classifiers

- Good generalization, compatible with online requirement

- Invariant to affine transformation

<figure><img src="image_27.png" /><figcaption>Minimum Distance to Mean (MDM)</figcaption></figure>

$$ \delta(\Sigma_1, \Sigma_2) = \left\| \log(\Sigma_1^{-\frac{1}{2}} \Sigma_2 \Sigma_1^{-\frac{1}{2}}) \right\|_{\mathcal{F}} $$

$$ \bar{\Sigma} = \mu(\{\Sigma_i\}) = \arg\min_{\Sigma} \sum_{i=1}^{N} \delta^2(\Sigma_i, \Sigma) $$
Geometric methods, gold standard in BCI (Yger et al, 2017)

- Covariance matrices, $\Sigma = \frac{1}{N-1}XX^T$, $\Sigma$ captures some invariants of the EEG signal

- Manifold of Symmetric Positive Definite (SPD) matrices

- Robust & effective classifiers

- Good generalization, compatible with online requirement

- Invariant to affine transformation

<figure><img src="image_28.png" /><figcaption>Minimum Distance to Mean (MDM)</figcaption></figure>

$$ \delta(\Sigma_1, \Sigma_2) = \left\| \log(\Sigma_1^{-\frac{1}{2}} \Sigma_2 \Sigma_1^{-\frac{1}{2}}) \right\|_{\mathcal{F}} $$

$$ \bar{\Sigma} = \mu(\{\Sigma_i\}) = \arg\min_{\Sigma} \sum_{i=1}^{N} \delta^2(\Sigma_i, \Sigma) $$

**Problem:** RG approaches in BCI rely only on covariance matrices without considering synchronous interactions between brain areas
# Riemannian Geometry (RG) & Functional connectivity (FC)

## Geometric methods, gold standard in BCI (Yger et al, 2017)

* Covariance matrices, $\Sigma = \frac{1}{N-1}XX^T$, $\Sigma$ captures some invariants of the EEG signal

* Manifold of Symmetric Positive Definite (SPD) matrices

* Robust & effective classifiers

* Good generalization, compatible with online requirement

* Invariant to affine transformation

## Functional connectivity

* Interconnected nature of brain connectivity & alternative features in BCI

* Use of all the possible information from connectivity matrices

* Regularization methods to make the matrices SPD

<figure><img src="image_29.png" /><figcaption>Minimum Distance to Mean (MDM)</figcaption></figure>

$$ \delta(\Sigma_1, \Sigma_2) = \left\| \log(\Sigma_1^{-\frac{1}{2}} \Sigma_2 \Sigma_1^{-\frac{1}{2}}) \right\|_{\mathcal{F}} \\ \bar{\Sigma} = \mu(\{\Sigma_i\}) = \arg \min_{\Sigma} \sum_{i=1}^{N} \delta^2(\Sigma_i, \Sigma) $$

<figure><img src="image_30.png" /><figcaption>Amplitude synchronization</figcaption></figure>

<figure><img src="image_31.png" /><figcaption>Phase synchronization</figcaption></figure>

FC contrast between conditions, adapted from [Cattai, Colonnese, Corsi, et al, IEEE TNSRE, 2021]
# Riemannian Geometry (RG) & Functional connectivity (FC)

## Geometric methods, gold standard in BCI (Yger et al, 2017)

* Covariance matrices, $\Sigma = \frac{1}{N-1}XX^T$, $\Sigma$ captures some invariants of the EEG signal

* Manifold of Symmetric Positive Definite (SPD) matrices

* Robust & effective classifiers

* Good generalization, compatible with online requirement

* Invariant to affine transformation

## Functional connectivity

* Interconnected nature of brain connectivity & alternative features in BCI

* Use of all the possible information from connectivity matrices

* Regularization methods to make the matrices SPD

<figure><img src="image_32.png" /><figcaption>Minimum Distance to Mean (MDM)</figcaption></figure>

$$ \delta(\Sigma_1, \Sigma_2) = \left\| \log\left(\Sigma_1^{-\frac{1}{2}} \Sigma_2 \Sigma_1^{-\frac{1}{2}}\right) \right\|_{\mathcal{F}} $$

$$ \bar{\Sigma} = \mu(\{\Sigma_i\}) = \arg\min_{\Sigma} \sum_{i=1}^{N} \delta^2(\Sigma_i, \Sigma) $$

<figure><img src="image_33.png" /><figcaption>Amplitude synchronization</figcaption></figure>

<figure><img src="image_34.png" /><figcaption>Phase synchronization</figcaption></figure>

FC contrast between conditions, adapted from [Cattai, Colonnese, Corsi, et al, IEEE TNSRE, 2021]

**Hypothesis** – Combining functional connectivity estimators and Riemannian geometry will lead to an improvement of classification performance.
# Riemannian Geometry (RG) & Functional connectivity (FC)

## Motivation

* Considering subjects' specificity

* Improving the robustness

## Approach

* Ensemble learning approach – Riemannian Geometry framework relying on covariance matrices & connectivity matrices

<figure><img src="image_35.png" /></figure>
<figure><img src="image_36.png" /><figcaption>FUnctional CONnectivity Ensemble (FUCONE) approach</figcaption></figure>
RegCSP+shLDA better

FUCONE better

<figure><img src="image_37.png" /></figure>

CSP+optSVM better

FUCONE better

<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

Common Spatial Patterns + Shrinkage Linear Discriminant Analysis

<figure><img src="image_40.png" /></figure>

Common Spatial Patterns + Support Vector Machine

FgMDM better

FUCONE better

<figure><img src="image_41.png" /></figure>

FUCONE better

Cov+EN better

<figure><img src="image_42.png" /></figure>

<figure><img src="image_43.png" /></figure>

Geodesic filtering + Minimum Distance to Mean classification

<figure><img src="image_44.png" /></figure>

Covariance + Elastic-Net

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>
# FUCONE – A replicable approach

RegCSP+shLDA better

FUCONE better

<figure><img src="image_47.png" /></figure>

CSP+optSVM better

FUCONE better

<figure><img src="image_48.png" /></figure>

<figure><img src="image_49.png" /></figure>

Common Spatial Patterns + Shrinkage Linear Discriminant Analysis

<figure><img src="image_50.png" /></figure>

Common Spatial Patterns + Support Vector Machine

FgMDM better

FUCONE better

<figure><img src="image_51.png" /></figure>

FUCONE better

Cov+EN better

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

Geodesic filtering + Minimum Distance to Mean classification

<figure><img src="image_54.png" /></figure>

Covariance + Elastic-Net

<figure><img src="image_55.png" /></figure>

<figure><img src="image_56.png" /></figure>
## Motivation

* Complementarity between MEG and EEG data [Sharon et al, NeuroImage, 2007]

* M/EEG integration led to an improvement of the BCI classification performance (power spectra) [Corsi et al, IJNS, 2018]

<figure><img src="image_57.png" /></figure>

$$ \lambda_i = \frac{p_i}{P_{EEG} + P_{MAG} + P_{GRAD}} $$

<figure><img src="image_58.png" /></figure>

<footnote>*** p < 10⁻⁶</footnote>

<footnote>* p < 0.05</footnote>
## Motivation

* Complementarity between MEG and EEG data [Sharon et al, NeuroImage, 2007]

* M/EEG integration led to an improvement of the BCI classification performance (power spectra) [Corsi et al, IJNS, 2018]

## Approach

* Generalization to M/EEG data to exploit their complementarities

* To limit volume conduction effect, work in the source space [Corsi et al, NeuroImage, 2020]

<figure><img src="image_59.png" /></figure>
<figure><img src="image_60.png" /></figure>
<figure><img src="image_61.png" /></figure>
## FUCONE approach

* Possibility to consider the users' specificity

* Replicable approach

* From 2 to 4-classes configuration

* Within and Cross-sessions

## Extension to multimodal data, M/EEG

* Possibility to consider their complementarities

* Outperforms the state-of-the art method

## Next steps

* Dimensionality reduction & interpretability of the chosen ROIs

* Features extraction
Paris Brain Institute

Denis Schwartz,
Nathalie George,
Laurent Hugueville,
Christophe Gitton
Sophie Dupont,
Juliana Gonzalez-Astudillo,
Fabrizio De Vico Fallani (PI)

Penn University

Ari E. Kahn,
Ankit Khambhati,
Jennifer Stiso,
Arnold Campbell,
Danielle S. Bassett (PI)

Univ. Paris Dauphine – GREYC

Florian Yger

CNRS, Univ. Paris Saclay

Sylvain Chevallier
Interested in this study?

Scan the QR code to get access to the associated paper!

<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

mccorsi/FUCONE

Thank you for your attention!

<figure><img src="image_64.png" /></figure>

marie-constance.corsi@inria.fr

<figure><img src="image_65.png" /></figure>

MConstanceCorsi