# Workshop: "Designing Brain-Computer Interfaces, from theory to real-life scenarios" - HappyFeat

Arthur Desbois

► To cite this version:

Arthur Desbois. Workshop: "Designing Brain-Computer Interfaces, from theory to real-life scenarios" - HappyFeat. 9th Graz Brain-Computer Interface Conference 2024, Sep 2024, Graz, Austria. hal-04854478

HAL Id: hal-04854478

https://inria.hal.science/hal-04854478v1

Submitted on 23 Dec 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Inria

<figure><img src="image_2.png" /></figure>

Paris Brain
Institute

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

Graz'24 Workshop 2 - Sept. 9th 2024

HappyFeat: an interactive and
efficient BCI framework
for clinical applications

Arthur Desbois

Inria Paris, NERV team, Paris Brain Institute (ICM)
1. Context

- What is BCI? What are we looking for?

- Feature-based classification vs. BCI variabilities

2. HappyFeat

- Key points

- Example through typical use-case

3. Perspectives & conclusion
# 1. Context

* What is BCI? What are we looking for?

* Feature-based classification vs. BCI variabilities

<figure><img src="image_5.png" /></figure>

http://mmspg.epfl.ch/research

<figure><img src="image_6.png" /></figure>

SSVEP – Inria/Hybrid
<figure><img src="image_7.png" /><figcaption>Protocole BRACCIO – Venot et al., ICM/NERV</figcaption></figure>
<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

Objective – online classification
& live feedback
<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>
<figure><img src="image_12.png" /></figure>

Acquisition

Pre-processing

Features extraction

Features selection

Phase 1 – training the classifier

TRAINED CLASSIFIER

Classifier

weights

Acquisition

Pre-processing

Features extraction

Online Classification

<figure><img src="image_13.png" /></figure>

Phase 2 – online classification

Feedback
Motor execution VS Rest

<figure><img src="image_14.png" /></figure>

Motor imagery VS Rest

<figure><img src="image_15.png" /></figure>

Spectral Power Decrease

<figure><img src="image_16.png" /></figure>

Desynchronization effect
(Pfurtscheller et al, 1999)
* Features of interest (FOI)

* Selecting adequate FOIs is a crucial step for BCI performance.

* After EEG signals acquisition, an analysis phase is needed to select best FOIs.

* Scientific softwares (i.e., MATLAB)

* Manual step, expertise needed

* If this analysis phase is too long, a lot can change in the meantime:

* EEG sensors impedances

* Subject's brain behaviors

* Subject's attention & motivation

* Signal characteristics might be very different between Acq/Training phase and Online phase...

<figure><img src="image_17.png" /></figure>
The “analysis phase” involves many
manipulations. An example:

* Setting up “feature extraction” pipelines (e.g. OpenViBE)

* Finding FOIs through visualization (e.g. with Matlab, Python, etc.)

* Setting up & running training pipelines (e.g. OpenViBE)

* ... and maybe going again through those steps multiple times until “correct” features have been found, or to account for inter-run variability

→ Tedious, error-prone, hard to achieve in a limited time

<figure><img src="image_18.png" /></figure>
# Python-based framework for facilitating MI pipelines

**Main focus:**

- Making Feature Selection
- & Classif. Training phases
- easy & fast

- Analyze your data,
- select your Features
- & train your classifier
- in less than 5 minutes!

<figure><img src="image_19.png" /></figure>
## 1. Context

- What is BCI? What are we looking for?

- Feature-based classification vs. BCI variabilities

## 2. HappyFeat

- Key points

- Example through typical use-case
<figure><img src="image_20.png" /></figure>

Unified, dashboard-like GUI
- Classical **local** measures

<figure><img src="image_21.png" /></figure>

- Novel **network-based** approaches

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

HappyFeat
<figure><img src="image_27.png" /></figure>

**Feature Extraction**

<figure><img src="image_28.png" /></figure>

**Feature Visualization**

<figure><img src="image_29.png" /></figure>

- Contrast maps

- Time-frequency maps

- Topographies
<figure><img src="image_30.png" /></figure>

Tool for the decision
Automatic manipulation of
pipelines & scenarii

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

Support for
BCI Setup

<figure><img src="image_35.png" /></figure>

Feature
Extraction

<figure><img src="image_36.png" /></figure>

Classifier
Training

<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /></figure>

HappyFeat

Feature
Visualization

<figure><img src="image_39.png" /></figure>
<figure><img src="image_40.png" /></figure>

<figure><img src="image_41.png" /></figure>

Workspace to ensure
**reproducibility** and
**replicability** of all the
manipulations
* **Key feats & mechanisms:**

* Clean, risk-free environment
    → avoid unnecessary & error-prone manipulations.

* Trial-and-error oriented workflow
    → all steps can be repeated quickly & as many times as needed

* Unified “dashboard” GUI + Workspace management system

* Direct link with BCI software(s): OpenViBE (TimeFlux in the near future!)
    → no scenario edition/manipulation necessary: everything is automated!

<figure><img src="image_42.png" /></figure>

* **Two main use cases:**

* Make BCI pipelines smoother/easier to use and allow reproducibility of exps.

* Prospective works & comparison of alternative features of interest (connectivity, networks...)
* **Efficient processing pipelines**

* Available features for classif.: *Spectral Power (PSD), Connectivity-based network metrics*

* ... It's also possible to train the classifier using a fusion of both features.

* **Feature extraction**

* Easy access to all experiment & signal processing parameters.

* Use pre-recorded signals or on-the-fly during acquisition phase.

<figure><img src="image_43.png" /></figure>

* **Assisting feature selection: Visualization, AutoFeat**

* Viz: $R^2$ maps, PSD comparison across trials, time/freq. ERD/ERS analysis, brain topography

* Select features manually using visualization tools...

* ...or a set of the channel/freq couples with the highest $R^2$, automatically
* Classifier training

* Run as many training attempts as needed, using different features, in only a few clicks.

* Concatenate trials from multiple recorded sessions

* Automatically try different feature combinations with one click

* Onwards to online classification

* Automatically generate/update the
  "online classification" OpenViBE / TimeFlux scenario
  using selected classification feats. and trained classifier.

<figure><img src="image_44.png" /></figure>
## 1. Workspace creation/selection

* Create a new environment (file tree, configuration, data structures) for the experiment/analysis

Ex: **sub23_session01**

* ... or select an already existing one.

<figure><img src="image_45.png" /></figure>

→ All user settings & manipulations are saved and available in the workspace for future usage or sharing:

* Extraction parameters (spectral analysis, list of processed files...)

* Training attempts (runs & features used, training accuracy...)
## 1. Workspace creation/selection

* When creating a new workspace:

* the feature type (PSD, Connectivity...) is chosen

* experimental/acquisition parameters are entered (electrode montage, stimulation names, experimental layout...)

* This step allows to automatically set up the OpenViBE acquisition scenario.

These experimental parameters are also important for the different processing steps of HappyFeat.

<figure><img src="image_46.png" /></figure>
## 1. Workspace creation/selection

## 2. Feature Extraction

* Files in the “signals” folder of the workspace are listed in the left panel

* Select one or multiple files...

* Enter the parameters for feature extraction...

* Press the button!

<figure><img src="image_47.png" /></figure>

Along the workspace mechanism, HappyFeat also uses work sessions to keep track of extraction parameters.

Therefore, all steps realized with a set of extraction parameters (which files have undergone extraction + training attempts) are stored and can be recovered.

<figure><img src="image_48.png" /></figure>
1. Workspace creation/selection

2. Feature Extraction

3. Analysis & Feature Selection

* "Extracted" files (runs) appear in the central panel.

* Select which runs to analyze...

<figure><img src="image_49.png" /></figure>
<figure><img src="image_50.png" /></figure>

1. Workspace creation/selection

2. Feature Extraction

3. Analysis & Feature Selection

* “Extracted” files (runs) appear in the central panel.

* Select which runs to analyze...

* And use the different visualization tools to select adequate features:

* Channel/frequency $R^2$ map

* Per-channel metric comparison

* Per-frequency $R^2$ topography

* More coming soon...
<figure><img src="image_51.png" /></figure>

1. Workspace creation/selection

2. Feature Extraction

3. Analysis & Feature Selection

* “Extracted” files (runs) appear in the central panel.

* Select which runs to analyze...

* And use the different visualization tools to select adequate features.

* ... or use the “AutoFeat” mechanism to automatically select a set of (channel/frequency) pairs with the highest R2
1. Workspace creation/selection

2. Feature Extraction

3. Analysis & Feature Selection

4. Classifier training

* "Extracted" files (runs) also appear in the right panel.

* Select one or multiple runs...
  ... their trials will all be used for training.

* Select one or multiple feature(s) of interest (channel;frequency)

* Click the button!

<figure><img src="image_52.png" /></figure>
1. Workspace creation/selection

2. Feature Extraction

3. Analysis & Feature Selection

4. Classifier training

* "Extracted" files (runs) also appear in the right panel.

* Select one or multiple runs...
  * ... their trials will all be used for training.

* Select one or multiple feature(s) of interest (channel;frequency)

* Click the button!

→ A training accuracy is provided

→ All previous training results are available (displayed in the GUI, and classif. weights in the workspace)

<figure><img src="image_53.png" /></figure>
1. Workspace creation/selection

2. Feature Extraction

3. Analysis & Feature Selection

4. Classifier training

5. Online classification

A satisfactory set of classification features has been found!

→ Generate a ready-to-use OpenViBE (soon TimeFlux) scenario

- trained classifier

- selected classif features

<figure><img src="image_54.png" /></figure>
1. Context

- What is BCI? What are we looking for?

- Feature-based classification vs. BCI variabilities

2. HappyFeat

- Key points

- Example through typical use-case

3. Perspectives & conclusion
* Current limitations

* Only one type of classification algo. proposed (LDA)

* Pipelines are “fixed”:

- trading OpenViBE's high level of flexibility...

+ ...for a high comfort of pipeline settings customization

+ & “trial-and-error” workflow

* Only three types of pipelines/feature types:

* Power Spectrum Density

* Connectivity (coherence & its variants)

* Dual (mixing PSD & Connectivity)

→ In project (long term!) for more “prospective power”:
    allow the user to choose btw. 1 and 3 feature-types & network metrics
    to mix as they see fit (MSC/node strength + Imag(Coh)/Laterality + ...)

<figure><img src="image_55.png" /></figure>
* Future works

* Fully autonomous 100% Python version, without OpenViBE (in progress: T. Venot)

- No acquisition/online possibilities

- Obviously slower... (no C++ optimizations! ... or maybe via pybind?)

+ More portable, all types of platforms supported (MacOS!)

+ Other (more flexible) formats for I/O and work files (CSV, EDF...)

+ Directly use great OS packages (MNE, scikit-learn...)

+ Research oriented app → integrating NERV's works on graphs
* Future works

* 2024 internship in progress: allowing other BCI softwares: Timeflux

- Slower... (no C++ optimizations! ... or maybe?)

+ More portable, all types of platforms

+ TimeFlux also 100% Python, easier communication between softwares

* Tutorials using MOABB datasets

* General GUI improvements

« Click & play » philosophy for feature selection
Version 0.2.0 (since June 2024!)

PyPi package / github repo

Testers, feedback & contributions are welcome (via github issues and PR...)

Full list of dependencies:

* Python 3.9

* shutils>=0.1.0

* Mne==1.5.1

* Numpy==1.26.4

* Pandas==2.2.1

* PySide2==5.15.2.1

* statsmodels>=0.13.1

* scipy>=1.7.1

* spectrum>=0.8.0

* Matplotlib>=3.8.4

* OpenViBE v3.6.0
## Available online:

github: https://github.com/Inria-NERV/happyFeat

PyPI: https://pypi.org/project/happyfeat/

documentation: https://happyfeat.readthedocs.io/

## To be continued...

* `+ +` flexibility (pipeline options, mixing metrics...)

* `+ +` network metrics (based on connectivity) (and associated viz. tools)

* `+ +` classification algorithms options

* Fully autonomous Python version, “sandbox” for research purposes

* Support for other BCI SW (starting with TimeFlux)

* GUI & general interactivity improvement

Stay tuned!
<figure><img src="image_56.png" /></figure>

<figure><img src="image_57.png" /></figure>

github.com/Inria-NERV/happyFeat

<figure><img src="image_58.png" /></figure>

https://doi.org/10.1016/j.simpa.2023.100610

<figure><img src="image_59.png" /></figure>

Thanks for your attention!
<figure><img src="image_60.png" /></figure>

BCI Motor Imagery with OpenViBE in X-Men: First Class

Thanks for your attention! Any questions?
<figure><img src="image_61.png" /></figure>

Paris Brain
Institute

<figure><img src="image_62.png" /></figure>

Graz'24 - WS2 - Sept. 9th 2024

HappyFeat: an interactive and efficient BCI
framework for clinical applications

**Additional material**

Arthur Desbois

Inria Paris, NERV team, Paris Brain Institute (ICM)
<figure><img src="image_63.png" /></figure>
<figure><img src="image_64.png" /></figure>

<figure><img src="image_65.png" /></figure>

Adapted from (Ahn & Jun, 2015)

Current BCIs fail to detect the mental intentions in ~30% of users – **BCI inefficiency**
* Sources of variabilities

* Data acquisition (eg EEG sensors impedances)

* Tiredness

* Brain signatures (eg spatial shifts)

→ Need to reduce the calibration time

→ Need to identify the features that best capture the patients' intent

**HappyFeat** is a software aiming to simplify the use of BCI pipelines
in **clinical settings by non-expert users** by
**assisting the extraction and the selection of the classification features.**
## 1. Workspace creation/selection

* Create a new environment (file tree, configuration, data structures) for the experiment/analysis

* ... or select an already existing one.

* All user settings & manipulations are saved and available in the workspace for future usage or sharing:

* Extraction parameters (spectral analysis, list of processed files...)

* Training attempts (runs & features used, training accuracy...)

<figure><img src="image_66.png" /></figure>
# HappyFeat - How? (with openvibe)

## 1. Workspace creation/selection

## 2. MI Pipeline / “Feature type” selection

* Selecting btw. multiple “template” scenarios depending on the use case (power spectrum, connectivity type...)

* Edit basic/common parameters (acquisition, extraction, training...)

<figure><img src="image_67.png" /></figure>
1. Workspace creation/selection

2. MI Pipeline / “Feature type” selection

3. Feature Extraction

* Select signal files, and extraction parameters (lengths and overlap of windows, FFT size...)

* Run the generated extraction OpenViBE scenario (in the background) for all selected signal files:
  → Extract metadata (sampling freq, electrodes...)

→ Cut the signal to regions of interest (MI trials & baseline portions), generate CSV file with only these chunks (for the training step)

→ Apply a signal processing pipeline (PSD computation, connectivity measure...) to the signal chunks of interest, generate CSV files for future analysis

* Runs in an autonomous thread:
  You can do visualizations and training attempts for signals already processed in the meantime.

<figure><img src="image_68.png" /></figure>
1. Workspace creation/selection

2. MI Pipeline / “Feature type” selection

3. Feature Extraction

4. Analysis, Feature Selection

* Select one or multiple signals & load their spectral/connectivity data (CSV work files generated during “Feature Extraction”)

* Use different Visualization Tools to help find & select features of interest (FOIs) for training

* Frequency/channel R² map

* PSD (or connect. metric) comparison btw. the 2 conditions (MI/REST) for a given electrode

* Time/frequency ERD/ERS analysis for each condition

* R² mapped as a brain topography for a given frequency (or range)

## Combine as many visualization windows as you need

A “Dual metric” pipeline allows to show (for ex.) R² maps for both Power Spectrum and Connectivity in parallel

<figure><img src="image_69.png" /></figure>
1. Workspace creation/selection

2. MI Pipeline / “Feature type” selection

3. Feature Extraction

4. Analysis, Feature Selection

5. Classifier Training

* Set one or more **Features of Interest** (Electrode/Freq.)

* Select file(s) with which you want to train your classifier
  If > 1 file: their trials are automatically concatenated

* Run the Classifier Training scenario
  (Auto. generated in step 1 + auto. edited with FOIs)
  → Classification ACCURACY + WEIGHTS

**Disappointing results?** ("My accuracy is 50% 😞")

Maybe try again with other features. It only takes a few seconds...

**Satisfying results?!** ("OMG 95%"!)

Good news! The “Online Classification” scenario has already been automatically updated with:

- Classifier training weights

- Features of interest used for training

<figure><img src="image_70.png" /></figure>

Training accuracy “score”

+ Classifier Weights

+ Online scenario updated