"Designing Brain-Computer Interfaces with
Open-Source Tools: from theory to real-life scenarios":
Happyfeat

Arthur Desbois

<figure><img src="image_1.png" /></figure>

To cite this version:

Arthur Desbois. "Designing Brain-Computer Interfaces with Open-Source Tools: from theory to real-life scenarios": Happyfeat. 5th International Neuroergonomics Conference, Jul 2024, Bordeaux, France. hal-04854431

HAL Id: hal-04854431

https://inria.hal.science/hal-04854431v1

Submitted on 23 Dec 2024

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Inria

<figure><img src="image_3.png" /></figure>

Paris Brain
Institute

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

# NEC24 Workshop 1 - July 8th 2024

## HappyFeat: an interactive and
efficient BCI framework
for clinical applications

Arthur Desbois

Inria Paris, NERV team, Paris Brain Institute (ICM)
1. Context

- What is BCI? What are we looking for?

- Feature-based classification vs. BCI variabilities

2. HappyFeat

- Key points

- Example through typical use-case

3. Perspectives & conclusion
<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

Objective – online classification
& live feedback
<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>
<figure><img src="image_10.png" /></figure>

Acquisition

Pre-processing

Features extraction

Features selection

Phase 1 – training the
classifier

TRAINED CLASSIFIER

Classifier
weights

Acquisition

Pre-processing

Features extraction

Online Classification

<figure><img src="image_11.png" /></figure>

Phase 2 – online
classification

Feedback
Motor execution VS Rest

<figure><img src="image_12.png" /></figure>

Spectral Power Decrease

<figure><img src="image_13.png" /></figure>

Desynchronization effect
(Pfurtscheller et al, 1999)

Motor imagery VS Rest

<figure><img src="image_14.png" /></figure>
* Features of interest (FOI)

* Selecting adequate FOIs is a crucial step for BCI performance.

* After EEG signals acquisition, an analysis phase is needed to select best FOIs.

* Scientific softwares (i.e., MATLAB)

* Manual step, expertise needed

* If this analysis phase is too long, a lot can change in the meantime:

* EEG sensors impedances

* Subject's brain behaviors

* Subject's attention & motivation

* Signal characteristics might be very different between Acq/Training phase and Online phase...

<figure><img src="image_15.png" /></figure>
The “analysis phase” involves many
manipulations. An example:

* Setting up “feature extraction” pipelines (e.g. OpenViBE)

* Finding FOIs through visualization (e.g. with Matlab, Python, etc.)

* Setting up & running training pipelines (e.g. OpenViBE)

* ... and maybe going again through those steps multiple times until “correct” features have been found, or to account for inter-run variability

→ Tedious, error-prone, hard to achieve in a limited time

<figure><img src="image_16.png" /></figure>
# Python-based framework for facilitating MI pipelines

**Main focus:**

- Making Feature Selection
- & Classif. Training phases
- easy & fast

- Analyze your data,
- select your Features
- & train your classifier
- in less than 5 minutes!

<figure><img src="image_17.png" /></figure>
## 1. Context

- What is BCI? What are we looking for?

- Feature-based classification vs. BCI variabilities

## 2. HappyFeat

- Key points

- Example through typical use-case
<figure><img src="image_18.png" /></figure>

Unified, dashboard-like GUI
- Classical **local** measures

<figure><img src="image_19.png" /></figure>

- Novel **network-based** approaches

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

HappyFeat
<figure><img src="image_25.png" /></figure>

**Feature Extraction**

<figure><img src="image_26.png" /></figure>

**Feature Visualization**

<figure><img src="image_27.png" /></figure>

- Contrast maps

- Time-frequency maps

- Topographies
<figure><img src="image_28.png" /></figure>

Tool for the decision
Automatic manipulation of

OpenViBE scenarios

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

Support for
BCI Setup

<figure><img src="image_33.png" /></figure>

Feature
Extraction

<figure><img src="image_34.png" /></figure>

Classifier
Training

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>

HappyFeat

Feature
Visualization

<figure><img src="image_37.png" /></figure>
<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

Workspace to ensure **reproducibility** and **replicability** of all the manipulations
## 1. Workspace creation/selection

* Create a new environment (file tree, configuration, data structures) for the experiment/analysis

Ex: **sub23_session01**

* ... or select an already existing one.

<figure><img src="image_40.png" /></figure>

→ All user settings & manipulations are saved and available in the workspace for future usage or sharing:

* Extraction parameters (spectral analysis, list of processed files...)

* Training attempts (runs & features used, training accuracy...)
## 1. Workspace creation/selection

* When creating a new workspace:

* the feature type (PSD, Connectivity...) is chosen

* experimental/acquisition parameters are entered (electrode montage, stimulation names, experimental layout...)

* This step allows to automatically set up the OpenViBE acquisition scenario.

These experimental parameters are also important for the different processing steps of HappyFeat.

<figure><img src="image_41.png" /></figure>
## 1. Workspace creation/selection

## 2. Feature Extraction

* Files in the “signals” folder of the workspace are listed in the left panel

* Select one or multiple files...

* Enter the parameters for feature extraction...

* Press the button!

<figure><img src="image_42.png" /></figure>

Along the workspace mechanism, HappyFeat also uses work sessions to keep track of extraction parameters.

Therefore, all steps realized with a set of extraction parameters (which files have undergone extraction + training attempts) are stored and can be recovered.

<figure><img src="image_43.png" /></figure>
1. Workspace creation/selection

2. Feature Extraction

3. Analysis & Feature Selection

* "Extracted" files (runs) appear in the central panel.

* Select which runs to analyze...

<figure><img src="image_44.png" /></figure>
1. Workspace creation/selection

2. Feature Extraction

3. Analysis & Feature Selection

* “Extracted” files (runs) appear in the central panel.

* Select which runs to analyze...

* And use the different visualization tools to select adequate features:

* Channel/frequency $R^2$ map

* Per-channel metric comparison

* Per-frequency $R^2$ topography

* More coming soon...

<figure><img src="image_45.png" /></figure>
<figure><img src="image_46.png" /></figure>

1. Workspace creation/selection

2. Feature Extraction

3. Analysis & Feature Selection

* “Extracted” files (runs) appear in the central panel.

* Select which runs to analyze...

* And use the different visualization tools to select adequate features.

* ... or use the “AutoFeat” mechanism to automatically select a set of (channel/frequency) pairs with the highest R2
1. Workspace creation/selection

2. Feature Extraction

3. Analysis & Feature Selection

4. Classifier training

* "Extracted" files (runs) also appear in the right panel.

* Select one or multiple runs...
  ... their trials will all be used for training.

* Select one or multiple feature(s) of interest (channel;frequency)

* Click the button!

<figure><img src="image_47.png" /></figure>
1. Workspace creation/selection

2. Feature Extraction

3. Analysis & Feature Selection

4. Classifier training

* "Extracted" files (runs) also appear in the right panel.

* Select one or multiple runs...
  * ... their trials will all be used for training.

* Select one or multiple feature(s) of interest (channel;frequency)

* Click the button!

→ A training accuracy is provided

→ All previous training results are available (displayed in the GUI, and classif. weights in the workspace)

<figure><img src="image_48.png" /></figure>
1. Workspace creation/selection

2. Feature Extraction

3. Analysis & Feature Selection

4. Classifier training

5. Online classification

A satisfactory set of classification features has been found!

→ Generate a ready-to-use OpenViBE (soon TimeFlux) scenario

- trained classifier

- selected classif features

<figure><img src="image_49.png" /></figure>
## 1. Workspace creation/selection

* Create a new environment (file tree, configuration, data structures) for the experiment/analysis

* ... or select an already existing one.

* All user settings & manipulations are saved and available in the workspace for future usage or sharing:

* Extraction parameters (spectral analysis, list of processed files...)

* Training attempts (runs & features used, training accuracy...)

<figure><img src="image_50.png" /></figure>
# HappyFeat - How? (with openvibe)

## 1. Workspace creation/selection

## 2. MI Pipeline / “Feature type” selection

* Selecting btw. multiple “template” scenarios depending on the use case (power spectrum, connectivity type...)

* Edit basic/common parameters (acquisition, extraction, training...)

<figure><img src="image_51.png" /></figure>
1. Workspace creation/selection

2. MI Pipeline / “Feature type” selection

3. Feature Extraction

* Select signal files, and extraction parameters (lengths and overlap of windows, FFT size...)

* Run the generated extraction OpenViBE scenario (in the background) for all selected signal files:
  → Extract metadata (sampling freq, electrodes...)

→ Cut the signal to regions of interest (MI trials & baseline portions), generate CSV file with only these chunks (for the training step)

→ Apply a signal processing pipeline (PSD computation, connectivity measure...) to the signal chunks of interest, generate CSV files for future analysis

* Runs in an autonomous thread:
  You can do visualizations and training attempts for signals already processed in the meantime.

<figure><img src="image_52.png" /></figure>
1. Workspace creation/selection

2. MI Pipeline / “Feature type” selection

3. Feature Extraction

4. Analysis, Feature Selection

* Select one or multiple signals & load their spectral/connectivity data (CSV work files generated during “Feature Extraction”)

* Use different Visualization Tools to help find & select features of interest (FOIs) for training

* Frequency/channel R² map

* PSD (or connect. metric) comparison btw. the 2 conditions (MI/REST) for a given electrode

* Time/frequency ERD/ERS analysis for each condition

* R² mapped as a brain topography for a given frequency (or range)

## Combine as many visualization windows as you need

A “Dual metric” pipeline allows to show (for ex.) R² maps for both Power Spectrum and Connectivity in parallel

<figure><img src="image_53.png" /></figure>
1. Workspace creation/selection

2. MI Pipeline / “Feature type” selection

3. Feature Extraction

4. Analysis, Feature Selection

5. Classifier Training

* Set one or more **Features of Interest** (Electrode/Freq.)

* Select file(s) with which you want to train your classifier
  If > 1 file: their trials are automatically concatenated

* Run the Classifier Training scenario
  (Auto. generated in step 1 + auto. edited with FOIs)
  → Classification ACCURACY + WEIGHTS

**Disappointing results?** ("My accuracy is 50% 😞")

Maybe try again with other features. It only takes a few seconds...

**Satisfying results?!** ("OMG 95%"!)

Good news! The “Online Classification” scenario has already been automatically updated with:

- Classifier training weights

- Features of interest used for training

<figure><img src="image_54.png" /></figure>

Training accuracy “score”

+ Classifier Weights

+ Online scenario updated
1. Context

- What is BCI? What are we looking for?

- Feature-based classification vs. BCI variabilities

2. HappyFeat

- Key points

- Example through typical use-case

3. Perspectives & conclusion
* Current limitations

* Only one type of classification algo. proposed (LDA)

* Pipelines are “fixed”:

- trading OpenViBE's high level of flexibility...

+ ...for a high comfort of pipeline settings customization

+ & “trial-and-error” workflow

* Only three types of pipelines/feature types:

* Power Spectrum Density

* Connectivity (coherence & its variants)

* Dual (mixing PSD & Connectivity)

→ In project (long term!) for more “prospective power”:
    allow the user to choose btw. 1 and 3 feature-types & network metrics
    to mix as they see fit (MSC/node strength + Imag(Coh)/Laterality + ...)

<figure><img src="image_55.png" /></figure>

<figure><img src="image_56.png" /></figure>

<figure><img src="image_57.png" /></figure>
* Future works

• Fully autonomous 100% Python version, without OpenViBE (in progress: T. Venot)

- No acquisition/online possibilities

- Obviously slower... (no C++ optimizations!)

+ More portable, all types of platforms supported (MacOS!)

+ Other (more flexible) formats for I/O and work files (CSV, EDF...)

+ Directly use great OS packages (MNE, scikit-learn...)

+ Research oriented app → integrating NERV's works on graphs
* Future works

• 2024 internship in progress: allowing other BCI softwares: Timeflux

– Slower... (no C++ optimizations!)

+ More portable, all types of platforms

+ TimeFlux also 100% Python, easier communication between softwares

• Tutorials using MOABB datasets

• General GUI improvements

« Click & play » philosophy for feature selection
## Version 0.2.0 (since June 2024!)

## PyPi package / github repo

Testers, feedback & contributions are welcome (via github issues and PR...)

Full list of dependencies:

* Python 3.9

* shutils>=0.1.0

* Mne==1.5.1

* Numpy==1.26.4

* Pandas==2.2.1

* PySide2==5.15.2.1

* statsmodels>=0.13.1

* scipy>=1.7.1

* spectrum>=0.8.0

* Matplotlib>=3.8.4

* OpenViBE v3.6.0
## Available online:

github: https://github.com/Inria-NERV/happyFeat

PyPI: https://pypi.org/project/happyfeat/

documentation: https://happyfeat.readthedocs.io/

## To be continued...

* `+ +` flexibility (pipeline options, mixing metrics...)

* `+ +` network metrics (based on connectivity) (and associated viz. tools)

* `+ +` classification algorithms options

* Fully autonomous Python version, “sandbox” for research purposes

* Support for other BCI SW (starting with TimeFlux)

* GUI & general interactivity improvement

Stay tuned!
<figure><img src="image_58.png" /></figure>

<figure><img src="image_59.png" /></figure>

github.com/Inria-NERV/happyFeat

<figure><img src="image_60.png" /></figure>

https://doi.org/10.1016/j.simpa.2023.100610

<figure><img src="image_61.png" /></figure>

Thanks for your attention!