# HappyFeat-An interactive and efficient BCI framework for clinical applications

Arthur Desbois, Tristan Venot, Fabrizio de Vico Fallani, Marie-Constance

Corsi

► To cite this version:

Arthur Desbois, Tristan Venot, Fabrizio de Vico Fallani, Marie-Constance Corsi. HappyFeat-An in-teractive and efficient BCI framework for clinical applications. Software Impacts, 2024, 19, pp.100610. 10.1016/j.simpa.2023.100610. hal-04343247

HAL Id: hal-04343247

https://hal.science/hal-04343247v1

Submitted on 13 Dec 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# HappyFeat - An interactive and efficient BCI framework for clinical applications

Arthur Desbois¹, Tristan Venot¹, Fabrizio De Vico Fallani¹, Marie-Constance Corsi¹,*

¹ Sorbonne Universite, Paris Brain Institute - ICM, CNRS, Inria, Inserm, AP-HP, Hopital Pitie Salpetriere, F-75013, Paris, France

* Corresponding authors: arthur.desbois@gmail.com; marie-constance.corsi@inria.fr

## Abstract

Brain-Computer Interface (BCI) systems allow to perform actions by translating brain activity into commands. Such systems require training a classification algorithm to discriminate between mental states, using specific features from the brain signals. This step is crucial and presents specific constraints in clinical contexts.

HappyFeat is an open-source software making BCI experiments easier in such contexts: effortlessly extracting and selecting adequate features for training, in a single GUI. Novel features based on Functional Connectivity can be used, allowing graph-oriented approaches. We describe HappyFeat's mechanisms, showing its performances in typical use cases, and showcasing how to compare different types of features.

**Keywords** - BCI, Signal Processing, Classification, Motor Imagery, Functional Connectivity, Brain Networks
Introduction

Brain-Computer Interface (BCI) systems allow to transcribe brain signals into commands. For this purpose, classification algorithms are used to discriminate between different mental states [1]. The field of possible applications is vast, ranging from communication to prosthesis control and post-stroke rehabilitation[2]. Multiple BCI paradigms exist, such as P300 and Steady-State Visual Evoked Potentials (SSVEP)[3]. We choose to focus on Motor Imagery (MI), as it is prominent in prospective therapeutic usages (e.g. post-stroke rehabilitation[4]), which aligns with our current research efforts aiming to improve the use BCI in clinical settings. In MI, the subject actively imagines a movement without actually performing it, in order to command a virtual or real device (e.g. moving an object on a screen, controlling a robotic arm). Consequently, MI offers a high level of interactivity and requires a strong active implication from the subject.

MI protocols consists of multiple phases[5][6] (illustrated in Figure 1): an acquisition phase of training data; an offline analysis phase usually including pre-processing, extraction of features of interest (based on e.g. power spectra, functional connectivity), feature selection and classification algorithm training; a closed-loop online BCI usage using the trained classification algorithm.

The performance of a BCI system, while dependent on internal (e.g. concentration, fatigue[7] and ease with BCI) and external factors (e.g. montage of electrodes), is strongly linked to the correct training of the classification algorithm. Therefore, the choice of adequate features that capture the user's intent is crucial.

The offline analysis phase, leading to the choice of those features, should be made as short and efficient as possible for two reasons:

* A complete MI experiment is a long and strenuous process lasting multiple hours, along which the subject's concentration and motivation can drop. Patients' conditions increase the need for a fast feature selection process.

* Too long a time between the different phases may result in low classification accuracy, and therefore poor BCI performance. Indeed, as time passes, many physical parameters may change: the impedance or localization of Electroencephalographic (EEG) sensors (leading to change in EEG signals characteristics and quality), the subject's motivation, but also their mental activity itself. These changes mean that at the time of the final online classification phase, the subject's EEG signals and brain activity might not match with the ones used by the BCI experimenter to select features and train the classifier anymore.

Therefore, there is a crucial need for providing assistance to the experimenter during the training phase, to identify the most relevant and robust features.

Software solutions already exist today to manipulate BCI systems, or to analyze acquired EEG data. For creating and manipulating complete BCI systems, one can cite OpenViBE[8], which offers compatibility with a wide range of EEG hardware, practical modular tools to edit BCI systems as "scenarios", and powerful signal processing capabilities. It also allows interfacing with Virtual or Augmented Reality headsets and other virtual environments such as video games. Timeflux[9] is another software solution, allowing experimenters to acquire and process EEG signals in real-time, with a high degree of scenario and interface flexibility. Another existing solution is BCI2000[10], which allows a lower level of customization and modularity. However, none of the aforementioned solutions provide tools to interactively identify the best features to use for training and fine-tuning a BCI system.

Tools for offline analysis of EEG signals also exist today, such as Brainstorm [11], EegLab[12] or FieldTrip[13]. While these software embark wide selections of statistical and signal processing methods and are useful for research on neurophysiological phenomena, they lack capabilities in the domain of machine-learning, and the possibility to interface with BCI in a direct workflow.

Finally, the open-source software package MNE-Python[14] provides extensive functionalities allowing experimenters to manipulate EEG data and analyze them with signal-processing and machine-learning methods using scikit-learn[15], some of which **HappyFeat** makes use of.
# Description and features

**HappyFeat** is a Python framework, consisting in a main application and various scripts and automation routines, allowing to facilitate the use of MI-based BCI pipelines. This is achieved by grouping all necessary manipulations and parameters in a unified graphical user interface (GUI), and making the steps of feature extraction, feature selection and classifier training as effortless and seamless as possible, so that experimenters may focus on building an efficient BCI.

The following contains a description of HappyFeat's main features and mechanisms.

## Key features

* HappyFeat is a **software assistant for feature extraction and selection**. It proposes an exploration-oriented workflow, where experimenters can extract, visualize and select Features of Interest (FOI) for training as many times as needed, in a short time, until a satisfying classification training accuracy is reached.

* Every operation from signal loading and feature extraction to classifier training is handled from a **unified, dashboard-like GUI**, removing the need to use different softwares for data acquisition, feature analysis, classifier training and online classification, and to manage data formatting across the different environments.

* Along with the commonly used **Power Spectral Density (PSD)**, HappyFeat enables to work with **Functional Connectivity**, allowing to use **novel network-based approaches** based on recent research [16][17]. This approach is illustrated in Figure 2)

* HappyFeat uses **OpenViBE** in the background for the extraction and training parts, as a fast and efficient processing engine, taking advantage of its optimized C++ implementation of signal processing methods (notably using the Eigen<fnref n="1" /> library). The generation and manipulation of OpenViBE scenarios is entirely **automated** via scripts and templates, removing the inherent risk of mistakes in a time constrained environment. Feature visualization and selection use tools from MNE-Python[14] and scipy[18]. The GUIs are built with PyQt<fnref n="2" />.

* HappyFeat puts the emphasis on **reproducibility**, by keeping track of all manipulations (EEG sessions file lists, signal processing steps and parameters, classification attempts) and allowing to save, load and export previous work.

Two main use-cases are targeted:

* Using MI in a clinical setting (e.g. stroke rehabilitation), by greatly reducing the risks of mistakes during the offline analysis and the time needed to perform this step, quickly bridging the gap between EEG data acquisition and online BCI usage.

* Exploring new, alternative metrics for discriminating between mental states. To this aim, prototypes for prospective methods need to be validated on signal databases, before moving on to experimental conditions. HappyFeat helps bridging this gap, and provides a framework in which such methods can be tested, after implementation.

## Mechanisms

HappyFeat's main GUI and mechanisms are shown in Figure 3

* HappyFeat allows the experimenter to choose between different metrics for discriminating between mental states, such as Power Spectral Density (PSD) [19] or network estimators based on Functional Connectivity (e.g. Node Strength) [20, 17, 21], both of which are described in the Annex section. Experimenters can also use a mix of two different estimators, allowing to classify using both PSD and Node-strength features for example.

<footnote n="1">¹https://eigen.tuxfamily.org/index.php?title=Main_Page</footnote>

<footnote n="2">²https://docovi.io/qtforpython-5/</footnote>
*   *Data loading:* Experimenters can select different EEG recordings (either directly after acquisition during a BCI experiment, or using pre-recorded signals) from which to compute metrics and extract features. Parameters relevant to the chosen classification metric can be edited. OpenViBE scenarios are automatically created, updated and ran in the background using relevant information provided by the experimenter, without needing any additional manipulation.

*   *Visualization tools:* allow to analyze and select features of interest, accumulating statistics across selected EEG runs. Such figures allow for a comfortable and easy selection of adequate features for training the classification algorithm. Experimenters may open any number of visualization windows, allowing to compare $R^2$ values between MI conditions (e.g. *MI* vs *REST*) as a channel-frequency map, as power densities for a given channel, as a topography map for a given frequency range, etc. Examples of such visualizations are given in Figures 4 and 5.

*   *Classifier training:* can be done iteratively in a trial-and-error way. Using a set of EEG runs (from which features have been extracted previously in *HappyFeat*) selected by the experimenter, and FOIs selected in the previous step, a classification algorithm (such as Linear Discriminant Analysis (LDA)[22]) is trained in a few seconds, and the application provides a training accuracy score.

At this point, if the training accuracy score is satisfactory, experimenters can proceed with the last step of the experiment (i.e. the closed-loop online BCI), using an OpenViBE scenario automatically updated with the trained classifier and the selected features. In the case of an insufficient score, experimenters may simply go back to the previous steps, either directly trying other features to try training the classifier again; or modify their visualizations to select other features; or extract features from other EEG signals. Going back and forth between these steps only takes a few seconds, and manipulations are limited to the strict minimum.

*   A *Session & Settings Management System* allows to export and import "workspaces", allowing experimenters to keep track of manipulations previously done on EEG recording sessions: extraction results and corresponding parameters, training attempts and accuracy scores with the corresponding feature set, etc.

## Example of metric comparison

The common metric used in MI protocols to discriminate between mental tasks is the Power Spectral Density (PSD), which is available in *HappyFeat*. Alternative metrics based on functional connectivity[17, 21] are proposed. As the raw connectivity matrices are difficult to interpret to physically store, having a dimension of (*nb channels* × *nb channels* × *nb frequencies*), the metrics proposed in *HappyFeat* are network metrics, such as the *node-strength*, obtained by summing the weights of all connections for each channel. The resulting matrix of node-strengths has a dimension of (*nb channels* × *nb frequencies*), which can be analyzed using the same tools as the PSD. Similarly to the PSD, sets of (channel, frequency) can be used as features for training the classification algorithm. *HappyFeat* also proposes to use a mix of different metrics (i.e. PSD and Node-Strength) to train the classifier.

To illustrate this, two subjects from Venot et al.'s BRACCIO protocol[23] were selected (both male, aged 25, right-handed). In this protocol, subjects were asked to perform either MI of the right hand, or no MI ("Rest"), with visual feedback in the form of a moving robotic arm. 120 trials (60 per class) were used.

Table 1 gives the accuracies obtained for both subjects when training a classifier with particular features, using PSD, coherence-based node strength (NS-COH), imaginary part of coherence-based node strength (NS-iCOH) and mixing metrics.

Figures 4 and 5 show the channel-frequency $R^2$ maps and metric comparison for Subject 1, obtained with *HappyFeat*'s visualization tools.

A high inter-subject variability can be observed. Subject 1 shows similar performances between metrics, with a slight advantage for PSD. Subject 2 illustrates how using NS-COH, NS-iCOH and mixing them with PSD can lead to performance improvement.
<table><caption>Table 1: Comparison of training accuracies when using connectivity-based features.</caption><thead><tr><th>Metric</th><th>Feature (channel;frequency)</th><th>Training Accuracy (%)</th></tr></thead><tbody><tr><td><strong>Subject 1</strong></td><td></td><td></td></tr><tr><td>PSD</td><td>C3;13</td><td>91.67</td></tr><tr><td>NS-COH</td><td>C3;10</td><td>81.67</td></tr><tr><td>PSD + NS-COH</td><td>C3;13 + C3;10</td><td>86.67</td></tr><tr><td>NS-iCOH</td><td>C1;11</td><td>80.00</td></tr><tr><td>PSD + NS-iCOH</td><td>C3;13 + C1;11</td><td>91.65</td></tr><tr><td><strong>Subject 2</strong></td><td></td><td></td></tr><tr><td>PSD</td><td>C3;11</td><td>83.35</td></tr><tr><td>NS-COH</td><td>C1;12</td><td>96.65</td></tr><tr><td>PSD + NS-COH</td><td>C3;11 + C1;12</td><td>93.35</td></tr><tr><td>NS-iCOH</td><td>P5;11</td><td>80.00</td></tr><tr><td>PSD + NS-iCOH</td><td>C3;11 + P5;11</td><td>88.35</td></tr></tbody></table>

## Impacts

*HappyFeat* offers a novel complete integrated workflow, allowing to perform all steps of the offline analysis leading to feature selection in BCI setup. Two main use-cases are targeted, whose impacts are methodological and scientific on one side, and clinical and therapeutic on the other.

* On the one hand, *HappyFeat*'s potential impact on research using BCI is noteworthy. Novel and innovative algorithms (e.g. based on graph-theory and functional connectivity[17, 21]) can be validated on pre-recorded data, and compared to one another or to state of the art techniques (PSD). *HappyFeat* provides an efficient framework in which new methods for discriminating between mental states can be tested and benchmarked.

* On the other hand, *HappyFeat* helps using MI in a clinical setting (e.g. stroke rehabilitation), by greatly reducing the risks of mistakes during the offline analysis and the time needed to perform this step, quickly bridging the gap between EEG data acquisition and online BCI usage. *HappyFeat*'s mechanisms and graphical interface have been designed to be easily used by experimenters and clinicians without strong programming skills, in order to facilitate the introduction of BCI methods in healthcare. By making therapeutic protocols using BCI feasible and realistically applicable in real-life, *HappyFeat* aims to facilitate the design of innovative training programs to improve neuro-rehabilitation, in order to improve patients' quality of life in the long run.

*HappyFeat* has been instrumental in a real-life experimental study using BCI to control a robotic arm[24][23], making the acquisition, analysis and BCI steps feasible in the same session, and allowing in a further study on recorded signals to compare training performance of PSD and Spectral Coherence. Such a comparison is illustrated in *figure 4*, *figure 5* and *table 1* with two subjects from this study.

As another example of real-life impact, in the context of the BCINET research project[25], *HappyFeat* will serve as the cornerstone software for feature analysis with Functional Connectivity. This protocol aims to evaluate training effects and recovery in stroke patients, using BCI and non-invasive stimulation techniques.

*HappyFeat*'s workspace management system allows for reproducible research, by enabling to import or share work sessions between different users. The flexibility offered by metric comparison and the open-source nature of the project will help to develop the use of BCI in multiple research domains and applications not limited to healthcare, and as a pedagogic tools for users new to BCI.
# Limitations & Perspectives

## Flexibility

In order to offer a safe and risk-free workflow, we chose to build *HappyFeat* around the concept of fixed pipelines, trading OpenViBE's high level of flexibility in designing BCI systems for fixed, efficient pipelines with reproducible results. Nevertheless, even though HappyFeat is designed as a turnkey software solution, more experienced experimenters are free to modify scenarios run along the pipeline and templates used to generate them if the need arises, for example to fine-tune the signal processing chain in the feature extraction step, or to edit the type of feedback provided in the online BCI scenario. An in-depth guide helping to edit *HappyFeat*'s pipelines is available in the software documentation.

## Proposed algorithms and methods

At the time of writing, the only machine-learning algorithm proposed for the classification step is *Linear Discriminant Analysis* (LDA). Other methods shall be made available in the future, such as *Support Vector Machine* (SVM) or *Riemannian Geometry* based methods[1].

Regarding discriminant metrics based on network and graph theory, the only method available is the node strength, calculated from the coherence-based connectivity matrix. Other network-based metrics will soon be available, such as network laterality or betweenness centrality[16]. Moreover, *HappyFeat* proposes to mix different metrics, but at the time of writing only PSD + NS-COH or PSD + NS-iCOH may be chosen. A mechanism allowing to mix NS-COH + NS-iCOH or any combination of future implemented metrics will be proposed in a later version of the software.

In practice, feature selection may be realized manually or using automatic method such as Common Spatial Pattern (CSP)[22]. However, while such methods offer simplicity and speed, they imply reducing control over the selection process, and a reduced level of interpretability, both of which are crucial when testing prospective feature such as connectivity-based metrics. The workflow proposed in HappyFeat is therefore a trade-off between speed, ease-of-use and human interpretability. It should be noted that such automatic methods could still be proposed in the interface in the future.

A time-frequency ERD/ERS analysis tool is provided in the visualization part of the GUI, but only allows to compare the averaged spectra of trial against a "baseline" defined as the EEG signal acquired right before the stimulation cue (i.e., MI trial vs. MI baseline, and REST trial vs. REST baseline). This tool will be improved in a future version to allow comparing conditions between themselves (i.e. REST vs. MI), and to be based on Morlet wavelets[26].

## Processing engine, BCI software dependency

At the time of writing, *HappyFeat* is built upon the manipulation of OpenViBE scenarios, taking advantage of this software's high level of modularity and its signal processing capabilities. However, work is ongoing to show that other modular BCI software can be used as processing engines (such as Timeflux[9]), without modifying the mechanisms of HappyFeat.

## Conclusions

We propose a software to facilitate usage of MI-based BCI for multiple types of experimenters: clinicians often without technical or programming background, and researchers whose focus is more targeted on exploring new features. By helping experimenters manage reproducible pipelines, and by reducing the time and effort necessary to select adequate features for classification, *HappyFeat* fills a gap in the BCI world between offline analysis of neurophysiological phenomena using pre-recorded signals, and live BCI applications.

*HappyFeat* was designed as an open-source project and its usage can be acknowledged by citing this article. *HappyFeat* is continuously updated with new features and regularly maintained. Suggestions of improvements, as well as further developments, can be addressed to the corresponding authors of this article.
Acknowledgements & Funding

The research leading to these results has received funding from the program "Investissements d'avenir" ANR-10-IAIHU-06. The authors acknowledge support from European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement No. 864729). The content is solely the responsibility of the authors and does not necessarily represent the official views of any of the funding agencies.
References

[1] F. Lotte, L. Bougrain, A. Cichocki, M. Clerc, M. Congedo, A. Rakotomamonjy, F. Yger, A review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update, Journal of Neural Engineering 15 (3) (2018) 031005. doi:10.1088/1741-2552/aab2f2.
URL https://doi.org/10.1088%2F1741-2552%2Faab2f2

[2] G. Pfurtscheller, C. Neuper, Future prospects of ERD/ERS in the context of brain-computer interface (BCI) developments, Progress in Brain Research 159 (2006) 433–437. doi:10.1016/S0079-6123(06)59028-4.

[3] F. Lotte, L. Bougrain, M. Clerc, Electroencephalography (EEG)-Based Brain-Computer Interfaces, in: Wiley Encyclopedia of Electrical and Electronics Engineering, John Wiley & Sons, Ltd, 2015, pp. 1–20, eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/047134608X.W8278.
doi:10.1002/047134608X.W8278.
URL https://onlinelibrary.wiley.com/doi/abs/10.1002/047134608X.W8278

[4] F. Pichiorri, N. Mrachacz-Kersting, M. Molinari, S. Kleih, A. Kübler, D. Mattia, Brain-computer interface based motor and cognitive rehabilitation after stroke – state of the art, opportunity, and barriers: summary of the BCI Meeting 2016 in Asilomar, Brain-Computer Interfaces 4 (1-2) (2017) 53–59, publisher: Taylor & Francis eprint: https://doi.org/10.1080/2326263X.2016.1246328. doi:10.1080/2326263X.2016.1246328.
URL https://doi.org/10.1080/2326263X.2016.1246328

[5] G. Pfurtscheller, C. Neuper, Motor imagery and direct brain-computer communication, Proceedings of the IEEE 89 (7) (2001) 1123–1134, conference Name: Proceedings of the IEEE.
doi:10.1109/5.939829.

[6] C. Jeunet, E. Jahanpour, F. Lotte, Why standard brain-computer interface (BCI) training protocols should be changed: an experimental study, Journal of Neural Engineering 13 (3) (2016) 036024. doi:10.1088/1741-2560/13/3/036024.
URL https://doi.org/10.1088%2F1741-2560%2F13%2F3%2F036024

[7] A. Myrden, T. Chau, Effects of user mental state on EEG-BCI performance, Frontiers in Human Neuroscience 9 (2015).
URL https://www.frontiersin.org/articles/10.3389/fnhum.2015.00308

[8] Y. Renard, F. Lotte, G. Gibert, M. Congedo, E. Maby, V. Delannoy, O. Bertrand, A. Lécuyer, OpenViBE: An Open-Source Software Platform to Design, Test and Use Brain-Computer Interfaces in Real and Virtual Environments, Presence: Teleoperators and Virtual Environments 19 (1) (2010) 35–53.
URL https://hal.archives-ouvertes.fr/hal-00477153

[9] P. Clisson, R. Bertrand-Lalo, M. Congedo, G. Victor-Thomas, J. Chatel-Goldman, Timeflux: an open-source framework for the acquisition and near real-time processing of signal streams, in: BCI 2019 - 8th International Brain-Computer Interface Conference, Graz, Austria, 2019.
doi:10.3217/978-3-85125-682-6-17.
URL https://hal.archives-ouvertes.fr/hal-02315098

[10] G. Schalk, D. J. McFarland, T. Hinterberger, N. Birbaumer, J. R. Wolpaw, BCI2000: a general-purpose brain-computer interface (BCI) system, IEEE transactions on bio-medical engineering 51 (6) (2004) 1034–1043. doi:10.1109/TBME.2004.827072.

[11] F. Tadel, S. Baillet, J. Mosher, D. Pantazis, R. Leahy, Brainstorm: A User-Firendly Application for MEG/EEG Analysis, Computational Intelligence and Neuroscience 2011 (Jan. 2011). doi:10.1155/2011/879716.
URL http://dx.doi.org/10.1155/2011/879716
[12] A. Delorme, S. Makeig, EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis, Journal of Neuroscience Methods 134 (1) (2004) 9–21. doi:10.1016/j.jneumeth.2003.10.009.

[13] R. Oostenveld, P. Fries, E. Maris, J.-M. Schoffelen, R. Oostenveld, P. Fries, E. Maris, J.-M. Schoffelen, FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data, FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data, Computational Intelligence and Neuroscience, Computational Intelligence and Neuroscience 2011, 2011 (2010) e156869. doi:10.1155/2011/156869, 10.1155/2011/156869. URL http://www.hindawi.com/journals/cin/2011/156869/abs/, http://www.hindawi.com/journals/cin/2011/156869/abs/

[14] A. Gramfort, M. Luessi, E. Larson, D. A. Engemann, D. Strohmeier, C. Brodbeck, L. Parkkonen, M. S. Hämäläinen, MNE software for processing MEG and EEG data, NeuroImage 86 (2014) 446–460. doi:10.1016/j.neuroimage.2013.10.027.

[15] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine Learning in Python, Journal of Machine Learning Research 12 (85) (2011) 2825–2830. URL http://jmlr.org/papers/v12/pedregosa11a.html

[16] J. Gonzalez-Astudillo, T. Cattai, G. Bassignana, M.-C. Corsi, F. D. V. Fallani, Network-based brain–computer interfaces: principles and applications, Journal of Neural Engineering 18 (1) (2021) 011001, publisher: IOP Publishing. doi:10.1088/1741-2552/abc760. URL https://doi.org/10.1088/1741-2552/abc760

[17] T. Cattai, S. Colonnese, M.-C. Corsi, D. S. Bassett, G. Scarano, F. De Vico Fallani, Characterization of Mental States through Node Connectivity between Brain Signals, in: 2018 26th European Signal Processing Conference (EUSIPCO), 2018, pp. 1377–1381, iSSN: 2076-1465. doi:10.23919/EUSIPCO.2018.8553000.

[18] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, I. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, SciPy 1.0: fundamental algorithms for scientific computing in Python, Nature Methods 17 (3) (2020) 261–272, number: 3 Publisher: Nature Publishing Group. doi:10.1038/s41592-019-0686-2. URL https://www.nature.com/articles/s41592-019-0686-2

[19] P. F. Diez, E. Laciar, V. Mut, E. Avila, A. Torres, A comparative study of the performance of different spectral estimation methods for classification of mental tasks, Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference 2008 (2008) 1155–1158. doi: 10.1109/IEMBS.2008.4649366.

[20] G. Nolte, O. Bai, L. Wheaton, Z. Mari, S. Vorbach, M. Hallett, Identifying true brain interaction from EEG data using the imaginary part of coherency, Clinical Neurophysiology 115 (10) (2004) 2292–2307. doi:10.1016/j.clinph.2004.04.029.

[21] T. Cattai, S. Colonnese, M.-C. Corsi, D. S. Bassett, G. Scarano, F. De Vico Fallani, Phase/Amplitude Synchronization of Brain Signals During Motor Imagery BCI Tasks, IEEE
Transactions on Neural Systems and Rehabilitation Engineering 29 (2021) 1168–1177, conference Name: IEEE Transactions on Neural Systems and Rehabilitation Engineering. doi: 10.1109/TNSRE.2021.3088637.

[22] S. Sanei, Classification and Clustering of Brain Signals, in: Adaptive Processing of Brain Signals, John Wiley & Sons, Ltd, 2013, pp. 101–117, section: 6 eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781118622162.ch6. doi:10.1002/9781118622162.ch6 URL https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118622162.ch6

[23] T. Venot, A. Desbois, M.-C. Corsi, L. Hugueville, L. Saint-Bauzel, F. D. V. Fallani, Mental imagery timing affects hybrid BCI control of robotic arms, number: arXiv:2309.12195 arXiv:2309.12195 [q-bio] (Sep. 2023). doi:10.48550/arXiv.2309.12195. URL http://arxiv.org/abs/2309.12195

[24] T. Venot, A. Desbois, M. C. Corsi, L. Hugueville, L. Saint-Bauzel, F. De Vico Fallani, Exploring strategies for multimodal BCIs in an enriched environment, in: 2022 IEEE International Conference on Metrology for Extended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE), 2022, pp. 685–690. doi:10.1109/MetroXRAINE54828.2022.9967624.

[25] European research council (erc) funded project. "bcinet: Non-invasive decoding of brain communication patterns to ease motor restoration after stroke". doi:10.3030/864729. URL https://cordis.europa.eu/project/id/864729

[26] N. Brodu, F. Lotte, A. Lécuyer, Comparative study of band-power extraction techniques for Motor Imagery classification, in: 2011 IEEE Symposium on Computational Intelligence, Cognitive Algorithms, Mind, and Brain (CCMB), 2011, pp. 1–6. doi:10.1109/CCMB.2011.5952105. URL https://ieeexplore.ieee.org/document/5952105

[27] J. González Astudillo, Development of Network Features for Brain-Computer Interfaces, These de doctorat, Sorbonne université (Oct. 2022). URL https://www.theses.fr/2022SORUS286

[28] G. Pfurtscheller, F. H. Lopes da Silva, Event-related EEG/MEG synchronization and desynchronization: basic principles, Clinical Neurophysiology 110 (11) (1999) 1842–1857. doi: 10.1016/S1388-2457(99)00141-8. URL http://www.sciencedirect.com/science/article/pii/S1388245799001418

[29] H.-J. Hwang, S. Kim, S. Choi, C.-H. Im, EEG-Based Brain-Computer Interfaces: A Thorough Literature Survey, International Journal of Human-Computer Interaction 29 (12) (2013) 814–826, publisher: Taylor & Francis eprint: https://doi.org/10.1080/10447318.2013.780869. doi: 10.1080/10447318.2013.780869. URL https://doi.org/10.1080/10447318.2013.780869

[30] F. De Vico Fallani, J. Richiardi, M. Chavez, S. Achard, Graph analysis of functional brain networks: practical issues in translational neuroscience, Philosophical Transactions of the Royal Society B: Biological Sciences 369 (1653) (Oct. 2014). doi:10.1098/rstb.2013.0521. URL http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4150298/

[31] A. Schlögl, G. Supp, Analyzing event-related EEG data with multivariate autoregressive parameters, in: C. Neuper, W. Klimesch (Eds.), Progress in Brain Research, Vol. 159 of Event-Related Dynamics of Brain Oscillations, Elsevier, 2006, pp. 135–147. doi:10.1016/S0079-6123(06) 59009-0. URL https://www.sciencedirect.com/science/article/pii/S0079612306590090
Figures

<figure><img src="image_2.png" /><figcaption>Figure 1: Typical BCI protocol.</figcaption></figure>
<figure><img src="image_3.png" /><figcaption>Figure 2: **From brain activity to Functional Connectivity and Networks.** Spectral coherence (or imaginary coherence) between EEG signals from electrodes $n_i$ and $n_j$ is computed from their Power Spectral Densities and Cross Spectral Density. Applying this operation to every electrode pairs yields a connectivity matrix, visualized here at frequency $f$. From this matrix, network estimators can be extracted. The "Node strength" of a given node is obtained by summing all connections to this node. Computing the node strength for all frequencies in the considered range yields a "Node Strength Density" which can be analyzed and manipulated similarly to the usual Power Spectral Density. Courtesy of J. Gonzalez-Astudillo [27]</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Figure 3: **HappyFeat main GUI.** The interface is split in 3 parts: in the leftmost part, experimenters can perform *Feature Extraction* from recorded EEG signals available in the workspace. The central part is dedicated to *Feature Visualization* for signals that have undergone extraction, allowing to select adequate features for training. The last and rightmost part allows the experimenter to train the classifier using selected features and signal files. Along the way, certain parameters can be set, while other parts of the interface help to remind acquisition settings or last training attempts.</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>Figure 4: Comparison of MI conditions "Rest" and "MI" in Subject 1, using PSD. $R^2$ values represented as a frequency-sensor map (A), as a "brain topography" mapped on a scalp (at frequency 12Hz) (B), and direct comparison of PSD averaged over trials (at sensor C1) (C). The PSD is obtained using Burg's AR method. Figures A and B share the same color scale for $R^2$ values, and show desynchronization in C1, C3, CP3, CP5 sensors, in the alpha and low beta bands (from roughly 8Hz to 18HZ). Fig. C highlights this desynchronization with more precision in the 11 to 15Hz band. The black curve in C shows the $R^2$ values, scaled from 0 to 1.</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>Figure 5: Comparison of MI conditions "Rest" and "MI" in Subject 1, using NS-COH (top row) and NS-iCOH (bottom row). $R^2$ values represented as a frequency-sensor map (A, D), as a "brain topography" mapped on a scalp (B, E) at frequency 12Hz, and direct comparison of NS averaged over trials for sensor C3 (C) and C1 (F). Figures A and B share the same color scale for $R^2$ values, as do figures D and E. The black curves in C and F show the $R^2$ values, scaled from 0 to 1.</figcaption></figure>
# Annex

## Installation & Requirements

HappyFeat is available as a PyPI Python package: `python -m pip install happyfeat`. Alternatively, the source code can be downloaded at https://github.com/Inria-NERV/happyFeat. It requires Python 3.9 and OpenViBE 3.5.0, which can be downloaded at https://openvibe.inria.fr/. Further information can be found in the official documentation (https://happyfeat.readthedocs.io/) and in the README.md file at the top level of the repository.

## Classification metrics

Motor Imagery BCI is based on Event-Related Desynchronization/Synchronization (ERD/ERS)[28], which corresponds to a decrease/increase of signal power between different motor-related tasks in specific frequency bands, for EEG signals in the sensorimotor cortex. This phenomenon is observable in both motor execution and motor imagery.

The discrimination between mental states, and therefore between MI tasks, is done by using a classification algorithm, with adequately selected classification features. Here, we will describe two types of features: the first is the signal's Power Spectral Density (PSD), which is the most commonly used feature [29]; the second is Coherence-based Functional Connectivity[17].

## Power Spectral Density

The PSD of the recorded EEG signals is widely used as a discriminant feature in MI BCIs. One of the most common ways of estimating the PSD is by using Welch's method[19], which consists in splitting the signal into overlapping segments, applying a window function on the segments, computing the periodograms of the windowed segments (via a Discrete Fourier Transform), and finally averaging the squared magnitude of the individual periodograms.

As described in Diez et al.[19], Burg's method is more relevant in the context of EEG signals. Notably, it allows high frequency resolution for short data inputs, which makes more sense in the context of mental tasks lasting a few seconds. It implies fitting an autoregressive (AR) model to the signal, by minimizing the forward and backward prediction errors, then computing the DFT of the AR coefficients.

## Functional Connectivity

As an alternative to PSD, metrics based on Functional Connectivity (FC) have been studied in the last years. FC allows the observation of brain activity as a network of time-varying connections between areas. Algorithms and analysis from network and graph theory can be applied on the connectivity matrix.

In order to estimate the degree of interaction between electrodes, the simplest measure is the coherence, which can be seen as a generalization of correlation in the frequency domain.

With $S_{xy}(f)$ is the cross-spectrum of complex signals $x$ and $y$ at frequency $f$, and $S_{xx}(f)$ and $S_{yy}(f)$ the spectra of $x$ and $y$ at frequency $f$, we define coherence and its variants[20, 17, 21] as:

* Spectral Coherence:

$$COH_{xy}(f) = \frac{|S_{xy}(f)|}{\sqrt{S_{xx}(f) \cdot S_{yy}(f)}}$$

* Imaginary part of Coherence:

$$iCOH_{xy}(f) = \frac{|Im(S_{xy}(f))|}{\sqrt{S_{xx}(f) \cdot S_{yy}(f)}}$$

Computing one of those metrics yields a connectivity (or adjacency) matrix, with (nb channels × nb channels × nb frequencies) coefficients, which can also be seen as a weighted network. The information contained in this matrix can be exploited or re-formatted in many ways: for example summing all the weights associated to one edge (or channel) yields the node strength. Other metrics can be explored, such as for example laterality, centrality, or betweenness[30][16].
## Processing speed

One way of evaluating how *HappyFeat* can help in a BCI experiment is by measuring the processing times of each segment.

The *feature extraction* step is the most time consuming, and its processing time depends on the metric chosen, as computing Connectivity matrices using Autoregressive (AR) models[31] is more costly than PSDs, but also on the used parameters: the higher the AR model order, or the lower the time between two connectivity measurements, the costlier it gets. Of course, the number of channels has an important impact on processing times, linear for PSD, and quadratic for Connectivity.

Table 2 shows a summary of the measured times, using a computer with 3 GHz 4 core CPU (8 threads) and 32 GB of RAM. Signal and extraction parameters can be found in the same table. PSD estimation is performed with Burg's method[19], Node strength is computed from Connectivity matrices using Coherence (NS-COH)[17].

<table><caption>Table 2: HappyFeat's main operations processing times.</caption><thead><tr><th></th><th>Param 1</th><th>Param 2</th></tr></thead><tbody><tr><td>Full signal length</td><td>7min49 (469s)</td><td></td></tr><tr><td>Number of channels</td><td>64</td><td></td></tr><tr><td>Number of trials per class</td><td>20</td><td></td></tr><tr><td>Trial duration</td><td>3s</td><td></td></tr><tr><td>Sampling frequency</td><td>500Hz</td><td></td></tr><tr><td>AR model order</td><td>19</td><td></td></tr><tr><td>FFT Size</td><td>512</td><td></td></tr><tr><td>Estimation window length</td><td>0.25s</td><td>0.5s</td></tr><tr><td>Overlap btw. windows</td><td>36%</td><td>20%</td></tr><tr><th colspan="3">Processing times (s)</th></tr><tr><td></td><td>PSD</td><td>NS-COH</td></tr><tr><td>Feature extraction</td><td>13.50</td><td>50.50</td></tr><tr><td>Load data for visualization</td><td>7.70</td><td>10.60</td></tr><tr><td>Classifier Training</td><td>4.30</td><td>7.80</td></tr></tbody></table>