# The largest EEG-based BCI reproducibility study for open science: the MOABB benchmark

Sylvain Chevallier, Igor Carrara, Bruno Aristimunha, Pierre Guetschel, Sara Sedlar, Bruna Junqueira Lopes, Sébastien Velut, Salim Khazem, Thomas Moreau

<figure><img src="image_1.png" /></figure>

To cite this version:

Sylvain Chevallier, Igor Carrara, Bruno Aristimunha, Pierre Guetschel, Sara Sedlar, et al.. The largest EEG-based BCI reproducibility study for open science: the MOABB benchmark. 2024. hal-04537061

HAL Id: hal-04537061

https://universite-paris-saclay.hal.science/hal-04537061

Preprint submitted on 8 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The largest EEG-based BCI reproducibility study for open science: the MOABB benchmark

Sylvain Chevallier, Igor Carrara, Bruno Aristimunha, Pierre Guetschel, Sara Sedlar, Bruna Junqueira Lopes, Sébastien Velut, Salim Khazem, Thomas Moreau

► To cite this version:

Sylvain Chevallier, Igor Carrara, Bruno Aristimunha, Pierre Guetschel, Sara Sedlar, et al.. The largest EEG-based BCI reproducibility study for open science: the MOABB benchmark. 2024. hal-04537061

HAL Id: hal-04537061

https://universite-paris-saclay.hal.science/hal-04537061v1

Preprint submitted on 8 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The largest EEG-based BCI reproducibility study for open science: the MOABB benchmark

Sylvain Chevallier, Igor Carrara, Bruno Aristimunha, Pierre Guetschel, Sara Sedlar, Bruna Lopes, Sebastien Velut, Salim Khazem, Thomas Moreau

Inria TAU, LISN-CNRS, Université Paris-Saclay, 91405, Orsay, France

Université Côte d'Azur, Inria Cronos Team, Sophia Antipolis, France

Donders Institute, Radboud University, Nijmegen, Netherlands

University of São Paulo, São Paulo, Brazil

Federal University of ABC, Santo Andre, Brazil

GeorgiaTech-CNRS IRL 2958, Centralesupelec, Metz, France

Inria Mind team, Université Paris-Saclay, CEA, Palaiseau, 91120, France

E-mail: sylvain.chevallier@universite-paris-saclay.fr

## Abstract.

*Objective.* This study conduct an extensive Brain-computer interfaces (BCI) reproducibility analysis on open electroencephalography datasets, aiming to assess existing solutions and establish open and reproducible benchmarks for effective comparison within the field. The need for such benchmark lies in the rapid industrial progress that has given rise to undisclosed proprietary solutions. Furthermore, the scientific literature is dense, often featuring challenging-to-reproduce evaluations, making comparisons between existing approaches arduous. *Approach.* Within an open framework, 30 machine learning pipelines (separated into raw signal: 11, Riemannian: 13, deep learning: 6) are meticulously re-implemented and evaluated across 36 publicly available datasets, including motor imagery (14), P300 (15), and SSVEP (7). The analysis incorporates statistical meta-analysis techniques for results assessment, encompassing execution time and environmental impact considerations. *Main results.* The study yields principled and robust results applicable to various BCI paradigms, emphasizing motor imagery, P300, and SSVEP. Notably, Riemannian approaches utilizing spatial covariance matrices exhibit superior performance, underscoring the necessity for significant data volumes to achieve competitive outcomes with deep learning techniques. The comprehensive results are openly accessible, paving the way for future research to further enhance reproducibility in the BCI domain. *Significance.* The significance of this study lies in its contribution to establishing a rigorous and transparent benchmark for BCI research, offering insights into optimal methodologies and highlighting the importance of reproducibility in driving advancements within the field.

*Keywords:* Brain-computer interface, EEG, Reproducibility, Riemannian classifier, Deep learning

Submitted to: *J. Neural Eng.*
# The largest EEG-based BCI reproducibility study for open science: the MOABB benchmark

Sylvain Chevallier, Igor Carrara, Bruno Aristimunha, Pierre Guetschel, Sara Sedlar, Bruna Lopes, Sebastien Velut, Salim Khazem, Thomas Moreau

Inria TAU, LISN-CNRS, Université Paris-Saclay, 91405, Orsay, France

Université Côte d'Azur, Inria Cronos Team, Sophia Antipolis, France

Donders Institute, Radboud University, Nijmegen, Netherlands

University of São Paulo, São Paulo, Brazil

Federal University of ABC, Santo Andre, Brazil

GeorgiaTech-CNRS IRL 2958, Centralesupelec, Metz, France

Inria Mind team, Université Paris-Saclay, CEA, Palaiseau, 91120, France

E-mail: sylvain.chevallier@universite-paris-saclay.fr

## Abstract.

*Objective.* This study conduct an extensive Brain-computer interfaces (BCI) reproducibility analysis on open electroencephalography datasets, aiming to assess existing solutions and establish open and reproducible benchmarks for effective comparison within the field. The need for such benchmark lies in the rapid industrial progress that has given rise to undisclosed proprietary solutions. Furthermore, the scientific literature is dense, often featuring challenging-to-reproduce evaluations, making comparisons between existing approaches arduous. *Approach.* Within an open framework, 30 machine learning pipelines (separated into raw signal: 11, Riemannian: 13, deep learning: 6) are meticulously re-implemented and evaluated across 36 publicly available datasets, including motor imagery (14), P300 (15), and SSVEP (7). The analysis incorporates statistical meta-analysis techniques for results assessment, encompassing execution time and environmental impact considerations. *Main results.* The study yields principled and robust results applicable to various BCI paradigms, emphasizing motor imagery, P300, and SSVEP. Notably, Riemannian approaches utilizing spatial covariance matrices exhibit superior performance, underscoring the necessity for significant data volumes to achieve competitive outcomes with deep learning techniques. The comprehensive results are openly accessible, paving the way for future research to further enhance reproducibility in the BCI domain. *Significance.* The significance of this study lies in its contribution to establishing a rigorous and transparent benchmark for BCI research, offering insights into optimal methodologies and highlighting the importance of reproducibility in driving advancements within the field.

*Keywords:* Brain-computer interface, EEG, Reproducibility, Riemannian classifier, Deep learning

Submitted to: *J. Neural Eng.*
# 1. Introduction

The field of Brain-Computer Interface (BCI) aims at developing methodologies to allow interactions with devices, like prostheses or computer environments, from decoding brain signals. It is a very good candidate technology to assist people with motor disability, as it requires very limited motor capabilities of the subject. To go from brain signals to decision-making, BCI systems are defined by several choices: a paradigm, that specifies the cognitive tasks to perform to control the interface, an acquisition device, to record the brain activity, and an algorithmic pipeline, that processes the acquired data and predicts which action the subject intends to perform. As such, BCI is at the forefront of interdisciplinary research, integrating expertise from diverse fields such as electronics, neuroscience, human-machine interaction (HMI), signal processing, and machine learning.

## 1.1. Open data

To fasten the emergence of BCI system, the field has organized to decouple the lengthy paradigm design and data acquisition processes from the development of algorithmic pipelines to process them. The creation and publication of many accessible and openly available datasets allow for offline development and evaluation of novel processing pipelines. They also improve experiment replication and ensure the replicability of published results.

Due to the constraints on real-time acquisitions in open environments, Electroencephalography (EEG) has become the leading device to develop BCI systems with its high-frequency acquisition and limited constraints on its deployment. In EEG-based BCI, many real-world competitions and open datasets ex-

ist on a world scale to design and evaluate the best BCI systems. BCI datasets come with various paradigms, depending on the decoding task to perform from brain signals. Motor Imagery (MI) is a common paradigm choice to control a BCI, with different imagined movements, but other paradigms are also very efficient like Event Related Potential (ERP) generated by oddball stimulus or Steady State Visually Evoked Potential (SSVEP) produced by repetitive stimulations. Many open datasets are thus openly available for EEG-based BCI.

Yet, open data is not solely about availability. It also requires the use of open formats that enable easy reading and exploitation of metadata. Indeed, there exists a wide variety of EEG acquisition devices with various hardware specifications, as well as a plethora of paradigm specifications, with their experimental design, shared annotations, and code for interpreting the cued events. Retrieving this information is critical to developing valid systems that can correctly process the retrieved data. Drawing inspiration from brain imaging techniques like fMRI or MEG that rely on the Brain Imaging Data Structure (BIDS), an extension of this format has been proposed for EEG-based research [91]. Despite this proposed format, most EEG open data available online are stored in diverse structures and formats, which hinders the automation of data collection, as each dataset requires specific processing scripts.

Moreover, while numerous books on EEG data acquisition and BCI exist, it remains challenging to find well-founded guidelines for hardware requirements and BCI design. These guidelines are needed to allow practitioners to make informed decisions regarding experimental design and hardware selection. In the worst-case scenario, a poorly designed BCI or an inappropriate hardware choice can result
# 1. Introduction

The field of Brain-Computer Interface (BCI) aims at developing methodologies to allow interactions with devices, like prostheses or computer environments, from decoding brain signals. It is a very good candidate technology to assist people with motor disability, as it requires very limited motor capabilities of the subject. To go from brain signals to decision-making, BCI systems are defined by several choices: a paradigm, that specifies the cognitive tasks to perform to control the interface, an acquisition device, to record the brain activity, and an algorithmic pipeline, that processes the acquired data and predicts which action the subject intends to perform. As such, BCI is at the forefront of interdisciplinary research, integrating expertise from diverse fields such as electronics, neuroscience, human-machine interaction (HMI), signal processing, and machine learning.

## 1.1. Open data

To fasten the emergence of BCI system, the field has organized to decouple the lengthy paradigm design and data acquisition processes from the development of algorithmic pipelines to process them. The creation and publication of many accessible and openly available datasets allow for offline development and evaluation of novel processing pipelines. They also improve experiment replication and ensure the replicability of published results.

Due to the constraints on real-time acquisitions in open environments, Electroencephalography (EEG) has become the leading device to develop BCI systems with its high-frequency acquisition and limited constraints on its deployment. In EEG-based BCI, many real-world competitions and open datasets ex-

ist on a world scale to design and evaluate the best BCI systems. BCI datasets come with various paradigms, depending on the decoding task to perform from brain signals. Motor Imagery (MI) is a common paradigm choice to control a BCI, with different imagined movements, but other paradigms are also very efficient like Event Related Potential (ERP) generated by oddball stimulus or Steady State Visually Evoked Potential (SSVEP) produced by repetitive stimulations. Many open datasets are thus openly available for EEG-based BCI.

Yet, open data is not solely about availability. It also requires the use of open formats that enable easy reading and exploitation of metadata. Indeed, there exists a wide variety of EEG acquisition devices with various hardware specifications, as well as a plethora of paradigm specifications, with their experimental design, shared annotations, and code for interpreting the cued events. Retrieving this information is critical to developing valid systems that can correctly process the retrieved data. Drawing inspiration from brain imaging techniques like fMRI or MEG that rely on the Brain Imaging Data Structure (BIDS), an extension of this format has been proposed for EEG-based research [91]. Despite this proposed format, most EEG open data available online are stored in diverse structures and formats, which hinders the automation of data collection, as each dataset requires specific processing scripts.

Moreover, while numerous books on EEG data acquisition and BCI exist, it remains challenging to find well-founded guidelines for hardware requirements and BCI design. These guidelines are needed to allow practitioners to make informed decisions regarding experimental design and hardware selection. In the worst-case scenario, a poorly designed BCI or an inappropriate hardware choice can result
in unusable data. The unique characteristics of EEG-based BCIs could be guided by meta-analyses of existing datasets, leading to recommendations on the number and positioning of electrodes, sampling frequency, the required number of subject trials, and the influence of the number of classes. However, to the best of our knowledge, there is no comprehensive and systematic evaluation of these parameters available in the existing literature.

## 1.2. BCI pipelines

Based on these open datasets, the development of BCI classification pipelines for EEG signals has a long-standing research history and a very active community [75]. Pipelines initially focused on methods based on feature extraction from raw signals, such as channel variance or local variation [74], combined with complex classifiers. The first breakthrough came with the introduction of *spatial filtering* methods based on covariance estimation, such as XDAWN [95], CSP [2], or CCA [72]. These methods significantly improved classification accuracy when paired with simple classifiers. By using supervised filters, the separability of different BCI classes was improved while reducing the dimensionality of the input signal. As a result, these raw-signal-based pipelines with spatial filtering became the top-ranked approaches in BCI competitions. The second advancement in BCI classification involved a reformulation of spatial filtering based on covariance matrices using *Riemannian geometry* [12, 124]. Leveraging the inherent manifold structure of symmetric covariance matrices, this approach provides robust classification by exploiting the invariance properties of the covariance manifold. This method has demonstrated remarkable performance across various BCI tasks, even with noisy EEG signals,

making it the preferred choice and quickly outperforming other pipelines in BCI competitions [33]. Pipelines based on Riemannian geometry are considered state-of-the-art methods in the current literature.

Recently, *deep learning methods* have been explored for EEG-based BCI classification [99]. Although deep learning has enjoyed success in computer vision tasks, it is not straightforward to adapt these techniques to handle strongly correlated time series data, such as EEG signals. As a result, the deep learning approaches performances on EEG signals have not outperformed existing approaches [103]. Indeed, the main workhorse for the success of deep learning lies in the availability of vast amounts of data. However, in the context of BCI and EEG data, there is a scarcity of data at subject-level, which could potentially explain the limited performance of deep neural networks in this domain. To address this issue, incorporating auxiliary tasks such as self-supervised learning [7] or data augmentation [97] are promising approaches. However, further investigation is still required to explore the efficacy of these methods.

## 1.3. Evaluation and Reproducibility in BCI

While the literature on EEG-based BCI pipelines is very dense, their evaluation and the interpretation of the results is a major issue for BCI. Indeed, for many studies, it is very difficult, if not impossible to compare the produced results. This stems from the fact that various factors hinder the experimental result comparison: specific preprocessing, cherry-picking datasets, subjects, or classes on a dataset, missing pipelines in the comparison, or lack of statistical analysis.

On the one hand, this issue is compounded
in unusable data. The unique characteristics of EEG-based BCIs could be guided by meta-analyses of existing datasets, leading to recommendations on the number and positioning of electrodes, sampling frequency, the required number of subject trials, and the influence of the number of classes. However, to the best of our knowledge, there is no comprehensive and systematic evaluation of these parameters available in the existing literature.

## 1.2. BCI pipelines

Based on these open datasets, the development of BCI classification pipelines for EEG signals has a long-standing research history and a very active community [75]. Pipelines initially focused on methods based on feature extraction from raw signals, such as channel variance or local variation [74], combined with complex classifiers. The first breakthrough came with the introduction of *spatial filtering* methods based on covariance estimation, such as XDAWN [95], CSP [2], or CCA [72]. These methods significantly improved classification accuracy when paired with simple classifiers. By using supervised filters, the separability of different BCI classes was improved while reducing the dimensionality of the input signal. As a result, these raw-signal-based pipelines with spatial filtering became the top-ranked approaches in BCI competitions. The second advancement in BCI classification involved a reformulation of spatial filtering based on covariance matrices using *Riemannian geometry* [12, 124]. Leveraging the inherent manifold structure of symmetric covariance matrices, this approach provides robust classification by exploiting the invariance properties of the covariance manifold. This method has demonstrated remarkable performance across various BCI tasks, even with noisy EEG signals,

making it the preferred choice and quickly outperforming other pipelines in BCI competitions [33]. Pipelines based on Riemannian geometry are considered state-of-the-art methods in the current literature.

Recently, *deep learning methods* have been explored for EEG-based BCI classification [99]. Although deep learning has enjoyed success in computer vision tasks, it is not straightforward to adapt these techniques to handle strongly correlated time series data, such as EEG signals. As a result, the deep learning approaches performances on EEG signals have not outperformed existing approaches [103]. Indeed, the main workhorse for the success of deep learning lies in the availability of vast amounts of data. However, in the context of BCI and EEG data, there is a scarcity of data at subject-level, which could potentially explain the limited performance of deep neural networks in this domain. To address this issue, incorporating auxiliary tasks such as self-supervised learning [7] or data augmentation [97] are promising approaches. However, further investigation is still required to explore the efficacy of these methods.

## 1.3. Evaluation and Reproducibility in BCI

While the literature on EEG-based BCI pipelines is very dense, their evaluation and the interpretation of the results is a major issue for BCI. Indeed, for many studies, it is very difficult, if not impossible to compare the produced results. This stems from the fact that various factors hinder the experimental result comparison: specific preprocessing, cherry-picking datasets, subjects, or classes on a dataset, missing pipelines in the comparison, or lack of statistical analysis.

On the one hand, this issue is compounded
by the need for shared resources and comprehensive evaluations. Thorough evaluations often exceed the scope of a single research work, and it is thus crucial to find rigorous methods to assess the performance of BCI pipelines and ensure the reproducibility of their results. On the other hand, the scientific community has been grappling with a reproducibility crisis across various domains [6], and the field of BCI is not exempt. Addressing this crisis in BCI research is particularly pertinent due to the domain's unique requirements and complexities. BCI studies involve complex methodologies, intricate data acquisition techniques, and multifaceted analysis pipelines, making it challenging to replicate and validate experiments. The transdisciplinary nature of BCI research necessitates specialized knowledge from multiple disciplines, further complicating the ability to reproduce results.

Meta-analysis, a statistical technique for combining the results of multiple studies, is a powerful tool for synthesizing findings and deriving insights from a large body of research [48]. However, due to non-standardized evaluation protocols, conducting meta-analyses in the field of BCI has proven to be challenging. The variability in datasets, experimental tasks, and analysis pipelines inhibits the aggregation of results, despite efforts to establish standardization protocols. Consequently, it is difficult to obtain a comprehensive understanding of the performance of BCIs pipelines across different paradigms through the existing literature.

In response to these challenges, the field has seen the emergence of the Mother of All BCI Benchmarks (MOABB). MOABB [4] was developed as an open-source platform to facilitate the benchmarking and assessment of new datasets and classifiers in major BCI paradigms. The first version of MOABB initiated

a community-driven effort to define rigorous and expert methods for conducting proper benchmarking assessments. By providing standardized evaluation procedures, MOABB enables researchers to compare and evaluate the performance of BCI methods in a transparent and reproducible manner. While its adoption by the community is broadening, the systematic evaluation of existing pipelines proposed in [54] is now outdated because it is focused on a specific BCI modality, and new pipelines have emerged in the literature.

## 1.4. Environmental impact

Addressing climate change requires comprehensive actions across all facets of human societies to adhere to the Paris Climate Agreement. Research, including the field of AI, plays a significant role in achieving this objective. Assessing the environmental impact of machine learning techniques is a challenging task [71], which has been pioneered by the natural language processing community with the prominence of conversational agents [76]. While machine learning models used in BCI are smaller in scale, assessing the performance of various models in link with their environmental impact is critical to promoting virtuous and sustainable models.

The costs associated with deploying computers or computer clusters for training machine learning models are dependent on infrastructure requirements. The environmental impact resulting from energy consumption during model training varies based on geographical criteria, as electricity production methods differ across countries. Several libraries have been developed to provide estimates of this environmental impact, measured in terms of grams of CO2 equivalent emissions [53]. Although these libraries have limitations, they offer a valuable
by the need for shared resources and comprehensive evaluations. Thorough evaluations often exceed the scope of a single research work, and it is thus crucial to find rigorous methods to assess the performance of BCI pipelines and ensure the reproducibility of their results. On the other hand, the scientific community has been grappling with a reproducibility crisis across various domains [6], and the field of BCI is not exempt. Addressing this crisis in BCI research is particularly pertinent due to the domain's unique requirements and complexities. BCI studies involve complex methodologies, intricate data acquisition techniques, and multifaceted analysis pipelines, making it challenging to replicate and validate experiments. The transdisciplinary nature of BCI research necessitates specialized knowledge from multiple disciplines, further complicating the ability to reproduce results.

Meta-analysis, a statistical technique for combining the results of multiple studies, is a powerful tool for synthesizing findings and deriving insights from a large body of research [48]. However, due to non-standardized evaluation protocols, conducting meta-analyses in the field of BCI has proven to be challenging. The variability in datasets, experimental tasks, and analysis pipelines inhibits the aggregation of results, despite efforts to establish standardization protocols. Consequently, it is difficult to obtain a comprehensive understanding of the performance of BCIs pipelines across different paradigms through the existing literature.

In response to these challenges, the field has seen the emergence of the Mother of All BCI Benchmarks (MOABB). MOABB [4] was developed as an open-source platform to facilitate the benchmarking and assessment of new datasets and classifiers in major BCI paradigms. The first version of MOABB initiated

a community-driven effort to define rigorous and expert methods for conducting proper benchmarking assessments. By providing standardized evaluation procedures, MOABB enables researchers to compare and evaluate the performance of BCI methods in a transparent and reproducible manner. While its adoption by the community is broadening, the systematic evaluation of existing pipelines proposed in [54] is now outdated because it is focused on a specific BCI modality, and new pipelines have emerged in the literature.

### 1.4. Environmental impact

Addressing climate change requires comprehensive actions across all facets of human societies to adhere to the Paris Climate Agreement. Research, including the field of AI, plays a significant role in achieving this objective. Assessing the environmental impact of machine learning techniques is a challenging task [71], which has been pioneered by the natural language processing community with the prominence of conversational agents [76]. While machine learning models used in BCI are smaller in scale, assessing the performance of various models in link with their environmental impact is critical to promoting virtuous and sustainable models.

The costs associated with deploying computers or computer clusters for training machine learning models are dependent on infrastructure requirements. The environmental impact resulting from energy consumption during model training varies based on geographical criteria, as electricity production methods differ across countries. Several libraries have been developed to provide estimates of this environmental impact, measured in terms of grams of CO2 equivalent emissions [53]. Although these libraries have limitations, they offer a valuable
measurement to enhance our understanding of the training requirements for models. This facilitates a more comprehensive comparison of pipelines within a benchmark setting.

## 1.5. Contributions

This paper aims to address the lack of reproducibility studies in the field of EEG-based BCI and to go beyond a simple benchmark of existing machine learning pipelines. Indeed, it is important to provide an updated comparison of the previous benchmark [54], including deep learning pipelines. Also, new BCI paradigms for controlling systems could now be included in the evaluation to propose a global take on the current BCI state.

It is an impossible task to evaluate all openly available datasets and all published pipelines in the literature, as there are exotic or unreadable data formats, BCI protocols that are used for a unique dataset, and unavailable code. For this paper, we chose to restrict to three common BCI paradigms, namely MI, ERP, and SSVEP, that are well documented in the literature. The reason for this choice is to ensure reproducible evaluation with enough datasets and subjects to achieve decent statistical meta-analysis.

For the machine learning pipelines, we follow similar guidelines to consider only approaches that could be reimplemented with Python open-source libraries. Indeed, we could only reimplement part of the published pipelines, but we try to include to the best of our effort all pipelines that have been reused in several publications or that are often cited as reference. The MOABB framework is designed to allow seamless integration of novel pipelines for new contributors and facilitate as much as possible the reproducibility of a benchmark to compare a novel pipeline with the one

presented here.

We thought of the MOABB open science initiative as a long-term project. The objective is to endow the community with the ability to easily compare results on new datasets or new pipelines, that will be added after the paper publication. A website reproduces the results presented here and will be updated with novel additions. The goal is also to limit the environmental impact of research works by providing a simple means for scientists to copy/paste up-to-date results in their publication to compare with a reference benchmark.

Beyond the purely quantitative benchmark, it is difficult to have a good overview of the open EEG datasets available online. They have been recorded with different hardware, using similar experimental protocols with some variability in experiment design. As data sharing is becoming a common requirement in the scientific community, future works will often result in sharing new open data. It is important to be able to quickly identify existing datasets, and common design choices, to correctly position new experiments with well-informed knowledge. This task is difficult, as open data is scattered in the literature with no common format.

To summarize, in this paper we aim to address the following open questions:

* What is the most effective approach for classifying EEG? How do their computation time and energy consumption compare for training a model?

* What is the best deep learning method? The best Riemannian-BCI pipeline?

* How many trials or channels are required in a dataset to achieve correct performances?

* For MI, which motor imagery tasks give
measurement to enhance our understanding of the training requirements for models. This facilitates a more comprehensive comparison of pipelines within a benchmark setting.

## 1.5. Contributions

This paper aims to address the lack of reproducibility studies in the field of EEG-based BCI and to go beyond a simple benchmark of existing machine learning pipelines. Indeed, it is important to provide an updated comparison of the previous benchmark [54], including deep learning pipelines. Also, new BCI paradigms for controlling systems could now be included in the evaluation to propose a global take on the current BCI state.

It is an impossible task to evaluate all openly available datasets and all published pipelines in the literature, as there are exotic or unreadable data formats, BCI protocols that are used for a unique dataset, and unavailable code. For this paper, we chose to restrict to three common BCI paradigms, namely MI, ERP, and SSVEP, that are well documented in the literature. The reason for this choice is to ensure reproducible evaluation with enough datasets and subjects to achieve decent statistical meta-analysis.

For the machine learning pipelines, we follow similar guidelines to consider only approaches that could be reimplemented with Python open-source libraries. Indeed, we could only reimplement part of the published pipelines, but we try to include to the best of our effort all pipelines that have been reused in several publications or that are often cited as reference. The MOABB framework is designed to allow seamless integration of novel pipelines for new contributors and facilitate as much as possible the reproducibility of a benchmark to compare a novel pipeline with the one

presented here.

We thought of the MOABB open science initiative as a long-term project. The objective is to endow the community with the ability to easily compare results on new datasets or new pipelines, that will be added after the paper publication. A website reproduces the results presented here and will be updated with novel additions. The goal is also to limit the environmental impact of research works by providing a simple means for scientists to copy/paste up-to-date results in their publication to compare with a reference benchmark.

Beyond the purely quantitative benchmark, it is difficult to have a good overview of the open EEG datasets available online. They have been recorded with different hardware, using similar experimental protocols with some variability in experiment design. As data sharing is becoming a common requirement in the scientific community, future works will often result in sharing new open data. It is important to be able to quickly identify existing datasets, and common design choices, to correctly position new experiments with well-informed knowledge. This task is difficult, as open data is scattered in the literature with no common format.

To summarize, in this paper we aim to address the following open questions:

* What is the most effective approach for classifying EEG? How do their computation time and energy consumption compare for training a model?

* What is the best deep learning method? The best Riemannian-BCI pipeline?

* How many trials or channels are required in a dataset to achieve correct performances?

* For MI, which motor imagery tasks give
the best accuracy?

* What are the open datasets in different paradigms and how do they compare?

To investigate those research questions, we conducted many experiments built on open-source tools and with the help of a large community driven by open science guidelines. This paper describes the results obtained from a wide experimental campaign and their in-depth analysis, resulting in the following contributions:

* the largest benchmarking in EEG-based BCI in open science relying on open source libraries

* a fair and replicable evaluation with expert knowledge in BCI

* a deep analysis of the benchmark results, with guidelines for proposing new machine learning pipelines and new datasets or experiments.

## 2. Benchmark methodology

In this section, we describe the methodology for our benchmark, including which methods are considered, how they are trained and evaluated, and how we analyze the results.

### 2.1. Analysis pipelines inclusion

A major difficulty in the BCI literature, apart from the data and code availability, is the importance of signal processing in EEG. Many toolboxes are available, in different software platforms, like Matlab, Python, C/C++, Java, C#, Julia, Delphi, and many more. Some toolboxes are sold as commercial products, with undisclosed code or signal processing techniques that are hidden or obfuscated for intellectual property reason. The choices made in those toolboxes, let apart all single projects

maintained by only one person, are very different regarding signal filtering or electrode referencing and yield very different outcomes. Despite the fact that complex preprocessing pipelines, and often arguably overcomplicated ones, are detrimental for EEG interpretation and classification [37], most of the toolboxes include them and promote their application in tutorials and guidelines.

In this paper, our analysis pipeline relies on the Python language for its large adoption in the scientific computing, neuroscience and machine learning communities, supported by robust and extensively validated libraries such as numpy [47], scipy [118], MNE [43], scikit-learn [89] and pyriemann [14]. Manipulation of EEG recordings is facilitated through MNE, enabling the extraction of hardware events and the conversion of recordings into numpy arrays. As a light preprocessing, the signal is processed with a 4th-order Butterworth bandpass IIR filter, applied with a forward-backward pass, using standard MNE parametrization. The specific bandpass frequencies are subsequently provided as they depend on the chosen BCI paradigm. Machine learning pipelines are based on scikit-learn and pyriemann estimators.

### 2.2. Evaluation method

Another difficulty for anyone who wants to reproduce literature results is that reporting classification performances on EEG-based BCI tasks is not standardized. A crucial methodological consideration pertains to the selection of the evaluation metric (whether it be Area Under the Curve (AUC), precision/recall, F1-score, or accuracy), a decision that holds notable significance in the assessment of outcomes. Moreover, in many studies, the authors focus on specific subsets of subjects
the best accuracy?

* What are the open datasets in different paradigms and how do they compare?

To investigate those research questions, we conducted many experiments built on open-source tools and with the help of a large community driven by open science guidelines. This paper describes the results obtained from a wide experimental campaign and their in-depth analysis, resulting in the following contributions:

* the largest benchmarking in EEG-based BCI in open science relying on open source libraries

* a fair and replicable evaluation with expert knowledge in BCI

* a deep analysis of the benchmark results, with guidelines for proposing new machine learning pipelines and new datasets or experiments.

## 2. Benchmark methodology

In this section, we describe the methodology for our benchmark, including which methods are considered, how they are trained and evaluated, and how we analyze the results.

### 2.1. Analysis pipelines inclusion

A major difficulty in the BCI literature, apart from the data and code availability, is the importance of signal processing in EEG. Many toolboxes are available, in different software platforms, like Matlab, Python, C/C++, Java, C#, Julia, Delphi, and many more. Some toolboxes are sold as commercial products, with undisclosed code or signal processing techniques that are hidden or obfuscated for intellectual property reason. The choices made in those toolboxes, let apart all single projects

maintained by only one person, are very different regarding signal filtering or electrode referencing and yield very different outcomes. Despite the fact that complex preprocessing pipelines, and often arguably overcomplicated ones, are detrimental for EEG interpretation and classification [37], most of the toolboxes include them and promote their application in tutorials and guidelines.

In this paper, our analysis pipeline relies on the Python language for its large adoption in the scientific computing, neuroscience and machine learning communities, supported by robust and extensively validated libraries such as numpy [47], scipy [118], MNE [43], scikit-learn [89] and pyriemann [14]. Manipulation of EEG recordings is facilitated through MNE, enabling the extraction of hardware events and the conversion of recordings into numpy arrays. As a light preprocessing, the signal is processed with a 4th-order Butterworth bandpass IIR filter, applied with a forward-backward pass, using standard MNE parametrization. The specific bandpass frequencies are subsequently provided as they depend on the chosen BCI paradigm. Machine learning pipelines are based on scikit-learn and pyriemann estimators.

### 2.2. Evaluation method

Another difficulty for anyone who wants to reproduce literature results is that reporting classification performances on EEG-based BCI tasks is not standardized. A crucial methodological consideration pertains to the selection of the evaluation metric (whether it be Area Under the Curve (AUC), precision/recall, F1-score, or accuracy), a decision that holds notable significance in the assessment of outcomes. Moreover, in many studies, the authors focus on specific subsets of subjects
within established datasets, or selectively use a
restricted part of the cognitive tasks conducted
during experiments. These choices make com-
paring findings across papers becomes inher-
ently cumbersome.

To maintain consistent terminology con-
cerning EEG signals, we will establish the fol-
lowing definitions. A *session* refers to a series
of runs where EEG electrodes remain attached
to a subject's head, and the overarching experi-
mental parameters remain constant. The term
*run* denotes the period during which an ex-
periment is conducted without interruption or
pause. Within a session, several runs are per-
formed with potential intervals between them.
An *epoch* or a *trial* signifies a segment within
a run during which an atomic event occurs,
triggered by an external stimulus for event-
related potential or steady-state evoked po-
tential or internal volition for motor imagery.
These epochs are positioned in time relative to
an onset, which corresponds to the start of a
stimulus or a task.

In the BCI framework, different evalu-
ation methodologies exist for partitioning the
data into training and test datasets, each
tailored to address specific challenges. We
differentiate between *within-session* as shown
in Figure 1, *cross-session*, and *cross-subject*
evaluations, respectively illustrated in the an-
nexes in Figure A1 and Figure A2.

In the context of within-session evalu-
ation, the primary concern lies in identify-
ing algorithms that can effectively mitigate
overfitting within a single session. In line
with common practices in machine learning
for cross-validation, all trials from a session
are shuffled before being split in k-folds, to
evaluate the generalization performance on un-
seen epochs. Consequently, the pipelines are
trained on trials sampled throughout the ses-
sion duration, which helps mitigate the im-

pact of intra-session variability in an individ-
ual's EEG. While this allows a more statisti-
cally accurate benchmark of a pipeline, it pro-
vides an upper bound for the evaluation met-
rics when compared to online evaluation. De-
spite the difference with experimental applica-
tions of BCI system, this training methodology
is commonly employed in the existing litera-
ture [13, 111, 84, 75], influencing our decision
to adopt it for this reproducibility study.

In contrast, the cross-session (resp. cross-
subject) evaluation employs a leave-one-out
cross-validation technique, where only one
session (resp. subject) is designated as the
testing dataset. The results from cross-
session or cross-subject evaluation methods
put more emphasis on transfer learning to
cope with subjects’ variability. However, the
questions raised by transfer learning in a BCI
context [119, 121] are manyfold – dealing with

<figure><img src="image_3.png" /><figcaption>Figure 1: Within-session evaluation, small rectangles indicate a sample or EEG trial, pastel colors on the two top lines shows the chronological order, bright color on the last three lines indicates training and testing samples/trials.</figcaption></figure>
within established datasets, or selectively use a
restricted part of the cognitive tasks conducted
during experiments. These choices make com-
paring findings across papers becomes inher-
ently cumbersome.

To maintain consistent terminology con-
cerning EEG signals, we will establish the fol-
lowing definitions. A *session* refers to a series
of runs where EEG electrodes remain attached
to a subject’s head, and the overarching experi-
mental parameters remain constant. The term
*run* denotes the period during which an ex-
periment is conducted without interruption or
pause. Within a session, several runs are per-
formed with potential intervals between them.
An *epoch* or a *trial* signifies a segment within
a run during which an atomic event occurs,
triggered by an external stimulus for event-
related potential or steady-state evoked po-
tential or internal volition for motor imagery.
These epochs are positioned in time relative to
an onset, which corresponds to the start of a
stimulus or a task.

In the BCI framework, different evalu-
tion methodologies exist for partitioning the
data into training and test datasets, each
tailored to address specific challenges. We
differentiate between *within-session* as shown
in Figure 1, *cross-session*, and *cross-subject*
evaluations, respectively illustrated in the an-
nexes in Figure A1 and Figure A2.

In the context of within-session evalu-
ation, the primary concern lies in identify-
ing algorithms that can effectively mitigate
overfitting within a single session. In line
with common practices in machine learning
for cross-validation, all trials from a session
are shuffled before being split in k-folds, to
evaluate the generalization performance on un-
seen epochs. Consequently, the pipelines are
trained on trials sampled throughout the ses-
sion duration, which helps mitigate the im-

pact of intra-session variability in an individ-
ual's EEG. While this allows a more statisti-
cally accurate benchmark of a pipeline, it pro-
vides an upper bound for the evaluation met-
rics when compared to online evaluation. De-
spite the difference with experimental applica-
tions of BCI system, this training methodology
is commonly employed in the existing litera-
ture [13, 111, 84, 75], influencing our decision
to adopt it for this reproducibility study.

In contrast, the cross-session (resp. cross-
subject) evaluation employs a leave-one-out
cross-validation technique, where only one
session (resp. subject) is designated as the
testing dataset. The results from cross-
session or cross-subject evaluation methods
put more emphasis on transfer learning to
cope with subjects’ variability. However, the
questions raised by transfer learning in a BCI
context [119, 121] are manyfold – dealing with

<figure><img src="image_2.png" /><figcaption>Figure 1: Within-session evaluation, small rectangles indicate a sample or EEG trial, pastel colors on the two top lines shows the chronological order, bright color on the last three lines indicates training and testing samples/trials.</figcaption></figure>
subject alignment, training models for each subject or a single one, at a session, subject or dataset level – and are outside the scope of this article. For this reason, this paper focuses on the most common evaluation strategy in the literature, which is within-session evaluation.

Nonetheless, one should acknowledge that within-session evaluation has its limitations. It addresses only partly the complications stemming from variations in sessions and subjects. Such variations may arise from internal factors, like minor electrode displacements between sessions, different calibrations of EEG hardware, or external factors, like the dynamic nature of EEG measurements in an individual based on the cognitive states [98], such as alertness, drowsiness or fatigue.

Within-session evaluation of a pipeline is conducted for each subject and each session by splitting the epochs into training and test epoch sets using five stratified K-fold splits. In each fold, the testing set contains 20% of the session epochs while maintaining the class distribution. The final evaluation score corresponds to the average over the five splits. Depending on whether the classification problem is binary or involves more than two classes, evaluation scores correspond to the Area Under the Receiver Operating Characteristic Curve (ROC) or classification accuracy, respectively.

### 2.3. Grid search

Another important aspect in BCIs is how the hyperparameters are selected. They can significantly affect the performance of the algorithm and, consequently, the accuracy of the system's predictions. It is therefore crucial to employ a method that facilitates the search for optimal parameter values. Grid search stands out as a popular method for

hyperparameter tuning in machine learning algorithms. In particular, it is essential to search for the correct hyper-parameters for each scenario, considering variations in evaluation procedures, datasets, subjects, and sessions.

Using nested cross-validation for hyper-parameters selection, we mitigate the risk of overfitting, as recommended in existing literature [25]. This approach is implemented by using an inner 3-fold cross-validation. Additionally, we have devised tailored grid search functions for each evaluation procedure, as detailed in subsection Appendix A.1. This standardized framework streamlines hyperparameter tuning, enabling seamless performance comparison across diverse machine-learning models and promoting experiment reproducibility across varied datasets.

### 2.4. Statistical analysis

The statistical comparison of two different pipelines can be conducted either at a dataset-wise level or across multiple datasets.

The dataset-wise comparison involves assessing the statistical differences between two pipelines within the same dataset. This is done using effect sizes and p-values. To optimize computational efficiency, the number of subjects *N* in the dataset determines the method of estimating p-values. For datasets with *N* < 13, one-tailed permutation-based paired *t*-tests are used with all possible permutations. For datasets with 13 ≤ *N* ≤ 20, 10000 random permutations are employed. For datasets with *N* > 20, the Wilcoxon signed-rank test [122] is used. The effect size between two pipelines is measured via Standardized Mean Difference (SMD) [48] estimated over the subjects.

For multiple datasets, the comparison of
subject alignment, training models for each subject or a single one, at a session, subject or dataset level – and are outside the scope of this article. For this reason, this paper focuses on the most common evaluation strategy in the literature, which is within-session evaluation.

Nonetheless, one should acknowledge that within-session evaluation has its limitations. It addresses only partly the complications stemming from variations in sessions and subjects. Such variations may arise from internal factors, like minor electrode displacements between sessions, different calibrations of EEG hardware, or external factors, like the dynamic nature of EEG measurements in an individual based on the cognitive states [98], such as alertness, drowsiness or fatigue.

Within-session evaluation of a pipeline is conducted for each subject and each session by splitting the epochs into training and test epoch sets using five stratified K-fold splits. In each fold, the testing set contains 20% of the session epochs while maintaining the class distribution. The final evaluation score corresponds to the average over the five splits. Depending on whether the classification problem is binary or involves more than two classes, evaluation scores correspond to the Area Under the Receiver Operating Characteristic Curve (ROC) or classification accuracy, respectively.

### 2.3. Grid search

Another important aspect in BCIs is how the hyperparameters are selected. They can significantly affect the performance of the algorithm and, consequently, the accuracy of the system's predictions. It is therefore crucial to employ a method that facilitates the search for optimal parameter values. Grid search stands out as a popular method for

hyperparameter tuning in machine learning algorithms. In particular, it is essential to search for the correct hyper-parameters for each scenario, considering variations in evaluation procedures, datasets, subjects, and sessions.

Using nested cross-validation for hyper-parameters selection, we mitigate the risk of overfitting, as recommended in existing literature [25]. This approach is implemented by using an inner 3-fold cross-validation. Additionally, we have devised tailored grid search functions for each evaluation procedure, as detailed in subsection Appendix A.1. This standardized framework streamlines hyperparameter tuning, enabling seamless performance comparison across diverse machine-learning models and promoting experiment reproducibility across varied datasets.

### 2.4. Statistical analysis

The statistical comparison of two different pipelines can be conducted either at a dataset-wise level or across multiple datasets.

The dataset-wise comparison involves assessing the statistical differences between two pipelines within the same dataset. This is done using effect sizes and p-values. To optimize computational efficiency, the number of subjects *N* in the dataset determines the method of estimating p-values. For datasets with *N* < 13, one-tailed permutation-based paired *t*-tests are used with all possible permutations. For datasets with 13 ≤ *N* ≤ 20, 10000 random permutations are employed. For datasets with *N* > 20, the Wilcoxon signed-rank test [122] is used. The effect size between two pipelines is measured via Standardized Mean Difference (SMD) [48] estimated over the subjects.

For multiple datasets, the comparison of
statistical differences between two pipelines over *D* is done by combining effect sizes {$s_i$}$_{i=1}^D$ and p-values {$p_i$}$_{i=1}^D$ with Stouffer's Z-score method [96] that take in account the different sizes of the datasets. Combined p-values are obtained by estimating $Z = \sum_{i=1}^D w_i Z_i$, where $Z_i = \Phi^{-1}(1 - p_i)$ with $\Phi$ the standard normal cumulative distribution function. This weighted Z-score relies on weights $w_i = \sqrt{\frac{N_i}{\sum_{i=1}^D w_i^2}}$ that are proportional to $N_i$ the number of subjects in the dataset *i*. Combined measures of the effect sizes are obtained by $S = \sum_{i=1}^D w_i s_i$ weighted average of SMD.

## 2.5. Code Carbon

The assessment of the environmental impact of research is gathering momentum, yet the methodology remains a topic of heavy debate, particularly in computer science and machine learning [71].

Within these fields, a significant environmental impact stems from the production processes of devices (such as acquisition devices, computers, GPUs, and clusters) and the energy consumption during their usage. Evaluating the impact of device production poses challenges, given the limited information shared by manufacturers, potential inaccuracies in estimations, and the frequent sharing and reusing of research equipment across multiple projects. This issue is notably prevalent for CPU and GPU clusters, where numerous experiments run in parallel [76].

Energy consumption during usage is commonly acknowledged as a key indicator of the environmental impact, although it represents only a fraction of the overall impact. This measurement is typically expressed as grams of CO2 equivalent (gCO2) emissions released into the atmosphere and heavily relies

on factors such as the power grid setup and energy production localization. National power companies typically offer estimates of the carbon footprint of consumed energy, articulated in gCO2 per Watt-hour. This value can significantly vary between countries based on energy production sources; for instance, countries reliant on coal or fuel power plants tend to have a higher carbon footprint compared to those utilizing hydroelectric power or solar panels.

For optimal measurement of energy consumption and carbon footprint, power meters are ideal, albeit encompassing the entire computer's energy consumption. To gain more precise insights, especially for evaluating specific programs or machine learning pipelines, software-based power meters prove useful [53]. Various tools exist for this purpose, many of which are open source. In our study, we opted to employ Code Carbon [35], a Python-based tool that seamlessly integrates into experiments conducted on individual computers or clusters.

When estimating the carbon footprint of a machine learning pipeline, it is critical to analyze both the training and inference phases. During k-fold validation, inference constitutes a substantial portion of energy consumption. Consequently, the estimated carbon footprint offers valuable insights into the trade-offs between different algorithms, with considerations for CPU consumption, execution time, and the parallelization of algorithms. Notably, well-parallelized algorithms can achieve lower execution times when distributed across a large number of CPUs. The carbon footprint serves as an insightful metric for assessing the computational complexity of a pipeline, particularly when considering its adaptability to embedded or dedicated architectures, common in commercial neurotechnology products.
statistical differences between two pipelines over *D* is done by combining effect sizes {$s_i$}$_{i=1}^D$ and p-values {$p_i$}$_{i=1}^D$ with Stouffer's Z-score method [96] that take in account the different sizes of the datasets. Combined p-values are obtained by estimating $Z = \sum_{i=1}^D w_i Z_i$, where $Z_i = \Phi^{-1}(1 - p_i)$ with $\Phi$ the standard normal cumulative distribution function. This weighted Z-score relies on weights $w_i = \sqrt{\frac{N_i}{\sum_{i=1}^D w_i^2}}$ that are proportional to $N_i$ the number of subjects in the dataset *i*. Combined measures of the effect sizes are obtained by $S = \sum_{i=1}^D w_i s_i$ weighted average of SMD.

## 2.5. Code Carbon

The assessment of the environmental impact of research is gathering momentum, yet the methodology remains a topic of heavy debate, particularly in computer science and machine learning [71].

Within these fields, a significant environmental impact stems from the production processes of devices (such as acquisition devices, computers, GPUs, and clusters) and the energy consumption during their usage. Evaluating the impact of device production poses challenges, given the limited information shared by manufacturers, potential inaccuracies in estimations, and the frequent sharing and reusing of research equipment across multiple projects. This issue is notably prevalent for CPU and GPU clusters, where numerous experiments run in parallel [76].

Energy consumption during usage is commonly acknowledged as a key indicator of the environmental impact, although it represents only a fraction of the overall impact. This measurement is typically expressed as grams of CO2 equivalent (gCO2) emissions released into the atmosphere and heavily relies

on factors such as the power grid setup and energy production localization. National power companies typically offer estimates of the carbon footprint of consumed energy, articulated in gCO2 per Watt-hour. This value can significantly vary between countries based on energy production sources; for instance, countries reliant on coal or fuel power plants tend to have a higher carbon footprint compared to those utilizing hydroelectric power or solar panels.

For optimal measurement of energy consumption and carbon footprint, power meters are ideal, albeit encompassing the entire computer's energy consumption. To gain more precise insights, especially for evaluating specific programs or machine learning pipelines, software-based power meters prove useful [53]. Various tools exist for this purpose, many of which are open source. In our study, we opted to employ Code Carbon [35], a Python-based tool that seamlessly integrates into experiments conducted on individual computers or clusters.

When estimating the carbon footprint of a machine learning pipeline, it is critical to analyze both the training and inference phases. During k-fold validation, inference constitutes a substantial portion of energy consumption. Consequently, the estimated carbon footprint offers valuable insights into the trade-offs between different algorithms, with considerations for CPU consumption, execution time, and the parallelization of algorithms. Notably, well-parallelized algorithms can achieve lower execution times when distributed across a large number of CPUs. The carbon footprint serves as an insightful metric for assessing the computational complexity of a pipeline, particularly when considering its adaptability to embedded or dedicated architectures, common in commercial neurotechnology products.
## 3. Datasets

There are different types of BCI applications, referred to as *paradigms* throughout this article, each relying on distinctive neurological patterns to facilitate communication between the brain and the BCI system. Depending on the selected paradigm, the raw EEG recordings are transformed into trials for machine learning pipelines. These transformations include bandpass filtering, signal cropping based on stimulus events, and potential resampling to adjust the sampling frequency if needed by the machine learning pipelines. Employing standardized preprocessing procedures enables a fair comparison among different algorithms.

### 3.1. Motor Imagery

The MI paradigm involves a cognitive process where an individual mentally simulates the execution of a motor action without actually performing it. This paradigm is widely used in neuroscience to delve into the neural mechanisms governing motor control and learning. Moreover, it finds applications in neurorehabilitation to assist in restoring motor functions for individuals with neurological disorders or injuries. Various tasks are associated with this paradigm in Mother Of All BCI Benchmark (BCI), with common examples including left-hand, right-hand, and feet imagery. The choice of evaluation metrics for classification performance varies based on the number of tasks involved in the classification process: ROC-AUC is used for two-task classifications, whereas accuracy metrics are employed for multiclass scenarios. With the MI paradigm, signal processing includes bandpass filtering within the [8 – 32] Hz frequency range [86].

The Table 1 provides a comprehensive

overview of MI datasets, with class name abbreviations including RH (Right Hand), LH (Left Hand), F (Feet), H (Hands), T (Tongue), R (Resting State), LHRF (Left Hand Right Foot), and RHLF (Right Hand Left Foot). The column 'No. trials' denotes the number of trials per class, session, and run. For instance, BNCI2014_001 comprises 12 trials per class across 4 classes, 6 runs, and 2 sessions, resulting in a total of 12 × 4 × 6 × 2 = 576 trials in the dataset. The only exception is for the PhysionetMI dataset, where in the first 3 runs, there are RH, LH, and R events; in the last 3 runs, there are H, F, and R events.

### 3.2. P300/ERP

The P300 paradigm, which is a specific ERP paradigm, serves as a framework for categorizing psychophysical experiments [77]. It provides a visual representation of a distinct component characterized by a prominent positive deflection occurring approximately 300 ms after stimulus onset. Typically, this component is elicited in cognitive tasks involving unpredictable and infrequent changes in stimuli. This is improperly called P300 ERP in the BCI community (see P300 BCI or P300 speller), whereas the cognitive components are more diverse than just the P300 wave. See [77] for a detailed discussion about this point.

In the present study, our primary focus is on P3b, a specific subtype that examines stimulus changes relevant to the task at hand. This particular brain wave response manifests when the cognitive tasks involve stimuli that are predictable to some extent but are still imbued with an element of unpredictability. As is commonly the case in the literature, we classify events as targets or non-targets, i.e., at the epoch level, resulting in a binary classification task. We evaluate
## 3. Datasets

There are different types of BCI applications, referred to as *paradigms* throughout this article, each relying on distinctive neurological patterns to facilitate communication between the brain and the BCI system. Depending on the selected paradigm, the raw EEG recordings are transformed into trials for machine learning pipelines. These transformations include bandpass filtering, signal cropping based on stimulus events, and potential resampling to adjust the sampling frequency if needed by the machine learning pipelines. Employing standardized preprocessing procedures enables a fair comparison among different algorithms.

### 3.1. Motor Imagery

The MI paradigm involves a cognitive process where an individual mentally simulates the execution of a motor action without actually performing it. This paradigm is widely used in neuroscience to delve into the neural mechanisms governing motor control and learning. Moreover, it finds applications in neurorehabilitation to assist in restoring motor functions for individuals with neurological disorders or injuries. Various tasks are associated with this paradigm in Mother Of All BCI Benchmark (BCI), with common examples including left-hand, right-hand, and feet imagery. The choice of evaluation metrics for classification performance varies based on the number of tasks involved in the classification process: ROC-AUC is used for two-task classifications, whereas accuracy metrics are employed for multiclass scenarios. With the MI paradigm, signal processing includes bandpass filtering within the [8 – 32] Hz frequency range [86].

The Table 1 provides a comprehensive

overview of MI datasets, with class name abbreviations including RH (Right Hand), LH (Left Hand), F (Feet), H (Hands), T (Tongue), R (Resting State), LHRF (Left Hand Right Foot), and RHLF (Right Hand Left Foot). The column 'No. trials' denotes the number of trials per class, session, and run. For instance, BNCI2014_001 comprises 12 trials per class across 4 classes, 6 runs, and 2 sessions, resulting in a total of 12 × 4 × 6 × 2 = 576 trials in the dataset. The only exception is for the PhysionetMI dataset, where in the first 3 runs, there are RH, LH, and R events; in the last 3 runs, there are H, F, and R events.

### 3.2. P300/ERP

The P300 paradigm, which is a specific ERP paradigm, serves as a framework for categorizing psychophysical experiments [77]. It provides a visual representation of a distinct component characterized by a prominent positive deflection occurring approximately 300 ms after stimulus onset. Typically, this component is elicited in cognitive tasks involving unpredictable and infrequent changes in stimuli. This is improperly called P300 ERP in the BCI community (see P300 BCI or P300 speller), whereas the cognitive components are more diverse than just the P300 wave. See [77] for a detailed discussion about this point.

In the present study, our primary focus is on P3b, a specific subtype that examines stimulus changes relevant to the task at hand. This particular brain wave response manifests when the cognitive tasks involve stimuli that are predictable to some extent but are still imbued with an element of unpredictability. As is commonly the case in the literature, we classify events as targets or non-targets, i.e., at the epoch level, resulting in a binary classification task. We evaluate
<table><caption>Table 1: Overview of the Motor Imagery EEG datasets available in MOABB.</caption><thead><tr><th colspan="10">Motor imagery</th></tr><tr><th>Dataset</th><th>No. subj.</th><th>No. ch.</th><th>No. classes</th><th>No. trials /session/class</th><th>Trial len.(s)</th><th>S.freq. (Hz)</th><th>No. sessions</th><th>No. runs</th><th>Classes</th></tr></thead><tbody><tr><td>AlexMI [8]</td><td>8</td><td>16</td><td>2(3)</td><td>20 ± 0</td><td>3</td><td>512</td><td>1</td><td>1</td><td>RH, F, (R)</td></tr><tr><td>BNCI2014_001 [111]</td><td>9</td><td>22</td><td>3(4)</td><td>72 ± 0</td><td>4</td><td>250</td><td>2</td><td>6</td><td>RH, LH, F, (T)</td></tr><tr><td>BNCI2014_002 [109]</td><td>14</td><td>15</td><td>2</td><td>80 ± 0</td><td>5</td><td>512</td><td>1</td><td>8</td><td>RH, F</td></tr><tr><td>BNCI2014_004 [70]</td><td>9</td><td>3</td><td>2</td><td>72.4 ± 9.5</td><td>4.5</td><td>250</td><td>5</td><td>1</td><td>RH, LH</td></tr><tr><td>BNCI2015_001 [38]</td><td>12</td><td>13</td><td>2</td><td>100 ± 0</td><td>5</td><td>512</td><td>3 subj. 8-11<br>2 others</td><td>1</td><td>RH, F</td></tr><tr><td>BNCI2015_004 [102]</td><td>9</td><td>30</td><td>2(5)</td><td>39.4 ± 1.6</td><td>7</td><td>256</td><td>2</td><td>1</td><td>RH, F</td></tr><tr><td>Cho2017 [30]</td><td>52</td><td>64</td><td>2</td><td>101.2 ± 4.7</td><td>3</td><td>512</td><td>1</td><td>1</td><td>RH, LH</td></tr><tr><td>Lee2019_MI [69]</td><td>54</td><td>62</td><td>2</td><td>50</td><td>4</td><td>1000</td><td>2</td><td>1</td><td>RH, LH</td></tr><tr><td>GrosseWentrup2009 [44]</td><td>10</td><td>128</td><td>2</td><td>150 ± 0</td><td>7</td><td>500</td><td>1</td><td>1</td><td>RH, LH</td></tr><tr><td>PhysionetMI [101]</td><td>109</td><td>64</td><td>4(5)</td><td>22.6 ± 1.3</td><td>3</td><td>160</td><td>1</td><td>6***</td><td>RH, LH, H, F, (R)</td></tr><tr><td>Schirrmeister2017 [103]</td><td>14</td><td>128</td><td>3(4)</td><td>240.8 ± 37.7</td><td>4</td><td>500</td><td>1</td><td>2</td><td>RH, LH, F, (R)</td></tr><tr><td>Shin2017A [105]</td><td>29</td><td>30</td><td>2</td><td>10 ± 0</td><td>10</td><td>200</td><td>3</td><td>1</td><td>RH, LH</td></tr><tr><td>Weibo2014 [125]</td><td>10</td><td>60</td><td>4(7)</td><td>79 ± 3</td><td>4</td><td>200</td><td>1</td><td>1</td><td>RH, LH, H, F, (LHRF), (RHLF), (R)</td></tr><tr><td>Zhou2016 [128]</td><td>4</td><td>14</td><td>3</td><td>50 ± 3.5</td><td>5</td><td>250</td><td>3</td><td>2</td><td>RH, LF, F</td></tr></tbody></table>

its performance with the ROC-AUC metric, which handles the inherent imbalance in the problem at hand. With the ERP paradigm, the signal is bandpass filtered to the 1-24 Hz frequency band [77].

The Table 2 shows an overview of all the ERP datasets. The column “No. epochs NT/T” indicates the number of *NonTarget* and *Target* epochs per session and run.

### 3.3. SSVEP

Steady State Visually Evoked Potentials are generated when presenting repetitive sensory stimuli to the subject. While tactile and auditory stimulations are seldom used, visual stimulation is very common, both for control [28] or cognitive probes [123]. The frequency of the stimulus repetition induces a brain oscillation in the associated sensory area. The amplitude of the generated oscillation follows the 1/f law,

meaning that stimulation in low frequency (5-7 Hz) induces responses of higher amplitude than higher frequency (20-25 Hz). Stimulation above 40 Hz could be difficult to detect due to the weak generated oscillations. In BCI applications, SSVEPs have been used for building spellers [83] and for button-pressing [56], but those applications are limited by the number of available frequencies of stimulation. It is possible, using systems with precisely synchronized stimulation and recording, to encode information both in frequency and phase, therefore multiplying the choices possible [85]. Similarly to the MI paradigm, we evaluate the classifiers using the ROC-AUC metric if only two classes are used and the accuracy metric if there are more. With the SSVEP paradigm, the signal is bandpass filtered to the 7-45 Hz frequency band [29]. The Table 3 encompasses all the SSVEP datasets considered in this study.
<table><caption>Table 1: Overview of the Motor Imagery EEG datasets available in MOABB.</caption><thead><tr><th colspan="10">Motor imagery</th></tr><tr><th>Dataset</th><th>No. subj.</th><th>No. ch.</th><th>No. classes</th><th>No. trials /session/class</th><th>Trial len.(s)</th><th>S.freq. (Hz)</th><th>No. sessions</th><th>No. runs</th><th>Classes</th></tr></thead><tbody><tr><td>AlexMI [8]</td><td>8</td><td>16</td><td>2(3)</td><td>20 ± 0</td><td>3</td><td>512</td><td>1</td><td>1</td><td>RH, F, (R)</td></tr><tr><td>BNCI2014_001 [111]</td><td>9</td><td>22</td><td>3(4)</td><td>72 ± 0</td><td>4</td><td>250</td><td>2</td><td>6</td><td>RH, LH, F, (T)</td></tr><tr><td>BNCI2014_002 [109]</td><td>14</td><td>15</td><td>2</td><td>80 ± 0</td><td>5</td><td>512</td><td>1</td><td>8</td><td>RH, F</td></tr><tr><td>BNCI2014_004 [70]</td><td>9</td><td>3</td><td>2</td><td>72.4 ± 9.5</td><td>4.5</td><td>250</td><td>5</td><td>1</td><td>RH, LH</td></tr><tr><td>BNCI2015_001 [38]</td><td>12</td><td>13</td><td>2</td><td>100 ± 0</td><td>5</td><td>512</td><td>3 subj. 8-11<br>2 others</td><td>1</td><td>RH, F</td></tr><tr><td>BNCI2015_004 [102]</td><td>9</td><td>30</td><td>2(5)</td><td>39.4 ± 1.6</td><td>7</td><td>256</td><td>2</td><td>1</td><td>RH, F</td></tr><tr><td>Cho2017 [30]</td><td>52</td><td>64</td><td>2</td><td>101.2 ± 4.7</td><td>3</td><td>512</td><td>1</td><td>1</td><td>RH, LH</td></tr><tr><td>Lee2019_MI [69]</td><td>54</td><td>62</td><td>2</td><td>50</td><td>4</td><td>1000</td><td>2</td><td>1</td><td>RH, LH</td></tr><tr><td>GrosseWentrup2009 [44]</td><td>10</td><td>128</td><td>2</td><td>150 ± 0</td><td>7</td><td>500</td><td>1</td><td>1</td><td>RH, LH</td></tr><tr><td>PhysionetMI [101]</td><td>109</td><td>64</td><td>4(5)</td><td>22.6 ± 1.3</td><td>3</td><td>160</td><td>1</td><td>6***</td><td>RH, LH, H, F, (R)</td></tr><tr><td>Schirrmeister2017 [103]</td><td>14</td><td>128</td><td>3(4)</td><td>240.8 ± 37.7</td><td>4</td><td>500</td><td>1</td><td>2</td><td>RH, LH, F, (R)</td></tr><tr><td>Shin2017A [105]</td><td>29</td><td>30</td><td>2</td><td>10 ± 0</td><td>10</td><td>200</td><td>3</td><td>1</td><td>RH, LH</td></tr><tr><td>Weibo2014 [125]</td><td>10</td><td>60</td><td>4(7)</td><td>79 ± 3</td><td>4</td><td>200</td><td>1</td><td>1</td><td>RH, LH, H, F, (LHRF), (RHLF), (R)</td></tr><tr><td>Zhou2016 [128]</td><td>4</td><td>14</td><td>3</td><td>50 ± 3.5</td><td>5</td><td>250</td><td>3</td><td>2</td><td>RH, LF, F</td></tr></tbody></table>

its performance with the ROC-AUC metric, which handles the inherent imbalance in the problem at hand. With the ERP paradigm, the signal is bandpass filtered to the 1-24 Hz frequency band [77].

The Table 2 shows an overview of all the ERP datasets. The column “No. epochs NT/T” indicates the number of *NonTarget* and *Target* epochs per session and run.

### 3.3. SSVEP

Steady State Visually Evoked Potentials are generated when presenting repetitive sensory stimuli to the subject. While tactile and auditory stimulations are seldom used, visual stimulation is very common, both for control [28] or cognitive probes [123]. The frequency of the stimulus repetition induces a brain oscillation in the associated sensory area. The amplitude of the generated oscillation follows the 1/f law,

meaning that stimulation in low frequency (5-7 Hz) induces responses of higher amplitude than higher frequency (20-25 Hz). Stimulation above 40 Hz could be difficult to detect due to the weak generated oscillations. In BCI applications, SSVEPs have been used for building spellers [83] and for button-pressing [56], but those applications are limited by the number of available frequencies of stimulation. It is possible, using systems with precisely synchronized stimulation and recording, to encode information both in frequency and phase, therefore multiplying the choices possible [85]. Similarly to the MI paradigm, we evaluate the classifiers using the ROC-AUC metric if only two classes are used and the accuracy metric if there are more. With the SSVEP paradigm, the signal is bandpass filtered to the 7-45 Hz frequency band [29]. The Table 3 encompasses all the SSVEP datasets considered in this study.
<table><caption>Table 2: Overview of the ERP EEG datasets available in MOABB.</caption><thead><tr><th colspan="10">P300 / ERP</th></tr><tr><th>Dataset</th><th>No. subj.</th><th>No. ch.</th><th>No. epochs NT/T /session</th><th>Epoch len.(s)</th><th>S.freq. (Hz)</th><th>No. sessions</th><th>No. runs</th><th>Keyboard</th></tr></thead><tbody><tr><td>BI2012 [116]</td><td>25</td><td>16</td><td>638.2 ± 1.9/127.6 ± 0.7</td><td>1</td><td>128</td><td>1</td><td>1</td><td>36 aliens</td></tr><tr><td>BI2013a [115, 10, 32]</td><td>24</td><td>16</td><td>400.3 ± 2.3/80.1 ± 0.5</td><td>1</td><td>512</td><td>8 subj. 1-7<br>1 subj. 8-24</td><td>1</td><td>36 aliens</td></tr><tr><td>BI2014a [64]</td><td>64</td><td>16</td><td>794.5 ± 276.7/158.9 ± 55.3</td><td>1</td><td>512</td><td>1</td><td>1</td><td>36 aliens</td></tr><tr><td>BI2014b [65]</td><td>37</td><td>32</td><td>201.3 ± 61.5/40.3 ± 12.3</td><td>1</td><td>512</td><td>1</td><td>1</td><td>36 aliens</td></tr><tr><td>BI2015a [62]</td><td>43</td><td>32</td><td>461.8 ± 220.9/92.3 ± 44.1</td><td>1</td><td>512</td><td>3</td><td>1</td><td>36 aliens</td></tr><tr><td>BI2015b [63]</td><td>44</td><td>32</td><td>2158.7 ± 6.3/479.9 ± 0.3</td><td>1</td><td>512</td><td>1</td><td>4</td><td>36 aliens</td></tr><tr><td>BNCI2014_008 [94, 39]</td><td>8</td><td>8</td><td>3500 ± 0/700 ± 0</td><td>1</td><td>256</td><td>1</td><td>1</td><td>36 char.</td></tr><tr><td>BNCI2014_009 [3]</td><td>10</td><td>16</td><td>480 ± 0/96 ± 0</td><td>0.8</td><td>256</td><td>3</td><td>1</td><td>36 char.</td></tr><tr><td>BNCI2015_003 [46]</td><td>10</td><td>8</td><td>2250 ± 1500/270 ± 60</td><td>0.8</td><td>256</td><td>1</td><td>2</td><td>36 char.</td></tr><tr><td>EPFLP300 [49]</td><td>8</td><td>32</td><td>685.2 ± 16.9/137.2 ± 3.5</td><td>1</td><td>2048</td><td>4</td><td>6</td><td>6 images</td></tr><tr><td>Huebner2017 [50]</td><td>13</td><td>31</td><td>3275.3 ± 2.1/1007.8 ± 0.6</td><td>0.9</td><td>1000</td><td>2 subj. 6<br>3 others</td><td>9</td><td>42 char.</td></tr><tr><td>Huebner2018 [51]</td><td>12</td><td>31</td><td>3638.4 ± 7.7/1119.6 ± 2.5</td><td>0.9</td><td>1000</td><td>3</td><td>10</td><td>42 char.</td></tr><tr><td>Lee2019_ERP [69]</td><td>54</td><td>62</td><td>3450/690</td><td>1</td><td>1000</td><td>2</td><td>1</td><td>36 char.</td></tr><tr><td>Sosulski2019 [106, 108]</td><td>13</td><td>31</td><td>75 ± 0/15 ± 0</td><td>1.2</td><td>1000</td><td>4 subj. 1<br>3 others</td><td>20</td><td>2 tones</td></tr><tr><td>Cattan2019_VR [24]</td><td>21</td><td>16</td><td>600 ± 0/120 ± 0</td><td>1</td><td>512</td><td>2</td><td>60</td><td>36 crosses</td></tr></tbody></table>

<table><caption>Table 3: Overview of the SSVEP EEG datasets available in MOABB.</caption><thead><tr><th colspan="10">SSVEP</th></tr><tr><th>Dataset</th><th>No. subj.</th><th>No. channels</th><th>No. classes</th><th>No. trials /session/class</th><th>Trial len.(s)</th><th>S.freq. (Hz)</th><th>No. sess.</th><th>No. runs</th><th>Classes</th></tr></thead><tbody><tr><td>Lee2019 SSVEP [69]</td><td>54</td><td>62</td><td>4</td><td>25</td><td>1</td><td>1000</td><td>2</td><td>1</td><td>4 (5.45-12)</td></tr><tr><td>MAMEM1 [87]</td><td>10</td><td>256</td><td>5</td><td>16.8 ± 3.5 classes 8.57,10.0<br>21.0 ± 4.4 classes 6.66,7.5,12.0</td><td>3</td><td>250</td><td>1</td><td>3 subj. 1,3,8; 4 subj. 4,6<br>5 others</td><td>5 (6.66-12.00)</td></tr><tr><td>MAMEM2 [87]</td><td>10</td><td>256</td><td>5</td><td>20 class 12.0; 30 class 8.57<br>25 others</td><td>3</td><td>250</td><td>1</td><td>5</td><td>5 (6.66-12.00)</td></tr><tr><td>MAMEM3 [87]</td><td>10</td><td>14</td><td>4</td><td>20.0 ± 0.0 class 6.66; 25.0 ± 0.0 class 8.57<br>30.0 ± 0.0 class 10.0; 25.0 ± 0.0 class 12.0</td><td>3</td><td>128</td><td>1</td><td>10</td><td>4 (6.66-12.00)</td></tr><tr><td>Nakanishi2015 [84]</td><td>9</td><td>8</td><td>12</td><td>15.0 ± 0.0</td><td>4.15</td><td>256</td><td>1</td><td>1</td><td>12 (9.25-14.75)</td></tr><tr><td>Kalunga2016 [56]</td><td>12</td><td>8</td><td>4</td><td>20.0 ± 7.7</td><td>2</td><td>256</td><td>1</td><td>5 subj. 12; 4 subj.10<br>3 subj. 7; 2 others</td><td>4 (13,17,21,rest)</td></tr><tr><td>Wang2016 [120]</td><td>34</td><td>62</td><td>40</td><td>6.0 ± 0.0</td><td>5</td><td>250</td><td>1</td><td>1</td><td>40 (8-15.8)</td></tr></tbody></table>

The Figure 2 displays an embedding of all the datasets in 2 dimensions, using UMAP for dimensionality reduction on all feature information regarding the datasets, as listed in Tables 1, 2 and 3. The color indicates the paradigms (blue for Motor Imagery, green for Event Related Potential, and red for Steady State Visually Evoked Potential), and the name of each dataset is written on top of the circle. The color intensity is related to

the number of electrodes, datasets with a low number of electrodes are in lighter color, and the size of the circles is proportional to the number of subjects. The UMAP embedding preserves the local topology. This highlights that, despite different paradigms, datasets with many electrodes and many subjects are in a central position. Datasets with fewer subjects and fewer electrodes are closer to the border of the figure. The
<table><caption>Table 2: Overview of the ERP EEG datasets available in MOABB.</caption><thead><tr><th colspan="10">P300 / ERP</th></tr><tr><th>Dataset</th><th>No. subj.</th><th>No. ch.</th><th>No. epochs NT/T /session</th><th>Epoch len.(s)</th><th>S.freq. (Hz)</th><th>No. sessions</th><th>No. runs</th><th>Keyboard</th></tr></thead><tbody><tr><td>BI2012 [116]</td><td>25</td><td>16</td><td>638.2 ± 1.9/127.6 ± 0.7</td><td>1</td><td>128</td><td>1</td><td>1</td><td>36 aliens</td></tr><tr><td>BI2013a [115, 10, 32]</td><td>24</td><td>16</td><td>400.3 ± 2.3/80.1 ± 0.5</td><td>1</td><td>512</td><td>8 subj. 1-7<br>1 subj. 8-24</td><td>1</td><td>36 aliens</td></tr><tr><td>BI2014a [64]</td><td>64</td><td>16</td><td>794.5 ± 276.7/158.9 ± 55.3</td><td>1</td><td>512</td><td>1</td><td>1</td><td>36 aliens</td></tr><tr><td>BI2014b [65]</td><td>37</td><td>32</td><td>201.3 ± 61.5/40.3 ± 12.3</td><td>1</td><td>512</td><td>1</td><td>1</td><td>36 aliens</td></tr><tr><td>BI2015a [62]</td><td>43</td><td>32</td><td>461.8 ± 220.9/92.3 ± 44.1</td><td>1</td><td>512</td><td>3</td><td>1</td><td>36 aliens</td></tr><tr><td>BI2015b [63]</td><td>44</td><td>32</td><td>2158.7 ± 6.3/479.9 ± 0.3</td><td>1</td><td>512</td><td>1</td><td>4</td><td>36 aliens</td></tr><tr><td>BNCI2014_008 [94, 39]</td><td>8</td><td>8</td><td>3500 ± 0/700 ± 0</td><td>1</td><td>256</td><td>1</td><td>1</td><td>36 char.</td></tr><tr><td>BNCI2014_009 [3]</td><td>10</td><td>16</td><td>480 ± 0/96 ± 0</td><td>0.8</td><td>256</td><td>3</td><td>1</td><td>36 char.</td></tr><tr><td>BNCI2015_003 [46]</td><td>10</td><td>8</td><td>2250 ± 1500/270 ± 60</td><td>0.8</td><td>256</td><td>1</td><td>2</td><td>36 char.</td></tr><tr><td>EPFLP300 [49]</td><td>8</td><td>32</td><td>685.2 ± 16.9/137.2 ± 3.5</td><td>1</td><td>2048</td><td>4</td><td>6</td><td>6 images</td></tr><tr><td>Huebner2017 [50]</td><td>13</td><td>31</td><td>3275.3 ± 2.1/1007.8 ± 0.6</td><td>0.9</td><td>1000</td><td>2 subj. 6<br>3 others</td><td>9</td><td>42 char.</td></tr><tr><td>Huebner2018 [51]</td><td>12</td><td>31</td><td>3638.4 ± 7.7/1119.6 ± 2.5</td><td>0.9</td><td>1000</td><td>3</td><td>10</td><td>42 char.</td></tr><tr><td>Lee2019_ERP [69]</td><td>54</td><td>62</td><td>3450/690</td><td>1</td><td>1000</td><td>2</td><td>1</td><td>36 char.</td></tr><tr><td>Sosulski2019 [106, 108]</td><td>13</td><td>31</td><td>75 ± 0/15 ± 0</td><td>1.2</td><td>1000</td><td>4 subj. 1<br>3 others</td><td>20</td><td>2 tones</td></tr><tr><td>Cattan2019_VR [24]</td><td>21</td><td>16</td><td>600 ± 0/120 ± 0</td><td>1</td><td>512</td><td>2</td><td>60</td><td>36 crosses</td></tr></tbody></table>

<table><caption>Table 3: Overview of the SSVEP EEG datasets available in MOABB.</caption><thead><tr><th colspan="10">SSVEP</th></tr><tr><th>Dataset</th><th>No. subj.</th><th>No. channels</th><th>No. classes</th><th>No. trials /session/class</th><th>Trial len.(s)</th><th>S.freq. (Hz)</th><th>No. sess.</th><th>No. runs</th><th>Classes</th></tr></thead><tbody><tr><td>Lee2019 SSVEP [69]</td><td>54</td><td>62</td><td>4</td><td>25</td><td>1</td><td>1000</td><td>2</td><td>1</td><td>4 (5.45-12)</td></tr><tr><td>MAMEM1 [87]</td><td>10</td><td>256</td><td>5</td><td>16.8 ± 3.5 classes 8.57,10.0<br>21.0 ± 4.4 classes 6.66,7.5,12.0</td><td>3</td><td>250</td><td>1</td><td>3 subj. 1,3,8; 4 subj. 4,6<br>5 others</td><td>5 (6.66-12.00)</td></tr><tr><td>MAMEM2 [87]</td><td>10</td><td>256</td><td>5</td><td>20 class 12.0; 30 class 8.57<br>25 others</td><td>3</td><td>250</td><td>1</td><td>5</td><td>5 (6.66-12.00)</td></tr><tr><td>MAMEM3 [87]</td><td>10</td><td>14</td><td>4</td><td>20.0 ± 0.0 class 6.66; 25.0 ± 0.0 class 8.57<br>30.0 ± 0.0 class 10.0; 25.0 ± 0.0 class 12.0</td><td>3</td><td>128</td><td>1</td><td>10</td><td>4 (6.66-12.00)</td></tr><tr><td>Nakanishi2015 [84]</td><td>9</td><td>8</td><td>12</td><td>15.0 ± 0.0</td><td>4.15</td><td>256</td><td>1</td><td>1</td><td>12 (9.25-14.75)</td></tr><tr><td>Kalunga2016 [56]</td><td>12</td><td>8</td><td>4</td><td>20.0 ± 7.7</td><td>2</td><td>256</td><td>1</td><td>5 subj. 12; 4 subj.10<br>3 subj. 7; 2 others</td><td>4 (13,17,21,rest)</td></tr><tr><td>Wang2016 [120]</td><td>34</td><td>62</td><td>40</td><td>6.0 ± 0.0</td><td>5</td><td>250</td><td>1</td><td>1</td><td>40 (8-15.8)</td></tr></tbody></table>

The Figure 2 displays an embedding of all the datasets in 2 dimensions, using UMAP for dimensionality reduction on all feature information regarding the datasets, as listed in Tables 1, 2 and 3. The color indicates the paradigms (blue for Motor Imagery, green for Event Related Potential, and red for Steady State Visually Evoked Potential), and the name of each dataset is written on top of the circle. The color intensity is related to

the number of electrodes, datasets with a low number of electrodes are in lighter color, and the size of the circles is proportional to the number of subjects. The UMAP embedding preserves the local topology. This highlights that, despite different paradigms, datasets with many electrodes and many subjects are in a central position. Datasets with fewer subjects and fewer electrodes are closer to the border of the figure. The
BNCI2014.001 dataset, commonly called BCI Competition IV dataset 2a, is the most widely used in BCI literature. There are closely related datasets with roughly the same number of subjects, electrodes, and trials (BNCI2015_004, BNCI2015_001), with more subjects (Shin2017A) or with fewer subjects (Zhou2016). This group of datasets is useful for the fast evaluation of new pipelines and to see how well results generalize with the same kind of dataset. It is also possible to ensure a good coverage of the dataset features, using only a few datasets. With MI, a selection of BNCI2014_001, BNCI2014_004, Schirrmeister2017, and PhysioNetMI might be sufficient to evaluate an approach on datasets with very different configurations.

# 4. Pipelines

As discussed in section 1, most classification algorithms in BCI research for EEG signals fall into one of three main categories: those based on raw signals (referred to as “Raw” hereafter), algorithms relying on covariance matrices seen as elements of a Riemannian manifold (denoted “Riemannian”) and the Deep Learning (DL) approaches.

The Raw signal methods typically employ supervised spatial filters to simultaneously enhance the component related to the cognitive task while reducing the dimensionality of the EEG data. In contrast, Riemannian pipelines consider the signal through its estimated covariance matrices, leveraging the natural metric acting on the curved geometry of SPD matrices, which remains invariant by congruence transformations [124, 33]. Those approaches are thus mostly invariant to any spatial transformations applied on the signal, making them highly effective in BCI applications.

Lastly, deep neural networks learn spatial

and temporal filters directly from raw EEG data. Although there is a wealth of literature on deep learning models, there are few models available or evaluated with reproducible frameworks. Initiatives like Braindecode [103] or, to a lesser extent for BCI, torchEEG [127], offer an open-source implementation of the most efficient models. This study considers a set of raw, Riemannian, and deep learning models, which are detailed in Table 4, along with the hyperparameters used for the grid-search approach, listed in the annexes within Table A1.

## 4.1. Raw signal

The category of pipelines referred to as Raw consists of BCI classifiers employing traditional statistical analysis, as well as temporal and/or spatial filtering tools to extract features. Variance-based pipelines represent one of the initial BCI pipeline concepts proposed for motor imagery. Several approaches have highlighted the value of utilizing intra-channel variance for online decoding tasks. These pipelines calculate the variance of each electrode within an epoch to create a positive definite real-valued vector. To address noise or artifacts, it is common practice to logarithmically transform the observed variance [74]. This approach essentially boils down to only considering the diagonal elements of the covariance matrices. Classifiers such as Linear Discriminant Analysis (LDA) or Support Vector Machine (SVM) can be utilized on the resulting vector to predict the label of the epoch.

Common Spatial Pattern (CSP) approach learns spatial filtering matrices in a supervised manner, minimizing the variance of the band power feature vectors within the same
BNCI2014.001 dataset, commonly called BCI Competition IV dataset 2a, is the most widely used in BCI literature. There are closely related datasets with roughly the same number of subjects, electrodes, and trials (BNCI2015_004, BNCI2015_001), with more subjects (Shin2017A) or with fewer subjects (Zhou2016). This group of datasets is useful for the fast evaluation of new pipelines and to see how well results generalize with the same kind of dataset. It is also possible to ensure a good coverage of the dataset features, using only a few datasets. With MI, a selection of BNCI2014_001, BNCI2014_004, Schirrmeister2017, and PhysioNetMI might be sufficient to evaluate an approach on datasets with very different configurations.

# 4. Pipelines

As discussed in section 1, most classification algorithms in BCI research for EEG signals fall into one of three main categories: those based on raw signals (referred to as “Raw” hereafter), algorithms relying on covariance matrices seen as elements of a Riemannian manifold (denoted “Riemannian”) and the Deep Learning (DL) approaches.

The Raw signal methods typically employ supervised spatial filters to simultaneously enhance the component related to the cognitive task while reducing the dimensionality of the EEG data. In contrast, Riemannian pipelines consider the signal through its estimated covariance matrices, leveraging the natural metric acting on the curved geometry of SPD matrices, which remains invariant by congruence transformations [124, 33]. Those approaches are thus mostly invariant to any spatial transformations applied on the signal, making them highly effective in BCI applications.

Lastly, deep neural networks learn spatial

and temporal filters directly from raw EEG data. Although there is a wealth of literature on deep learning models, there are few models available or evaluated with reproducible frameworks. Initiatives like Braindecode [103] or, to a lesser extent for BCI, torchEEG [127], offer an open-source implementation of the most efficient models. This study considers a set of raw, Riemannian, and deep learning models, which are detailed in Table 4, along with the hyperparameters used for the grid-search approach, listed in the annexes within Table A1.

## 4.1. Raw signal

The category of pipelines referred to as Raw consists of BCI classifiers employing traditional statistical analysis, as well as temporal and/or spatial filtering tools to extract features. Variance-based pipelines represent one of the initial BCI pipeline concepts proposed for motor imagery. Several approaches have highlighted the value of utilizing intra-channel variance for online decoding tasks. These pipelines calculate the variance of each electrode within an epoch to create a positive definite real-valued vector. To address noise or artifacts, it is common practice to logarithmically transform the observed variance [74]. This approach essentially boils down to only considering the diagonal elements of the covariance matrices. Classifiers such as Linear Discriminant Analysis (LDA) or Support Vector Machine (SVM) can be utilized on the resulting vector to predict the label of the epoch.

Common Spatial Pattern (CSP) approach learns spatial filtering matrices in a supervised manner, minimizing the variance of the band power feature vectors within the same
Dataset visualization

<figure><img src="image_4.png" /><figcaption>Figure 2: Visualization of the MOABB datasets, with Motor Imagery in green, Event Related Potential in pink/purple and Steady State Visually Evoked Potential in yellow/brown. The size of the circle is proportional to the number of subjects and the contrast depends on the number of electrodes.</figcaption></figure>

<table><caption>Table 4: Pipelines considered in this study, the color indicates the paradigm. Green is for motor imagery, pink for P300 and orange for SSVEP.</caption><thead><tr><th>Pipeline Name</th><th>Category</th><th>References</th><th>Pipeline Name</th><th>Category</th><th>References</th></tr></thead><tbody><tr><td>LogVar + LDA</td><td>Raw</td><td></td><td>TS + LR</td><td>Riemannian</td><td>[12]</td></tr><tr><td>LogVar + SVM</td><td>Raw</td><td></td><td>TS + SVM</td><td>Riemannian</td><td>[12]</td></tr><tr><td>CSP + LDA</td><td>Raw</td><td>[61, 19]</td><td>ACM + TS + SVM</td><td>Riemannian</td><td>[22]</td></tr><tr><td>CSP + SVM</td><td>Raw</td><td>[61, 19]</td><td>ShallowConvNet</td><td>Deep Learning</td><td>[103]</td></tr><tr><td>TRCSP + LDA</td><td>Raw</td><td>[73]</td><td>DeepConvNet</td><td>Deep Learning</td><td>[103]</td></tr><tr><td>DLCSPauto + shLDA</td><td>Raw</td><td>[61, 19]</td><td>EEGNet 8 2</td><td>Deep Learning</td><td>[67]</td></tr><tr><td>FBCSP+SVM</td><td>Raw</td><td>[2]</td><td>EEGTCNet</td><td>Deep Learning</td><td>[52]</td></tr><tr><td>FgMDM</td><td>Riemannian</td><td>[12]</td><td>EEGITNet</td><td>Deep Learning</td><td>[100]</td></tr><tr><td>MDM</td><td>Riemannian</td><td>[13]</td><td>EEGNeX 8 32</td><td>Deep Learning</td><td>[27]</td></tr><tr><td>TS + EL</td><td>Riemannian</td><td>[34]</td><td></td><td></td><td></td></tr><tr><td>XDAWN + LDA</td><td>Raw</td><td>[95]</td><td>XDAWNCov + TS + SVM</td><td>Riemannian</td><td>[28]</td></tr><tr><td>XDAWNCov + MDM</td><td>Riemannian</td><td>[9]</td><td>ERPCov + MDM</td><td>Riemannian</td><td>[10]</td></tr><tr><td>ERPCov(svd<sub>n</sub> = 4) + MDM</td><td>Riemannian</td><td>[10]</td><td></td><td></td><td></td></tr><tr><td>TRCA</td><td>Raw</td><td>[85]</td><td>SSVEP MDM</td><td>Riemannian</td><td>[29]</td></tr><tr><td>CCA</td><td>Raw</td><td>[72]</td><td>SSVEP TS + LR</td><td>Riemannian</td><td>[29]</td></tr><tr><td>MsetCCA</td><td>Raw</td><td>[126]</td><td>SSVEP TS + SVM</td><td>Riemannian</td><td>[29]</td></tr></tbody></table>
Dataset visualization

<figure><img src="image_3.png" /><figcaption>Figure 2: Visualization of the MOABB datasets, with Motor Imagery in green, Event Related Potential in pink/purple and Steady State Visually Evoked Potential in yellow/brown. The size of the circle is proportional to the number of subjects and the contrast depends on the number of electrodes.</figcaption></figure>

<table><caption>Table 4: Pipelines considered in this study, the color indicates the paradigm. Green is for motor imagery, pink for P300 and orange for SSVEP.</caption><thead><tr><th>Pipeline Name</th><th>Category</th><th>References</th><th>Pipeline Name</th><th>Category</th><th>References</th></tr></thead><tbody><tr><td>LogVar + LDA</td><td>Raw</td><td></td><td>TS + LR</td><td>Riemannian</td><td>[12]</td></tr><tr><td>LogVar + SVM</td><td>Raw</td><td></td><td>TS + SVM</td><td>Riemannian</td><td>[12]</td></tr><tr><td>CSP + LDA</td><td>Raw</td><td>[61, 19]</td><td>ACM + TS + SVM</td><td>Riemannian</td><td>[22]</td></tr><tr><td>CSP + SVM</td><td>Raw</td><td>[61, 19]</td><td>ShallowConvNet</td><td>Deep Learning</td><td>[103]</td></tr><tr><td>TRCSP + LDA</td><td>Raw</td><td>[73]</td><td>DeepConvNet</td><td>Deep Learning</td><td>[103]</td></tr><tr><td>DLCSPauto + shLDA</td><td>Raw</td><td>[61, 19]</td><td>EEGNet 8 2</td><td>Deep Learning</td><td>[67]</td></tr><tr><td>FBCSP+SVM</td><td>Raw</td><td>[2]</td><td>EEGTCNet</td><td>Deep Learning</td><td>[52]</td></tr><tr><td>FgMDM</td><td>Riemannian</td><td>[12]</td><td>EEGITNet</td><td>Deep Learning</td><td>[100]</td></tr><tr><td>MDM</td><td>Riemannian</td><td>[13]</td><td>EEGNeX 8 32</td><td>Deep Learning</td><td>[27]</td></tr><tr><td>TS + EL</td><td>Riemannian</td><td>[34]</td><td></td><td></td><td></td></tr><tr><td>XDAWN + LDA</td><td>Raw</td><td>[95]</td><td>XDAWNCov + TS + SVM</td><td>Riemannian</td><td>[28]</td></tr><tr><td>XDAWNCov + MDM</td><td>Riemannian</td><td>[9]</td><td>ERPCov + MDM</td><td>Riemannian</td><td>[10]</td></tr><tr><td>ERPCov(svd<sub>n</sub> = 4) + MDM</td><td>Riemannian</td><td>[10]</td><td></td><td></td><td></td></tr><tr><td>TRCA</td><td>Raw</td><td>[85]</td><td>SSVEP MDM</td><td>Riemannian</td><td>[29]</td></tr><tr><td>CCA</td><td>Raw</td><td>[72]</td><td>SSVEP TS + LR</td><td>Riemannian</td><td>[29]</td></tr><tr><td>MsetCCA</td><td>Raw</td><td>[126]</td><td>SSVEP TS + SVM</td><td>Riemannian</td><td>[29]</td></tr></tbody></table>
class, while maximizing the between-class variance [81, 19]. To enhance the robustness of the CSP against noise and overfitting, Tikhonov Regularized CSP (CSP) approach has been proposed in [73]. In contrast to the classical CSP, which uses band-pass filtered EEG signals that may vary per subject, Filter-Bank CSP (CSP) addresses this issue by extracting CSP features for each band-pass filter from the filter bank [2]. The subsection Appendix B.1 details the CSP algorithm.

Canonical Correlation Analysis (CCA) has emerged as a prominent approach for classifying SSVEP signals, initially introduced in the work by [72]. Subsequently, CCA has been successfully employed in numerous notable SSVEP-based BCI studies, such as those conducted by [18] and [83]. SSVEP signals exhibit correlation to flickering visual stimuli, with their signal phase and frequency corresponding to stimulus characteristics. Leveraging this relationship, CCA aids in extracting EEG spatial components with the strongest correlation to SSVEP stimuli. Further details on the different pipelines can be found in subsection Appendix B.2.

## 4.2. Riemannian geometry

The introduction of Riemannian geometry into BCI processing marked a pivotal moment for the BCI community [13]. The fundamental concept underlying this approach is to represent the signal using covariance matrices or their derivatives. As covariance matrices are Symmetric Positive Definite (SPD) matrices, they live in a Riemannian space [124]. We provide here a general description of the framework needed to define algorithms based on the Riemann distance for classification tasks. For a more in-depth description of the Riemannian

framework, we refer the reader to [20] which gives a pedagogical introduction to these concepts.

Due to the curvature of the SPD matrix space, traditional Euclidean geometry is ill-suited and introduces a swelling effect. Particularly, the Euclidean distance could result in a wrong characterization of the relationship between SPD matrices. Instead, Riemannian methods rely on a distance that respects the geometry of the SPD matrices space, based on geodesics, i.e., the shortest path that connects 2 elements and stays in the space of SPD matrices. A common choice of distance is the affine-invariant one [79]. Considering the manifold of SPD matrices $M_n = \{\mathbf{P} \in \mathbb{R}^{n \times n} | \mathbf{P} = \mathbf{P}^{\top} \text{ and } x^{\top} \mathbf{P} x > 0, \forall x \in \mathbb{R}^n\}$, the affine-invariant distance for $\mathbf{P}_1, \mathbf{P}_2 \in M$ is defined as

$$ \delta_R(\mathbf{P}_1, \mathbf{P}_2) = \left[ \sum_{i=1}^{n} \log^2 \lambda_i (\mathbf{P}_1^{-1} \mathbf{P}_2) \right]^{1/2} \quad (1) $$

where $\lambda_i(\mathbf{P})$ is the i-th eigenvalues of $\mathbf{P}$.

Similarly, the concept of the mean in Riemannian geometry must be redefined to ensure it belongs to the manifold; it is known as the Frechet mean [79] and is defined as:

$$ \hat{\mathbf{G}} = \underset{\mathbf{P} \in P(n)}{\operatorname{argmin}} \sum_{i=1}^{m} \delta_{R}^{2}(\mathbf{P}, \mathbf{P}_{i}) \quad (2) $$

As we are operating within a Riemannian manifold, traditional machine-learning classification algorithms cannot be directly applied. Instead, there are two options: either create new algorithms to classify SPD matrices on the Riemannian manifold or map the matrices to an associated Euclidean space and then apply standard classification algorithms. In the former scenario, algorithms like Minimum Distance to Mean give robust accuracy. In the latter scenario, it is possible to rely on the tangent
class, while maximizing the between-class variance [81, 19]. To enhance the robustness of the CSP against noise and overfitting, Tikhonov Regularized CSP (CSP) approach has been proposed in [73]. In contrast to the classical CSP, which uses band-pass filtered EEG signals that may vary per subject, Filter-Bank CSP (CSP) addresses this issue by extracting CSP features for each band-pass filter from the filter bank [2]. The subsection Appendix B.1 details the CSP algorithm.

Canonical Correlation Analysis (CCA) has emerged as a prominent approach for classifying SSVEP signals, initially introduced in the work by [72]. Subsequently, CCA has been successfully employed in numerous notable SSVEP-based BCI studies, such as those conducted by [18] and [83]. SSVEP signals exhibit correlation to flickering visual stimuli, with their signal phase and frequency corresponding to stimulus characteristics. Leveraging this relationship, CCA aids in extracting EEG spatial components with the strongest correlation to SSVEP stimuli. Further details on the different pipelines can be found in subsection Appendix B.2.

## 4.2. Riemannian geometry

The introduction of Riemannian geometry into BCI processing marked a pivotal moment for the BCI community [13]. The fundamental concept underlying this approach is to represent the signal using covariance matrices or their derivatives. As covariance matrices are Symmetric Positive Definite (SPD) matrices, they live in a Riemannian space [124]. We provide here a general description of the framework needed to define algorithms based on the Riemann distance for classification tasks. For a more in-depth description of the Riemannian

framework, we refer the reader to [20] which gives a pedagogical introduction to these concepts.

Due to the curvature of the SPD matrix space, traditional Euclidean geometry is ill-suited and introduces a swelling effect. Particularly, the Euclidean distance could result in a wrong characterization of the relationship between SPD matrices. Instead, Riemannian methods rely on a distance that respects the geometry of the SPD matrices space, based on geodesics, i.e., the shortest path that connects 2 elements and stays in the space of SPD matrices. A common choice of distance is the affine-invariant one [79]. Considering the manifold of SPD matrices $M_n = \{\mathbf{P} \in \mathbb{R}^{n \times n} | \mathbf{P} = \mathbf{P}^{\top} \text{ and } x^{\top} \mathbf{P} x > 0, \forall x \in \mathbb{R}^n\}$, the affine-invariant distance for $\mathbf{P}_1, \mathbf{P}_2 \in M$ is defined as

$$ \delta_R(\mathbf{P}_1, \mathbf{P}_2) = \left[ \sum_{i=1}^{n} \log^2 \lambda_i (\mathbf{P}_1^{-1} \mathbf{P}_2) \right]^{1/2} \quad (1) $$

where $\lambda_i(\mathbf{P})$ is the i-th eigenvalues of $\mathbf{P}$.

Similarly, the concept of the mean in Riemannian geometry must be redefined to ensure it belongs to the manifold; it is known as the Frechet mean [79] and is defined as:

$$ \hat{\mathbf{G}} = \underset{\mathbf{P} \in P(n)}{\operatorname{argmin}} \sum_{i=1}^{m} \delta_{R}^{2}(\mathbf{P}, \mathbf{P}_{i}) \quad (2) $$

As we are operating within a Riemannian manifold, traditional machine-learning classification algorithms cannot be directly applied. Instead, there are two options: either create new algorithms to classify SPD matrices on the Riemannian manifold or map the matrices to an associated Euclidean space and then apply standard classification algorithms. In the former scenario, algorithms like Minimum Distance to Mean give robust accuracy. In the latter scenario, it is possible to rely on the tangent
space to a point of the manifold. It is possible to circulate between the manifold and the tangent space using the Log and Exp map functions. The Log (resp. Exp) maps the manifold to the tangent space (resp. the tangent space to the manifold):

$$ \mathrm{Exp}_{\mathbf{P}}(\mathbf{S}_i) = \mathbf{P}^{1/2} \mathrm{Exp}(\mathbf{P}^{-1/2} \mathbf{S}_i \mathbf{P}^{-1/2}) \mathbf{P}^{1/2} \quad (3) $$

$$ \mathrm{Log}_{\mathbf{P}}(\mathbf{P}_i) = \mathbf{P}^{1/2} \mathrm{Log}(\mathbf{P}^{-1/2} \mathbf{P}_i \mathbf{P}^{-1/2}) \mathbf{P}^{1/2} \quad (4) $$

Projected in the tangent space, any machine learning algorithm could be applied. One limitation is the size of the considered space, that is $\frac{n(n+1)}{2}$ for $M_n$. Machine learning algorithms like Support Vector Machine (SVM), ElasticNet (EL) or Logistic Regression (LR) are among the most popular for classification in the tangent space. Details regarding the pipelines implementation are available in subsection Appendix B.3.

### 4.3. Deep learning

DL methods have demonstrated in recent years considerable promise in various tasks that involve handling massive volumes of digital data and those in different fields such as computer vision [66] and Natural Language Processing [117, 104].

This also holds for BCI applications. The BCI field has been significantly impacted by the integration of DL techniques [99, 36], which exhibit strong generalization capabilities, with transfer learning emerging as a key focus in BCI research. One notable advantage of DL models is their capacity to leverage vast datasets, a task typically challenging for classical Machine Learning (ML) algorithms. Moreover, by conducting all processing and classification steps within a neural network, DL models enable optimized end-to-end

learning. In this paper, we will focus specifically on DL methods that have been applied to MI paradigms.

The predominant Python libraries for DL are TENSORFLOW [1] and PyTorch [88]. To facilitate broader access to MOABB, we developed integration for both libraries using wrappers from SCIKERAS [42] (for TENSORFLOW) and SKORCH [114] (for PyTorch). We were also keen on integrating the BRAINDECODE [103] library, which incorporates several DL algorithms for EEG processing using PYTORCH.

Most DL architectures for EEG decoding operate on minimally pre-processed (bandpass filters) or raw epochs. To address the spatio-temporal complexity of EEG signals, convolutional layers are commonly employed with separable 1D convolution along the temporal and spatial dimension (i.e., EEG channels). By segregating the spatial and temporal convolutions, these models account for the fact that all EEG channels observe all brain sources.

Although the goal is to minimize pre-processing steps before passing data to DL pipelines, a few steps are usually necessary. Bandpass filtering is applied to ensure a fair comparison between ML and DL methods. Neural network architectures reviewed here are designed for decoding EEG signals at a certain sampling frequency. Resampling datasets to match DL architectures' expected frequencies is performed to avoid interfering with the relative temporal length of their kernels. Standard rescaling of the signal is implemented before DL pipelines in adherence to common deep learning practices [68].

Data augmentation, a technique generating synthetic training samples through transformations, is well-established in computer vision for producing state-of-the-art results [26]. However, in the realm of EEG applications,
space to a point of the manifold. It is possible to circulate between the manifold and the tangent space using the Log and Exp map functions. The Log (resp. Exp) maps the manifold to the tangent space (resp. the tangent space to the manifold):

$$ \mathrm{Exp}_{\mathbf{P}}(\mathbf{S}_i) = \mathbf{P}^{1/2} \mathrm{Exp}(\mathbf{P}^{-1/2} \mathbf{S}_i \mathbf{P}^{-1/2}) \mathbf{P}^{1/2} \quad (3) $$

$$ \mathrm{Log}_{\mathbf{P}}(\mathbf{P}_i) = \mathbf{P}^{1/2} \mathrm{Log}(\mathbf{P}^{-1/2} \mathbf{P}_i \mathbf{P}^{-1/2}) \mathbf{P}^{1/2} \quad (4) $$

Projected in the tangent space, any machine learning algorithm could be applied. One limitation is the size of the considered space, that is $\frac{n(n+1)}{2}$ for $M_n$. Machine learning algorithms like Support Vector Machine (SVM), ElasticNet (EL) or Logistic Regression (LR) are among the most popular for classification in the tangent space. Details regarding the pipelines implementation are available in subsection Appendix B.3.

### 4.3. Deep learning

DL methods have demonstrated in recent years considerable promise in various tasks that involve handling massive volumes of digital data and those in different fields such as computer vision [66] and Natural Language Processing [117, 104].

This also holds for BCI applications. The BCI field has been significantly impacted by the integration of DL techniques [99, 36], which exhibit strong generalization capabilities, with transfer learning emerging as a key focus in BCI research. One notable advantage of DL models is their capacity to leverage vast datasets, a task typically challenging for classical Machine Learning (ML) algorithms. Moreover, by conducting all processing and classification steps within a neural network, DL models enable optimized end-to-end

learning. In this paper, we will focus specifically on DL methods that have been applied to MI paradigms.

The predominant Python libraries for DL are TENSORFLOW [1] and PyTorch [88]. To facilitate broader access to MOABB, we developed integration for both libraries using wrappers from SCIKERAS [42] (for TENSORFLOW) and SKORCH [114] (for PyTorch). We were also keen on integrating the BRAINDECODE [103] library, which incorporates several DL algorithms for EEG processing using PYTORCH.

Most DL architectures for EEG decoding operate on minimally pre-processed (bandpass filters) or raw epochs. To address the spatio-temporal complexity of EEG signals, convolutional layers are commonly employed with separable 1D convolution along the temporal and spatial dimension (i.e., EEG channels). By segregating the spatial and temporal convolutions, these models account for the fact that all EEG channels observe all brain sources.

Although the goal is to minimize pre-processing steps before passing data to DL pipelines, a few steps are usually necessary. Bandpass filtering is applied to ensure a fair comparison between ML and DL methods. Neural network architectures reviewed here are designed for decoding EEG signals at a certain sampling frequency. Resampling datasets to match DL architectures' expected frequencies is performed to avoid interfering with the relative temporal length of their kernels. Standard rescaling of the signal is implemented before DL pipelines in adherence to common deep learning practices [68].

Data augmentation, a technique generating synthetic training samples through transformations, is well-established in computer vision for producing state-of-the-art results [26]. However, in the realm of EEG applications,
data augmentation poses unique challenges. Despite its potential to reduce overfitting and enable complex algorithms, it is still an active area of research. Assessing whether a generated signal accurately captures the physiological attributes of EEG remains an open question. Thus, this study abstained from employing data augmentation procedures in the context of EEG decoding. For a comprehensive examination of various existing techniques for EEG, we refer the reader to [97].

Consistent parameters were used across all DL experiments, training networks for 300 epochs with a batch size of 64, Adam optimizer [60] with a base learning rate of 0.001 and cross-entropy as the loss function. An early stopping strategy with a patience parameter of 75 epochs was crucial to prevent overfitting. While the considered DL architecture hyperparameters align with state-of-the-art standards through signal resampling, further optimization could enhance model performance via a grid-search approach tailored to individual scenarios.

# 5. Experimental results

The experimental results outlined in this section encapsulate the key insights gathered during this benchmark study. While we provide only the most important findings to the reader in this section, we also report all the raw results to ensure proper reproducibility. Due to space limitations, all the detailed evaluations, including ROC-AUC or accuracy scores, are available in the appendix. Tables D1, D2, D3, D4 and D5 display the average scores of each pipeline across all subjects and sessions within a specific dataset. Additionally, Figures C1, C2, C3, C4 and C5 present the pipeline groups' scores for each subject and session across all

datasets to aid in result interpretation.

## 5.1. Riemannian pipelines outperforms others pipelines

The Riemannian distance-based classification pipeline consistently outperforms results obtained through DL and Raw pipelines across all datasets, on all paradigms. Figure 3 illustrates this superiority in the context of right-hand vs left-hand classification, SSVEP and P300. The ROC-AUC and accuracy results are shown based on each pipeline's performance relative to its category (Raw, Riemannian, DL). It should be emphasized that each dot on the plots represents the average of all pipelines from a category on a dataset, each dataset encompassing 10 to 100 subjects. In the distributions, each dataset has the same weight, regardless of the number of subjects or sessions. The results presented here are thus summarizing the largest BCI study to date.

The plots demonstrate the dominance of the Riemannian approaches, showcasing its superior performance not only in overall averages but also consistently across all datasets examined. This trend is further reinforced by the noticeable peak shift in the distribution of results, providing additional confirmation of the effectiveness of the Riemannian approaches. Similar findings are observed across various classification tasks within the Motor Imagery paradigm, as could be seen in the appendix on Figures C1 and C2.

The suboptimal performance of DL pipelines can be attributed to two main factors. Firstly, the hyperparameters of DL pipelines were not fine-tuned for each dataset; instead, the parameters described in the original articles were employed. This marks a significant divergence from the Riemannian and Raw approaches, which had their hyperpa-
data augmentation poses unique challenges. Despite its potential to reduce overfitting and enable complex algorithms, it is still an active area of research. Assessing whether a generated signal accurately captures the physiological attributes of EEG remains an open question. Thus, this study abstained from employing data augmentation procedures in the context of EEG decoding. For a comprehensive examination of various existing techniques for EEG, we refer the reader to [97].

Consistent parameters were used across all DL experiments, training networks for 300 epochs with a batch size of 64, Adam optimizer [60] with a base learning rate of 0.001 and cross-entropy as the loss function. An early stopping strategy with a patience parameter of 75 epochs was crucial to prevent overfitting. While the considered DL architecture hyperparameters align with state-of-the-art standards through signal resampling, further optimization could enhance model performance via a grid-search approach tailored to individual scenarios.

# 5. Experimental results

The experimental results outlined in this section encapsulate the key insights gathered during this benchmark study. While we provide only the most important findings to the reader in this section, we also report all the raw results to ensure proper reproducibility. Due to space limitations, all the detailed evaluations, including ROC-AUC or accuracy scores, are available in the appendix. Tables D1, D2, D3, D4 and D5 display the average scores of each pipeline across all subjects and sessions within a specific dataset. Additionally, Figures C1, C2, C3, C4 and C5 present the pipeline groups' scores for each subject and session across all

datasets to aid in result interpretation.

## 5.1. Riemannian pipelines outperforms others pipelines

The Riemannian distance-based classification pipeline consistently outperforms results obtained through DL and Raw pipelines across all datasets, on all paradigms. Figure 3 illustrates this superiority in the context of right-hand vs left-hand classification, SSVEP and P300. The ROC-AUC and accuracy results are shown based on each pipeline's performance relative to its category (Raw, Riemannian, DL). It should be emphasized that each dot on the plots represents the average of all pipelines from a category on a dataset, each dataset encompassing 10 to 100 subjects. In the distributions, each dataset has the same weight, regardless of the number of subjects or sessions. The results presented here are thus summarizing the largest BCI study to date.

The plots demonstrate the dominance of the Riemannian approaches, showcasing its superior performance not only in overall averages but also consistently across all datasets examined. This trend is further reinforced by the noticeable peak shift in the distribution of results, providing additional confirmation of the effectiveness of the Riemannian approaches. Similar findings are observed across various classification tasks within the Motor Imagery paradigm, as could be seen in the appendix on Figures C1 and C2.

The suboptimal performance of DL pipelines can be attributed to two main factors. Firstly, the hyperparameters of DL pipelines were not fine-tuned for each dataset; instead, the parameters described in the original articles were employed. This marks a significant divergence from the Riemannian and Raw approaches, which had their hyperpa-
rameters optimized through a nested cross-
validation strategy. We could not complete a
similar hyperparameter search for DL as the
search space is too large and it involves com-
plex changes in the architecture shape, such as
the kernel size, or the activation functions that
could not be automatized easily. Secondly, we
chose not to include any data augmentation
steps in our research, as explained in Sect. 4.3.
It is noteworthy that such procedures have
been demonstrated to exert a substantial im-
pact on performance, as evidenced by previous
studies [97]. Still, it required a lengthy and
dataset-specific parametrization. It is impor-
tant to recall that the objective of this study is
to assess existing and published algorithms, to
evaluate their off-the-shelve performances, and
not to investigate how to properly tune them.

## 5.2. Riemannian pipelines work well with limited number of electrodes

Riemannian pipelines perform best in scenar-
ios involving a reduced number of channels for
MI tasks. Employing a limited number of elec-
trodes in EEG recordings offers numerous ad-
vantages for practical BCI experiments. This
approach simplifies the setup, reducing both
complexity and cost. Additionally, it enhances
user comfort, providing greater flexibility and
ease of use in diverse settings, including home-
based or mobile applications. However, using
fewer electrodes may compromise signal qual-
ity and spatial resolution, potentially resulting
in lower classification accuracy and reduced ro-
bustness against artifacts and noise.

To address these challenges, it becomes
crucial to design classification algorithms
capable of delivering high performance with
a limited number of electrodes. As depicted
in Figure 4-(a), the Riemannian pipelines
excel in performance with datasets containing

[0, 25] electrodes. Notably, their performances
tend to decrease as the number of channels
increases. Conversely, DL pipelines require a
substantial amount of information to achieve
satisfactory performance, while facing the
limitations described earlier, emphasizing the
importance of balancing electrode count and
classification effectiveness.

As observed, there is a decline in
performance in settings with a moderate
number of channels. This decrease in
effectiveness can be attributed to several
factors. Primarily, an increased number of
electrodes elevates the problem's complexity,
as the ML algorithm needs to extract relevant
information more efficiently. Furthermore, this
category encompasses datasets with varying
average scores, impacting overall performance.
Conversely, scenarios with a high number of
electrodes exhibit less pronounced detrimental
effects. This could be a side effect; a
large number of electrodes implies higher-
grade equipment and specialized technicians.
The enhanced data recording procedures and
superior data quality associated with a larger
number of channels might thus alleviate
previous issues.

Classification algorithms based on Rie-
mannian distances can be implemented in two
distinct approaches. The first method in-
volves performing classification directly on the
Riemannian manifold, using algorithms like
the MDM (Minimum Distance to Mean) algo-
rithm. The alternative approach involves pro-
jecting data onto the tangent space and con-
ducting classification using conventional ML
algorithms (SVM, LR, EL). Comparing the
two strategies, it is observed that the Rie-
mannian method based on Tangent space pro-
jection consistently outperforms the approach
centered on the Riemannian surface as shown
in Figure 4-(b).
rameters optimized through a nested cross-
validation strategy. We could not complete a
similar hyperparameter search for DL as the
search space is too large and it involves com-
plex changes in the architecture shape, such as
the kernel size, or the activation functions that
could not be automatized easily. Secondly, we
chose not to include any data augmentation
steps in our research, as explained in Sect. 4.3.
It is noteworthy that such procedures have
been demonstrated to exert a substantial im-
pact on performance, as evidenced by previous
studies [97]. Still, it required a lengthy and
dataset-specific parametrization. It is impor-
tant to recall that the objective of this study is
to assess existing and published algorithms, to
evaluate their off-the-shelve performances, and
not to investigate how to properly tune them.

## 5.2. Riemannian pipelines work well with limited number of electrodes

Riemannian pipelines perform best in scenar-
ios involving a reduced number of channels for
MI tasks. Employing a limited number of elec-
trodes in EEG recordings offers numerous ad-
vantages for practical BCI experiments. This
approach simplifies the setup, reducing both
complexity and cost. Additionally, it enhances
user comfort, providing greater flexibility and
ease of use in diverse settings, including home-
based or mobile applications. However, using
fewer electrodes may compromise signal qual-
ity and spatial resolution, potentially resulting
in lower classification accuracy and reduced ro-
bustness against artifacts and noise.

To address these challenges, it becomes
crucial to design classification algorithms
capable of delivering high performance with
a limited number of electrodes. As depicted
in Figure 4-(a), the Riemannian pipelines
excel in performance with datasets containing

[0, 25] electrodes. Notably, their performances
tend to decrease as the number of channels
increases. Conversely, DL pipelines require a
substantial amount of information to achieve
satisfactory performance, while facing the
limitations described earlier, emphasizing the
importance of balancing electrode count and
classification effectiveness.

As observed, there is a decline in
performance in settings with a moderate
number of channels. This decrease in
effectiveness can be attributed to several
factors. Primarily, an increased number of
electrodes elevates the problem's complexity,
as the ML algorithm needs to extract relevant
information more efficiently. Furthermore, this
category encompasses datasets with varying
average scores, impacting overall performance.
Conversely, scenarios with a high number of
electrodes exhibit less pronounced detrimental
effects. This could be a side effect; a
large number of electrodes implies higher-
grade equipment and specialized technicians.
The enhanced data recording procedures and
superior data quality associated with a larger
number of channels might thus alleviate
previous issues.

Classification algorithms based on Rie-
mannian distances can be implemented in two
distinct approaches. The first method in-
volves performing classification directly on the
Riemannian manifold, using algorithms like
the MDM (Minimum Distance to Mean) algo-
rithm. The alternative approach involves pro-
jecting data onto the tangent space and con-
ducting classification using conventional ML
algorithms (SVM, LR, EL). Comparing the
two strategies, it is observed that the Rie-
mannian method based on Tangent space pro-
jection consistently outperforms the approach
centered on the Riemannian surface as shown
in Figure 4-(b).
<figure><img src="image_5.png" /><figcaption>Figure 3: Average performance of pipelines grouped by category (Deep Learning, Riemannian, and Raw) across the MI (right-hand vs left-hand), SSVEP, and ERP paradigms displayed as raincloud plots. Each point in the plot corresponds to the average score of one dataset across all pipelines within a specific category, encompassing all subjects and sessions.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 4: (a) AUC scores are averaged across all sessions, subjects, and datasets within the right-hand vs left-hand MI paradigm for each category (Deep Learning, Riemannian, Raw), segmented by the number of channels on the y-axis. Box plots overlaid with strip plots show individual ROC-AUC scores. (b) Distribution of AUC scores for the Riemannian MI pipelines is depicted for the right-hand vs left-hand classification task. The boxes and horizontal black bars denote quartile ranges.</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Figure 3: Average performance of pipelines grouped by category (Deep Learning, Riemannian, and Raw) across the MI (right-hand vs left-hand), SSVEP, and ERP paradigms displayed as raincloud plots. Each point in the plot corresponds to the average score of one dataset across all pipelines within a specific category, encompassing all subjects and sessions.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4: (a) AUC scores are averaged across all sessions, subjects, and datasets within the right-hand vs left-hand MI paradigm for each category (Deep Learning, Riemannian, Raw), segmented by the number of channels on the y-axis. Box plots overlaid with strip plots show individual ROC-AUC scores. (b) Distribution of AUC scores for the Riemannian MI pipelines is depicted for the right-hand vs left-hand classification task. The boxes and horizontal black bars denote quartile ranges.</figcaption></figure>
### 5.3. Deep learning requires a high number of trials

It is essential to emphasize that the lower performance of DL pipelines, in comparison to the other pipeline categories, is closely linked to the specific DL architecture under consideration. This observation is illustrated in Figure 5-(a). Amongst DL pipelines, Shallow-ConvNet stands out with the highest AUC. Interestingly, a distinctive dichotomy appears within DL models. The first group, consisting of ShallowConvNet, EEGNet-8.2, and Deep-ConvNet, exhibits superior performance. In contrast, the second group, gathering EEGTC-Net, EEGITNet, and EEGNeX, shows lower performance levels. This divergence in performance is likely attributed to the optimization of hyperparameters in the models. It demonstrates that DL models that have been more extensively tested, and hence correctly parametrized, yield higher classification performance on all datasets.

The number of trials employed in training algorithms carries particular importance, specifically in the context of DL pipelines. A clear trend emerges when evaluating DL algorithm performance, indicating that achieving satisfactory results typically requires more than 150 trials per class, as shown in Figure 5-(b). Again we observe the same two distinct groups described above, the group comprising ShallowConvNet, EEGNet-8.2, and DeepConvNet exhibits a higher resilience to the impact of the number of trials. Notably, this group manages to achieve already satisfactory performance with as few as 50 trials per class, showcasing a relatively robust response to variations in the training dataset size, highlighting the distinct capabilities of certain architectures to yield superior performance with a more limited number of trials.

### 5.4. Recommended number of trials depends on the task complexity

In scenarios where tasks exhibit a clear distinguishability, such as the MI task involving right-hand/feet movements, achieving impressive AUC performance with a limited number of trials is feasible, as clearly depicted in Figure 6a. Notably, both the Raw and Rieman-nian pipelines demonstrate exceptional classification scores even with fewer than 50 trials. Increasing the trial count beyond 50 does not yield significant improvements in AUC results for these pipelines. On the other hand, for DL pipelines to reach good AUC, datasets associ-ated with more than 150 trials are imperative to achieve satisfactory AUC scores, highlight-ing the pivotal role trial quantity plays in DL model performance. This discrepancy high-lights the varying requirements and efficiencies of different pipeline approaches based on the task complexity and nature of the data.

When dealing with tasks of higher com-plexity, such as MI paradigms involving 3 to 7 distinct classes (refer to Section 3.1 for de-tailed explanation), obtaining optimal accu-racy becomes considerably more challenging. In such intricate tasks, a larger number of tri-als is imperative for pipelines to achieve the highest levels of accuracy. This phenomenon is illustrated in Figure 6b, where the accu-racy variability across different datasets be-comes apparent. The fluctuation in accuracy levels observed here is intricately tied to the unique characteristics and quality of individ-ual datasets, portraying the nuanced dynamic between dataset quality, trial quantity, and the performance of the classification pipelines. Specifically, the noticeable decline in accuracy for datasets with over 200 trials compared to datasets with 100 to 200 trials underscores the diverse demands and responses of different
### 5.3. Deep learning requires a high number of trials

It is essential to emphasize that the lower performance of DL pipelines, in comparison to the other pipeline categories, is closely linked to the specific DL architecture under consideration. This observation is illustrated in Figure 5-(a). Amongst DL pipelines, Shallow-ConvNet stands out with the highest AUC. Interestingly, a distinctive dichotomy appears within DL models. The first group, consisting of ShallowConvNet, EEGNet-8.2, and Deep-ConvNet, exhibits superior performance. In contrast, the second group, gathering EEGTC-Net, EEGITNet, and EEGNeX, shows lower performance levels. This divergence in performance is likely attributed to the optimization of hyperparameters in the models. It demonstrates that DL models that have been more extensively tested, and hence correctly parametrized, yield higher classification performance on all datasets.

The number of trials employed in training algorithms carries particular importance, specifically in the context of DL pipelines. A clear trend emerges when evaluating DL algorithm performance, indicating that achieving satisfactory results typically requires more than 150 trials per class, as shown in Figure 5-(b). Again we observe the same two distinct groups described above, the group comprising ShallowConvNet, EEGNet-8.2, and DeepConvNet exhibits a higher resilience to the impact of the number of trials. Notably, this group manages to achieve already satisfactory performance with as few as 50 trials per class, showcasing a relatively robust response to variations in the training dataset size, highlighting the distinct capabilities of certain architectures to yield superior performance with a more limited number of trials.

### 5.4. Recommended number of trials depends on the task complexity

In scenarios where tasks exhibit a clear distinguishability, such as the MI task involving right-hand/feet movements, achieving impressive AUC performance with a limited number of trials is feasible, as clearly depicted in Figure 6a. Notably, both the Raw and Rieman-nian pipelines demonstrate exceptional classification scores even with fewer than 50 trials. Increasing the trial count beyond 50 does not yield significant improvements in AUC results for these pipelines. On the other hand, for DL pipelines to reach good AUC, datasets associ-ated with more than 150 trials are imperative to achieve satisfactory AUC scores, highlight-ing the pivotal role trial quantity plays in DL model performance. This discrepancy high-lights the varying requirements and efficiencies of different pipeline approaches based on the task complexity and nature of the data.

When dealing with tasks of higher com-plexity, such as MI paradigms involving 3 to 7 distinct classes (refer to Section 3.1 for de-tailed explanation), obtaining optimal accu-racy becomes considerably more challenging. In such intricate tasks, a larger number of tri-als is imperative for pipelines to achieve the highest levels of accuracy. This phenomenon is illustrated in Figure 6b, where the accu-racy variability across different datasets be-comes apparent. The fluctuation in accuracy levels observed here is intricately tied to the unique characteristics and quality of individ-ual datasets, portraying the nuanced dynamic between dataset quality, trial quantity, and the performance of the classification pipelines. Specifically, the noticeable decline in accuracy for datasets with over 200 trials compared to datasets with 100 to 200 trials underscores the diverse demands and responses of different
<figure><img src="image_7.png" /><figcaption>Figure 5: (a) Distributions of AUC scores averaged over all datasets for the right-hand vs left-hand classification task within the DL pipelines. (b) AUC scores averaged across all sessions, subjects, and datasets within the right-hand vs left-hand MI paradigm for the DL pipelines, segmented based on the number of epochs on the y-axis.</figcaption></figure>

pipelines to varying trial quantities in complex classification tasks.

## 5.5. Best practices in motor imagery

Motor imagery is a widely utilized paradigm in the BCI community, and a detailed analysis was conducted on the MI results obtained in this benchmark to assist practitioners in experimental design and pipeline selection.

The highest performance levels in MI are achieved in the binary classification distinguishing between right-hand and feet movements. This task notably surpasses the performance of the right-hand/left-hand classification, highlighting a significant disparity in performance between the two tasks. This performance discrepancy is visible when comparing the results of right-hand/left-hand and right-hand/feet tasks in the supplementary Figure C1 and C2, respectively.

This trend is consistent across all pipeline categories – Raw, Riemannian and DL – and is prevalent across datasets, with 9 datasets for right-hand/feet tasks and 10 for right-hand/left-hand tasks. Five datasets are common to both tasks, with the AUC notably higher for the right-hand/feet task.

This result holds significant implications for the BCI community, as it provides for the first time valuable insight into selecting the task that yields the highest accuracy in MI. The findings presented here are highly reliable, drawn from a diverse range of subjects recorded under various protocols and using different hardware setups. Furthermore, since this trend is observable even when analyzing subjects performing both tasks, any confounding effects related to recording
<figure><img src="image_6.png" /><figcaption>Figure 5: (a) Distributions of AUC scores averaged over all datasets for the right-hand vs left-hand classification task within the DL pipelines. (b) AUC scores averaged across all sessions, subjects, and datasets within the right-hand vs feet MI paradigm for the DL pipelines, segmented based on the number of epochs on the y-axis.</figcaption></figure>

pipelines to varying trial quantities in complex classification tasks.

## 5.5. Best practices in motor imagery

Motor imagery is a widely utilized paradigm in the BCI community, and a detailed analysis was conducted on the MI results obtained in this benchmark to assist practitioners in experimental design and pipeline selection.

The highest performance levels in MI are achieved in the binary classification distinguishing between right-hand and feet movements. This task notably surpasses the performance of the right-hand/left-hand classification, highlighting a significant disparity in performance between the two tasks. This performance discrepancy is visible when comparing the results of right-hand/left-hand and right-hand/feet tasks in the supplementary Figure

C1 and C2, respectively. This trend is consistent across all pipeline categories – Raw, Riemannian and DL – and is prevalent across datasets, with 9 datasets for right-hand/feet tasks and 10 for right-hand/left-hand tasks. Five datasets are common to both tasks, with the AUC notably higher for the right-hand/feet task.

This results holds significant implications for the BCI community, as it provides for the first time valuable insight into selecting the task that yields the highest accuracy in MI. The findings presented here are highly reliable, drawn from a diverse range of subjects recorded under various protocols and using different hardware setups. Furthermore, since this trend is observable even when analyzing subjects performing both tasks, any confounding effects related to recording
<figure><img src="image_8.png" /><figcaption>Figure 6: (a) AUC scores for datasets segmented by the number of trials in the right-hand vs feet MI paradigm for each pipeline category (Deep learning, Riemannian, Raw). (b) Same results for MI using all available classes and the accuracy metric.</figcaption></figure>

conditions can be effectively eliminated. This insight holds particular importance for BCI practitioners when designing experimental protocols.

To delve deeper into how different pipelines compare in the right-hand/feet classification task, Figure 7-(a) illustrates the AUC scores of the top three pipelines in each category, arranged by their average scores per dataset. The very good performances of Raw and Riemannian pipelines are visible on this plot. The performances of the DL pipelines are below Raw and Riemannian pipelines for the reasons outlined previously.

While the AUC performance measurements offer valuable insights, Figure 7-(a) reveals significant subject variability. In practical BCI applications, the choice of the pipeline is often influenced by the algorithm's ranking, with the best-performing algorithm selected for each subject. To evaluate how pipelines fare based on this criterion, pipelines were ranked in each session for all subjects, ranging from 1 (the best) to 16 (the worst) based on

their relative scores. Figure 7-(b) visualizes the frequency of sessions (y-axis) where a pipeline achieves a specific rank (x-axis), using distinct colors for each pipeline.

While the results align with the average AUC scores presented in Figure 7-(a), they provide different qualitative insights. Firstly, the ACM+TS+SVM is outperforming other pipelines, consistently securing top positions and rarely falling below the 7th spot. Secondly, the TS+LR and TS+EL Riemannian pipelines exhibit comparable AUC scores, yet TS+EL consistently outperforms other pipelines, whereas TS+LR seldom claims the top spot but frequently ranks among the top 3 pipelines. Lastly, DL pipelines generally rank below the 9th position, excluding the notable exception of the ShallowConvNet pipeline, which attains top 3 rankings in several sessions, despite its lower average AUC score placing it behind CSP-based pipelines.

Accuracy is a primary criterion for selecting a BCI pipeline, but it's also essential to consider calibration time. Pipelines
<figure><img src="image_7.png" /><figcaption>Figure 6: (a) AUC scores for datasets segmented by the number of trials in the right-hand vs feet MI paradigm for each pipeline category (Deep learning, Riemannian, Raw). (b) Same results for MI using all available classes and the accuracy metric.</figcaption></figure>

conditions can be effectively eliminated. This insight holds particular importance for BCI practitioners when designing experimental protocols.

To delve deeper into how different pipelines compare in the right-hand/feet classification task, Figure 7-(a) illustrates the AUC scores of the top three pipelines in each category, arranged by their average scores per dataset. The very good performances of Raw and Riemannian pipelines are visible on this plot. The performances of the DL pipelines are below Raw and Riemannian pipelines for the reasons outlined previously.

While the AUC performance measurements offer valuable insights, Figure 7-(a) reveals significant subject variability. In practical BCI applications, the choice of the pipeline is often influenced by the algorithm's ranking, with the best-performing algorithm selected for each subject. To evaluate how pipelines fare based on this criterion, pipelines were ranked in each session for all subjects, ranging from 1 (the best) to 16 (the worst) based on

their relative scores. Figure 7-(b) visualizes the frequency of sessions (y-axis) where a pipeline achieves a specific rank (x-axis), using distinct colors for each pipeline.

While the results align with the average AUC scores presented in Figure 7-(a), they provide different qualitative insights. Firstly, the ACM+TS+SVM is outperforming other pipelines, consistently securing top positions and rarely falling below the 7th spot. Secondly, the TS+LR and TS+EL Riemannian pipelines exhibit comparable AUC scores, yet TS+EL consistently outperforms other pipelines, whereas TS+LR seldom claims the top spot but frequently ranks among the top 3 pipelines. Lastly, DL pipelines generally rank below the 9th position, excluding the notable exception of the ShallowConvNet pipeline, which attains top 3 rankings in several sessions, despite its lower average AUC score placing it behind CSP-based pipelines.

Accuracy is a primary criterion for selecting a BCI pipeline, but it's also essential to consider calibration time. Pipelines
are often trained immediately following a calibration run or updated after running multiple trials, making execution time a crucial factor in pipeline selection. User interaction is typically paused during pipeline training on the training dataset, emphasizing the significance of efficient execution. This aspect is crucial not only for real-time BCI operation but also for offline evaluation and hyperparameter optimization, as it directly impacts experiment duration.

To address this issue, the average execution times for pipeline categories (Raw, Riemannian, and DL) are presented in Figure 8-(a). These findings, focused on MI right-hand vs feet classification, are applicable across other tasks and paradigms as well. The measurements were conducted using the French Jean Zay CPU/GPU HPC environment, featuring Intel Cascade Lake 6248 CPUs and Nvidia V100 16 GB RAM, encompassing both training and inference phases for a single fold of cross-validation.

Raw pipelines demonstrate the shortest computational time requirements, closely followed by Riemannian pipelines. DL pipelines, on the other hand, exhibit longer training durations on average but remain within an acceptable 30-second range for experimental systems. It's noteworthy that these results were obtained using an early stopping strategy to prevent overfitting, which also contributes to reducing training time.

Environmental impact assessment is crucial in AI-related domains given the exponential growth in these areas. In this study, the direct environmental impact, measured in gCO2 equivalent generated during training and inference phases, is evaluated. The absolute values are significantly influenced by the country's energy production methods, hence the CPU/GPU server localization are important to

measure the generated gCO2 equivalent. Precisely measuring algorithm energy consumption is challenging, as existing libraries differ in solutions and measurements. Using Code Carbon [35], the environmental footprint, expressed in gCO2 equivalent emissions, for Riemannian TS+EL, Raw CSP+SVM, and ShallowConvNet pipelines is documented in Figure 8-(b). This provides a unified measure of required computational resources, illustrating that the Riemannian pipeline consumes less energy despite its longer training times as shown in Figure 8-(a).

## 6. Future Directions for reproducible BCI machine learning pipelines

The path towards open and reproducible approaches in BCI improved with initiatives in open EEG hardware [40, 21], libraries for experimental design [93, 90, 31] and, indeed, machine learning pipelines [4, 14]. For the latter, there is still room for improvements, along two main axes. The first one is to get closer to experimental situations and the second one is to allow fast benchmarking of the most recent BCI decoding techniques. An initiative‡ to tackle the second axes relies on BENCHOPT [80] and aims to provide an easy environment to evaluate novel BCI techniques along with reproducible evaluation conditions.

### 6.1. Pseudo-online benchmarking

The first limitation is that, in order to maintain compatibility with numerous datasets, the inherent chronology between epochs is disregarded, and the evaluation within each session consists of a simple 5-fold cross-validation over shuffled epochs. However, by leveraging historical knowledge, certain unsupervised

‡ https://github.com/benchopt/benchmark_bci
are often trained immediately following a calibration run or updated after running multiple trials, making execution time a crucial factor in pipeline selection. User interaction is typically paused during pipeline training on the training dataset, emphasizing the significance of efficient execution. This aspect is crucial not only for real-time BCI operation but also for offline evaluation and hyperparameter optimization, as it directly impacts experiment duration.

To address this issue, the average execution times for pipeline categories (Raw, Riemannian, and DL) are presented in Figure 8-(a). These findings, focused on MI right-hand vs feet classification, are applicable across other tasks and paradigms as well. The measurements were conducted using the French Jean Zay CPU/GPU HPC environment, featuring Intel Cascade Lake 6248 CPUs and Nvidia V100 16 GB RAM, encompassing both training and inference phases for a single fold of cross-validation.

Raw pipelines demonstrate the shortest computational time requirements, closely followed by Riemannian pipelines. DL pipelines, on the other hand, exhibit longer training durations on average but remain within an acceptable 30-second range for experimental systems. It's noteworthy that these results were obtained using an early stopping strategy to prevent overfitting, which also contributes to reducing training time.

Environmental impact assessment is crucial in AI-related domains given the exponential growth in these areas. In this study, the direct environmental impact, measured in gCO2 equivalent generated during training and inference phases, is evaluated. The absolute values are significantly influenced by the country's energy production methods, hence the CPU/GPU server localization are important to

measure the generated gCO2 equivalent. Precisely measuring algorithm energy consumption is challenging, as existing libraries differ in solutions and measurements. Using Code Carbon [35], the environmental footprint, expressed in gCO2 equivalent emissions, for Riemannian TS+EL, Raw CSP+SVM, and ShallowConvNet pipelines is documented in Figure 8-(b). This provides a unified measure of required computational resources, illustrating that the Riemannian pipeline consumes less energy despite its longer training times as shown in Figure 8-(a).

# 6. Future Directions for reproducible BCI machine learning pipelines

The path towards open and reproducible approaches in BCI improved with initiatives in open EEG hardware [40, 21], libraries for experimental design [93, 90, 31] and, indeed, machine learning pipelines [4, 14]. For the latter, there is still room for improvements, along two main axes. The first one is to get closer to experimental situations and the second one is to allow fast benchmarking of the most recent BCI decoding techniques. An initiative‡ to tackle the second axes relies on BENCHOPT [80] and aims to provide an easy environment to evaluate novel BCI techniques along with reproducible evaluation conditions.

## 6.1. Pseudo-online benchmarking

The first limitation is that, in order to maintain compatibility with numerous datasets, the inherent chronology between epochs is disregarded, and the evaluation within each session consists of a simple 5-fold cross-validation over shuffled epochs. However, by leveraging historical knowledge, certain unsupervised

‡ https://github.com/benchopt/benchmark_bci
<figure><img src="image_9.png" /><figcaption>Figure 7: (a) AUC scores are presented for the best three motor imagery pipelines in each category for the right-hand vs feet classification task, ordered by their average score per dataset. (b) Pipeline rankings within individual sessions for the right-hand vs feet task, with each pipeline color-coded. The x-axis displays different ranks achieved by pipelines, while the y-axis indicates the number of sessions each pipeline achieves a specific rank.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 8: (a) Average execution times in seconds per dataset for the MI right-hand vs feet paradigm, segmented by pipeline category (DL, Riemannian, Raw). (b) Carbon emissions in gCO2 equivalent for the high-ranking pipelines (Raw CSP+SVM, Riemannian TS+EL, DL ShallowConvNet) in the MI right-hand vs feet task.</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Figure 7: (a) AUC scores are presented for the best three motor imagery pipelines in each category for the right-hand vs feet classification task, ordered by their average score per dataset. (b) Pipeline rankings within individual sessions for the right-hand vs feet task, with each pipeline color-coded. The x-axis displays different ranks achieved by pipelines, while the y-axis indicates the number of sessions each pipeline achieves a specific rank.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 8: (a) Average execution times in seconds per dataset for the MI right-hand vs feet paradigm, segmented by pipeline category (DL, Riemannian, Raw). (b) Carbon emissions in gCO2 equivalent for the high-ranking pipelines (Raw CSP+SVM, Riemannian TS+EL, DL ShallowConvNet) in the MI right-hand vs feet task.</figcaption></figure>
classification techniques can rival supervised ones [51, 107]. This evaluation presents a significant challenge as it completely overlooks the non-stationarity of the data, causing some classifiers to perform well under these conditions but fail in an online scenario. Consequently, in the near future, it is important to integrate pseudo-online benchmarking of algorithms, like along the lines proposed in [23].

In real online experiments, we anticipate a decrease in accuracy with excessively long calibration periods. User feedback could prove to be highly motivating, a factor that remains undisclosed in offline evaluations.

Regarding pseudo-online evaluation, we hypothesize that a gradual distribution shift occurs during the session. Achieving high accuracy becomes considerably more challenging as training data only capture a limited range of subject variability. Additionally, we anticipate observing learning curves that ascend and level off due to this shift; a plateau phenomenon not observed in offline analyses, where accuracy appears to increase as more data is amassed.

## 6.2. CVEP paradigm

Most open data in BCI include MI, ERP, and SSVEP paradigms. A recent addition to the landscape of evoked potentials, alongside the well-established ERP and SSVEP scenario, is the emerging Code-modulated Visually Evoked Potentials (c-VEP) paradigm [78]. Drawing an analogy to telecommunications, these three evoked paradigms can be likened to time-domain, frequency-domain, and code-domain multiple access schemes [41]. Notably, research has demonstrated that the c-VEP paradigm exhibits superior performance compared to both ERP and SSVEP paradigms [17], garnering increasing attention and leading to the development of high-speed

BCIs measured through Information Transfer Rate (ITR); see, for instance, [82, 113, 110]).

## 6.3. Character-level benchmarking of ERP and c-VEP

The decoding algorithms for ERP (and soon c-VEP) are mostly benchmarked at the epoch classification level, typically addressing a binary problem where the algorithms need to predict whether an epoch corresponds to a *target* or a *non-target* in the original application. However, this decoding approach overlooks the specific character or target being attended to in the original application, despite the availability of information regarding the stimulus sequence used for each character. Recent advancements have introduced methods that leverage application-level information to reduce the number of target hypotheses [59, 16, 15]. Moreover, unsupervised classifiers have been proposed that capitalize on the structural sequence information, showcasing the potential to surpass established, even supervised algorithms [107, 112]. Unfortunately, the emphasis on binary decoding has led to incomplete availability of the necessary structural information in all existing ERP datasets.

## 6.4. Cross-dataset transfer learning

Transfer learning has consistently posed a significant challenge in the realm of BCI [55, 57]. While current support includes benchmarks for cross-session and cross-subject scenarios, the absence of benchmarks across datasets remains a gap. Recent advancements in DL models have yielded remarkably high performances in solving cross-dataset transfer learning challenges, particularly in MI paradigms [58, 45, 121, 5]. This surge in interest toward cross-dataset transfer no longer stems solely from
classification techniques can rival supervised ones [51, 107]. This evaluation presents a significant challenge as it completely overlooks the non-stationarity of the data, causing some classifiers to perform well under these conditions but fail in an online scenario. Consequently, in the near future, it is important to integrate pseudo-online benchmarking of algorithms, like along the lines proposed in [23].

In real online experiments, we anticipate a decrease in accuracy with excessively long calibration periods. User feedback could prove to be highly motivating, a factor that remains undisclosed in offline evaluations.

Regarding pseudo-online evaluation, we hypothesize that a gradual distribution shift occurs during the session. Achieving high accuracy becomes considerably more challenging as training data only capture a limited range of subject variability. Additionally, we anticipate observing learning curves that ascend and level off due to this shift; a plateau phenomenon not observed in offline analyses, where accuracy appears to increase as more data is amassed.

## 6.2. CVEP paradigm

Most open data in BCI include MI, ERP, and SSVEP paradigms. A recent addition to the landscape of evoked potentials, alongside the well-established ERP and SSVEP scenario, is the emerging Code-modulated Visually Evoked Potentials (c-VEP) paradigm [78]. Drawing an analogy to telecommunications, these three evoked paradigms can be likened to time-domain, frequency-domain, and code-domain multiple access schemes [41]. Notably, research has demonstrated that the c-VEP paradigm exhibits superior performance compared to both ERP and SSVEP paradigms [17], garnering increasing attention and leading to the development of high-speed

BCIs measured through Information Transfer Rate (ITR); see, for instance, [82, 113, 110]).

## 6.3. Character-level benchmarking of ERP and c-VEP

The decoding algorithms for ERP (and soon c-VEP) are mostly benchmarked at the epoch classification level, typically addressing a binary problem where the algorithms need to predict whether an epoch corresponds to a *target* or a *non-target* in the original application. However, this decoding approach overlooks the specific character or target being attended to in the original application, despite the availability of information regarding the stimulus sequence used for each character. Recent advancements have introduced methods that leverage application-level information to reduce the number of target hypotheses [59, 16, 15]. Moreover, unsupervised classifiers have been proposed that capitalize on the structural sequence information, showcasing the potential to surpass established, even supervised algorithms [107, 112]. Unfortunately, the emphasis on binary decoding has led to incomplete availability of the necessary structural information in all existing ERP datasets.

## 6.4. Cross-dataset transfer learning

Transfer learning has consistently posed a significant challenge in the realm of BCI [55, 57]. While current support includes benchmarks for cross-session and cross-subject scenarios, the absence of benchmarks across datasets remains a gap. Recent advancements in DL models have yielded remarkably high performances in solving cross-dataset transfer learning challenges, particularly in MI paradigms [58, 45, 121, 5]. This surge in interest toward cross-dataset transfer no longer stems solely from
fundamental research but also unfolds as a
promising avenue for future BCIs.

7. Conclusion

This contribution represents the largest repro-
ducibility study in EEG-based BCI, leverag-
ing the MOABB library. By utilizing openly
available data collected from different hard-
ware sources in varied formats and structures,
a systematic benchmarking process was under-
taken. Machine learning pipelines from pub-
lished works were re-implemented in a uni-
fied and open framework, aligning with the
established standards of the machine learn-
ing community. This effort extends to DL
pipelines, considering the rapidly evolving pro-
cessing standards for time series data.

The study's strength lies in the extensive
number of subjects analyzed across diverse
datasets, enabling robust assessments through
meta-analysis statistical techniques. The
pipelines undergo evaluation using 5-fold cross-
validation, employing the AUC metric for
binary classification tasks and accuracy for
datasets with multiple classes. Furthermore,
the environmental impact of the pipelines is
assessed and factored into the reported results.

The primary outcome is a comprehensive
and meticulous benchmarking of prominent
pipelines from within the BCI literature. Re-
sources are provided to reproduce these results
and facilitate comparisons with future works,
including result tables in the annexes and
on a dedicated online platform to streamline
comparisons and avoid unnecessary duplica-
tions. The Riemannian pipelines demonstrate
the highest accuracy, whereas DL pipelines,
while achieving admirable accuracy with ex-
tensive trial data, show limitations across most
datasets. Although data augmentation tech-
niques and advanced parameterization can en-

hance their performance, improvements are
still required for these off-the-shelf pipelines.

As the benchmark incorporates various
pipelines and datasets, recommendations can
be formulated regarding the optimal number
of trials or channels for designing BCI
experiments to achieve peak performance. A
detailed analysis of all considered datasets
is presented, aiding practitioners in tailoring
their experimental designs or selecting specific
datasets for evaluating novel pipelines.

Future development avenues are outlined
along two key directions. Firstly, benchmarks
could progress towards evaluations that mir-
ror real-world experimental conditions, aim-
ing to narrow the disparity between offline
assessment and practical online BCI applica-
tions. Secondly, the integration of novel BCI
paradigms like CVEP and transfer learning ap-
proaches across datasets is suggested for fur-
ther exploration and integration.

8. Acknowledgements

SC, BA and SS were supported by DATAIA
Convergence Institute as part of the “Pro-
gramme d’Investissement d’Avenir”, (ANR-
17-CONV-0003) operated by LISN-CNRS.
This work was granted access to the HPC re-
sources of IDRIS under the allocation 2023-
AD011014322 made by GENCI.

References

[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens,
fundamental research but also unfolds as a
promising avenue for future BCIs.

7. Conclusion

This contribution represents the largest repro-
ducibility study in EEG-based BCI, leverag-
ing the MOABB library. By utilizing openly
available data collected from different hard-
ware sources in varied formats and structures,
a systematic benchmarking process was under-
taken. Machine learning pipelines from pub-
lished works were re-implemented in a uni-
fied and open framework, aligning with the
established standards of the machine learn-
ing community. This effort extends to DL
pipelines, considering the rapidly evolving pro-
cessing standards for time series data.

The study's strength lies in the extensive
number of subjects analyzed across diverse
datasets, enabling robust assessments through
meta-analysis statistical techniques. The
pipelines undergo evaluation using 5-fold cross-
validation, employing the AUC metric for
binary classification tasks and accuracy for
datasets with multiple classes. Furthermore,
the environmental impact of the pipelines is
assessed and factored into the reported results.

The primary outcome is a comprehensive
and meticulous benchmarking of prominent
pipelines from within the BCI literature. Re-
sources are provided to reproduce these results
and facilitate comparisons with future works,
including result tables in the annexes and
on a dedicated online platform to streamline
comparisons and avoid unnecessary duplica-
tions. The Riemannian pipelines demonstrate
the highest accuracy, whereas DL pipelines,
while achieving admirable accuracy with ex-
tensive trial data, show limitations across most
datasets. Although data augmentation tech-
niques and advanced parameterization can en-

hance their performance, improvements are
still required for these off-the-shelf pipelines.

As the benchmark incorporates various
pipelines and datasets, recommendations can
be formulated regarding the optimal number
of trials or channels for designing BCI
experiments to achieve peak performance. A
detailed analysis of all considered datasets
is presented, aiding practitioners in tailoring
their experimental designs or selecting specific
datasets for evaluating novel pipelines.

Future development avenues are outlined
along two key directions. Firstly, benchmarks
could progress towards evaluations that mir-
ror real-world experimental conditions, aim-
ing to narrow the disparity between offline
assessment and practical online BCI applica-
tions. Secondly, the integration of novel BCI
paradigms like CVEP and transfer learning ap-
proaches across datasets is suggested for fur-
ther exploration and integration.

8. Acknowledgements

SC, BA and SS were supported by DATAIA
Convergence Institute as part of the “Pro-
gramme d’Investissement d’Avenir”, (ANR-
17-CONV-0003) operated by LISN-CNRS.
This work was granted access to the HPC re-
sources of IDRIS under the allocation 2023-
AD011014322 made by GENCI.

References

[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens,
B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow.org/.

[2] K. K. Ang, Z. Y. Chin, H. Zhang, and C. Guan. Filter bank common spatial pattern (FBCSP) in brain-computer interface. In IEEE IJCNN, pages 2390–2397, 2008.

[3] P. Aricò, F. Aloise, F. Schettini, S. Salinari, D. Mattia, and F. Cincotti. Influence of P300 latency jitter on event related potential-based brain–computer interface performance. Journal of neural engineering, 11(3):035008, 2014.

[4] B. Aristimunha, I. Carrara, P. Guetschel, S. Sedlar, P. Rodrigues, J. Sosulski, D. Narayanan, E. Bjareholt, B. Quentin, R. T. Schirrmeister, E. Kalunga, L. Darmet, C. Gregoire, A. Abdul Hussain, R. Gatti, V. Goncharenko, J. Thielen, T. Moreau, Y. Roy, V. Jayaram, A. Barachant, and S. Chevallier. Mother of all BCI Benchmarks, 2023. URL https://github.com/NeuroTechX/moabb.

[5] B. Aristimunha, R. Y. de Camargo, W. H. L. Pinaya, S. Chevallier, A. Gramfort, and C. Rommel. Evaluating the structure of cognitive tasks with transfer learning. arXiv preprint arXiv:2308.02408, 2023.

[6] M. Baker. 1,500 scientists lift the lid on reproducibility. Nature, 533(7604):452–454, May 2016.

[7] H. Banville, O. Chehab, A. Hyvärinen, D.-A. Engemann, and A. Gramfort.

Uncovering the structure of clinical EEG signals with self-supervised learning. Journal of Neural Engineering, 18(4): 046020, 2021.

[8] A. Barachant. Commande robuste d'un effecteur par une interface cerveau machine EEG asynchrone. PhD thesis, Grenoble, 2012.

[9] A. Barachant. MEG decoding using Riemannian geometry and unsupervised classification. Grenoble University: Grenoble, France, 2014.

[10] A. Barachant and M. Congedo. A plug&play P300 BCI using information geometry. arXiv preprint arXiv:1409.0107, 2014.

[11] A. Barachant, S. Bonnet, M. Congedo, and C. Jutten. Common spatial pattern revisited by riemannian geometry. In IEEE International Workshop on Multimedia Signal Processing, pages 472–476, 2010. doi: 10.1109/MMSP.2010. 5662067.

[12] A. Barachant, S. Bonnet, M. Congedo, and C. Jutten. Riemannian geometry applied to BCI classification. Lva/Ica, 10:629–636, 2010.

[13] A. Barachant, S. Bonnet, M. Congedo, and C. Jutten. Multiclass brain–computer interface classification by Riemannian geometry. IEEE Transactions on Biomedical Engineering, 59(4):920–928, 2011.

[14] A. Barachant, Q. Barthélemy, J.-R. King, A. Gramfort, S. Chevallier, P. L. C. Rodrigues, E. Olivetti, V. Goncharenko, G. W. vom Berg, G. Reguig, A. Lebeurrier, E. Bjäreholt, M. S. Yamamoto, P. Clisson, and M.-C. Corsi. pyriemann, June 2023. URL https://doi.org/10.5281/zenodo.593816.
B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow.org/.

[2] K. K. Ang, Z. Y. Chin, H. Zhang, and C. Guan. Filter bank common spatial pattern (FBCSP) in brain-computer interface. In IEEE IJCNN, pages 2390–2397, 2008.

[3] P. Aricò, F. Aloise, F. Schettini, S. Salinari, D. Mattia, and F. Cincotti. Influence of P300 latency jitter on event related potential-based brain–computer interface performance. Journal of neural engineering, 11(3):035008, 2014.

[4] B. Aristimunha, I. Carrara, P. Guetschel, S. Sedlar, P. Rodrigues, J. Sosulski, D. Narayanan, E. Bjareholt, B. Quentin, R. T. Schirrmeister, E. Kalunga, L. Darmet, C. Gregoire, A. Abdul Hussain, R. Gatti, V. Goncharenko, J. Thielen, T. Moreau, Y. Roy, V. Jayaram, A. Barachant, and S. Chevallier. Mother of all BCI Benchmarks, 2023. URL https://github.com/NeuroTechX/moabb.

[5] B. Aristimunha, R. Y. de Camargo, W. H. L. Pinaya, S. Chevallier, A. Gramfort, and C. Rommel. Evaluating the structure of cognitive tasks with transfer learning. arXiv preprint arXiv:2308.02408, 2023.

[6] M. Baker. 1,500 scientists lift the lid on reproducibility. Nature, 533(7604):452–454, May 2016.

[7] H. Banville, O. Chehab, A. Hyvärinen, D.-A. Engemann, and A. Gramfort.

Uncovering the structure of clinical EEG signals with self-supervised learning. Journal of Neural Engineering, 18(4): 046020, 2021.

[8] A. Barachant. Commande robuste d'un effecteur par une interface cerveau machine EEG asynchrone. PhD thesis, Grenoble, 2012.

[9] A. Barachant. MEG decoding using Riemannian geometry and unsupervised classification. Grenoble University: Grenoble, France, 2014.

[10] A. Barachant and M. Congedo. A plug&play P300 BCI using information geometry. arXiv preprint arXiv:1409.0107, 2014.

[11] A. Barachant, S. Bonnet, M. Congedo, and C. Jutten. Common spatial pattern revisited by riemannian geometry. In IEEE International Workshop on Multimedia Signal Processing, pages 472–476, 2010. doi: 10.1109/MMSP.2010. 5662067.

[12] A. Barachant, S. Bonnet, M. Congedo, and C. Jutten. Riemannian geometry applied to BCI classification. Lva/Ica, 10:629–636, 2010.

[13] A. Barachant, S. Bonnet, M. Congedo, and C. Jutten. Multiclass brain–computer interface classification by Riemannian geometry. IEEE Transactions on Biomedical Engineering, 59(4):920–928, 2011.

[14] A. Barachant, Q. Barthélemy, J.-R. King, A. Gramfort, S. Chevallier, P. L. C. Rodrigues, E. Olivetti, V. Goncharenko, G. W. vom Berg, G. Reguig, A. Lebeurrier, E. Bjäreholt, M. S. Yamamoto, P. Clisson, and M.-C. Corsi. pyriemann, June 2023. URL https://doi.org/10.5281/zenodo.593816.
[15] Q. Barthélemy, S. Chevallier, R. Bertrand-Lalo, and P. Clisson. End-to-end P300 BCI using Bayesian accumulation of riemannian probabilities. *Brain-Computer Interfaces*, 10(1): 50–61, 2023.

[16] L. Bianchi, C. Liti, G. Liuzzi, V. Piccialli, and C. Salvatore. Improving P300 speller performance by means of optimization and machine learning. *Annuals of Operations Research*, pages 1–39, 2021.

[17] G. Bin, X. Gao, Y. Wang, B. Hong, and S. Gao. VEP-based brain-computer interfaces: time, frequency, and code modulations [research frontier]. *IEEE Computational Intelligence Magazine*, 4 (4):22–26, 2009.

[18] G. Bin, X. Gao, Z. Yan, B. Hong, and S. Gao. An online multi-channel SSVEP-based brain–computer interface using a canonical correlation analysis method. *Journal of neural engineering*, 6(4):046002, 2009.

[19] B. Blankertz, R. Tomioka, S. Lemm, M. Kawanabe, and K.-R. Muller. Optimizing spatial filters for robust EEG single-trial analysis. *IEEE Signal processing magazine*, 25(1):41–56, 2007.

[20] N. Boumal. *An introduction to optimization on smooth manifolds*. Cambridge University Press, 2023.

[21] Y. N. Cardona-Álvarez, A. M. Álvarez-Meza, D. A. Cárdenas-Peña, G. A. Castaño-Duque, and G. Castellanos-Dominguez. A novel OpenBCI framework for EEG-based neurophysiological experiments. *Sensors*, 23(7):3763, 2023.

[22] I. Carrara and T. Papadopoulo. Classification of BCI-EEG based on augmented

covariance matrix. *arXiv preprint arXiv:2302.04508*, 2023.

[23] I. Carrara and T. Papadopoulo. Pseudoinline framework for BCI evaluation: a MOABB perspective using various MI and SSVEP datasets. *Journal of Neural Engineering*, 21(1):016003, 2024.

[24] G. Cattan, A. Andreev, P. Rodrigues, and M. Congedo. Dataset of an EEG-based BCI experiment in virtual reality and on a personal computer. *arXiv preprint arXiv:1903.11297*, 2019.

[25] G. C. Cawley and N. L. Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation. *The Journal of Machine Learning Research*, 11:2079–2107, 2010.

[26] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In *International conference on machine learning*, pages 1597–1607. PMLR, 2020.

[27] X. Chen, X. Teng, H. Chen, Y. Pan, and P. Geyer. Toward reliable signals decoding for electroencephalogram: A benchmark study to EEGNeX. *arXiv preprint arXiv:2207.12369*, 2022.

[28] S. Chevallier, G. Bao, M. Hammami, F. Marlats, L. Mayaud, D. Annane, F. Lofaso, and E. Azabou. Brain-machine interface for mechanical ventilation using respiratory-related evoked potential. In *Artificial Neural Networks and Machine Learning-ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part III 27*, pages 662–671. Springer, 2018.

[29] S. Chevallier, E. Kalunga, Q. Barthélemy, and F. Yger. Rie-
[15] Q. Barthélemy, S. Chevallier, R. Bertrand-Lalo, and P. Clisson. End-to-end P300 BCI using Bayesian accumulation of riemannian probabilities. *Brain-Computer Interfaces*, 10(1): 50–61, 2023.

[16] L. Bianchi, C. Liti, G. Liuzzi, V. Piccialli, and C. Salvatore. Improving P300 speller performance by means of optimization and machine learning. *Annuals of Operations Research*, pages 1–39, 2021.

[17] G. Bin, X. Gao, Y. Wang, B. Hong, and S. Gao. VEP-based brain-computer interfaces: time, frequency, and code modulations [research frontier]. *IEEE Computational Intelligence Magazine*, 4 (4):22–26, 2009.

[18] G. Bin, X. Gao, Z. Yan, B. Hong, and S. Gao. An online multi-channel SSVEP-based brain–computer interface using a canonical correlation analysis method. *Journal of neural engineering*, 6(4):046002, 2009.

[19] B. Blankertz, R. Tomioka, S. Lemm, M. Kawanabe, and K.-R. Muller. Optimizing spatial filters for robust EEG single-trial analysis. *IEEE Signal processing magazine*, 25(1):41–56, 2007.

[20] N. Boumal. *An introduction to optimization on smooth manifolds*. Cambridge University Press, 2023.

[21] Y. N. Cardona-Álvarez, A. M. Álvarez-Meza, D. A. Cárdenas-Peña, G. A. Castaño-Duque, and G. Castellanos-Dominguez. A novel OpenBCI framework for EEG-based neurophysiological experiments. *Sensors*, 23(7):3763, 2023.

[22] I. Carrara and T. Papadopoulo. Classification of BCI-EEG based on augmented

covariance matrix. *arXiv preprint arXiv:2302.04508*, 2023.

[23] I. Carrara and T. Papadopoulo. Pseudocode online framework for BCI evaluation: a MOABB perspective using various MI and SSVEP datasets. *Journal of Neural Engineering*, 21(1):016003, 2024.

[24] G. Cattan, A. Andreev, P. Rodrigues, and M. Congedo. Dataset of an EEG-based BCI experiment in virtual reality and on a personal computer. *arXiv preprint arXiv:1903.11297*, 2019.

[25] G. C. Cawley and N. L. Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation. *The Journal of Machine Learning Research*, 11:2079–2107, 2010.

[26] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In *International conference on machine learning*, pages 1597–1607. PMLR, 2020.

[27] X. Chen, X. Teng, H. Chen, Y. Pan, and P. Geyer. Toward reliable signals decoding for electroencephalogram: A benchmark study to EEGNeX. *arXiv preprint arXiv:2207.12369*, 2022.

[28] S. Chevallier, G. Bao, M. Hammami, F. Marlats, L. Mayaud, D. Annane, F. Lofaso, and E. Azabou. Brain-machine interface for mechanical ventilation using respiratory-related evoked potential. In *Artificial Neural Networks and Machine Learning-ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part III 27*, pages 662–671. Springer, 2018.

[29] S. Chevallier, E. Kalunga, Q. Barthélemy, and F. Yger. Rie-
mannian classification for SSVEP based BCI: offline versus online implementations. In *Brain-Computer Interfaces Handbook: Technological and Theoretical Advances*. Taylor & Francis, 2018.

[30] H. Cho, M. Ahn, S. Ahn, M. Kwon, and S. C. Jun. EEG datasets for motor imagery brain–computer interface. *Giga-Science*, 6(7):gix034, 2017.

[31] P. Clisson, R. Bertrand-Lalo, M. Congedo, G. Victor-Thomas, and J. Chatel-Goldman. Timeflux: an open-source framework for the acquisition and near real-time processing of signal streams. In *BCI 2019-8th International Brain-Computer Interface Conference*, 2019.

[32] M. Congedo, M. Goyat, N. Tarrin, G. Ionescu, L. Varnet, B. Rivet, R. Phlypo, N. Jrad, M. Acquadro, and C. Jutten. Brain invaders: a prototype of an open-source P300-based video game working with the OpenViBE platform. In *BCI 2011-5th International Brain-Computer Interface Conference*, pages 280–283, 2011.

[33] M. Congedo, A. Barachant, and R. Bhatia. Riemannian geometry for EEG-based brain-computer interfaces; a primer and a review. *Brain-Computer Interfaces*, 4(3):155–174, 2017.

[34] M.-C. Corsi, S. Chevallier, F. D. V. Fal-lani, and F. Yger. Functional connec-tivity ensemble method to enhance BCI performance (FUCONE). *IEEE Transactions on Biomedical Engineering*, 69 (9):2826–2838, 2022.

[35] B. Courty, V. Schmidt, Goyal-Kamal, MarionCoutarel, B. Feld, J. Lecourt, LiamConnell, SabAmine, kngoyal, inimaz, M. Léval, L. Blanche, A. Cru-veiller, ouminasara, F. Zhao, A. Joshi,

A. Bogroff, A. Saboni, H. de Lavoreille, N. Laskaris, E. Abati, D. Blank, Z. Wang, A. Catovic, alencon, M. Stechly, JPW, MinervaBooks, N. Carkaci, and J. Crall. Codecarbon, 2024.

[36] A. Craik, Y. He, and J. L. Contreras-Vidal. Deep learning for electroencephalogram EEG classification tasks: a review. *Journal of neural engineering*, 16 (3):031001, 2019.

[37] A. Delorme. EEG is better left alone. *Scientific reports*, 13(1):2372, 2023.

[38] J. Faller, C. Vidaurre, T. Solis-Escalante, C. Neuper, and R. Scherer. Autocalibration and recurrent adaptation: Towards a plug and play online ERD-BCI. *IEEE Transactions on Neural Systems and Rehabilitation Engineering*, 20(3):313–319, 2012.

[39] L. A. Farwell and E. Donchin. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. *Electroencephalography and clinical Neurophysiology*, 70(6):510–523, 1988.

[40] J. Frey. Comparison of an open-hardware electroencephalography amplifier with medical grade device in brain-computer interface applications. In *PhyCS-International Conference on Physiological Computing Systems*. SCITEPRESS, 2016.

[41] S. Gao, Y. Wang, X. Gao, and B. Hong. Visual and auditory brain–computer interfaces. *IEEE Transactions on Biomedical Engineering*, 61(5):1436–1447, 2014.

[42] A. Garcia Badaracco. *SciKeras*, 2020. URL https://github.com/adriangb/scikeras.
mannian classification for SSVEP based BCI: offline versus online implementations. In *Brain-Computer Interfaces Handbook: Technological and Theoretical Advances*. Taylor & Francis, 2018.

[30] H. Cho, M. Ahn, S. Ahn, M. Kwon, and S. C. Jun. EEG datasets for motor imagery brain–computer interface. *Giga-Science*, 6(7):gix034, 2017.

[31] P. Clisson, R. Bertrand-Lalo, M. Congedo, G. Victor-Thomas, and J. Chatel-Goldman. Timeflux: an open-source framework for the acquisition and near real-time processing of signal streams. In *BCI 2019-8th International Brain-Computer Interface Conference*, 2019.

[32] M. Congedo, M. Goyat, N. Tarrin, G. Ionescu, L. Varnet, B. Rivet, R. Phlypo, N. Jrad, M. Acquadro, and C. Jutten. Brain invaders: a prototype of an open-source P300-based video game working with the OpenViBE platform. In *BCI 2011-5th International Brain-Computer Interface Conference*, pages 280–283, 2011.

[33] M. Congedo, A. Barachant, and R. Bhatia. Riemannian geometry for EEG-based brain-computer interfaces; a primer and a review. *Brain-Computer Interfaces*, 4(3):155–174, 2017.

[34] M.-C. Corsi, S. Chevallier, F. D. V. Fal-lani, and F. Yger. Functional connec-tivity ensemble method to enhance BCI performance (FUCONE). *IEEE Transactions on Biomedical Engineering*, 69 (9):2826–2838, 2022.

[35] B. Courty, V. Schmidt, Goyal-Kamal, MarionCoutarel, B. Feld, J. Lecourt, LiamConnell, SabAmine, kngoyal, inimaz, M. Léval, L. Blanche, A. Cru-veiller, ouminasara, F. Zhao, A. Joshi,

A. Bogroff, A. Saboni, H. de Lavoreille, N. Laskaris, E. Abati, D. Blank, Z. Wang, A. Catovic, alencon, M. Stechly, JPW, MinervaBooks, N. Carkaci, and J. Crall. Codecarbon, 2024.

[36] A. Craik, Y. He, and J. L. Contreras-Vidal. Deep learning for electroencephalogram EEG classification tasks: a review. *Journal of neural engineering*, 16 (3):031001, 2019.

[37] A. Delorme. EEG is better left alone. *Scientific reports*, 13(1):2372, 2023.

[38] J. Faller, C. Vidaurre, T. Solis-Escalante, C. Neuper, and R. Scherer. Autocalibration and recurrent adaptation: Towards a plug and play online ERD-BCI. *IEEE Transactions on Neural Systems and Rehabilitation Engineering*, 20(3):313–319, 2012.

[39] L. A. Farwell and E. Donchin. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. *Electroencephalography and clinical Neurophysiology*, 70(6):510–523, 1988.

[40] J. Frey. Comparison of an open-hardware electroencephalography amplifier with medical grade device in brain-computer interface applications. In *PhyCS-International Conference on Physiological Computing Systems*. SCITEPRESS, 2016.

[41] S. Gao, Y. Wang, X. Gao, and B. Hong. Visual and auditory brain–computer interfaces. *IEEE Transactions on Biomedical Engineering*, 61(5):1436–1447, 2014.

[42] A. Garcia Badaracco. *SciKeras*, 2020. URL https://github.com/adriangb/scikeras.
[43] A. Gramfort, M. Luessi, E. Larson, D. A. Engemann, D. Strohmeier, C. Brodbeck, R. Goj, M. Jas, T. Brooks, L. Parkkonen, and M. S. Hämäläinen. MEG and EEG data analysis with MNE-Python. *Frontiers in Neuroscience*, 7(267):1–13, 2013. doi: 10.3389/fnins.2013.00267.

[44] M. Grosse-Wentrup, C. Liefhold, K. Grammann, and M. Buss. Beamforming in noninvasive brain–computer interfaces. *IEEE Transactions on Biomedical Engineering*, 56(4):1209–1219, 2009.

[45] P. Guetschel and M. Tangermann. Transfer learning between motor imagery datasets using deep learning - validation of framework and comparison of datasets, Nov. 2023.

[46] C. Guger, S. Daban, E. Sellers, C. Holzner, G. Krausz, R. Carabalona, F. Gramatica, and G. Edlinger. How many people are able to control a P300-based brain–computer interface (BCI)? *Neuroscience letters*, 462(1):94–98, 2009.

[47] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant. Array programming with NumPy. *Nature*, 585(7825): 357–362, Sept. 2020.

[48] L. V. Hedges and I. Olkin. *Statistical methods for meta-analysis*. Academic press, 2014.

[49] U. Hoffmann, J.-M. Vesin, T. Ebrahimi, and K. Diserens. An efficient P300-based brain–computer interface for disabled subjects. *Journal of Neuroscience methods*, 167(1):115–125, 2008.

[50] D. Hübner, T. Verhoeven, K. Schmid, K.-R. Müller, M. Tangermann, and P.-J. Kindermans. Learning from label proportions in brain-computer interfaces: Online unsupervised learning with guarantees. *PloS one*, 12(4):e0175856, 2017.

[51] D. Hüebner, T. Verhoeven, K.-R. Müller, P.-J. Kindermans, and M. Tangermann. Unsupervised learning for brain-computer interfaces based on event-related potentials: Review and online comparison. *IEEE Computational Intelligence Magazine*, 13(2): 66–77, 2018.

[52] T. M. Ingolfsson, M. Hersche, X. Wang, N. Kobayashi, L. Cavigelli, and L. Benini. EEG-TCNet: An accurate temporal convolutional network for embedded motor-imagery brain–machine interfaces. In *IEEE International Conference on Systems, Man, and Cybernetics (SMC)*, pages 2958–2965. IEEE, 2020.

[53] M. Jay, V. Ostapenco, L. Lefèvre, D. Trystram, A.-C. Orgerie, and B. Fichel. An experimental comparison of software-based power meters: focus on CPU and GPU. In *IEEE/ACM international symposium on cluster, cloud and internet computing*, pages 1–13, 2023.

[54] V. Jayaram and A. Barachant. MOABB: trustworthy algorithm benchmarking for bcis. *Journal of neural engineering*, 15 (6):066011, 2018.

[55] V. Jayaram, M. Alamgir, Y. Altun, B. Scholkopf, and M. Grosse-Wentrup. Transfer learning in brain-computer interfaces.
[43] A. Gramfort, M. Luessi, E. Larson, D. A. Engemann, D. Strohmeier, C. Brodbeck, R. Goj, M. Jas, T. Brooks, L. Parkkonen, and M. S. Hämäläinen. MEG and EEG data analysis with MNE-Python. *Frontiers in Neuroscience*, 7(267):1–13, 2013. doi: 10.3389/fnins.2013.00267.

[44] M. Grosse-Wentrup, C. Liefhold, K. Grammann, and M. Buss. Beamforming in noninvasive brain–computer interfaces. *IEEE Transactions on Biomedical Engineering*, 56(4):1209–1219, 2009.

[45] P. Guetschel and M. Tangermann. Transfer learning between motor imagery datasets using deep learning - validation of framework and comparison of datasets, Nov. 2023.

[46] C. Guger, S. Daban, E. Sellers, C. Holzner, G. Krausz, R. Carabalona, F. Gramatica, and G. Edlinger. How many people are able to control a P300-based brain–computer interface (BCI)? *Neuroscience letters*, 462(1):94–98, 2009.

[47] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant. Array programming with NumPy. *Nature*, 585(7825): 357–362, Sept. 2020.

[48] L. V. Hedges and I. Olkin. *Statistical methods for meta-analysis*. Academic press, 2014.

[49] U. Hoffmann, J.-M. Vesin, T. Ebrahimi, and K. Diserens. An efficient P300-based brain–computer interface for disabled subjects. *Journal of Neuroscience methods*, 167(1):115–125, 2008.

[50] D. Hübner, T. Verhoeven, K. Schmid, K.-R. Müller, M. Tangermann, and P.-J. Kindermans. Learning from label proportions in brain-computer interfaces: Online unsupervised learning with guarantees. *PloS one*, 12(4):e0175856, 2017.

[51] D. Hüebner, T. Verhoeven, K.-R. Müller, P.-J. Kindermans, and M. Tangermann. Unsupervised learning for brain-computer interfaces based on event-related potentials: Review and online comparison. *IEEE Computational Intelligence Magazine*, 13(2): 66–77, 2018.

[52] T. M. Ingolfsson, M. Hersche, X. Wang, N. Kobayashi, L. Cavigelli, and L. Benini. EEG-TCNet: An accurate temporal convolutional network for embedded motor-imagery brain–machine interfaces. In *IEEE International Conference on Systems, Man, and Cybernetics (SMC)*, pages 2958–2965. IEEE, 2020.

[53] M. Jay, V. Ostapenco, L. Lefèvre, D. Trystram, A.-C. Orgerie, and B. Fichel. An experimental comparison of software-based power meters: focus on CPU and GPU. In *IEEE/ACM international symposium on cluster, cloud and internet computing*, pages 1–13, 2023.

[54] V. Jayaram and A. Barachant. MOABB: trustworthy algorithm benchmarking for bcis. *Journal of neural engineering*, 15 (6):066011, 2018.

[55] V. Jayaram, M. Alamgir, Y. Altun, B. Scholkopf, and M. Grosse-Wentrup. Transfer learning in brain-computer interfaces.
terfaces. *IEEE Computational Intelligence Magazine*, 11(1):20–31, 2016.

[56] E. K. Kalunga, S. Chevallier, Q. Barthélemy, K. Djouani, E. Monacelli, and Y. Hamam. Online SSVEP-based BCI using Riemannian geometry. *Neurocomputing*, 191:55–68, 2016.

[57] E. K. Kalunga, S. Chevallier, and Q. Barthélemy. Transfer learning for SSVEP-based BCI using riemannian similarities between users. In *EUSIPCO*, pages 1685–1689, 2018.

[58] S. Khazem, S. Chevallier, Q. Barthélemy, K. Haroun, and C. Noûs. Minimizing subject-dependent calibration for BCI with Riemannian transfer learning. In *International IEEE/EMBS Conference on Neural Engineering (NER)*, pages 523–526. IEEE, 2021.

[59] P.-J. Kindermans, H. Verschore, D. Verslaeten, and B. Schrauwen. A P300 BCI for the masses: Prior information enables instant unsupervised spelling. *Advances in neural information processing systems*, 25, 2012.

[60] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. *arXiv preprint arXiv:1412.6980*, 2014.

[61] Z. J. Koles, M. S. Lazar, and S. Z. Zhou. Spatial patterns underlying population differences in the background EEG. *Brain topography*, 2:275–284, 1990.

[62] L. Korczowski, M. Cederhout, A. Andreev, G. Cattan, P. L. C. Rodrigues, V. Gautheret, and M. Congedo. *Brain Invaders calibration-less P300-based BCI with modulation of flash duration Dataset (bi2015a)*, 2019.

[63] L. Korczowski, M. Cederhout, A. Andreev, G. Cattan, P. L. C. Rodrigues,

V. Gautheret, and M. Congedo. *Brain Invaders Cooperative versus Competitive: Multi-User P300-based Brain-Computer Interface Dataset (bi2015b)*, 2019.

[64] L. Korczowski, E. Ostaschenko, A. Andreev, G. Cattan, P. L. C. Rodrigues, V. Gautheret, and M. Congedo. *Brain Invaders calibration-less P300-based BCI using dry EEG electrodes Dataset (bi2014a)*, 2019.

[65] L. Korczowski, E. Ostaschenko, A. Andreev, G. Cattan, P. L. C. Rodrigues, V. Gautheret, and M. Congedo. *Brain Invaders Solo versus Collaboration: Multi-User P300-based Brain-Computer Interface Dataset (bi2014b)*, 2019.

[66] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. *Communications of the ACM*, 60(6):84–90, 2017.

[67] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. *Journal of neural engineering*, 15(5):056013, 2018.

[68] Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller. Efficient backprop. In *Neural networks: Tricks of the trade*, pages 9–50. Springer, 2002.

[69] M.-H. Lee, O.-Y. Kwon, Y.-J. Kim, H.-K. Kim, Y.-E. Lee, J. Williamson, S. Fazli, and S.-W. Lee. EEG dataset and openbmi toolbox for three BCI paradigms: An investigation into BCI illiteracy. *GigaScience*, 8(5):giz002, 2019.
terfaces. *IEEE Computational Intelligence Magazine*, 11(1):20–31, 2016.

[56] E. K. Kalunga, S. Chevallier, Q. Barthélemy, K. Djouani, E. Monacelli, and Y. Hamam. Online SSVEP-based BCI using Riemannian geometry. *Neurocomputing*, 191:55–68, 2016.

[57] E. K. Kalunga, S. Chevallier, and Q. Barthélemy. Transfer learning for SSVEP-based BCI using riemannian similarities between users. In *EUSIPCO*, pages 1685–1689, 2018.

[58] S. Khazem, S. Chevallier, Q. Barthélemy, K. Haroun, and C. Noûs. Minimizing subject-dependent calibration for BCI with Riemannian transfer learning. In *International IEEE/EMBS Conference on Neural Engineering (NER)*, pages 523–526. IEEE, 2021.

[59] P.-J. Kindermans, H. Verschore, D. Verslaeten, and B. Schrauwen. A P300 BCI for the masses: Prior information enables instant unsupervised spelling. *Advances in neural information processing systems*, 25, 2012.

[60] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. *arXiv preprint arXiv:1412.6980*, 2014.

[61] Z. J. Koles, M. S. Lazar, and S. Z. Zhou. Spatial patterns underlying population differences in the background EEG. *Brain topography*, 2:275–284, 1990.

[62] L. Korczowski, M. Cederhout, A. Andreev, G. Cattan, P. L. C. Rodrigues, V. Gautheret, and M. Congedo. *Brain Invaders calibration-less P300-based BCI with modulation of flash duration Dataset (bi2015a)*, 2019.

[63] L. Korczowski, M. Cederhout, A. Andreev, G. Cattan, P. L. C. Rodrigues,

V. Gautheret, and M. Congedo. *Brain Invaders Cooperative versus Competitive: Multi-User P300-based Brain-Computer Interface Dataset (bi2015b)*, 2019.

[64] L. Korczowski, E. Ostaschenko, A. Andreev, G. Cattan, P. L. C. Rodrigues, V. Gautheret, and M. Congedo. *Brain Invaders calibration-less P300-based BCI using dry EEG electrodes Dataset (bi2014a)*, 2019.

[65] L. Korczowski, E. Ostaschenko, A. Andreev, G. Cattan, P. L. C. Rodrigues, V. Gautheret, and M. Congedo. *Brain Invaders Solo versus Collaboration: Multi-User P300-based Brain-Computer Interface Dataset (bi2014b)*, 2019.

[66] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. *Communications of the ACM*, 60(6):84–90, 2017.

[67] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. *Journal of neural engineering*, 15(5):056013, 2018.

[68] Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller. Efficient backprop. In *Neural networks: Tricks of the trade*, pages 9–50. Springer, 2002.

[69] M.-H. Lee, O.-Y. Kwon, Y.-J. Kim, H.-K. Kim, Y.-E. Lee, J. Williamson, S. Fazli, and S.-W. Lee. EEG dataset and openbmi toolbox for three BCI paradigms: An investigation into BCI illiteracy. *GigaScience*, 8(5):giz002, 2019.
[70] R. Leeb, F. Lee, C. Keinrath, R. Scherer, H. Bischof, and G. Pfurtscheller. Brain-computer communication: motivation, aim, and impact of exploring a virtual apartment. *IEEE Transactions on Neural Systems and Rehabilitation Engineering*, 15(4):473-482, 2007.

[71] A.-L. Ligozat, J. Lefevre, A. Bugeau, and J. Combaz. Unraveling the hidden environmental impacts of AI solutions for environment life cycle assessment of AI solutions. *Sustainability*, 14(9):5172, 2022.

[72] Z. Lin, C. Zhang, W. Wu, and X. Gao. Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs. *IEEE transactions on biomedical engineering*, 53(12):2610-2614, 2006.

[73] F. Lotte and C. Guan. Regularizing common spatial patterns to improve BCI designs: unified theory and new algorithms. *IEEE Transactions on biomedical Engineering*, 58(2):355-362, 2010.

[74] F. Lotte, M. Congedo, A. Lécuyer, F. Lamarche, and B. Arnaldi. A review of classification algorithms for EEG-based brain-computer interfaces. *Journal of neural engineering*, 4(2):R1, 2007.

[75] F. Lotte, L. Bougrain, A. Cichocki, M. Clerc, M. Congedo, A. Rakotomamonjy, and F. Yger. A review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update. *Journal of neural engineering*, 15(3): 031005, 2018.

[76] A. S. Luccioni, S. Viguier, and A.-L. Ligozat. Estimating the carbon footprint of bloom, a 176b parameter language

model. *Journal of Machine Learning Research*, 24(253):1-15, 2023.

[77] S. J. Luck. *An introduction to the event-related potential technique*. The MIT Press, 2014.

[78] V. Martínez-Cagigal, J. Thielen, E. Santamaria-Vazquez, S. Pérez-Velasco, P. Desain, and R. Hornero. Brain-computer interfaces based on code-modulated visual evoked potentials (c-VEP): A literature review. *Journal of Neural Engineering*, 18(6):061002, 2021.

[79] M. Moakher. A differential geometric approach to the geometric mean of symmetric positive-definite matrices. *SIAM journal on matrix analysis and applications*, 26(3):735-747, 2005.

[80] T. Moreau, M. Massias, A. Gramfort, P. Ablin, P.-A. Bannier, B. Charlier, M. Dagréou, T. D. la Tour, G. Durif, C. F. Dantas, Q. Klopfenstein, J. Larson, E. Lai, T. Lefort, B. Malézieux, B. Moufad, B. T. Nguyen, A. Rakotomamonjy, Z. Ramzi, J. Salmon, and S. Vaiter. Benchopt: Reproducible, efficient and collaborative optimization benchmarks. In *Advances in Neural Information Processing Systems (NeurIPS)*, volume 36, New-Orleans, LA, USA, Nov. 2022. Curran Associates, Inc.

[81] J. Müller-Gerking, G. Pfurtscheller, and H. Flyvbjerg. Designing optimal spatial filters for single-trial EEG classification in a movement task. *Clinical neurophysiology*, 110(5):787-798, 1999.

[82] S. Nagel and M. Spüler. World's fastest brain-computer interface: combining EEG2Code with deep learning. *PloS one*, 14(9):e0221909, 2019.