Package: moabb
Mother of all BCI BenchmarksBuild a comprehensive benchmark of popular Brain-Computer Interface (BCI) algorithms applied on an extensive list of freely available EEG datasets.DisclaimerThis is an open science project that may evolve depending on the need of the
community.Welcome!First and foremost, Welcome! :tada: Willkommen! :confetti_ball: Bienvenue!
:balloon::balloon::balloon:Thank you for visiting the Mother of all BCI Benchmark repository.This document is a hub to give you some information about the project. Jump straight to
one of the sections below, or just scroll down to find out more.What are we doing? (And why?)InstallationRunningSupported datasetsWho are we?Get in touchDocumentationArchitecture and main conceptsCiting MOABB and related publicationsWhat are we doing?The problemBrain-Computer Interfacesallow to interact with a computer using brain signals. In this project, we focus mostly on
electroencephalographic signals
(EEG), that is a very active
research domain, with worldwide scientific contributions. Still:Reproducible Research in BCI has a long way to go.While many BCI datasets are made freely available, researchers do not publish code, and
reproducing results required to benchmark new algorithms turns out to be trickier than
it should be.Performances can be significantly impacted by parameters of the preprocessing steps,
toolboxes used and implementation “tricks” that are almost never reported in the
literature.As a result, there is no comprehensive benchmark of BCI algorithms, and newcomers are
spending a tremendous amount of time browsing literature to find out what algorithm works
best and on which dataset.The solutionThe Mother of all BCI Benchmarks allows to:Build a comprehensive benchmark of popular BCI algorithms applied on an extensive list
of freely available EEG datasets.The code is available on GitHub, serving as a reference point for the future algorithmic
developments.Algorithms can be ranked and promoted on a website, providing a clear picture of the
different solutions available in the field.This project will be successful when we read in an abstract “ … the proposed method
obtained a score of 89% on the MOABB (Mother of All BCI Benchmarks), outperforming the
state of the art by 5% ...”.InstallationPip installationTo use MOABB, you could simply do:pip install MOABBSeeTroubleshootingsection if you have a problem.Manual installationYou could fork or clone the repository and go to the downloaded directory, then run:installpoetry(only once per machine):curl -sSL https://install.python-poetry.org | python3 -orcheckout installation instructionor
useconda forge version(Optional, skip if not sure) Disable automatic environment creation:poetry config virtualenvs.create falseinstall all dependencies in one command (have to be run in the project directory):poetry installSeecontributors' guidelinesfor detailed explanation.Requirements we useSeepyproject.tomlfile for full list of dependenciesRunningVerify InstallationTo ensure it is running correctly, you can also runpython -m unittest moabb.testsonce it is installed.Use MOABBFirst, you could take a look at ourtutorialsthat cover the most important
concepts and use cases. Also, we have a severalexamplesavailable.You might be interested inMOABB documentationMoabb and dockerMoabb has a default image to run the benchmark. You have two options to download this
image: build from scratch or pull from the docker hub.We recommend pulling from the
docker hub.If this were your first time using docker, you would need toinstall the dockerandloginon docker hub. We recommend theofficialdocker documentation
for this step, it is essential to follow the instructions.After installing docker, you can pull the image from the docker hub:dockerpullbaristimunha/moabb# rename the tag to moabbdockertagbaristimunha/moabbmoabbIf you want to build the image from scratch, you can use the following command at the
root. You may have to login with the API key in theNGC Catalogto run this command.bashdocker/create_docker.shWith the image downloaded or rebuilt from scratch, you will have an image calledmoabb.
To run the default benchmark, still at the root of the project, and you can use the
following command:mkdirdataset
mkdirresults
mkdiroutput
bashdocker/run_docker.shPATH_TO_ROOT_FOLDERAn example of the command is:cd/home/user/project/moabb
mkdirdataset
mkdirresults
mkdiroutput
bashdocker/run_docker.sh/home/user/project/moabbNote: It is important to use an absolute path for the root folder to run, but you can
modify the run_docker.sh script to save in another path beyond the root of the project. By
default, the script will save the results in the project's root in the folderresults,
the datasets in the folderdatasetand the output in the folderoutput.TroubleshootingCurrently pip install moabb fails when pip version < 21, e.g. with 20.0.2 due to anidnapackage conflict. Newer pip versions resolve this conflict automatically. To fix this you
can upgrade your pip version using:pip install -U pipbefore installingmoabb.Supported datasetsThe list of supported datasets can be found here :https://neurotechx.github.io/moabb/datasets.htmlDetailed information regarding datasets (electrodes, trials, sessions) are indicated on
the wiki:https://github.com/NeuroTechX/moabb/wiki/Datasets-SupportSubmit a new datasetyou can submit a new dataset by mentioning it to thisissue. The datasets currently on our radar
can be seenhere.Who are we?The founders of the Mother of all BCI Benchmarks areAlexander BarachantandVinay Jayaram. This project is under the umbrella ofNeuroTechX, the international community for NeuroTech enthusiasts. The
project is currently maintained bySylvain Chevallier.What do we need?You! In whatever way you can help.We need expertise in programming, user experience, software sustainability, documentation
and technical writing and project management.We'd love your feedback along the way.Our primary goal is to build a comprehensive benchmark of popular BCI algorithms applied
on an extensive list of freely available EEG datasets, and we're excited to support the
professional development of any and all of our contributors. If you're looking to learn to
code, try out working collaboratively, or translate your skills to the digital domain,
we're here to help.Get involvedIf you think you can help in any of the areas listed above (and we bet you can) or in any
of the many areas that we haven't yet thought of (and here we'resureyou can) then
please check out ourcontributors' guidelinesand ourroadmap.Please note that it's very important to us that we maintain a positive and supportive
environment for everyone who wants to participate. When you join us we ask that you follow
ourcode of conductin all interactions both on and offline.Contact usIf you want to report a problem or suggest an enhancement, we'd love for you toopen an issueat this GitHub repository because then we can get right on
it.For a less formal discussion or exchanging ideas, you can also reach us on theGitter
channelor join our weekly office hours! This an open video meeting
happening on aregular basis, please ask
the link on the gitter channel. We are also onNeuroTechX Slack #moabb
channel.Architecture and Main ConceptsThere are 4 main concepts in the MOABB: the datasets, the paradigm, the evaluation, and the pipelines. In addition, we offer statistical and visualization utilities to simplify the workflow.DatasetsA dataset handles and abstracts low-level access to the data. The dataset will read data
stored locally, in the format in which they have been downloaded, and will convert them
into a MNE raw object. There are options to pool all the different recording sessions per
subject or to evaluate them separately.ParadigmA paradigm defines how the raw data will be converted to trials ready to be processed by a
decoding algorithm. This is a function of the paradigm used, i.e. in motor imagery one can
have two-class, multi-class, or continuous paradigms; similarly, different preprocessing
is necessary for ERP vs ERD paradigms.EvaluationsAn evaluation defines how we go from trials per subject and session to a generalization
statistic (AUC score, f-score, accuracy, etc) -- it can be either within-recording-session
accuracy, across-session within-subject accuracy, across-subject accuracy, or other
transfer learning settings.PipelinesPipeline defines all steps required by an algorithm to obtain predictions. Pipelines are
typically a chain of sklearn compatible transformers and end with a sklearn compatible
estimator. SeePipelinesfor more info.Statistics and visualizationOnce an evaluation has been run, the raw results are returned as a DataFrame. This can be
further processed via the following commands to generate some basic visualization and
statistical comparisons:from moabb.analysis import analyze

results = evaluation.process(pipeline_dict)
analyze(results)Citing MOABB and related publicationsTo cite MOABB, you could use the following paper:Vinay Jayaram and Alexandre Barachant."MOABB: trustworthy algorithm benchmarking for BCIs."Journal of neural engineering 15.6 (2018): 066011.DOIIf you publish a paper using MOABB, please contact us ongitteror open an
issue, and we will add your paper to thededicated wiki page.Thank YouThank you so much (Danke schön! Merci beaucoup!) for visiting the project and we do hope
that you'll join us on this amazing journey to build a comprehensive benchmark of popular
BCI algorithms applied on an extensive list of freely available EEG datasets.