Package: benchnirs
BenchNIRSBenchmarking framework for machine learning with fNIRSQuick links→Journal article→BenchNIRS repository→Install BenchNIRS→Documentation→Issue trackerFeaturesloading of open access datasetssignal processing and feature extraction on fNIRS datatraining, optimisation and evaluation of machine learning models (including deep learning)production of training graphs, metrics and other useful figures for evaluationbenchmarking and comparison of machine learning modelsmuch more!DocumentationThe documentation of the framework with examples can be foundhere.Recommendation checklistA checklist of recommendations towards good practice for machine learning with fNIRS (for brain-computer interface applications) can be foundhere. We welcome contributions from the community in order to improve it, please see below for more information on how to contribute.Minimum tested requirementsPython 3.8with the following libraries:matplotlib 3.3mne 0.23nirsimple 0.1numpy 1.19pandas 1.0scikit-learn 0.24scipy 1.8seaborn 0.11statsmodels 0.12.2torch 1.5Downloading the datasetsHerff et al. 2014(n-back task): you can download the dataset by making a requesthere. In the examples, the unzipped folder has been renamed todataset_herff_2014for convenience.Shin et al. 2018(n-back and word generation tasks): you can download the datasethere. In the examples, the unzipped folder has been renamed todataset_shin_2018for convenience.Shin et al. 2016(mental arithmetic task): you can download the dataset by filling the formhere. Then click onNIRS_01-29to download the fNIRS data. In the examples, the unzipped folder has been renamed todataset_shin_2016for convenience.Bak et al. 2019(motor execution task): you can download the datasethere. In the examples, the unzipped folder has been renamed todataset_bak_2019for convenience.SetupDownload and install Python 3.8 or greater, for example withMiniconda.Download and unzip theBenchNIRSrepository.In a terminal or command prompt (eg. Anaconda Prompt), navigate to the directory containing therequirements.txtfile and run:python-mpipinstall-rrequirements.txt-fhttps://download.pytorch.org/whl/torch_stable.htmlAlternatively, theBenchNIRSlibrary containing the core functions (without main scripts) is available onPyPIand can be installed usingpip:pipinstallbenchnirsand updated to the newest version with:pipinstall--upgradebenchnirsRunning main scriptsgeneralised.pycompares the 6 models (LDA, SVC, kNN, ANN, CNN and LSTM) on the 5 datasets with a generalised approach (testing with unseen subjects)dataset_size.pyreproducesgeneralised.pybut with a range different dataset sizes (50% to 100% of dataset) to study the influence of this parameter on the classification accuracywindow_size.pyreproducesgeneralised.pybut with only the 4 models using feature extraction (LDA, SVC, kNN and ANN) and with a range different window sizes (2 to 10 seconds) to study the influence of this parameter on the classification accuracysliding_window.pyreproducesgeneralised.pybut with only the 4 models using feature extraction (LDA, SVC, kNN and ANN) and with a 2-second sliding window on the 10-second epochspersonalised.pycompares the 6 models (LDA, SVC, kNN, ANN, CNN and LSTM) on the 5 datasets with a personalised approach (training and testing with each subject individually)visualisation.pyenables to visualise the data from the datasets with various signal processingExampleAn example script showing how to use the framework with a custom deep learning model can be foundhere.Simple use caseBenchNIRSenables to evaluate your model in Python with simplicity on an open access dataset supported:importbenchnirsasbnepochs=bn.load_dataset('shin_2018_nb')data=bn.process_epochs(epochs['0-back','2-back','3-back'])results=bn.deep_learn(*data,my_model)print(results)Contributing to the repositoryContributions from the community to this repository are highly appreciated. We are mainly interested in contributions to:improving the recommendation checklistadding support for newopen access datasetsadding support for new machine learning modelsadding more fNIRS signal processing techniquesimproving the documentationtracking bugsContributions are encouraged under the form ofissues(for reporting bugs or requesting new features) andmerge requests(for fixing bugs and implementing new features).
Please refer tothis tutorialfor creating merge requests from a fork of the repository.AcknowledgementsIf you are usingBenchNIRS, please citethis article:@article{benerradi2023benchmarking,
  title={Benchmarking framework for machine learning classification from fNIRS data},
  author={Benerradi, Johann and Clos, Jeremie and Landowska, Aleksandra and Valstar, Michel F and Wilson, Max L},
  journal={Frontiers in Neuroergonomics},
  volume={4},
  year={2023},
  publisher={Frontiers Media},
  url={https://www.frontiersin.org/articles/10.3389/fnrgo.2023.994969},
  doi={10.3389/fnrgo.2023.994969},
  issn={2673-6195}
}If you are using the datasets of the framework, please also cite those related works.Herff et al. 2014:@article{herff2014mental,
	title={Mental workload during n-back task—quantified in the prefrontal cortex using fNIRS},
	author={Herff, Christian and Heger, Dominic and Fortmann, Ole and Hennrich, Johannes and Putze, Felix and Schultz, Tanja},
	journal={Frontiers in human neuroscience},
	volume={7},
	pages={935},
	year={2014},
	publisher={Frontiers}
}Shin et al. 2018:@article{shin2018simultaneous,
	title={Simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset},
	author={Shin, Jaeyoung and Von L{\"u}hmann, Alexander and Kim, Do-Won and Mehnert, Jan and Hwang, Han-Jeong and M{\"u}ller, Klaus-Robert},
	journal={Scientific data},
	volume={5},
	pages={180003},
	year={2018},
	publisher={Nature Publishing Group}
}Shin et al. 2016:@article{shin2016open,
	title={Open access dataset for EEG+NIRS single-trial classification},
	author={Shin, Jaeyoung and von L{\"u}hmann, Alexander and Blankertz, Benjamin and Kim, Do-Won and Jeong, Jichai and Hwang, Han-Jeong and M{\"u}ller, Klaus-Robert},
	journal={IEEE Transactions on Neural Systems and Rehabilitation Engineering},
	volume={25},
	number={10},
	pages={1735--1745},
	year={2016},
	publisher={IEEE}
}Bak et al. 2019:@article{bak2019open,
	title={Open-Access fNIRS Dataset for Classification of Unilateral Finger-and Foot-Tapping},
	author={Bak, SuJin and Park, Jinwoo and Shin, Jaeyoung and Jeong, Jichai},
	journal={Electronics},
	volume={8},
	number={12},
	pages={1486},
	year={2019},
	publisher={Multidisciplinary Digital Publishing Institute}
}