dataesr/scientific-paragraphs-categorization huggingface dataset:
To evaluate the benefit of a template-based anatomy, we included a filterbank model using source localization based on the fsaverage subject from FreeSurfer  (Fischl 2012 ). The forward model was computed with a 3-layer Boundary Element Method (BEM) model. Source spaces were equipped with a set of 4098 candidate dipole locations per hemisphere. Source points closer than 5mm from the inner skull surface were excluded. The noise covariance matrices used along with forward solutions to compute minimum-norm estimates inverse operators were taken as data-independent diagonal matrices. Diagonal values defaulted to the M/EEG-specific expected scale of noise (obtained via the "make_ad_hoc_cov" function from MNE-Python). All computations were done with MNE  (Gramfort et al. 2014 (Gramfort et al. , 2013)).
For computational efficiency, source power estimates were obtained by applying the inverse operators to the subjects' covariance data (MNE-Python function "apply_inverse_cov"). Dimensionality reduction was carried out with a parcellation containing 448 ROIs  (Khan et al. 2018 ). This procedure closely followed the one from  (Engemann et al. 2020) , with the difference that here an MRI template was used instead of subject-specific MRIs. Finally, the 448 ROI-wise source power estimates represented as diagonal matrices were the inputs of the log-diag pipeline from  (Sabbagh et al. 2020; D. Sabbagh et al. 2019). Features were computed using the coffeine package 2. 2 https://github. com/coffeine-labs/coffeine.
Deep learning approaches This category concerns modeling strategies in which the outcome is mapped directly from the raw signals without employing separate a priori feature-defining procedures. Instead, multiple layers of nonlinear but parametric transformations are estimated end-to-end to successively summarize and compress the input data. This process is controlled by supervision and enabled by a coherent single optimization objective. In many fields, emerging deep learning methods keep defining the state of the art in generalization performance, often outperforming humans. Deep learning models are however greedy for data, and it may take hundreds of thousands if not millions of training examples until these models show a decisive advantage over classical machine-learning pipelines. Applied to neuroscience, where the bulk of datasets is small to medium-sized, deep learning models may or may not outperform classical approaches  (Poldrack, Huckins, and Varoquaux 2020; Schulz et al.
2020; Roy et al. 2019; He et al. 2020). The success of using a deep-learning model may, eventually, depend on the amount of energy and resources invested in its development  (Gemein et al. 2020). Apart from high performance on standard laboratory M/EEG datasets and decoding tasks, deep learning models are attractive for other reasons. First, when very specific hypotheses about data generators or noise generators are available  (Kietzmann, McClure, and Kriegeskorte 2019). In this setting, the model architecture can be designed to implement this knowledge, e. g. to explicitly extract band power features in a motor decoding task. Second, these models have a strategic advantage when the data generating mechanism is not known at all, hence, few hypotheses about classes of features are available  (Schirrmeister et al.
(Banville et al. 2021). Based on prior work, here we benchmarked two battle-tested general architectures  (Gemein et al. 2020 ) implemented using the Braindecode package 3  (Schirrmeister et al. 2017; Gramfort et al. 2013). Braindecode is an open-source library for end-to-end learning on EEG 3 https://braindecode. org signals. It is closely intertwined with other libraries. One of them is Mother of all BCI Benchmarks (MOABB)  (Jayaram and Barachant 2018) , which allows for convenient EEGdata fetching, MNE  (Gramfort et al. 2013 (Gramfort et al. , 2014)) , implements well established data structures, preprocessing functionality, and more. A second key dependency is Skorch  (Tietz et al. 2017) , which implements the commonly known scikit-learn  (Pedregosa et al. 2011 ) API for neural network training  (Buitinck et al.
While BD-Deep has around 276k trainable parameters and has therefore more learning capacity, BD-Shallow has only about 36k parameters. It is important to note, that we did neither adjust the model architectures (apart from those changes required by the regression task) nor run task-specific hyperparameter optimization. Both ConvNets were used as implemented in Braindecode with hyperparameters that were already successfully applied to pathology decoding from the TUH Abnormal EEG Corpus  (Gemein et al. 2020; van Leeuwen et al. 2019; Tibor Schirrmeister et al. 2017). For more information on Braindecode or the ConvNets, please refer to the original publication  (Schirrmeister et al. 2017). For decoding, we converted the MEG input data from Tesla to Femtotesla, the EEG input data from Volts to Microvolts, and additionally rescaled the data, such that it has roughly zero mean and unit variance by dividing by the standard deviation of each dataset (see Section Datasets).
General data processing strategy using BIDS and the MNE-BIDS pipeline Neuroimaging and behavioral data are stored in many different complex formats, potentially hampering efforts of building widely usable methods, hence, impeding reproducible research. Our goal was to provide brain-age prediction models that can be directly applicable to any new electrophysiological dataset. For this purpose, we used the Brain Imaging Data Structure (BIDS)  (Gorgolewski et al. 2016 ) which allows us to organize neuroimaging data in a standardized way supporting interoperability between programming languages and software tools. We used the MNE-BIDS software  (Appelhoff et al. 2019)  for programmatically converting M/EEG datasets into the BIDS format  (Pernet et al. 2019; Niso et al. 2018 ). This has allowed us to access all datasets included in this work in the same way, enabling data analysis for all these datasets with the same code.
We will now summarize the general workflow (cf. Fig. 1 ). For this study, we used the MNE-BIDS-Pipeline for automatic preprocessing of MEG and EEG data stored in BIDS format 4  (Jas et al. 2018). Its main advantage is that we can implement various custom analyses for different datasets without having to write any elaborate code. Modifying the overall processing pipeline or adapting a given pipeline to a new dataset only requires few edits. Controlling the pipeline is achieved through datasetspecific configuration files that specify the desired processing steps and options of the MNE-BIDS-Pipeline while dealing with the peculiarities of the data. The MNE-BIDS-Pipeline scripts themselves do not need to be modified and are readily applicable on diverse datasets. We designed configuration files to implement data processing steps common to all datasets analyzed in this benchmark while handling dataset-specific details.
Raw signals bandpass-filtered between 0. 1 and 49Hz using a zero-phase finite impulse response (FIR) filter with Hamming window. Window length and transition bandwidth were automatically controlled by default settings of MNE-Python (v0. 24). We considered epochs of 10-second length without overlap. These epochs coincided with eyes-closed or eyes-open resting-state conditions in some of the datasets. As additional channels measuring ocular and cardiac activity were not consistently available across datasets, we only implemented amplitudebased artifact rejection using the local autoreject method  (Jas et al. 2017). Through 5-fold cross-validation, autoreject chose channel-specific rejection peak-to-peak-amplitude thresholds and then decided if a given epoch could be repaired using interpolation, or if it should be rejected to obtain clean data.
We kept the default grid of candidate values for the 4 https://github. com/mne-tools/mne-bids-pipeline hyperparameters 'rho' (the consensus proportion of bad channels leading to rejection of an epoch) and 'kappa' (maximum number of channels allowed to be interpolated). For 'rho' we considered a linearly spaced grid of 11 points between 0 and 1. For 'kappa' we considered 1, 4, or 32 channels. As the local autoreject is not yet supported in the MNE-BIDS pipeline, this step was implemented in a custom script (see the "compute_autoreject. py" in the code repository). Apart from preprocessing, we also made use of the MNE-BIDS-Pipeline to generate forward solutions and inverse operators for the source localization approach based on template MRI (see section Covariance-based filterbank approaches for detailed explanations).
Datasets Large datasets and biobanks are the backbone of population modeling. In the past 10 years, this has led to a wealth of publications in cognitive neuroscience on modeling biomedical outcomes and individual differences in cognition from MRI data  (Kernbach et al. 2018; James H. Cole 2020; Smith et al. 2015). This has been enabled by consortia and large-scale institutional collaborations  (Bycroft et al. 2018; Van Essen et al. 2013 ) that aim at recontextualizing existing data for open-ended future usage  (Leonelli 2016). More recently, the first M/EEG datasets have emerged with a focus on characterizing populations (Taylor et     al. 2017; Larson-Prior et al. 2013; Babayan et al. 2019; Obeid and Picone 2016; Niso et al. 5 https://github. com/meeg-ml-benchmarks/meeg-brain-age-benchmark-paper 2016;  Valdes-Sosa et al.
2021; Bosch-Bayard et al. 2020). The selection of datasets for the present study did not aim at comprehensiveness but represents an attempt to secure a minimum degree of diversity. Social bias and fairness are important challenges, not only in the field of machine learning but also in biomedical research. It has been shown for modern biobanks that the sample deviates from the general population in important ways, oversampling Caucasian people with higher education degrees  (Fry et al. 2017; Henrich and Heine 2010). For deployment of predictive biomarkers, this can have tragic consequences as clinical utility may depend on sex and ethnicity  (Duncan et al. 2019 ). As a result, in EEG research, specific risks of racial bias have been recognized lately, highlighting the risk of selection bias and confounding, e.
g. , due to culture-specific hair style  (Choy, Baker, and Stavropoulos 2021). Taken together, this emphasizes the importance of benchmarking on socially and culturally different datasets. Our selection includes M/EEG datasets from four different countries representing culturally and socioeconomically diverse contexts. In the following we will provide a high-level introduction to the datasets, highlighting characteristic differences, challenges and opportunities for unique benchmarks.
Cam-CAN MEG data. The Cambridge Centre of Ageing and Neuroscience (Cam-CAN) dataset  (Taylor et al. 2017; Shafto et al. 2014 ) has been the starting point of our efforts in building brain age models  (Engemann et al. 2020; David Sabbagh et al. 2020 ) and we like to see it as a discovery context. The combination of a wide, almost uniformly distributed age range and MEG data alongside MRI and fine-grained neurobehavioral results make it a rich resource for exploring aging-related cortical dynamics. On the other hand, models developed on this dataset may not be generalizable to real-world contexts in which EEG is operated. The following two sections are based on the methods description from our previous publications  (Engemann et al. 2020; Sabbagh et al. 2020). Sample description. The present work was based on the latest BIDS release of the Cam-CAN dataset (downloaded February 2021).
We included resting-state MEG recordings from 646 participants (female = 319, male = 327). The age of the participants ranged from 18. 5 to 88. 9 years with a mean age of 54. 9 (female = 54. 5, male = 55. 4) and a standard deviation of 18. 4 years. Data is provided in Tesla and has a standard deviation of 369. 3 Femtotesla. We did not apply any data exclusion. Final numbers of samples reflect successful preprocessing and feature extraction. For technical details regarding the MEG instrumentation and data acquisition, please consider the reference publications by the Cam-CAN  (Taylor et al. 2017; Shafto et al. 2014 ). In the following we highlight a few points essential for understanding our benchmarks on the Cam-CAN MEG data. Data acquisition and processing. MEG was recorded with a 306 VectorView system (Elekta Neuromag, Helsinki).
As a result of this procedure, the signal was high pass filtered at 0. 1Hz and the dimensionality of the data was reduced to 65, approximately. It is worthwhile to note that Maxwell filtering methods like tSSS merge the signal from magnetometers and gradiometers into one common low-rank representation. As a result, after tSSS, the signal displayed on magnetometers becomes a linear transformation of the signals displayed on the gradiometers. This leads to virtually identical results when conducting analyses exclusively on magnetometers versus gradiometers  (Garcés et al. 2017). To reduce computation time, we analyzed the magnetometers for our benchmark. To deal with the reduced data rank, a PCA projection to the common rank of 65 was applied whenever the machine learning pipeline was sensitive to the rank (e.
LEMON EEG data. The Leipzig Mind-Brain-Body (LEMON) dataset offers rich multimodal EEG, MRI and fMRI data for a well characterized group of young and elderly adults sampled from the general population  (Babayan et al. 2019). As it was the case for the Cam-CAN data, here the research was conducted in a research context using high-end equipment accompanied by rich and finegrained neurocognitive and behavioral assessments. Sample description. EEG resting-state data from 227 healthy individuals from the LEMON dataset were included in this study. This sample contains 82 females (mean age = 44. 2) and 145 males (mean age = 36), representing a clearly visible difference in the composition of the sample (Fig. 2 ). Their age distribution went from 20 to 77 years old with an average of 38. 9 +-20. 3 years.
Our sample covers the whole available dataset (downloaded September 2021) as we did not apply any exclusion criteria. It is a peculiarity of this dataset is that it is divided into 2 distinct age subpopulations, one between 20-35, the second between 55-77 (Fig. 2 ), rendering the mean a bad representation of the age distribution. Moreover, the public version of the datasets only provides ages in a granularity of 5 years to mitigate the risk of identifying participants. For the purpose of this study, we included the precise ages obtained through institutional collaboration. The impact on average modeling results turned out negligible, however. Data is provided in Volts and has a standard deviation of 9. 1 Microvolts. Data acquisition and processing. EEG was recorded with 62-channel active ActiCAP electrodes and a bandpass filter between 0.
015Hz and 1kHz. We applied additional bandpass filtering between 0. 1Hz and 49Hz. The channel placement implemented the 10-5 system  (Oostenveld and Praamstra 2001). EEG data were sampled at 2500Hz. After bandpass filtering (0. 1 -49Hz), data were decimated by a factor of 5, yielding a final sampling frequency of 500Hz. As a peculiarity of the dataset, resting-state recordings encompass samples from two conditions: eyes-closed and eyes-open. Our pipeline explicitly respected these different conditions. To include a maximum of data and, potentially, a larger set of distinguishable EEG sources, we pooled the data prior to feature extraction. For the full specification of the preprocessing, please refer to the "config_lemon_eeg. py" file in the code repository.
CHBP EEG data. The Cuban Human Brain Mapping Project (CHBP) provides rich multimodal EEG and MRI data sampled from young to middle-aged adults from the general population  (Valdes-Sosa et al. 2021; Hernandez-Gonzalez et al. 2011; Bosch-Bayard et al. 2020). As for the Cam-CAN and LEMON data, research was carried out using high-end electrophysiological equipment in a biomedical research context. However, the data was collected in a Latin American midincome country, (Valdes-Sosa et al. 2021), adding a much-needed opportunity for increasing the diversity in population-level neuroscience datasets. This diversity expresses itself in the composition of EEG protocols which contain elements of real-world neurology exams, e. g. , a hyperventilation task. Sample description. EEG resting-state data from 282 healthy individuals from the CHBP dataset were included in this study.
The sample contained 87 females (mean age = 36. 7) and 195 males (mean age = 29. 9), representing a clearly visible difference in the composition of the sample (Fig. 2 ). The overall age distribution went from 18 to 68 years with an average of 32 +/-9. 3 years. Data is provided in Volts and has a standard deviation of 6. 6 Microvolts. Our sample covers the whole available dataset (download June 2021) as we did not apply any exclusion criteria. Final numbers reflect successful processing of the data. Data acquisition and processing. EEG data were recorded using a MEDICID 5 system and two different electrode caps of either 64 or 128 channels. The channel placement implemented the 10-5 system  (Oostenveld and Praamstra 2001). Here we focused the analysis on the subset of common channels present in all recordings, leading to 53 channels.
We applied additional bandpass filtering between 0. 1Hz and 49Hz. As in the LEMON dataset, resting-state recordings encompassed samples from eyes-closed and eyes-open conditions. Again, we pooled both conditions prior to feature extraction. Note that for the data release (downloaded July 2021) used in this work, we could not benefit from the expert-based annotations of clean data. The results obtained on this dataset may therefore be impacted by quality issues to unknown extents. For the full specification of the preprocessing, please refer to the "config_chbp_eeg. py" file in the code repository.
TUAB EEG data. The Temple University Hospital Abnormal EEG Corpus (TUAB) provides socially and ethnically heterogeneous clinical EEG data (Obeid and Picone 2016) mostly from Latin-American and African American participants (personal communication, Joseph Picone). As a peculiarity, the EEG data is obtained from an archival effort of recovering different EEG exams from the Temple University Hospital in Philadelphia. The clinical and social diversity render the TUAB dataset an important resource for electrophysiological population modeling  (Gemein et al. 2020; David Sabbagh et al. 2020).
Sample description. Here, we focused exclusively on the EEG recordings labeled as not pathological by medical experts comprising a subsample of 1385 subjects (female = 775 and males = 610). This sample contained individuals ranging from newborn children (min age = 0 for female and min age = 1 for male) to elderly (max age = 95 for female and 90 for male) people (Fig. 2 ). The average age is 44. 4 +/-16. 5 years. Data is provided in Volts and has a standard deviation of 9. 7 Microvolts. The data processing closely followed our previous work on the TUAB data  (Sabbagh et al. 2020). For further details about the dataset, please refer to the reference publications  (Harati et al. 2014; Obeid and Picone 2016). Data acquisition and processing. EEG data were recorded using different Nicolet EEG devices (Natus Medical Inc.
), equipped with between 24 and 36 channels. For channel placement, the 10-5 system was applied  (Oostenveld and Praamstra 2001). All sessions have been recorded with an average reference. Here we considered a subset of 21 common channels. As channel numbers differed across recordings, re-referencing was necessary. For consistency, we also applied re-referencing with an average reference on all other EEG datasets. As sampling frequencies were inconsistent across recordings, we resampled the data to 200Hz. For many patients, multiple recordings were available. For simplicity we only considered the first recording. For the full specification of the preprocessing, please refer to the "config_tuab_eeg. py" file in the code repository. Individual observations are displayed by rug plots at the bottom of each panel.
The Cam-CAN data (MEG) show a wide age range with a quasi-uniform distribution and no obvious sex imbalance. This situation poses no a priori challenges for age prediction while, at the same time, analysis of MEG data may be more complex. The LEMON dataset included a group of young participants and a group of old participants, leading to a characteristic bi-modal distribution. Sex imbalance is clearly visible with more male participants in the group of young participants and fewer male participants in the group of older participants. This may lead to potential sex differences in prediction success and renders the average age a bad summary of the age distribution. The CHBP data shows a rather reduced age range with a right-skewed age distribution and some sex imbalance (again more young male participants).
Predicting the age can be expected to turn out more difficult on this dataset for the implied lack of density along the age range. Finally, the TUAB data present a symmetric age distribution with minor sex differences, however, a less uniform age distribution. This may lead to more pronounced errors in young and elderly participants. This may, however, be compensated for by the more generous sample size. To summarize, the four datasets investigated here pose unique challenges for M/EEG brain age modeling.
Model evaluation and comparison To gauge model performance, we first defined a baseline model that should not provide any intelligent prediction. As in previous work  (Sabbagh et al. 2020; Sabbagh et al. 2019; Engemann et al. 2020) , we employed a dummy regressor model as a low-level baseline in which the outcome is guessed from the average of the outcome on the training data. This approach is fast and typically converges with more computationally demanding procedures based on permutation testing that we shall briefly outline. This is particularly relevant for the present benchmark where the combinatorial matrix of machine learning models (including deep learning) versus datasets would lead to unpleasant computation times when applying tens of thousands of permutations. The same can be said for other approximations focusing on ranking statistics across hundreds of Monte Carlo cross-validation iterations  (Sabbagh et al.
2019). Finally, another approach relies on large left-out datasets, entirely independent from model construction, in which predictions can be treated like random variables, hence, classical inferential statistics are valid. In previous work  (Dadi et al. 2021) , permutation tests and the non-parametric bootstrap were employed on more than 4000 left-out data points to assess performance above chance and pairwise differences between models. Such generous held-out datasets are not available in the present setting, nor can we readily compute statistics across folds, as cross-validation iterations are not statistically independent. We therefore implemented a less formal approach comparing competing models against dummy regressors and against each other based on standard 10fold cross-validation based on fixed random seeds.
In contrast, the R 2 metric clearly penalizes systematically wrong predictions by assigning scores smaller than 0. Positive predictive success thus falls into a range of R 2 between 0 and 1. This facilitates comparisons across models within the same dataset while posing challenges when comparing models across datasets. We therefore considered the MAE which has the benefit of expressing prediction errors at the scale of the outcome. This is particularly convenient for scientific interpretation when the outcome has some practical meaning as is the case in the present benchmarks on age prediction. Importantly, the MAE does not per se resolve the problem of comparisons across datasets as the meaning of errors entirely depend on the distribution of the outcome: Small errors in years are good for datasets with wide age distributions but bad in datasets with narrow age distributions.