dataesr/scientific-paragraphs-categorization huggingface dataset:
Computational considerations and software M/EEG data processing. BIDS conversion and subsequent data analysis steps were carried out in Python 3. 7. 1, the MNE-Python software (v0. 24, Gramfort et al. 2014 Gramfort et al. , 2013)) , the MNE-BIDS package (v0. 9, Appelhoff et al. 2019 ) and the MNE-BIDS-pipeline on a 48-core Linux high-performance server with 504 GB RAM. The joblib library (v1. 0. 1) was used for parallel processing. For artifact removal, the latest development version (v0. 3dev) of the autoreject package  (Jas et al. 2017 ) was used. Classical machine learning benchmarks. For future computation, the mne-features (0. 2, Schiratti, Le Douget, Le  Van Quyen, et al. 2018), PyRiemann (v0. 2. 6 ) and the coffeine Machine-learning specific computation was composed using the scikit-learn package  (Pedregosa et al.
Results For the age prediction benchmark, we considered five alternative approaches: heterogeneous handcrafted features & random forest ('handcrafted'), filterbank features based on Riemannian embeddings & ridge regression ('filterbank-riemann'), filterbank features based on source localization with MRI-average template & ridge regression ('filterbank-source'), a shallow deep learning architecture ("shallow") and a 4-layer deep-learning architecture ('deep'). These approaches were benchmarked across four M/EEG datasets: The Cambridge Centre of Ageing and Neuroscience (Cam-CAN) dataset  (Taylor et al. 2017) , the Cuban Human Brain Mapping Project (CHBP) dataset  (Valdes-Sosa et al. 2021) , the Leipzig Mind-Brain-Body (LEMON) dataset  (Babayan et al. 2019 ) and the Temple University Hospital Abnormal EEG Corpus (TUAB) dataset  (Obeid and Picone 2016).
Generalization performance was estimated using 10-fold cross validation after shuffling the samples (fixed random seed). The coefficient of determination (R 2 ) was used as a metric enabling comparisons between datasets independently of the age distribution, mathematically quantifying the additional variance explained by predicting better than the average age. A dummy model empirically quantifies chance-level prediction by returning the average age of the training data as prediction. The results are displayed in Fig. 3. One can see that on most of the datasets all machine learning models achieved R 2 scores well beyond the dummy baseline. The highest scores were observed on the Cam-CAN MEG dataset, followed by the LEMON EEG dataset. Caution is warranted though to avoid premature conclusions: The R 2 offers a common scale that explicitly compares the incremental model performance over the average predictor.
This is achieved by dividing the sum of squares of the model's prediction by the sum of squares of the average predictor but, in turn, depends on the distribution of age. As a result, this can be misleading in cross-dataset comparisons when the variance of the outcome is not the same, which is the case here (cf. Fig. 2 ). We therefore also computed results using the mean absolute error as a performance metric (Fig  4 ). One can now see that the overall distribution of scores, including the scores of the dummy model, depend not only on the dataset but also on its age range. Where the range is small, improvements over the baseline models are harder to observe. Moreover, comparing MAE scores across datasets without taking into account the baseline can yield misleading conclusions. For example, the same score of e.
g. an MAE = 10 can be way above chance in one dataset (Cam-CAN) but below chance in another dataset (CHBP). To alleviate this problem, normalized MAE scores have been suggested in which the MAE scores are related to the range of the age distribution  (Cole, Franke, and Cherbuin 2019). This does not come without its own problems, as then outliers in non-uniform distributions could drive the scores. As research keeps evolving on this topic and the community has not yet agreed on the best metric, we recommend considering multiple classical machine learning metrics when comparing model performance -in critical awareness of their respective limitations. Confronting the relative performances of models to the dummy baseline in Fig. 3  and Fig. 4 , one can see overall similar performance rankings between the models, regardless of the metric.
See Table  1  for side-by-side comparisons of the aggregated cross-validation distributions. The big-picture results argue in favor of the importance of fine-grained spatial features for M/EEG prediction while considering important between-dataset heterogeneity. Both filterbank pipelines provide features based on spatially aware representations of the M/EEG signals, which either explicitly or implicitly deal with the spatial spread of electrical potentials and fields characteristic for M/EEG signals. The source-level filterbank approximates source localization using the average MRI template, whereas Riemannian embeddings provide non-linear spectral features that are affine invariant, hence, independent of linear mixing. The deep benchmarks, on the other hand, implied spatial-filtering layers capable of mimicking source localization by learning an unmixing function.
Surprisingly, using the average MRI template instead of the Riemannian embedding to construct a filterbank model did not lead to consistent improvements across datasets, suggesting that both approaches may be equally effective in practice. We would have conjectured that even an imprecise biophysical head model would provide inverse solutions leading to more accurate unmixing of M/EEG sources. Compared to our previous benchmarks  (Engemann et al. 2020; Sabbagh et al. 2020)  favoring filterbank models based on source-localization, one has to point out that this finding may reflect at least two differences: The use of an MRI template instead of individual co-registration and the use of empty-room-based suppression of environmental noise. The second factor may be less relevant for EEG though where empty room recordings are not available and data-based covariances are more common in event-related studies where brain activity induced by stimuli is compared against the background resting-state activity.
As a practical implication, and if inspection of the brain sources is not a priority, the purely data-driven pipelines may be more practical as no additional MRI-based data processing is needed (cf. Fig. 1 ). Fig. 3. Age prediction benchmarks across M/EEG datasets (R 2 score). Generalization performance was assessed by 10-fold cross-validation and the R 2 score for five machine learning strategies compared against a dummy model (rows) and four datasets (panels). Across datasets, dummy models were mostly well-calibrated with R 2 scores close to zero. The LEMON dataset was one exception as dummy scores were systematically worse than chance, which can be explained by the bimodal age distribution (cf. Fig. 2 ), rendering the average age a bad guess for the age. The 'handcrafted' benchmark delivered moderate but systematic prediction success across all datasets.
The two filterbank models performed well across datasets with similar performances, markedly higher than for the 'handcrafted' approach. The only exception was the CHBP benchmark for which neither the filterbank nor the deep models delivered useful predictions. Note that here, for the 'filterbank-source', a single fold with an abysmal R 2 score of -15 was obtained (x limits constrained to a range between -. 3 and 1. 0). Overall, the deep learning benchmarks performed similarly to the filterbank models. Interestingly, none of these approaches involving spatially fine-grained representations of the M/EEG signal worked well on the CHBP data, whereas the random forest on top of hand-crafted features scored systematically better than the dummy baseline. This may be related to three factors that come together in the CHBP benchmark dataset: Like the LEMON dataset, the sample size is relatively small.
Progress on this specific benchmark may therefore involve explicit consideration of the montage when selecting samples for cross-validation or even at the level of the machine learning model (e. g. , by including the number of electrodes or montage type as covariate). Moreover, future availability of samples from older populations in the CHBP dataset will help disambiguate this point. Finally, once the expert-based qualitycontrol annotations are considered for epochs-selection, the results obtained in this benchmark may change (see section Datasets/CHBP EEG data for details). A different type of challenge is illustrated by the benchmarks on the LEMON dataset. As the age distribution is bimodal here (Fig. 2 ), the R 2 score is not well calibrated as the average predictor will not provide a reasonable summary of the distribution.
This is not automatically mitigated by considering the MAE as a metric. On the other hand, it will not affect the ranking of the machine learning models, which compare overall well to results obtained on the Cam-CAN and the TUAB datasets. To obtain a more rigorous baseline, one could envision a group-wise average predictor that, depending on the age group, would return the groups' respective average age from the training data. We did not implement such a custom baseline here as it was our goal to stick to standard routines provided by the software libraries our benchmarks were based on. Second, it was our intention to expose such issues as this may stimulate future research and development. In the following we wish to point out a few imminent opportunities for turning the limitations of the present work into future research projects, potentially, enabled by the results and tools brought by the current benchmarks.
Specifically, it would be possible to use methods from neural architecture search, e. g. , AutoPyTorch  (Zimmer, Lindauer, and Hutter 2021) , to design better-performing architectures. Since this benchmark does not only provide access to diverse datasets in an identical file format, but also enables direct comparison to others, it is the optimal starting point for such an optimization while at the same time avoiding overfitting the architecture to a single dataset. The role of preprocessing. While data cleaning is of major importance for extracting physiologically interpretable biomarkers, predictions from machine learning models tend to be far less affected by noise  (Sabbagh et al. 2020). On the other hand, artifacts and noise may inform predictions, potentially reducing biological specificity.
Future studies could benefit from this benchmark to quantify the role of artifact signals for brain age predictions and develop de-confounding strategies  (Du et al. 2021; Mehrabi et al. 2021; Lu, Schölkopf, and Hernández-Lobato 2018; Bica, Alaa, and Van Der Schaar 2020). Eyes-open versus eyes-closed. Some of the datasets analyzed in this benchmark contain resting-state signals under different conditions. In the lack of strong a-priori hypotheses, here we simply pooled both conditions. It is currently unclear whether the relationship between eyes-closed versus eyes-open resting-state may contain valuable information about brain aging. It is imaginable, however, that signals induced by transient visual deprivation may reveal levels of vigilance  (Wong, DeYoung, and Liu 2016) , which in turn may be altered by neuropsychiatric conditions  (Hegerl et al.
2021; Cole et al. 2018; Liem et al. 2017). To further establish the relative merit of M/EEG over MRI, comparisons between the modalities are essential  (Engemann et al. 2020). Most of the datasets covered in this benchmark include MRI data, social details and psychometric scores next to the M/EEG data, providing a wealth of opportunities for deep cross-dataset validation of brain age measures.
Conclusion Computational benchmarks across M/EEG datasets and machine learning methods bear the potential to enhance applications of machine learning in clinical neuroscience in several ways. Standardization of data formats, software and analysis pipelines are important factors for the scalability of predictive modeling of M/EEG. For stimulating the development of more generalizable machine learning models it is crucial that a critical mass of M/EEG datasets be analyzed by the international community. As the diversity of the datasets increases, generalization gaps will manifest themselves, calling for computation methods for closing these gaps. The implied learning process may eventually lead to developing more widely applicable M/EEG-based biomarkers that are clinically robust across a wide range of sociocultural contexts, clinical populations, recording sites and measurement techniques.
This may lead to potential sex differences in prediction success and renders the average age a bad summary of the age distribution. The CHBP data shows a rather reduced age range with a right-skewed age distribution and some sex imbalance (again more young male participants). Predicting the age can be expected to turn out more difficult on this dataset for the implied lack of density along the age range. Finally, the TUAB data present a symmetric age distribution with minor sex differences, however, a less uniform age distribution. This may lead to more pronounced errors in young and elderly participants. This may, however, be compensated for by the more generous sample size. To summarize, the four datasets investigated here pose unique challenges for M/EEG brain age modeling.
(Ibanez et al. 2021;Shekh Ibrahim et al. 2020;et al. 2019datasets Similar performance may be explained by the fact that our filterbank models and the deep models imply similar spatially aware representations of the M/EEG data (see results section for detailed discussion in context). Moreover, given the relatively small training datasets, it can be considered good news that these parameter-rich models did not seem to overfit as was evidenced by comparisons against simpler classical models. Yet, it may be simply a matter of collecting larger samples until deep learning approaches may reveal their advantage at extracting more elaborate representations of M/EEG signals. This may lead to positioning M/EEG-based brain age prediction on par with MRI-based brain age prediction just as MRI-based deep learning models of brain age have defined state-of-the-art performance on large datasets(Cole et al.
2017;Bashyam et al. 2020;Jonsson et al. 2019). However, more importantly, the value of M/EEG-derived brain age models should not be defined in terms of incremental improvement over MRI-based models as M/EEG-based models may enhance MRI-derived information(Engemann et al. 2020)or may be the only option available(Sun et al. 2019). Despite potentially relevant similarities between brain development and aging, age prediction in developmental cohorts would have exceeded the scope of the present study. Even if we had integrated these resources in the present benchmark, this may have only marginally enhanced the diversity covered by the current selection datasets as most public neuroscience datasets come from the wealthiest nations. We hope that this situation will improve as new promising international consortia and efforts emerge that focus on curating large EEG datasets from diverse national and cultural contexts(Ibanez et al.
Acknowledgements We would like to thank  Pedro Valdés-Sosa  and  Jorge Bosch-Bayard  and their team for the support with the  Cuban Human Brain Project data  and for the stimulating discussion. We would also like to thank  Joseph Picone  for the support with  Temple University Hospital data. Finally, we would like to thank  Anahit Babyan ,  Vadim Nikulin  and  Arno Villringer  for the support with the LEMON dataset and the stimulating scientific discussions. We would like to thank  Rik Henson  and  Ethan Knights  for providing the Can-CAN datasets in BIDS format. We also thank  Ana Radanovic  and the other participants of the IDESSAI 2021 Inria-DFKI summer school for feedback and testing on earlier versions of the code and analysis developed in this project. This work was supported by the  ANR BrAIN AI chair  ( ANR-20-CHIA-0016 ) and  ANR AI-Cog  ( ANR-20-IADJ-0002 ) as well as by the  Freiburg Graduate School of Robotics.
2021;Shekh Ibrahim et al. 2020; "Global Brain Consortium Homepage"). A second limitation of the present study concerns the depth of validation. To advance our understanding of M/EEG-derived brain age, more systematic comparisons against MRI-derived brain age (Xifra-Porxas et al. 2021) and other measures of mental health and cognitive function are important objectives (Anatürk et al. 2021; Dadi et al. 2021). dataset benchmark R 2 (M) R 2 (SD) MAE(M) MAE(SD) Cam-CAN (MEG) deep 0. 66 0. 05 8. 29 0. 74 Cam-CAN (MEG) shallow 0. 69 0. 03 8. 14 0. 90 Cam-CAN (MEG) filterbank-source 0. 69 0. 06 8. 10 1. 11 Cam-CAN (MEG) filterbank-riemann 0. 72 0. 05 7. 65 0. 81 Cam-CAN (MEG) handcrafted 0. 49 0. 07 10. 65 0. 98 Cam-CAN (MEG) dummy -0. 02 0. 03 15. 90 1. 22 LEMON (EEG) deep 0. 65 0. 20 7. 78 2. 25 LEMON (EEG) shallow 0.
69 0. 08 8. 80 1. 58 LEMON (EEG) filterbank-source 0. 65 0. 12 8. 93 1. 56 (Alexander et al. 2017ust performance. In line with previous work(Gemein et al. 2020), these results suggest that deep learning methods do not necessarily show a consistent advantage over classical pipeline models:Our results nicely demonstrate a second critical merit of cross-model and crossdataset benchmarking. It was sufficient to analyze four different sources of data until we found a perfectly legitimate EEG dataset from an academic research context (CHBP) in which our previously favored modeling techniques developed on the Cam-CAN and the TUAB data did not perform well by default. There may be good reasons for these discrepancies related to the age distribution found in the CHBP data and the fact that multiple different EEG montages were used in that dataset (see results section for detailed discussion in context).
But more importantly, we did not anticipate this to happen and would have never learned about it had we confined the scope to previously analyzed datasets. Such discoveries are favored by systematic benchmarks with dataset-independent code implementation, which has the potential to lower the burden threshold for including always more datasets into model development. In the long run, we hope that this effort will stimulate new research leading to more generalizable models. cohort(Loeffleretal. 2015)includes a large number of EEG recordings of participants sampled from the general population yet follows a closed / controlled access scheme. The Healthy-Brain-Network EEG data(Alexander et al. 2017) concerns a developmental cohort.
To this end, a linear spatio-temporal model (f • g) is fitted to predict the brain activity of one voxel Y , given GPT-2 activations X as input. The degree of mapping, called "brain score" is defined for each voxel as the Pearson correlation between predicted and actual brain activity on held-out data (blue equation, cf. Methods). Finally, we test the correlation between the comprehension scores of the subjects and their corresponding brain scores using Pearson's correlation (red equation). A positive correlation means that the representations shared across the brain and GPT-2 are key for the subjects to understand a narrative. B. Brain scores (fMRI predictability) of the activations of the eighth layer of GPT-2. Scores are averaged across subjects, narratives, and voxels within brain regions (142 regions in each hemisphere, following a subdivision of Destrieux Atlas (10), cf.
Phonological features To account for low-level speech processing, we computed the alignment (Eq. (  1 )) between the fMRI brain record- In Figure  1G , we use the original parcellation for simplicity, and the following acronyms:  were assessed with a t-test, as implemented in statsmodels. Replication across single narratives. To further support that the R were not driven by the narratives' variability, we replicate the analysis of Figure  1D  within single narratives. In Figure  3 , we show that correlation scores between brain scores and comprehension scores are positive for each of the seven narratives. Noise Ceiling Estimates. fMRI recordings are inherently noisy. Thus, we estimate an upper bound of the best brain score that can be obtained given the level of noise in the Narrative dataset.
Then, we extract the activations corresponding 234 to w k. The procedure results in a vector of activations of size (w, d) 235 with w the number of tokens in the story and d the dimensionality 236 of the model. There are fewer fMRI scans than words. Thus, 237 the activation vectors between successive fMRI measurements are 238 summed to obtain one vector of size d per measurement. To match 239 the fMRI measurements and the GPT-2 vectors over time, we used 240 the speech-to-text correspondences provided in the fMRI dataset 241 (21).
ings Y and phonological features X: the word rate (of dimension d = 1, the number of words per fMRI scan), the phoneme rate (d = 1, the number of phonemes per fMRI scan) and the concatenation of phonemes, stresses and tones of the words in the stimuli (categorical feature, d = 117). The latter features are provided in the original Narratives database (21) , and computed using Gentle ¶ forced-alignment algorithm. SignificanceSignificance was either assessed by using either (i) a second-level Wilcoxon test (two-sided) across subject-narrative pairs, testing whether the mapping (one value per pair) was significantly different from zero (Figure1B), or (ii) by using the first-level Pearson p-value provided by SciPy ‖ (Figure 1D-G). In Figure 1B, E, F, pvalues were corrected for multiple comparison (2 × 142 ROIs) using False Discovery Rate (Benjamin/Hochberg) * *.
Acknowledgements Mattias Sjö thanks Torbjörn Lundberg for rewarding discussions resulting in the new treatment of propositions 4. 1 and 4. 2 as compared to  [20]. This work is supported in part by the Swedish Research Council grants contract numbers 2016-05996 and 2019-03779. Colour schemes for the figures, appropriate for colourblind people and monochrome printing, were chosen based on  [43, 44].
61) since n , r are the normals and offsets of the facets of the hull. 46 This is, by construction, a minimal representation. If we choose β j such that Hull {β j } j∈J is a suitable subset of A c (Ω), we will see that it is possible to write a simple rule that selects a subset M ⊆ L such that 44 The implementation code is available from Mattias Sjö upon request. 45 For up-to-date information about qhull, see http://www. qhull. org/. B ∈M n , r = A c (Ω). (B. 62) 46 qhull uses a different sign convention, but the conversion to the format given above is trivial. In order to do this, let Ω be given as in eq. (4. 9). Importantly, assume without loss of generality that α i , -1 = 0, -1 for some i ∈ I -1 , but that α i = 0 for all i ∈ I 0. 47 Then, define the following sets of points: P ± (Ω) = {α i | i ∈ I ±1 } , Z (p) ± (Ω) = {α i + (p -1)α j | i ∈ I ±1 , j ∈ I 0 } , N (p) ± (Ω) = {pα i + (p -1)α j | i ∈ I ±1 , j ∈ I ∓1 } (B.
4. 1; ONT) were loaded with library material. Run performance was monitored, and once the number of sequencing pores dropped below 10% of the starting number, the run was stopped and a nuclease flush was performed using the NFL_9076_v109_revA Nuclease Flush protocol from Oxford Nanopore. Additional library material was loaded onto flow-cells (by species) and sequencing was initiated. In total, three flow cells were used to sequence the Dwarf sample (with three reloads among them) and three flow cells for the Normal sample (with three reloads). Raw nanopore reads were basecalled using guppy (version 3. 0. 5. flip-flop HAC model). Data metrics before quality filtering were 72. 1 Gb (N50 = 27. 1 kb) for the Dwarf sample and 80 Gb (N50 = 27. 9 kb) for the Normal sample.