dataesr/scientific-paragraphs-categorization huggingface dataset:
Acknowledgments We are grateful to  Ludivine Brechard ,  Claudia Andrieu ,  Emilie Burel ,  Elsa Prudent ,  Céline Gazin , and  Marielle Bedotto  for their technical help. All authors have no conflicts of interest to declare. Didier Raoult has been a consultant for  Hitachi High-Technologies Corporation , Tokyo, Japan from 2018 to 2020. He is a scientific board member of  Eurofins company and a founder of a microbial culture company (Culture Top ). Funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.
80 -far superior to what can be currently obtained using genotyping data in cohorts 20-times larger. We observed that functional MRI was more important for prediction than anatomical MRI, and that increasing sample size steadily increased prediction accuracy, providing an efficient strategy to improve biomarkers. We also observed that despite a strong incentive to generalise to unseen data, model development on a given dataset faces the risk of overfitting: performing well in cross-validation on the data at hand, but not generalising. Finally, we were able.
, 2014b ; Di Martino et al. , 2017 ) , EU-AIMS (European Autism Interventions -A Multicentre Study for Developing New Medications,  Murphy and Spooren, 2012 )  or the IBIS Network  (Infant Brain Imaging Study, Hazlett et al. , 2012 ) , increasing sample sizes through data sharing. They extend the amount and quality of the data collected through harmonisation efforts. Recent analysis across cohorts has shown that ASD is significantly associated with changes in functional connectivity  ( Holiga et al. , 2019 ). But are these changes large enough to ground reliable prediction to new sites, despite heterogeneity in imaging techniques and populations recruited? The study we present built upon these large cohorts, and framed the extraction of biomarkers as an open, international challenge to predict ASD from the largest MRI dataset currently available -more than 2000 individuals.
Brain imaging dataset The brain imaging dataset combined data from the public Autism Brain Imaging Data Exchange (ABIDE) I and II datasets  ( Di Martino et al. , 2014a ; Di Martino et al. , 2017 )  and an unpublished dataset from the Robert Debré Hospital (RDB) in Paris, France (see Supplemental Table 1 for additional demographic information for the RDB site). ABIDE provides open access to functional and anatomical MRI data for 2156 subjects. The RDB dataset contained data from 247 subjects, 56 of whom were also part of the ABIDE II project: we excluded these duplicate subjects from the private dataset. With the exception of data coming from the RDB centre which was acquired in a 1. 5 Tesla scanner, all MRI data was acquired in 3 Tesla scanners. In all sites subjects were diagnosed using standard ADI/ADOS tools to support clinical assessment.
Most subjects had a full IQ > 75. We extracted anatomical features from the anatomical MRI: regional brain volumes, cortical thickness, and surface area; and extracted timeseries signals from the resting-state functional MRI. To derive these individual measurements, the data were processed with standard neuroimaging tools: Freesurfer  ( Fischl et al. , 1999 ) , FSL  ( Woolrich et al. , 2009 ) , and AFNI  ( Cox, 1996 ). We split the total dataset into public and private datasets, aiming at balancing the age and sex distributions ( Table  1  and Fig. 1  provide demographic information for the public and private datasets). A total of 2117 subjects were included: 947 ASD and 1170 controls. We did not exclude subjects based on quality control, but provided challengers with quality control scores obtained from visual inspection by 3 experts.
The public dataset contained data from 1150 subjects: 549 ASD and 601 controls. The private dataset contained data from 967 subjects: 398 ASD and 569 controls. The public dataset was a subset of ABIDE I and II, while the private dataset combined data from ABIDE I, II and RDB. The dataset should capture well the clinical and methodological heterogeneity of ASD neuroimaging. The ABIDE dataset was collected by 24 different centres worldwide, and spans an age range from 5 to 64 years old (median 13. 8 years old). All subjects had intellectual quotients within the normal range (97% have full IQ > 80), 80% of patients and 90% of controls were right-handed. The cohort was composed of 80% males. This diversity should allow classification algorithms to generalise, preventing them from specialising in a particular type of data or age range.
Description of the challenge: We launched the challenge inviting data scientists to submit algorithms to predict ASD diagnostic from provided MRI data. The challenge lasted 3 months and attracted 146 challengers. We awarded money prizes to the 10 best challengers. These were determined after the closure of the competition, by assessing how well the algorithms would predict ASD diagnostic in a private, unseen dataset. The ABIDE data can be openly distributed, which enabled us to provide a rich dataset on which challengers could tune their algorithms. To facilitate access to the data, we provided challengers with a "starting kit " giving a proof-of-concept predictive model on the data (extracted on standard brain atlases). Challengers were then able to develop their own prediction algorithms, which they submitted using a Web interface.
The code was executed on our central server. The challengers never had access to the private dataset. They used machine-learning techniques tuned on the public dataset and submitted the corresponding code to the central server which evaluated them in the private dataset. To measure the quality of the predictions, we used a standard metric: the area under the receiver operating characteristic curve (ROC-AUC). This measure summarises various detection tradeoffs, for example, favouring few false negatives to the cost of false positives in the case of a screening study, or the converse, in the case of a confirmatory study. Prediction at chance level gives an AUC of 0. 5, while perfect prediction gives an AUC of 1. Statististical analysis strategy: After the closure of the challenge, we analysed the 10 best submissions to understand the factors driving their predictions.
Considering that these machine-learning algorithms captured the best possible predictive biomarkers given our brainimaging cohort, we varied the data on which they were applied. Each time, we fit the algorithms anew on the data, to extract the corresponding biomarkers, and measured the prediction accuracy on the private dataset. First, we applied them to different imaging modalities: only functional MRI, or only anatomical MRI. Second, we varied the number of available subjects, to measure the importance of the sample size. Finally, we investigated the importance of different brain regions by removing those that appeared as most discriminative and attempting to extract biomarkers from the rest of the data. To compute which regions were the most discriminative, we used the absolute value of the model's coefficients when the algorithm used a linear model, and the feature importance when the algorithm used a random forest.
Challenge organisation Organisation of the challenge: We launched the challenge on May 1st 2018 and closed it on July 1st 2018. The challenge attracted about 146 participants accounting for a total of about 720 submissions. We awarded money prizes to the 10 best challengers. Challengers were ranked based on the ROC-AUC score of their submission computed on the private dataset. We framed the challenge problem by providing: (i) a public dataset, (ii) a standard way to assess the submission, and (iii) a starting kit. For this purpose, we used the RAMP (Rapid Analytics and Model Prototyping) workbench. Participants submitted their solutions on the RAMP website ( https://ramp. studio ). During the challenge we provided to the participants the cross-validated score computed on the public dataset.
At the end of the challenge, we asked each participant to select a single submission. This submission was trained (fitted) on the public dataset and evaluated on the private dataset hidden from the participants. Participants were ranked based on the score of these submissions computed on the private dataset. Challenge platform: RAMP is an online data science tool used to organise challenges. RAMP enables us to easily compare and reproduce predictive experiments. It can be used on the user's computer or online: the former is for developing predictive models while the latter assesses their predictive accuracy. A RAMP "starting kit " is a placeholder where we define the data-science problem: we provide the datasets, the metric, and the model validation technique. Participants can focus on the development of their machine-learning predictive model.
We also provide examples to help participants understand the challenge. The RAMP website was used to evaluate the solutions of the participants (i. e. , predictive models): participants submit their code and the website trains (fits) and evaluates them. We deployed, trained, and tested the full workflow on Amazon Web Services. Note that participants can also train and test their models locally. However, they only have access to the public dataset to test their models. We rely on Python and its rich ecosystem. Participants were free to use any Python library to build their machine learning model. Data and code availability: The public data, the code of the ten best submissions, the MRI preprocessing scripts and the scripts used to generate the figures are available at https://github. com/neuroa natomy/autism-challenge.
Participation to the challenge We received 589 submissions from 61 teams. To assess external validity of the biomarkers  ( Steyerberg and Harrell, 2016 ) , the private dataset contained > 200 subjects not included in ABIDE from the Robert Debré Hospital in Paris, France (RDB). We selected the 10 submissions which performed the best (taking only one submission per team) as the winning ones, and gave money prizes to the submitters. After close inspection we discovered that we forgot to remove 56 subjects of the RDB dataset which were also in the ABIDE dataset. For the post-hoc analyses presented here, we removed those duplicated subjects from the private dataset to avoid artificially inflating the prediction score. The ranking between the 10 best submissions remained relatively stable after the removal of the duplicate subjects (Spearman correlation: rs = 0.
Used as a confirmatory diagnostic test, the predictor would detect 25% of the individuals with ASD and only 3% of controls would be misclassified as patients. Predictions were also good (even slightly better) on the RDB subjects which were absent from the ABIDE dataset (median AUC = 0. 809 versus AUC = 0. 768 -Fig. 2 d ). After the challenge, we used an additional dataset from the EU-AIMS project to evaluate the ability of the blended predictor to generalise. Performance on this additional data was slightly worse than on our external site (AUC = 0. 721), revealing that the predictor was still fragile to distribution shifts despite having been extracted from multiple sites. Demographic information for this dataset can be found in Table  1  ; the most pronounced difference between EU-AIMS and our study cohort is on the IQ of participants and we speculate that this difference might drive the prediction performance loss.
Further increases in sample size should lead to increased prediction accuracy The analysis of the learning curve showed that prediction accuracy was not reaching a plateau and should keep improving by increasing the number of subjects ( Fig. 2 e ). Extrapolating on this increase, we estimate that prediction accuracy could reach an AUC = 0. 83 if a dataset of 10,000 subjects were available (see Supplemental Methods M3 for additional information). Increasing the number of subjects improves the discriminative power of the methods. The current trend suggests an increase of prediction performance to 0. 83 for a dataset of 10,000 subjects.
One submission performed dimensionality reduction using a principal component analysis (PCA). Six submissions used logistic regression as a first layer predictor, two used linear c-support vector classification (SVC) and the two others a combination of different methods. Three submissions obtained directly a single prediction whereas the remaining seven obtained several predictions, which were combined using a logistic regression for five of them, average for one them, and a majority vote for the last one. Challengers used a variety of methods, which included machinelearning techniques ranging from simple logistic regressions to complex graph-convolutional deep-learning models. Inspection of the individual scores showed that deep-learning techniques displayed strong overfits: good performance on the public dataset but poor generalisation to the private dataset.
On the opposite, many algorithms which used simpler approaches had a stable prediction performance when applied to new data, displaying prediction accuracies between AUC = 0. 7 and 0. 8 which did generalise to the private dataset. The submissions which led to reliable predictions had some methodology in common. In particular, they used linear support vector machines or logistic regression to compute the combination of the different signals -anatomical and functionalthat best discriminated patients from controls. In addition, several submissions combined signals from multiple atlases using a stacking strategy  ( Wolpert, 1992 ).
Evaluation without a blind validation set is at risk of severe optimism Our challenge highlights that a trustworthy development of biomarkers must include the evaluation on a new dataset, blinded to the analysts. In our challenge, participants had no interest in developing algorithms that would give optimistic results on the public dataset, as they knew that they would be evaluated on the private dataset. Nevertheless, comparing the prediction quality on the public dataset to that on the private dataset ( Figs. 4  and 5  ) clearly showed that the best predictions on the public dataset were too good to be true, and did not carry over to the unseen, private dataset. This is likely because the challengers made their analytic choices or trained hyperparameters to improve the prediction score that they measured on the public dataset ( Fig.
4  ). As a result, they obtained seemingly-excellent predictions, but which likely relied on noise in the public data and did not generalise to the private dataset. Indeed, the techniques used in machine-learning to measure prediction performance, such as cross-validation, are not completely robust to systematic exploration of analytic choices  ( Varoquaux, 2018 ) :  The score of each final submission to the challenge is represented as a point on the public and the private dataset. The prediction on the public dataset was measured with 8 random splits (with 20% of the data left out), the standard machine-learning procedure to measure prediction performance. Extreme prediction scores in the public dataset (AUC > 0. 8) did not generalise to the private dataset. By contrast, algorithms with conservative scores (0.
6 < AUC < 0. 8) did predict the private dataset equally well. To fully trust the prediction accuracy of biomarkers, these must be validated externally, as in our challenge. However, when testing external validity on a third, completely new data (EU-AIMS, see Fig. 5  ), the ranking of the models stayed roughly similar as on the private dataset, a pattern already reported on kaggle challenges  ( Roelofs et al. , 2019 ). The blind assessment was necessary to select methods that extract good ASD biomarkers, and to reliably quantify prediction accuracy.
We acquired clinical, cytokine and resting-state fMRI data before and after each treatment to investigate acute effects of pro/antiinflammatory therapies on regional global brain connectivity (GBC). In contrast to task-based fMRI, use of resting-state fMRI (rs-fMRI) allowed us to uncover basic pharmacological mechanisms that are not restricted to circuits engaged by any specific paradigm  (Khalili-Mahani et al. , 2017)  and provide a more accurate picture of regional brain vulnerability to treatment-induced functional changes. We hypothesized that interferon-α and anti-TNF treatments would result in divergent changes in GBC across the whole-brain (and predicted that the spatial patterns of changes would at least partially anticorrelate). Based on current models which posit that peripheral inflammation influences the brain by engaging local neuroinflammatory processes  (Miller and Raison, 2016) , we also hypothesized that the extent to which the GBC of a brain region in-/decreased in response to these treatments could be explained by spatial variation in the distribution of genes of the neuroimmune axis, such as those expressed in microglia and astrocytes, in the healthy human brain.
Image acquisition MR imaging was performed on a 1. 5 T Siemens Avanto (Siemens AG Medical Solutions, Erlangen, Germany) equipped with a 32-channel head-coil. Functional MRI data were obtained during rest using a T2*weighted EPI sequence (TR/TE = 2520/43 ms; flip angle = 90⁰; resolution = 3 × 3 × 3 mm, with 20% between-slice gap; matrix size = 64x64; 34 axial slices; 190 volumes). Functional data for four anti-TNF treated patients were acquired with a slightly longer TR (2620 ms) and consequently a slightly smaller number of volumes (180) to maintain the total duration the same. Excluding these four patients did not significantly change our estimates of treatment-induced changes in GBC; therefore, we kept these patients in all analyses to maximize power. A 3D T1-weighted anatomical scan was obtained for each participant in one session using an MPRAGE acquisition (TR = 2730 ms, TE = 3.
Imaging analysis The interferon-α and anti-TNF rs-fMRI datasets were pre-processed using the FMRIB Software Library (FSL). Pre-processing steps included volume re-alignment with MCFLIRT  (Jenkinson et al. , 2002) , non-brain tissue removal with the brain extraction tool (BET)  (Smith, 2002) , spatial smoothing with an 8-mm FWHM Gaussian kernel and de-noising with ICA-based Automatic Removal Of Motion Artifacts (ICA-AROMA)  (Pruim et al. , 2015). Additionally, subject-specific WM and CSF masks, obtained from the segmentation of the subjects' structural images and eroded to minimize the contribution of grey matter partial volume effects, were used to extract and regress out the mean WM and CSF signals from each participant's pre-processed dataset. A high-pass temporal filter with a cut-off frequency of 0.
Following previous recommendations, we excluded all negative correlations when calculating the GBC due to poor understanding of the biological meaning of negative correlations  (Cole et al. , 2012; Murphy et al. , 2009; Fox et al. , 2009). The python script can be downloaded from https://github. com/matteofr igo/gfcpy. We then investigated treatment effects by comparing baseline and post-administration GBC for each region using paired-T tests, applying false discovery rate (FDR) correction for the total number of regions tested.
Microarray expression data -Allen Human Brain Atlas (AHBA) Regional microarray expression data were obtained from the six post-mortem brains provided by the Allen Human Brain Atlas (AHBA; https://human. brain-map. org/) (ages 24-57 years)  (Hawrylycz et al. , 2012). The abagen toolbox (https://github. com/netneurolab/abagen) was used to process and map the transcriptomic data to the 83 parcellated regions of the DK brain atlas  (Desikan et al. , 2006). Briefly, genetic probes were reannotated using information provided by  Arnatkeviciute et al. , 2019)  instead of the default probe information from the AHBA dataset, hence discarding probes that cannot be reliably matched to genes. Following previously published guidelines for probe-to-gene mappings and intensity-based filtering  Arnatkeviciute et al.
(2019) , the reannotated probes were filtered based on their intensity relative to background noise level; probes with intensity lower than the background in ≥50% of samples were discarded. A single probe with the highest differential stability (highest pooled correlation across donors) was selected to represent each gene  (Hawrylycz et al. , 2015). This procedure retained 15,633 probes, each representing a unique gene. Differential stability was calculated as  (Hawrylycz et al. , 2015) : Δs(p) = 1 ( N 2 ) ∑ N-1 i=1 ∑ N j=i+1 ρ [ B i (p), B j (p) ] Here, ρ is Spearman's rank correlation of the expression of a single probe p across regions in two donor brains, Bi and Bj, and N is the total number of donor brains. This procedure retained 15,633 probes, each representing a unique gene. Next, tissue samples were assigned to brain regions using their corrected MNI coordinates (https://github.
Ensemble gene set enrichment analyses We then used the list of genes ranked by their respective weights in the PLS 1 component to perform enrichment analyses for biological pathways (gene ontology -GO) and genes expressed in different brain cell types, as identified in previous single-cell transcriptomic studies of the human brain  (Zhang et al. , 2016; Lake et al. , 2018; Habib et al. , 2017; Darmanis et al. , 2015; Li et al. , 2018). These analyses were implemented using the ensemble enrichment analysis toolbox of Fulcher et al. (https://github. com/benfulcher/GeneSetEnrichmentAnalysis) (Fulcher et al. , 2021). Significance was assessed against ensembles of 10,000 randomized brain markers where the brain's spatial autocorrelation was preserved (see below) enabling us to determine whether a given brain phenotype is significantly correlated to genes in a category beyond what would be expected from the null phenotypes and minimise the risk of false positives  (Fulcher et al.
Spatial alignment between individual connectivity changes and patterns of transcriptomic vulnerability correlates with inter-individual differences in peripheral inflammatory responses and depressive symptoms In the Interferon-α dataset, the extent to which each participant expressed patterns of GBC changes aligning with the PLS 1 pattern of    Genes were ranked according to their respective weights in the first component of our partial least square regression models. Negative enrichment ratios indicate enrichment for a certain gene set among genes with negative weights. The reverse applies to positive enrichment ratios. The colour scale depicts enrichment ratio. Note: because we were predominantly interested in genes with high expression in regions where GBC decreases and low expression in regions with negligible GBC changes, we only interpreted enrichment for negatively weighted genes in both datasets.
Abbreviations: Excexcitatory; Ininhibitory. transcriptomic vulnerability negatively correlated with: i) increases in circulating IL-6, 4 h after first interferon-α administration; ii) subsequent increases in HAM-D depression scores 4 weeks later (both p FDR < 0. 05). None of the remaining correlations reached statistical significance (Table  2 ). In the anti-TNF dataset, the extent to which each subject expressed patterns of GBC changes aligning with the PLS 1 pattern of transcriptomic vulnerability positively correlated with peripheral decreases in IL-6, 24 h after first anti-TNF treatment administration (p < 0. 05, uncorrected), though this correlation did not survive FDR correction for multiple testing. None of the remaining correlations reached statistical significance (Table  3 ).