dataesr/scientific-paragraphs-categorization huggingface dataset:
ADNI Three subsets were created from the ADNI dataset: ADNIT1w, ADNICLASS and ADNICLASS, Aß. ADNIT1w comprises all participants (N=1,628) for whom a T1-weighted (T1w) MR image was available at baseline. ADNICLASS comprises 1,159 participants for whom a T1w MR image and an FDG PET scan, with a known effective resolution, were available at baseline. ADNICLASS, Aß is a subset of ADNICLASS that comprises 918 participants with a known amyloid status determined from a PiB or an AV45 PET scan using 1. 47 and 1. 10 as cutoffs, respectively  (Landau et al. , 2013). For each ADNI subset, five diagnosis groups were considered: • CN: subjects who were diagnosed as CN at baseline; • AD: subjects who were diagnosed as AD at baseline; • MCI: subjects who were diagnosed as MCI, EMCI or LMCI at baseline; • pMCI: subjects who were diagnosed as MCI, EMCI or LMCI at baseline, were followed during at least 36 months and progressed to AD between their first visit and the visit at 36 months; • sMCI: subjects who were diagnosed as MCI, EMCI or LMCI at baseline, were followed during at least 36 months and did not progress to AD between their first visit and the visit at 36 months.
Methods We developed a unified set of tools for data management, image preprocessing, feature extraction, classification, and evaluation. These tools have been integrated into Clinica 5   (Routier et al. , 2018) , an open-source software platform that we developed. Conversion tools allow an easy update of the datasets as new subjects become available. The different components were designed in a modular way: processing pipelines using Nipype  (Gorgolewski et al. , 2011) , and classification and evaluation tools using the scikit-learn 6  library  (Pedregosa et al. , 2011). This allows the development and testing of other methods as replacement for a given step, and the objective measurement of the impact of each component on the results. A simple command line interface is provided and the code can also be used as a Python library.
Converting datasets to a standardized data structure Even though public datasets are extremely valuable, an important difficulty with these studies lies in the organization of the clinical and imaging data. As an example, the ADNI and AIBL imaging data, in the state they are downloaded, do not rely on community standards for data organization and lack of a clear structure. Multiple image acquisitions exist for a given visit of a participant and the complementary image information is contained in numerous csv files, making the exploration of the database and subject selection very complicated. To organize the data, we selected the BIDS format  (Gorgolewski et al. , 2016) , a community standard enabling the storage of multiple neuroimaging modalities. Being based on a file hierarchy rather than on a database management system, BIDS can be easily deployed in any environment.
Very importantly, we provide the code that automatically performs the conversion of the data as they were downloaded to the BIDS organized version, for all the datasets used: ADNI, AIBL and OASIS. This allows direct reproducibility by other groups without having to redistribute the dataset, which is not allowed in the case of ADNI and AIBL. We also provide tools for subject selection according to desired imaging modalities, duration of follow up and diagnoses, which makes possible the use of the same groups with the largest possible number of subjects across studies. Finally, we propose a BIDSinspired standardized structure for all the outputs of the experiments.
Conversion of the ADNI dataset to BIDS The ADNI to BIDS converter requires the user to have downloaded all the ADNI study data (tabular data in csv format) and the imaging data of interest. Note that the downloaded files must be kept exactly as they were downloaded. The following steps are performed by the automatic converter (no user intervention is required). To convert the imaging data to BIDS, a list of subjects with their sessions is first obtained from the ADNIMERGE spreadsheet. This list is compared for each modality of interest to the list of scans available, as provided by modality-specific csv files (e. g. MRILIST. csv). If the modality was acquired for a specific pair of subject-session, and several scans and/or preprocessed images are available, only one is converted. Regarding the T1 scans, when several are available for a single session, the preferred scan (as identified in MAYOADIRL_MRI_IMAGEQC_12_08_15.
Conversion of the AIBL dataset to BIDS The AIBL to BIDS converter requires the user to have downloaded the AIBL non-imaging data (tabular data in csv format) and the imaging data of interest. The conversion of the imaging data to BIDS relies on modality-specific csv files that provide the list of scans available. For each AIBL participant, the only T1w MR image available per session is converted. Note that even though they are not used in this work, we also convert the Florbetapir, PiB and Flutemetamol PET images (only one image per tracer is available for each session). Once the images of interest have been selected and the paths to the image files identified, the imaging data are converted to BIDS following the same steps as described in the above section. The conversion of the clinical data relies on the list of subjects and sessions obtained after the conversion of the imaging data and on the csv files containing the non-imaging data.
Conversion of the OASIS dataset to BIDS The OASIS to BIDS converter requires the user to have downloaded the OASIS-1 imaging data and the associated csv file. To convert the imaging data to BIDS, the list of subjects is obtained from the downloaded folders. For each subject, among the multiple T1w MR images available, we select the average of the motion-corrected co-registered individual images resampled to 1 mm isotropic voxels, located in the SUBJ_111 subfolder. Once the paths to the image files have been identified, the images in Analyse format are converted to nifti using the mri_convert tool of FreeSurfer  (Fischl, 2012) , the BIDS folder hierarchy is created, and the images are copied to the appropriate folder and renamed. The clinical data are converted using the list of subjects obtained after the conversion of the imaging data and the csv file containing the non-imaging data, as described in the previous section.
Finally, we mask the non-brain regions using a binary mask resulting from thresholding the sum of the GM, WM and 7 http://www. fil. ion. ucl. ac. uk/spm/software/spm12/ 8 https://github. com/UCL/PETPVC CSF tissue probability maps for the subject in MNI space. The resulting masked SUVR images are also in a common space and provide voxel-wise correspondence across subjects.
The individual subject parcellations were then registered to MNI space to generate a maximum probability map. It is composed of 140 regions covering the whole cortex as well as the main subcortical structures. Data were made available for the "MICCAI 2012 Grand Challenge and Workshop on Multi-Atlas Labeling". The main difference between the LBPA40, Hammers and Neuromorphometrics atlases is the degree of detail (i. e. the number of regions) of the anatomical parcellation.
Random forest The third classifier used is the random forest. Unlike both linear SVM and logistic regression, random forest is an ensemble method that fits a number of decision trees on various sub-samples of the dataset. The combined estimator prevents overfitting and improves the predictive accuracy. Based on the implementation provided by the scikit-learn library  (Pedregosa et al. , 2011) , there is a large number of parameters that can be optimized. After preliminary experiments to assess which had a larger influence, we selected the following two hyperparameters to optimize: i) the number of trees in the forest; ii) the number of features to consider when looking for the best split. Random forest was only used for regional features and not voxel features, due to its high computational cost.
Classification experiments The different classification tasks considered in our analyses for each dataset, driven by the data availability, are detailed in Table  6. Details regarding the group compositions can be found in Tables  2, 3 , 4 and 5. In general, we perform clinical diagnosis classification tasks, or "predictive" tasks of the evolution of MCI subjects. Note that tasks involving progression from MCI to AD were not performed for AIBL due to the small number of participants in the sMCI and pMCI categories. However, the framework would allow performing these experiments very easily when more progressive MCI subjects become publicly available in AIBL. Depending on the type of features, the performance of several classifiers with different parameters was tested. For voxel features, the only classifier was the linear SVM.
Four different levels of smoothing were applied to the images using a Gaussian kernel, from no smoothing to up to 12 mm full width at half maximum (FWHM). For region-based classification experiments, three classifiers were tested: linear SVM, logistic regression and random forest. The features were extracted using five atlases: AAL2, AICHA, Hammers, LPBA40 and Neuromorphometrics. This information is summarized in Table  7. For the datasets under study, different imaging modalities were available: while both T1w MRI and FDG PET images were available for the ADNI participants, only T1w MRI were available for AIBL and OASIS participants. For each modality considered, both voxel and region features were extracted using the different parameters detailed in Table  7. All the classification experiments tested in this work are summarized in Table  8.
Results Here, we present a selection of the results that we believe are the most valuable. The complete results of all experiments (including other tasks, preprocessing parameters, features or classifiers) are available in the Supplementary Material as well as in the repository containing all the code and experiments (https://gitlab. icm-institute. org/aramislab/AD-ML). In the following subsections, we present the results using the balanced accuracy as performance metric but all the other metrics are available in the results.
Influence of the smoothing T1w MRI and FDG PET images were not smoothed or smoothed using Gaussian kernels with FWHMs of 4 mm, 8 mm and 12 mm. To determine the influence of different smoothing degrees on the classification accuracy, a linear SVM classifier using voxel features was chosen. Three classification tasks were studied: CN vs AD, CN vs pMCI and sMCI vs pMCI. The results in Figure  2  show that, for most classification tasks, the balanced accuracy does not vary to a great extent with the smoothing kernel size. The only variations are observed for the CN vs pMCI and sMCI vs pMCI tasks when the features were extracted from T1w MR images: the balanced accuracy increases slightly with the kernel size. The same analysis was run using T1w MR images from the AIBL dataset. The mean balanced accuracy also increased slightly with the kernel size, but the standard deviations of the balanced accuracies are larger than for ADNI.
Influence of the type of features We compared the balanced accuracies obtained for the voxel features with reference smoothing (Gaussian kernel of 4 mm FWHM) to the ones obtained for the regional features with reference atlas (AAL2) when using linear SVM classifiers. These features were extracted from T1w MRI and FDG PET images of ADNI participants. The same three classification tasks as before were evaluated. The results, displayed in Table  9 , do not show notable differences between the mean balanced accuracies obtained using voxel or regional features. In the case of the AIBL dataset, the balanced accuracy is higher for the region-based classification (for AD vs CN: voxel-based 0. 79 [±0. 059], region-based 0. 86 [±0. 042]), but we can observe that the corresponding standard deviations are high.
Influence of the dataset We also wanted to know how consistent were the results across datasets, and thus we compared the classification performances obtained from ADNI, AIBL and OASIS, for the task of differentiating control subjects from patients with Alzheimer's disease. Voxel (4 mm smoothing) and regional (AAL2 atlas) features were extracted from T1w MR images and used with linear SVM classifiers. We tested two configurations: training and testing the classifiers on the same dataset, and training a classifier on ADNI and testing it on AIBL and OASIS. Results are displayed in Table  11. Performances obtained on ADNI and AIBL were comparable and much higher than those obtained on OASIS. When training on ADNI and testing on AIBL or OASIS, the balanced accuracy was at least as high as when training and testing on AIBL or OASIS respectively, suggesting that classifiers trained on ADNI generalized well to the other datasets.
In particular, training on ADNI substantially improved the classification performances on OASIS. We aimed to assess whether this was due to the larger number of subjects in ADNI. To that purpose, we performed the same experiments but with subsets of participants of equal size for each dataset. We randomly sampled populations of 70 AD patients and 70 CN participants from each of the datasets, ensuring that the demographic and clinical characteristics of the subpopulations did not differ from the original ones. As can be seen from Table  11 , using the subset, the improvement disappeared for the voxel-based but remained for the regional features. Table  11. Influence of dataset. Average ± SD of the balanced accuracy obtained for the reference linear SVM classifier when differentiating CN and AD subjects using voxel (4 mm smoothing) and regional (AAL2 atlas) features extracted from T1w MR images for three datasets: ADNI, AIBL and OASIS.
Upper rows display results for the full population. Lower rows display results for subsets of equal size for each dataset. The subsets were obtained by randomly sampling populations of 70 AD patients and 70 CN participants from each of the datasets. Note that for the "full dataset" experiment, a balanced subset of AIBL was used (i. e. 72 CN and 72 AD subjects). When the testing dataset differs from the training dataset, there is no CV and thus no empirical SD.
Influence of the training dataset size Learning curves were computed to assess how the performance of linear SVM classifiers varies depending on the size of the training dataset. Using only ADNI participants, we tested four scenarios: voxel and region features extracted from T1w MRI and FDG PET images. As crossvalidation, 250 iterations were run where the dataset was randomly split into a test dataset (30% of the samples) and a training dataset (70% of the samples). The maximum number of subjects used for training and testing for each of the different tasks is of 362 for CN vs AD, of 313 for CN vs pMCI and of 355 for sMCI vs pMCI. For each run, 10 classifiers were trained and evaluated on the same test set using from 10% to all of the training set (from 7% to up to 70% of the samples), increasing the number of samples used by 10% on each step.
Therefore, the number of participants used for training ranged from 20 to 197 for CN, 24 to 239 for sMCI , 12 to 117 for pMCI and 17 to 166 for AD. We can observe from the learning curves in Figure  6  that, as expected, the balanced accuracy increases with the number of training samples. Learning curves were also computed for the CN vs AD task when using larger datasets obtained by combining participants from ADNI and AIBL (balanced subset composed of 72 CN subjects and 72 AD subjects) and from ADNI, AIBL and OASIS. Results are displayed in Figure  7. We observe that for an equivalent number of subjects, combining ADNI and AIBL or only using ADNI leads to a similar balanced accuracy. For regional features, the performance is slightly higher when combining ADNI and AIBL compared to when only using ADNI, but the difference is largely within the standard deviation.
Discussion We presented an open-source framework for the reproducible evaluation of AD classification methods that contains the following components: i) converters to normalize three publicly available datasets into BIDS; ii) standardized preprocessing and feature extraction pipelines for T1w MRI and PET; iii) standard classification algorithms; iv) cross-validation procedures following recent best practices. We demonstrated its use for the assessment of different imaging modalities, preprocessing options, features and classifiers on three public datasets. In this work, we first aim to contribute to make evaluation of machine learning approaches in AD: i) more reproducible; ii) more objective. Reproducibility is the ability to reproduce results based on the same data and experimental procedures.
Calls to increase reproducibility have been made in different fields, including neuroimaging  (Poldrack et al. , 2017)  and machine learning  (Ke et al. , 2017). Reproducibility differs from replication, which is the ability to confirm results on independent data. Key elements of reproducible research include: data sharing, storing of data using community standards, fully automatic data manipulation, sharing of code. Our work can contribute to increase reproducibility of AD ML research through different aspects. A first component is the fully automatic conversion of three public datasets into the community standard BIDS. Indeed, ADNI and AIBL cannot be redistributed. Through these tools, we hope to make it easy to reproduce experiments based on these datasets without redistributing them. In particular, we offer a huge saving of time to users compared to simply making public the list of subjects used.
This is particularly true for complex multimodal datasets such as ADNI (with plenty of incomplete data, multiple instances of a given modality and complex metadata). The second key component is publicly available code for preprocessing, feature extraction and classification. These contributions are gathered in Clinica 10  , a freely available software platform for clinical neuroscience research studies. In addition to increased reproducibility, we hope that these tools will also make the work of researchers easier. We also hope to contribute to more objective evaluations. Objective evaluation of a new approach (classification algorithm, feature extraction method or other) requires to test this specific component without changing the others. Our framework includes standard approaches for preprocessing and feature extraction from T1-weighted MRI and FDG PET data, and standard classification tools.
These constitute a set of baseline approaches against which new methods can easily be compared. Researchers working on novel methods can then straightforwardly replace a given part of the pipeline (e. g. feature extraction, classification) with their own solution, and evaluate the added value of this specific new component over the baseline approach provided. We also propose tools for rigorous validation, largely based on recent guidelines of  (Varoquaux et al. , 2017)  and implemented based on the standard software scikit-learn  (Pedregosa et al. , 2011). These include: i) large number of repeated random split to extensively assess the variability of performances; ii) reporting the full distribution of accuracies and standard deviation rather than only mean accuracies; iii) adequate nested CV for hyperparameter tuning.
On the other hand, random forests or other ensemble methods could be useful when combining features from different modalities such as images, clinical data and cognitive scores, as done in  (Moradi et al. , 2015; Sørensen et al. , 2018). In other papers comparing several standard classification algorithms such as SVM, LDA or Naive Bayes  (Aguilar et al. , 2013; Cabral et al. , 2015; Sabuncu et al. , 2015) , results did not show differences between methods. We also assessed the influence of class imbalance, which in our datasets ranges from very mild (1. 2 times more CN than AD for ADNI) to moderate (1. 7 times more CN than pMCI and 2 times more sMCI than pMCI for ADNI) to very strong (6. 1 times more CN than AD in AIBL). In the case of voxel-based features, we found that a very strong class imbalance (as in the case of AIBL where the proportion is 6 to 1) leads to lower performance but that moderate class imbalance (up to 2 to 1 in ADNI) are adequately handled.
, 2012; Tohka et al. , 2016; Vemuri et al. , 2008)  and dimensionality reduction (e. g. principal component analysis, manifold learning)  (Beheshti et al. , 2015; Guerrero et al. , 2014; Liu et al. , 2015; Salvatore et al. , 2015). These approaches have the potential to improve the performance but they need to be validated using rigorous cross-validation procedures  (Eskildsen et al. , 2013; Maggipinto et al. , 2017). The evaluation of the added value of all these approaches could be done using our framework. This is out of the scope of the present paper and is left for future work. Using multiple datasets is important to assess if the performances are robust to different populations, acquired in different conditions. A first component consisted in performing the same experiments on different datasets.
We found that classification results were similar for ADNI and AIBL datasets, but much lower for OASIS. The lower performance for OASIS is likely due to the diagnosis criteria which are less rigorous (in OASIS, all participants with CDR>0 are considered AD). It is also valuable to know how a classifier will perform when trained on one dataset and tested on another one. The classifiers trained on ADNI data generalized well to AIBL and OASIS. Interestingly, for OASIS, the performances were substantially increased when training on ADNI compared to when training on OASIS. Such improvement may arise from several factors: larger training set size, higher image quality or stricter diagnostic criteria. When using subsets of equal size, the improvement obtained for voxel-based features disappeared, suggesting that increased training set size is important, in particular when using very high dimensional features.
On the other hand, for regional features, training on the ADNI subset improved performances compared to training on the OASIS subset, suggesting that other factors (image quality, stricter diagnostic criteria) contribute to the improvement. In general, we can say that classifiers are able to generalize across different datasets, as is also concluded in  (Dukart et al. , 2013; Sabuncu et al. , 2015)  particularly if they are obtained using large multicentric datasets with strict diagnostic criteria, as is the case for ADNI. Unsurprisingly, increased training set size led to increased classification performances. This improvement of the results depending on the training set size has also been found in other studies such as  (Abdulkadir et al. , 2011; Chu et al. , 2012; Franke et al. , 2010).
One can note that when combining multiple datasets, performances also increased with training set size. However, when combining OASIS together with ADNI and AIBL, the performance was lower than when using only AIBL and ADNI. This is consistent with the fact that performances for OASIS are systematically lower than those obtained on ADNI and AIBL. Again, this is likely due to diagnostic criteria which are less rigorous in OASIS. Interestingly, with the current number of samples available, the point where the results stop improving has not been reached. The performance of the classifier reaches a limit imposed by the number of images that have been provided for training, meaning that more data are necessary to find the top performance of a classifier. These results highlight the need for more publicly available datasets, on which most of the current research in the field relies.