dataesr/scientific-paragraphs-categorization huggingface dataset:
Datasets The data used in this study, consisting of 2106 subjects, were obtained from multiple cohorts: the Alzheimer's Disease Neuroimaging Initiative (ADNI)  [51] , the Open Access Series of Imaging Studies (OASIS)  [52] , the Australian Imaging, Biomarkers and Lifestyle (AIBL)  [53]  and the Minimal Interval Resonance Imaging in Alzheimer's Disease (MIRIAD)  [54]. We used the baseline T1-weighted MRI available in each of these studies. Each dataset contains AD patients and CN subjects. ADNI1 and AIBL datasets also include pMCI and sMCI patients. As in  [12] , patients were considered as pMCI if they were diagnosed as MCI at the baseline and progressed to AD within 36 months. By contrast, patients were considered as sMCI if they were diagnosed as MCI at the baseline and all of sessions in the following 36 months.
The group lists were obtained using ClinicaDL 1   [12]  and thus the selection criteria is similar. Table  1  summarizes the number of participants and their age distribution for each dataset used in this study. T-tests showed no statistical differences in terms of age between two groups of the same dataset. During our experiments, AD and CN subjects from ADNI1 were used for training and all the other subjects as testing set. To minimize possible bias learned through training, we selected the same number of AD/CN subjects from ADNI1 for training without significant differences between the two age distributions ( = 0. 27). The evaluation consisted of two different tasks: Diagnosis (main task) and Prognosis (unknown task).
Collective AI In medical analysis, high generalization capacity across domains and unknown tasks presents potential clinical value as real data is diverse and may come from any source. As recently shown in  [10, 12] , current deep learning methods for AD classification can well generalize to similar datasets but poorly perform on datasets having differences such as MRI protocols, age ranges, country of origin or inclusion criteria. In our testing datasets, different age range and MRI protocol were present in OASIS, different country of origin was present in AIBL and different inclusion criteria was present in MIRIAD. It should be noted that for OASIS, MCI and AD patients are mixed, so we used the ADNI inclusion criteria to separate AD patients and be able to assess the diagnosis of AD. In this work, we propose to use an innovative collective artificial intelligence strategy to improve the generalization across domains and to unseen tasks.
Implementation details For each of the 125 patch locations, 80% of the training dataset (i. e. , ADNI1) was used for training a 3D U-Net and the remaining 20% for validation. To avoid bias resulting from dataset imbalance, the training/validation sets employed the same number of AD and CN. As the number of images in ADNI1 dataset was small, the training/validation data was re-split for each patch location to exploit the maximum information possible. The model was trained with voxel-wise mean absolute error (MAE) loss and Adam optimizer with a learning rate of 0. 001. All voxels equally contribute to the loss function during training. The training process is stopped after 20 epochs without improvement in validation loss. We employed several data augmentation and sampling strategies to alleviate the overfitting issue during training.
The model was trained using the binary cross-entropy loss, Adam optimizer with a learning rate of 0. 0003. No data augmentation was applied during training. The training process was stopped after 20 epochs without improvement in validation loss. During testing, we randomly added noise ∼  (0, 0. 01) to the node features and computed the average of 3 predictions to get the global decision  [63]. Experiments showed that it helps our GCN to be more stable. For training and evaluating steps, we used a NVIDIA TITAN X with 12GB of memory. The total training time for = 125 U-Nets and the GCN model is about 23 hours. The total inference time of our method is about 1. 63 seconds per preprocessed image.
Performance study In this section, the 125 CNN grading models and the classifier were trained using AD and CN subjects of the ADNI1 dataset. Then, we assessed their generalization capacity to domain shift using AD and CN subjects from ADNI2, AIBL, OASIS and MIRIAD. The generalization capacity for unseen tasks was studied using pMCI, sMCI subjects (AD prognosis) from ADNI1 (same domain) and AIBL (out of domain). Due to the imbalanced nature of testing datasets, we used the balanced accuracy (BACC) and area under receiver operating characteristic curve (AUC) to measure the performance of different classifiers. The global BACC/AUC for diagnosis and prognosis was measured with all available testing images for each task. Each experiment was repeated ten times (to reduce bias related to random nature of DL training) and the average results was provided as final results.
Comparison of Grading vs. Volume As discussed previously, brain atrophy is an important biomarker of Alzheimer's disease. Many studies used structure volume for AD classification and achieved encouraging results  [65, 66, 67]. So, we compare the proposed biomarker (grading, exp. 3) and the classical one (volume, exp. 4) to assess the efficiency of our new biomarker. The additional evaluation using the age feature (exp. 5) was performed to confirm that no age bias was present in the training/testing partitions. The efficiency of (exp. 3) was clearly better than V (exp. 4). outperformed V in global diagnosis, global prognosis and all of the tests on an individual dataset (all < 0. 05). Thus, the proposed biomarker presents an important interest for AD classification. Moreover, we trained a UMAP  [68]  with AD/CN subjects from ADNI1 and visualized the transformed test set in 2D space (see Figure  2 ).
Comparison of Collective AI features vs. Individual AI features We aimed at assessing the efficiency of the collective AI strategy. To do this, we compared the efficiency of and features (exp. 1, 3) (see Table  2 ). Experimental results showed that (exp. 3) is significantly better than (exp. 1) for both global diagnosis ( = 0. 007) and global prognosis ( = 0. 001). Consequently, collective AI strategy offered a significant improvement for unseen domain (AD diagnosis) and unseen task (AD prognosis). In terms of generalization, alleviated the drop in performance in AIBL dataset for AD diagnosis.
Comparison with state-of-the-art methods Tables  5  and 6  summarize the current performance in BACC of state-of-the-art methods proposed for AD diagnosis and prognosis classification that have been validated on external datasets. A comparison of performance in AUC is provided in annexes (Tables  10  and 11 ). In this comparison we consider five categories of deep methods: patch-based strategy based on a single model (Patch-based CNN  [12] ), patch-based strategy based on multiple models (Landmarkbased CNN  [32] , Hierarchical FCN  [34] ), ROI-based strategy based on a single model focused on hippocampus (ROIbased CNN  [12] ), subject-based considering the whole image based on a single model (Subject-based CNN  [12] , Efficient 3D  [21]  and 2  [79] ) and a classical voxel-based model using a SVM (Voxel-based SVM  [12] ).
Comparison with methods under the same condition For a fair comparison, we retrained and evaluated four methods whose code is available: Patch-based CNN, ROIbased CNN, Subject-based CNN and Voxel-based SVM  [12]  with our training/testing data. The results are reported in Table  5. For AD diagnosis (i. e. , AD/CN), as ADNI2 and ADNI1 (training set) are very similar, we used the performance on ADNI2 as a reference to assess the capacity of generalization on other datasets (i. e. , AIBL, OASIS, MIRIAD). Based on that, we observed a major drop in performance in Patch-based CNN method for AIBL, OASIS and MIRIAD, ROI-based CNN method for AIBL (see Table  5 ). For AD prognosis (i. e. , pMCI/sMCI), we also observed a drop in performance between AIBL and ADNI1 (training domain) in Patch-based CNN method, ROI-based CNN method and Subject-based CNN method.
Literature comparison We also detail the results of four other methods without available implementation performing evaluation across different datasets. In this case, we present the results of the original papers in Table  6. Consequently, there are many different factors between methods: number of subjects in training/testing sets, selection criteria, etc. However, this could help to get an idea of the performance of current methods in the application of AD diagnosis/prognosis. Overall, our method has most of the time the best or the second best result. Furthermore, it should be noted that our model is trained using only 340 images (from ADNI1) without any domain adaptation technique but outperforms Efficient3D (trained on 2843 images) and 2 (with domain adaptation) in most of datasets/tasks.
Consistency study Thibeau-sutre et al. have recently shown that for the same CNN architecture, different training data or even training runs can lead to different explanations  [88]. They suggested that a good explanation method should not depend on training data or training initialization. In this study, we analyzed these two aspects for our grading maps (dependency to data training and model initialization) by performing two experiments. First, we trained two grading models (each one consisting of 125 U-Nets) on ADNI1 (model 1) and ADNI2 (model 2) datasets. For each model, we then calculated the vector from the grading map for all images in testing set (excluding ADNI1 and ADNI2). Finally, we measured the cosine similarity of two vectors obtained from each image. We obtained a median of 0.
92 as similarity between two vectors from two models training on different datasets that demonstrate the good robustness to domain shift of our method. For the second one, we trained the grading model twice using only ADNI1 as training set (models 1 & 3). Finally, we obtained a median of 0. 95 as similarity between vectors from two retrained models on the same dataset that demonstrate the good robustness to training initialization of our method. Figure  5  shows examples of individual grading maps for four considered populations (i. e. , CN, sMCI, pMCI, AD). We can visually see the similarities between grading models trained on different datasets (i. e. , models 1 & 2) and between grading models trained several times on the same dataset (i. e. , models 1 & 3). Overall, the three models identify AD-related areas in a similar way.
Discussion In this paper, we proposed a novel deep grading framework dedicated to Alzheimer's disease diagnosis and prognosis. Our framework was designed to overcome three main limitations of current deep learning methods for AD classification: performance compared to conventional machine learning method (i. e. , SVM), generalization to unseen datasets/tasks and interpretability. While many studies found that deep learning and SVM methods had a similar performance for AD classification problem  [10, 12] , the authors also suggested that better model design could improve the performance of DL methods. Indeed, Jo et al. indicate that hybrid methods using CNN features and a conventional classifier showed better accuracy than pure deep learning methods  [89]. In this study, we combined CNN features with a GCN classifier.
The use of a GCN also allows to combine additional demographic information to further improve the model performance. As a result, our model showed a better performance with a large margin compared to traditional methods (e. g. , SVM). Furthermore, a careful design of edge connectivity in the subject's graph may also boost the model performance. In this study, we propose to define the edge connection between two brain structures as the absolute difference of their volumes. This type of connectivity allows our model to make more accurate decisions. The analysis of this connectivity is provided in annexes. This study is one of the few assessing deep model performance on multiple independent datasets (i. e. , ADNI2, AIBL, OASIS, MIRIAD) and unseen tasks (i. e. , AD prognosis)  [10, 12, 21, 32, 34, 79].
For AD diagnosis, the result on ADNI2 dataset was 87. 6% in BACC which was competitive with the current performance reported in the literature. On OASIS, we achieved the second place with 89. 1% accuracy. On AIBL and MIRIAD, our model outperformed current state-of-the-art methods with respectively 92. 4% and 99. 6%. Besides several studies found a drop in performance when evaluating on independent datasets  [10, 12]. This performance drop could come from differences in MRI protocols, age ranges, country of origin and inclusion criteria. Our results demonstrated the high generalization capacity of our method against datasets with such differences. Especially, the use of collective AI enabled a better generalization to unseen tasks (i. e. , AD prognosis) than other deep learning methods. Finally, the use of the weighted fusion strategy could improve even more the model performance (in AD diagnosis).
In terms of interpretability, our framework provides 3D grading maps capable of indicating regions impacted by AD. The most important structures highlighted by our grading map were correlated with knowledge about the disease in the literature. Furthermore, our experiments showed that grading features were more efficient than volume features for both AD diagnosis and prognosis which confirmed the finding of  [43, 9]. When coupling the grading map with a GCN classifier, it yielded high performance across datasets. Hence, grading maps are not only an interpretable visualization but provide also discriminative features for AD classification. However, the use of GCN made our framework become not fully interpretable. The fully-interpretable framework can be done by replacing the final GCN by an SVM classifier or a simple threshold.
This paper is among a few studies proposing an interpretable model for AD classification problem. Another approach for an interpretable model is to carefully design the graph neural network classifier and its input. Li et al. define individual graphs using features extracted from neuroimaging data and a graph neural network with ROIaware convolutional layer and an appropriate loss function  [72]. Similar to our result, this approach can provide both salient brain regions at the subject level and the community level. However, similar to explainable methods discussed above, it cannot provide quantitative information on the disease severity for a given region. While the performance of our framework across unseen datasets and unknown tasks was quite high, there also exist some limitations. First, the ground truth used for grading was potentially not optimal due to a lack of consensus on structures relevant to Alzheimer's disease.
Indeed, there may be some structures that are not impacted by AD and the ground-truth of these structures should be zero. With our ground-truth annotation, small structures surrounding another one highly related to AD had a high chance to appear together in all patches. Thus, those structures would be also predicted as related to AD. Another direction should focus on an unsupervised learning manner to find only abnormalities caused by AD to improve the interpretability. Second, this study exploited only structural MRI while the performance could be improved using multi-modal inputs such as PET, functional MRI, diffusion MRI or perfusion MRI  [92]. Better disease patterns are expected to be learned with this kind of input. However, a multi-modal input implies even larger differences between different datasets.
Conclusion In this paper, we addressed three major limitations of CNN-based methods by introducing a novel interpretable, generalizable and accurate deep grading framework. First, deep grading offers a meaningful visualization of the disease progression. Second, we proposed a collective artificial intelligence strategy to improve the generalization on datasets owning differences such as MRI protocols, age ranges, country of origin and inclusion criteria. Finally, we proposed to use a graph-based modeling to better capture AD signature using both inter-subject similarity and intra-subject variability. Based on that, our DG method showed state-of-the-art performance in both AD diagnosis and prognosis. sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health (www.
fnih. org). The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer's Therapeutic Research Institute at the University of Southern California. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern California. The OASIS Cross-Sectional project (Principal Investigators: D. Marcus, R, Buckner, J, Csernansky J. Morris) was supported by the following grants: P50 AG05681, P01 AG03991, P01 AG026276, R01 AG021910, P20 MH071616, and U24 RR021382. The Australian Imaging, Biomarker & Lifestyle Flagship Study of Ageing (AIBL) was supported by the Science & Industry Endowment Fund. The MIRIAD dataset is made available through the support of the UK Alzheimer's Society (Grant RF116). The original data collection was funded through an unrestricted educational grant from GlaxoSmithKline (Grant 6GKC).
Comparison of different types of features for classification. All the edges are set to 1, the classifier used is GCN. Red: best result, Blue: second best result. The balanced accuracy (BACC) is used to assess the model performance. The results are the average accuracy of 10 repetitions and presented in percentage. All the methods were trained on the AD/CN subjects of the ADNI1 dataset. Value in bold: of one-sided Wilcoxon test comparing with our baseline (in gray) is lower than 0. 05, meaning a significantly superior performance is found compared to the baseline. A comparison using area under curve (AUC) is provided in annexes.
Comparison of different graph edge types. The classifier used is GCN and the input features is and A. Red: best result, Blue: second best result. The balanced accuracy (BACC) is used to assess the model performance. The results are the average accuracy of 10 repetitions and presented in percentage. All the methods were trained on the AD/CN subjects of the ADNI1 dataset. A comparison using area under curve (AUC) is provided in annexes.
Comparison of different classifiers. For graph-based approaches (i. e. , all the approaches except SVM and multi-layer perceptron), the edge based on structure volume difference is used and the input features is and A. Red: best result, Blue: second best result. The balanced accuracy (BACC) is used to assess the model performance. The results are the average accuracy of 10 repetitions and presented in percentage. All the methods were trained on the AD/CN subjects of the ADNI1 dataset. A comparison using area under curve (AUC) is provided in annexes.
Comparison of our method with state-of-the-art methods using published results. Red: best result, Blue: second best result. The balanced accuracy (BACC) is used to assess the model performance. All the methods are trained on the AD/CN subject of the ADNI1 dataset. However, there are many different factors: number of subjects in training/testing sets, selection criteria, etc. A comparison using area under curve (AUC) is provided in annexes.
Comparison of different types of features for classification. All the edges are set to 1, the classifier used is GCN. Red: best result, Blue: second best result. The Area Under the ROC Curve (AUC) is used to assess the model performance. The results are the average accuracy of 10 repetitions and presented in percentage. All the methods were trained on the AD/CN subjects of the ADNI1 dataset. Value in bold: of one-sided Wilcoxon test comparing with baseline (in gray) is lower than 0. 05, meaning a significantly superior performance is found compared to the baseline.
Comparison of different classifiers. For graph-based approaches (i. e. , all the approaches except SVM and multi-layer perceptron), the edge based on structure volume difference is used and the input features is and A. Red: best result, Blue: second best result. The Area Under the ROC Curve (AUC) is used to assess the model performance. The results are the average accuracy of 10 repetitions and presented in percentage. All the methods were trained on the AD/CN subjects of the ADNI1 dataset.
Acknowledgement This work benefited from the support of the project DeepvolBrain of the  French National Research Agency  ( ANR-18-CE45-0013 ). This study was achieved within the context of the  Laboratory of Excellence TRAIL   ANR-10-LABX-57  for the  BigDataBrain  project. Moreover, we thank the  Investments for the future Program IdEx Bordeaux  ( ANR-10-IDEX-03-02 ), the  French Ministry of Education and Research , and the  CNRS  for  DeepMultiBrain  project. Datasets  ADNI1-2  used for this project was funded by the  Alzheimer's Disease Neuroimaging Initiative (ADNI)  ( National Institutes of Health  Grant  U01 AG024904 ) and  DOD ADNI (Department of Defense  award number  W81XWH-12-2-0012 ). ADNI is funded by the  National Institute on Aging , the  National Institute of Biomedical Imaging and Bioengineering , and through generous contributions from the following:  AbbVie, Alzheimer's Association; Alzheimer's Drug Discovery Foundation ;  Araclon Biotech ;  BioClinica, Inc.
We integrated the different datasets using STACAS (Fig EV1A  -C )  (Andreatta & Carmona, 2021)  and performed unsupervised clustering using a graph-based approach with the Seurat package  (Hao et al, 2021) (Fig EV1D). We identified 3 clusters of contaminating B cells, NK cells, and T cells, that we excluded from subsequent analysis  (Fig EV1E and F). Cells from both donors were distributed homogenously (Fig  EV1F ). Unsupervised analysis grouped cells according to time points  (Figs 1A and EV1G). Analysis of differential gene expression between time points showed a temporal regulation of gene expression  (Fig EV1H) , validating the experimental set-up. We identified 14 different clusters in the dataset, most of them with mixed sample origins  (Fig 1B). Clusters 1 and 2 contained almost exclusively cells captured at 3 h, cluster 5 contained cells captured at 9 h, while clusters 11, 12, 13, and 14 displayed a majority of cells from the 48 h time point  (Fig 1B).
To decipher cluster identity, we analyzed the expression of the top differentially expressed genes between clusters (Fig 1C; Dataset EV1) and of canonical genes of monocytes (S100A9, S100A8, VCAN, CD14), mo-Mac (MAFB, CXCL8, THBS1, APOE) and mo-DC (IRF4, PRDM1, CD1B, CD1A; Fig 1D). Clusters 1, 2, 3 and 4 had the highest expression of monocyte genes. Consistent with this, clusters 1, 2, 3, 4, 5, and 6 were enriched for the monocyte signature compared to other clusters (Fig  1E ). Macrophage genes were predominantly expressed in clusters 4, 5, 6, 13, and 14, including APOC1, sterol 27-hydroxylase (CYP27A1), fatty acid binding proteins FABP4 and FABP5, and chemokine CCL22. DC genes were found in clusters 3, 7, 8, 11, and 12, including CD1E and CST3. Of note, cluster 10 contained cells from all time points and expressed genes consistent with a contaminating population of intermediate CD14 + CD16 + monocytes, including FCGR3A (encoding CD16) and AIF1 (Villani et al, 2017).