SlowGuess/NeurIPS_2023_OCR huggingface dataset:
brain features. Clustering-based parcellations partition the brain by computing the similarity in the BOLD signals between voxels and performing clustering on the voxels based on the similarities. Each subject directly generates its own set of ROIs.  $k$ -means and Ward clustering are the two methods used in the domain. We included them and set the number of ROIs as 100, which is close to the predefined value in atlas-based PMs. Our datasheet in the Supplementary details the file format and output files. Figure 2 shows examples of the ROIs extracted by these methods.
4.5 Brain Network Extraction
The BOLD signals from parcellated ROIs are used to compute the connectivity matrix of the brain network. The matrix captures functional relationships between parcellated regions and reflects their co-activations. Connectivity matrix is extracted using a built-in function of nilearn [2] called ConnectivityMeasures [55]. It takes 2 inputs, BOLD signals of the brain regions and the connectivity metric (e.g., correlation). The former is obtained by averaging the BOLD signals of all the voxels lying in the same ROI. Figure 1E shows an example of a connectivity matrix. Among existing connectivity measures (correlation, covariance, partial correlation, etc.), correlation is used by a majority of graph-based neuroscience research [34, 37, 44]. Thus, in our released collection, we select correlation as the connectivity measure and make the fully weighted matrices available without performing any thresholding on the correlation values. For the practitioners who are interested in alternative measures, we have uploaded preprocessed images (excluding the Mātai dataset) and our matrix generation code as part of our collection, which enables the researchers to generate their own matrices based on their desired measure. The datasheet in our Supplementary details our outputs and file formats. The connectivity matrix can be represented equivalently as a glass brain connectome. An example connectome plot on the AAL atlas is shown in the right bottom corner of Figure 2. The connectome plot shows top  $1\%$  of brain functional connectivities for visualization.
The brain network constructed by this pipeline has the set of ROIs as nodes and their co-activations (measured as Correlation) as edges. Each brain network is represented by a weighted adjacency matrix and a feature matrix. The former is the connectivity matrix extracted and the latter stores the BOLD signals of ROIs. Note that we include the BOLD signals as node feature matrices in our data release for data completeness: some analytical methods on brain networks may want to utilize such information directly or further extract features from them.
5 Data Quality Assessment and Baseline Comparisons
To assess if the graph representation maintains the data quality from neuroimages, we test our brain network data collection on two tasks, graph classification and graph ordinal regression. The aim of graph classification is to distinguish patients from healthy subjects for datasets with two classes (ABIDE, Mātai, TaoWu, and Neurocon); and to distinguish subjects among different disease stages and healthy groups by treating each class independently when it comes to multi-class datasets (ADNI and PPMI). In contrast, graph ordinal regression is only applied to multi-class datasets by differentiating stages based on the disease severity.
For classification, we test on both conventional ML models typically used in neuroscience and representative graph ML models. For conventional models, we follow the common practice in the domain [10, 30] to vectorize the input connectivity matrices by flattening them. For ordinal regression, we test on the classic logistic ordinal regression model. To further validate the quality of our data, we also test on a recent graph analysis model on functional networks for classification [34]. Finally, we also include a sensitivity study on the number of ROIs and the training set size. All these models only utilize the connectivity matrices for assessing the quality of our brain network construction. The data is split to 8:1:1 for training, validation, and testing with 10-fold cross-validation performed.
5.1 Results on Classification
We select 6 conventional machine learning models for this study: Logistic Regression (LR), Gaussian Naive Bayes (NB), Support Vector Machine Classifier (SVC),  $k$ -Nearest Neighbours (kNN), Random Forest (RF), and Multi-Layer Perceptron (MLP) [29]. We used the implementation from the scikit-learn library [45, 11] with Grid Search for model selection. We also select 6 typical graph-based machine learning models, including GCN [33], GraphSAGE [28], GIN [53], GAT [50], GatedGCN
Table 3: Classification accuracy (mean±standard deviation) on conventional ML methods. The best result at each parcellation is highlighted in bold. The best result in each dataset is underlined.
AAL	ABIDE	Ward	AAL	HO	ADNI
HO	Schaefer	k-means	Ward	Schaefer	k-means	Ward
LR	63.8±3.0	63.6±4.2	64.8±3.7	48.4±4.4	51.1±5.8	64.1±1.8	61.9±2.1	62.0±4.2	58.6±2.6	60.4±0.8
NB	60.4±5.5	59.2±5.4	61.6±3.6	51.6±3.7	54.2±5.0	53.6±4.4	52.7±6.7	48.9±3.3	32.5±3.6	55.0±2.6
SVC	65.7±3.3	62.9±3.6	64.4±5.1	49.3±4.1	53.2±4.9	63.4±1.9	66.2±2.9	61.5±5.0	61.8±0.3	61.8±0.3
kNN	58.1±5.3	56.1±4.5	59.7±3.5	50.7±2.9	49.3±2.3	60.6±2.1	62.9±3.8	63.1±2.4	31.4±22.9	59.5±3.9
RF	62.4±2.8	60.5±3.1	62.6±2.6	49.5±5.4	51.3±2.9	61.9±2.1	61.7±2.8	62.1±1.8	61.5±0.4	61.6±0.5
MLP	54.4±1.2	62.2±4.4	49.6±4.4	63.6±3.9	51.7±6.3	62.4±1.8	62.9±2.2	46.7±5.3	61.7±0.2	48.6±5.9
PPMI	Mātai
AAL	HO	Schaefer	k-means	Ward	AAL	HO	Schaefer	k-means	Ward
LR	56.0±7.8	56.0±9.2	56.5±6.8	60.3±7.4	60.3±8.8	66.7±21.1	58.3±13.4	60.0±20.0	55.0±18.3	58.3±20.1
NB	58.4±5.2	52.6±8.6	57.5±7.6	58.4±5.7	62.2±7.4	68.3±22.9	45.0±21.2	56.7±18.6	55.0±15.0	53.3±18.0
SVC	64.1±5.7	63.6±6.4	63.2±8.6	60.8±7.5	60.8±8.9	65.0±20.3	58.3±13.4	56.7±17.0	58.3±17.1	58.3±17.1
kNN	51.2±9.1	55.5±7.3	53.5±7.6	60.3±6.6	60.8±8.9	61.7±16.8	50.0±21.1	58.3±18.7	58.3±17.1	48.3±13.8
RF	61.6±8.8	62.6±9.9	62.6±8.1	58.4±9.2	61.7±7.3	60.0±18.6	53.3±20.0	63.3±24.5	48.3±17.4	53.3±20.8
MLP	57.8±10.4	62.2±8.0	57.9±5.0	57.4±9.2	52.6±5.7	45.0±19.8	48.3±21.7	53.3±18.0	63.3±18.0	50.0±22.4
TaoWu	Neurocon
AAL	HO	Schaefer	k-means	Ward	AAL	HO	Schaefer	k-means	Ward
LR	77.5±17.5	72.5±23.6	75.0±15.0	50.0±15.8	52.5±17.5	68.5±25.1	61.5±26.3	65.5±24.9	58.0±22.9	63.0±25.9
NB	65.0±15.8	60.0±24.5	62.5±23.6	37.5±30.1	50.0±19.4	58.0±22.9	59.0±28.4	63.5±23.3	48.0±24.2	55.5±22.3
SVC	67.5±17.5	67.5±17.5	65.0±15.0	52.5±7.5	50.0±19.4	63.0±25.9	68.5±25.9	69.0±25.9	63.0±25.9	63.0±25.9
kNN	55.0±21.8	60.0±16.6	65.0±16.6	42.5±16.0	50.0±0.0	65.0±25.5	49.0±27.6	55.5±24.9	63.0±25.9	63.0±25.9
RF	65.0±11.3	60.0±22.9	57.5±25.1	40.0±32.0	47.5±20.8	55.5±24.9	61.0±29.8	58.5±22.4	58.0±22.9	63.0±25.9
MLP	60.0±20.0	67.5±16.0	42.5±22.5	57.5±27.5	45.0±21.8	61.0±11.8	67.5±16.0	63.5±11.8	63.5±11.8	63.5±11.8
Table 4: Classification accuracy (mean±standard deviation) on graph ML methods and BOLD time series based methods with Schaefer parcellation. The best result in each dataset is in bold, with those underlined indicating superior performance to conventional ML methods.
ABIDE	ADNI	PPMI	Mātai	TaoWu	Neurocon
GCN	61.0±2.8	61.6±0.6	54.0±9.1	56.7±18.6	60.0±29.2	59.0±20.7
GraphSAGE	63.1±3.1	61.2±1.7	55.0±12.9	61.7±10.7	60.0±33.9	68.5±15.2
GIN	57.0±3.9	61.9±0.4	57.9±8.1	48.3±13.8	65.0±20.0	68.5±15.2
GAT	60.9±5.0	61.3±1.3	55.0±8.0	66.7±18.3	67.5±22.5	54.0±15.6
GatedGCN	63.6±4.7	62.1±4.5	52.6±11.5	58.3±8.3	65.0±22.9	69.0±25.5
BrainNetCNN	65.8±2.5	61.1±2.9	57.3±10.3	61.7±13.3	65.0±27.8	66.0±22.5
GRU	53.6±1.4	61.8±0.3	54.1±2.2	58.3±8.3	50.0±0.0	63.8±1.5
1D-CNN	55.5±3.0	50.2±4.0	55.0±11.5	61.7±13.0	45.0±15.0	58.5±11.2
[12], and BrainNetCNN [32]. The implementations mainly follow those in [18]. To compare the effectiveness of brain networks with that of BOLD signals in disease classification, we test on two methods based on BOLD signals: GRU [15] and 1D-CNN [35].
Table 3 reports the classification accuracy of conventional ML methods on the datasets using different parcellation methods. In most cases, the overall accuracy falls within the range of  $60\%$  to  $70\%$ , which is consistent with the results reported in previous studies [34, 10, 46, 37, 30], despite some differences in data filtering and preprocessing. With respect to parcellation methods, we observe that atlas-based parcellation methods in general outperform clustering-based methods. The best performance on each dataset always occurs when AAL, HO or Schaefer is applied. Particularly, AAL achieves the best performance in 4 out of 6 datasets. The inferiority of clustering-based parcellation is likely due to the effects from some randomness in the clustering process, e.g., initial centroid selection in  $k$ -means. Among different learning models, LR and SVC perform better in most cases. This also explains why they are commonly used as baseline models in the domain [34, 37].
Table 4 reports the classification results of graph-base ML methods and BOLD time series based methods on the datasets with the Schaefer parcellation. No single graph ML method can consistently dominate across datasets. In 3 out of 6 datasets, the best graph ML method outperforms the best of conventional ML methods, which shows the potential of graph-based models. Compared with both conventional and graph ML models, the model performance based on BOLD signals is significantly worse. This verifies the effectiveness/usefulness of brain networks in the task of disease classification.
Table 5: Accuracy (mean±standard deviation) on logistic ordinal regression (LOR) vs logistic regression (LR) on ADNI and PPMI. The best result at each parcellation is highlighted in bold.
AAL	ADNI	Ward	AAL	PPMI
HO	Schaefer	k-means	Schaefer	k-means	Ward
LR	64.1±1.8	61.9±2.1	62.0±4.2	58.6±2.6	60.4±0.8	56.0±7.8	56.0±9.2	56.5±6.8	60.3±7.4	60.3±8.8
LOR	65.6±2.2	62.1±3.0	62.8±2.2	58.5±5.5	60.2±3.3	54.1±13.0	57.5±8.1	55.6±8.9	59.4±8.5	60.8±9.1
Table 6: Results of CS-P1 [34] on the ABIDE dataset used in [34] and on our ABIDE dataset
Subgroup	# Graphs	ABIDE used in [34]	# Graphs	Our ABIDE
Accuracy in [34]	Reproduced Accuracy	Reproduced	Accuracy
Method 1	Method 1	Method 2	Method 1	Method 2
Adolescents	237	72.0±7.0	71.4±3.7	64.9±7.8	259	72.2±5.5	68.3±3.1
Children	101	86.0±7.0	83.0±9.6	69.5±8.6	96	80.1±2.6	67.1±14.0
Eyesclosed	294	71.0±3.0	68.7±4.6	60.9±3.7	312	68.6±3.5	64.8±6.2
Male	838	63.0±1.0	63.5±2.3	60.4±5.1	873	65.3±2.1	62.7±4.0
The accuracy scores in Tables 3 and 4 might appear relatively modest, raising concerns about the clinical applicability of solving the classification problem. It's important to note that disease classification through brain networks is a nascent and evolving research domain. Many of the ML methods tested are general-purpose and not specifically designed for brain networks. Our release of fully weighted brain network matrices in this collection is anticipated to stimulate and advance research in this field. This initiative is expected to foster the development of tailored learning models for brain networks, yielding clinically relevant outcomes. The current results can serve as foundational benchmarks for future research endeavors.
5.2 Results on Ordinal Regression
With the two multi-class datasets ADNI and PPMI, we study the problem of ordinal regression, in which the classes are ordered by the severity of the corresponding disease. We select the classic logistic regression model for this experiment and use the implementation in the scikit-learn library. Table 5 reports the results of the logistic ordinal regression (LOR) versus the logistic regression (LR). The results show that LOR does not exhibit superiority to LR. One potential reason may be its sensitivity to the class imbalance problem: ADNI and PPMI have very skewed class distribution as shown in Table II in the supplementary. With the availability of the datasets, researchers could further inspect this issue and design better ordinal regression solutions to handle them.
5.3 Data Quality Study on ABIDE Dataset with a Graph Analysis Approach for Classification
To evaluate the quality of our data collection, we performed a recent graph-based functional analysis approach [34] to both an existing brain network dataset derived from ABIDE and our processed brain connectivity networks of the ABIDE for disease classification. The approach in [34] (model version of CS-P1) first computes two summary connectivity matrices  $C$  and  $P$ , respectively from the Control group and the Patient group. It then extracts the dense contrast subgraphs from  $C$  and  $P$  and uses them as features for classification. Essentially, a dense contrast subgraph refers to a subset of nodes whose induced subgraph is dense in  $C$  and sparse in  $P$  (namely contrast), or vice versa. It can be considered an Optimal Quasi-Clique (OQC) problem [48] and solved by the DENSDP approach [13]. To make the classification fairer, we list two methods of finding the contrast dense subgraph for CS-P1 [34]. Method 1 uses the whole dataset (all the subjects in both training and test sets) to extract the contrast dense subgraph. The results in [34] used this method. Method 2 uses only the training set to extract the contrast dense subgraph.
We first applied the two methods of CS-P1 to the functional networks of different subgroups of the ABIDE dataset: both the code and data were provided by the authors of [34]. The results using Method 1 (left part of Table 4) are generally comparable with the results reported in [34]: the reproduced accuracy is  $3\%$  lower than the claimed accuracy in the subgroup of "Children", and  $2\%$  lower in the subgroup of "Eyesclosed"; the results on the other two subgroups are consistent. The accuracy results using Method 2 are much lower than the results using Method 1.
We then applied CS-P1 (the same code) on our functional networks of subgroups of ABIDE dataset. Table 6 (right) shows the result. Note that the subjects in different subgroups may overlap. Due
Table 7: Classification accuracy (mean±standard deviation) when tuning #ROIs in Schaefer parcellation. The best result in each method is in bold and the best result in each dataset is underlined.
ABIDE	TaoWu
100	200	500	1000	100	200	500	1000
LR	64.8±3.7	67.9±3.8	67.4±3.8	67.6±4.7	75.0±15.0	57.5±22.5	52.5±23.6	52.5±17.5
NB	61.6±3.6	62.4±3.6	61.9±3.0	60.5±3.5	62.5±23.6	62.5±23.1	60.0±20.0	60.0±20.0
SVC	64.4±5.1	63.7±4.4	62.4±3.8	63.1±4.0	65.0±15.0	57.5±22.5	55.0±24.5	50.0±19.4
kNN	59.7±3.5	59.5±5.1	58.0±3.8	55.5±5.8	65.0±16.6	52.5±23.6	50.0±25.0	50.0±25.0
RF	62.6±2.6	62.4±3.2	61.6±3.2	62.1±5.3	57.5±25.1	55.0±26.9	52.5±26.1	45.0±18.7
Figure 3: Test accuracy on ABIDE and ADNI with Schaefer when tuning training set size.
to different data quality filtering procedures, there is a small difference in the number of subjects in each subgroup. For both Method 1 and Method 2, the classification accuracy on our datasets is marginally better than those on the datasets provided by [34] in all subgroups except for "Children". The difference in performance is likely due to the slight difference in the number of subjects in the two datasets. The results validate that the quality of the brain networks generated by our pipeline is up to the standard of those used in other studies.
5.4 Sensitivity Study on Number of ROIs and Training Set Size
To assess the effect of the number of ROIs to model performance, we tune the number of ROIs on the Schaefer parcellation from \{100, 200, 500, 1000\} on the classification task. The results are reported in Table 7. Each model performs the best when the number of ROIs is set at 100 or 200.
We also test on different training set sizes by tuning  $k$  in  $k$ -fold CV and report the results in Figure 3. The performance of each method improves with the increase in the training set size (larger  $k$ ), which confirms the necessity of having datasets at this scale.
6 Conclusions, Limitations, and Future Directions
We release a functional brain network collection to the public at a large scale. The collection originates from 6 raw MRI image sources, covers 4 brain conditions, and totals to 2,702 subjects. Working with domain experts, we come up with a unified pipeline that converts raw fMRI and T1w images to brain networks. We tested the collection on 12 ML models and a recent graph analysis model to demonstrate that the data quality is not compromised while at the same time providing the results as domain baselines. We hope that the release of this collection of brain networks, together with the complete code of the processing pipeline, will promote both the development of graph-based models and the clinical advancement in the diagnosis and early intervention of neurodegenerative diseases.
We identify two limitations of this study: (1) Though the current brain network collection contains 2,702 subjects, it may still be considered small when applied to certain machine learning models (e.g., deep learning models). Due to the high computational cost, we are not able to produce a larger collection for now but will make continuous efforts in enriching the collection. (2) The current collection focuses on extracting brain networks from a single modality, that is, the functional MRI images. Recently, neuroimaging data lean towards the direction of combining different modalities such as diffusion and functional MRI for further integrated explorations of brain functionalities. The fusion of functional MRI and diffusion MRI can encode information regarding integrated activities of the human brain. In the future, we plan to work on the processing pipeline of diffusion MRI images to make multi-modal brain networks publicly accessible to the research community.
This research is supported by the MBIE Catalyst Strategic Fund UOAX2001 under the NZ-Singapore Data Science Research Programme, New Zealand, and the National Research Foundation, Singapore under its Industry Alignment Fund – Pre-positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
Neurocon data used in this article was obtained from the NEUROCON project (84/2012), financed by UEFISCDI.
ADNI data collection used in this article was funded by the ADNI (National Institutes of Health Grant U01 AG024904) and DOD ADNI (Department of Defense award number W81XWH-12-2-0012). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: AbbVie, Alzheimer's Association; Alzheimer's Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research & Development, LLC.; Johnson & Johnson Pharmaceutical Research & Development LLC.; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition Therapeutics. The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health (www.f NIH.org). The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer's Therapeutic Research Institute at the University of Southern California. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern California.
Mātai data collection was supported by Kānoa, New Zealand; the Hugh Green Foundation; an anonymous donation; and the MBIE Catalyst Strategic Fund UOAX2001 under the NZ-Singapore Data Science Research Programme, New Zealand. We are grateful to the Tairāwhiti Gisborne community and Mātai Ngā Māngai Maori for their guidance. We would like to sincerely thank our research participants for dedicating their time toward and support for this study. We would like to acknowledge the support from the Mātai Medical Research Institute imaging team, including Paul Condron, Taylor Emsden, Dr Patrick McHugh, Leigh Potter, Dr Samantha Holdsworth, Davidson Taylor, Dr Maryam Tayebi, Dr Daniel Cornfeld, and to the Mātai interns for assisting with the data collection and related community outreach and education activities. We are also grateful for the support of Dr Jerome Maller from GE Healthcare for his guidance on the imaging protocol.
PPMI data collection used in this article were obtained from the PPMI database (https://www.ppmi-info.org/accessdata-specimens/download-data). PPMI – a public-private partnership – is funded by The Michael J. Fox Foundation for Parkinson's Research and funding partners, including 4D Pharma, AbbVie Inc., AcureX Therapeutics, Allergan, Amathus Therapeutics, Aligning Science Across Parkinson's (ASAP), Avid Radiopharmaceuticals, Bial Biotech, Biogen, BioLegend, Bristol Myers Squibb, Calico Life Sciences LLC, Celgene Corporation, DaCapo Brainscience, Denali Therapeutics, The Edmond J. Safra Foundation, Eli Lilly and Company, GE Healthcare, GlaxoSmithKline, Golub Capital, Handl Therapeutics, Insitro, Janssen Pharmaceuticals, Lundbeck, Merck & Co., Inc, Meso Scale Diagnostics, LLC, Neurocrine Biosciences, Pfizer Inc., Piramal Imaging, Prevail Therapeutics, F. Hoffmann-La Roche Ltd and its affiliated company Genentech Inc., Sanofi Genzyme, Servier, Takeda Pharmaceutical Company, Teva Neuroscience, Inc., UCB, Vanquia Bio, Verily Life Sciences, Voyager Therapeutics, Inc., Yumanity Therapeutics, Inc.. For up-to-date information on the study, visit www.ppmi-info.org.
[1] A repository of benchmark graph datasets for graph classification. https://github.com/GRAND-Lab/graph_datasets.
[2] A. Abraham, F. Pedregosa, M. Eickenberg, P. Gervais, A. Mueller, J. Kossaifi, A. Gramfort, B. Thirion, and G. Varouquaux. Machine learning for neuroimaging with scikit-learn. Frontiers