SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Question 4.2 To what extent does the number of ROIs impact the performance of predictive modeling on graphs?
The second question delves into the influence of varying the number of ROIs on the performance of predictive modeling. The objective is to assess how the granularity of ROIs affects the quality and the performance of the predictive models. We evaluate the use of 100, 400 and 1000 ROIs.
Question 4.3 To what degree does sparsifying brain functional connectome graphs impact the performance of predictive modeling? What threshold yields optimal performance?
Our third question investigates the impact of sparsifying brain functional connectome graphs on the performance of the predictive models. It aims to establish a threshold that leads to optimal model performance in graph machine learning setting. In our exploration, we consider the top  $20\%$ ,  $10\%$ , and  $5\%$  percentile values from the correlation matrices to construct the graph edges.
Question 4.4 Which graph convolution approaches are preferable for the predictive modeling on brain graphs?
Our fourth and final question delves into the exploration of various graph convolution methods, assessing their suitability for predictive modeling on brain graphs. The aim here is not only to identify, but also to recommend the most effective techniques, considering the specific features and intricacies of neuroimaging data. In this endeavor, we have put to test over 12 GNNs, which include two of our own implemented frameworks, to gauge their comparative performance.
By addressing these questions, we aim to set a robust benchmarking framework for graph-based machine learning methods in neuroimaging, providing invaluable insights into their optimal application.
5 Benchmarking Results
In this section, we introduce the baseline models, describe our experimental setup, and present the results from our preliminary exploration study. Following this, we lay out our approach to benchmarking.
5.1 Baselines and Experimental Setup
This section outlines the specifics of our unique, generalized experimental setup designed to evaluate a range of GNN models. We consider 10 well-established GNN models:  $k$  GNN [22], GCN [18], GraphSAGE [19], Unified Message Passing (UniMP)[25], Residual GCN (ResGCN) [21], Graph Isomorphism Network (GIN), Chebyshev Convolution (Cheb) [24], Graph Attention Network (GAT) [23], Simplified GCN (SGC) [55], and General Convolution (General)  $[56]^2$ . We also consider 3-layered Neural Network (NN), 2D Convolutional Neural Network (CNN) and Random Forest (RF) for the comparison.
In our experimental setup, we devise a graph classification architecture comprising three layers of GNNs, followed by a sort pooling aggregator [57]. Sort pooling sorts the node features based on the last channel, selecting only the first  $k$  representations. Subsequently, sort pooling is advanced through two one-dimensional convolution layers, which are then succeeded by a two-layer Multi-Layer Perceptron (MLP). This architecture has been consistently utilized across all GNNs throughout the entire experimental setup. For the dynamic datasets, we utilize our baseline method with five different GNNs. For NN, we utilized 512, 256, and 128 hidden units in each layer, respectively. For the CNN, we utilized a four-layer model with a stride of 2, 64 kernels of size 5, and padding set to 2. This was complemented by three fully connected layers [58]. For the Random Forest (RF) [59], we opted for 100 estimators, leaving the remaining parameters at their Scikit-learn defaults. All of our experiments were carried out on a system equipped with an Intel(R) Xeon(R) Gold 6238R CPU operating at 2.20GHz with 112 cores, 512 GB of RAM, and an NVIDIA A40 GPU with 48GB of memory.
we use PyG implementations and default settings for running all these models
Table 2: Results of the gender classification using three distinct node feature configurations across three settings, evaluated on 10 GNNs. The configurations include CORRELATIONS (CORR), BOLD signals, and a combination of BOLD + CORRELATIONS, evaluated across 100ROIs, 400ROIs, and 1000ROIs. Avg. column indicates the average results across the row and numbers under ROIs indicates average results across each ROI. The blue notation highlights the overall best results for each GNN and red indicates average best performance across each ROI. Average best results are obtained through 1000 ROIs with sparser graphs.
Dataset	k-GNN	GCN	SAGE	UniMP	ResGCN	GIN	Cheb	GAT	SGC	General	Avg.
100ROIs	CORR	65.65	68.98	68.70	68.33	66.06	68.24	63.94	69.49	68.43	64.95	67.30
BOLD	49.58	50.97	51.67	51.30	51.34	55.09	53.19	49.95	51.90	51.11	51.11
CORR+BOLD	52.78	51.02	50.28	50.79	50.60	54.91	49.44	50.37	51.57	51.30	51.36
400ROIs	CORR	72.21	74.10	61.66	68.57	70.09	71.89	58.94	69.35	75.99	73.09	69.56
BOLD	51.16	51.62	53.94	51.39	52.31	55.09	49.07	50.46	53.24	53.94	52.22
CORR+BOLD	51.53	51.90	52.96	51.57	52.36	55.56	50.63	52.13	52.08	52.61	53.33
1000ROIs	CORR	78.80	75.19	71.71	75.14	78.75	77.22	64.77	71.34	73.75	63.13	72.98
BOLD	48.15	46.99	49.31	50.93	47.92	56.48	47.22	50.93	49.31	51.62	49.89
CORR+BOLD	51.30	51.81	51.25	51.11	49.86	54.35	49.66	51.22	51.34	51.37	51.33
Models training: We have carefully carried out the training and evaluation of each dataset in our study. Each dataset was partitioned randomly with  $70\%$  training,  $20\%$  testing, and  $10\%$  for validation. To ensure reproducibility and balance across the datasets, we employed a fixed seed, 123, for the split in a stratified setting. This stratified approach facilitated an equitable distribution of classes in each partition. Each model underwent training for 100 epochs with a learning rate of  $1e^{-5}$  for classification, and for 50 epochs with a learning rate of  $1e^{-3}$  for regression problem. Across all experiments, we set dropout to 0.5, weight decay to  $5e^{-4}$ , and designated 64 hidden dimensions for both the GNN convolution and MLP layers. Furthermore, for loss functions, we utilized cross entropy for classification and mean absolute error for regression problems. All benchmarks and their source codes can be accessed on GitHub<sup>3</sup>. The static benchmark datasets are also available at  $\mathrm{PyG}^4$ .
5.2 Exploratory Experiments and Results
Here we address the questions outlined earlier by conducting a series of experiments including the evaluation of different node feature configurations, the influence of varying numbers of ROIs, the implications of sparsity in brain graphs, and the effectiveness of diverse graph convolution approaches. Each experiment aligns with a question, thereby paving the way for comprehensive analysis and definitive conclusions.
Performance enhancement with correlations as node features: Our first step involves evaluating the interplay between the number of ROIs and the configuration of node features, with an aim to streamline the overall search space. For this purpose, we engage in the gender classification problem using 10 different GNNs. The results of these experiments are presented in Table 2. It is clear that employing correlations as node features consistently enhances the performance across all evaluated numbers of ROIs. However, what caught our attention was the significant variance in the results obtained through correlations and BOLD signals and the number of ROIs. The performance notably declines when correlations and BOLD signals are combined, and the number of ROIs are reduced. This motivates further investigation on how to leverage BOLD signal or perhaps obtain features from the BOLD signals to be used for learning. Furthermore, the performance of different GNNs baselines does not consistently correlate with the number of ROIs or node features.
Performance enhancement through large ROIs and sparse brain graphs: Our analysis extended to evaluating the efficacy of 10 GNNs on gender classification, using a varying number of ROIs and different graph densities. In addition to gender classification, we further incorporated task-state classification problem to strengthen our observations under different settings. For all the experiments, we opted for correlations as node features, a decision driven by the consistent boost they offer in performance from the last experiment. The results are presented in Table 3. An important observation from our findings reveals that larger numbers of ROIs, (1000) demonstrate superior performance in gender classification. Similarly, a significant number of GNNs exhibit improved results with the use of 1000 ROIs for the task-state classification problem. An analysis of the graph densities reveals an
<sup>3</sup>https://github.com/Anwar-Said/NeuroGraph
4https://pytorch-geometric.readthedocs.io/
Table 3: Performance comparison in terms of accuracy of 10 GNNs with different ROIs and varying graph densities on gender and task state classification problems. The blue highlights the overall best results per model in each classification problem.
Dataset	k-GNN	GCN	SAGE	UniMP	ResGCN	GIN	Cheb	GAT	SGC	General
Gender Classification	100ROIs	Sparse	63.33	72.96	69.35	69.72	68.06	69.72	63.70	70.28	70.37	67.22
Medium	65.65	68.98	68.70	68.33	66.06	68.24	63.94	69.49	68.43	64.95
Dense	64.44	68.52	65.00	68.06	63.70	66.39	64.26	69.72	68.43	61.76
400ROIs	Sparse	69.95	77.14	69.86	67.56	71.43	69.4	66.45	72.72	78.25	76.13
Medium	65.65	68.98	68.70	68.33	66.06	68.24	63.94	69.49	68.43	64.95
Dense	71.61	76.13	62.58	61.20	69.77	73.27	61.84	67.83	74.19	72.44
1000ROIs	Sparse	82.13	75.46	77.69	76.67	78.33	75.56	59.07	76.2	76.48	78.89
Medium	78.80	75.19	71.71	75.14	78.75	77.22	71.43	71.34	73.75	63.13
Dense	61.57	73.80	78.86	72.50	78.89	78.70	76.67	71.67	75.25	72.69
Task Classification	100ROIs	Sparse	91.50	91.56	91.43	92.73	92.14	88.31	92.55	92.91	91.40	91.52
Medium	90.91	90.80	91.81	92.75	92.25	88.01	93.06	93.15	91.40	91.22
Dense	90.30	91.15	93.15	93.28	93.02	87.12	93.18	93.08	90.49	89.47
400ROIs	Sparse	93.23	94.21	94.78	94.72	94.61	89.79	94.45	95.2	94.17	93.62
Medium	92.26	93.93	93.89	95.02	94.33	89.44	79.03	94.67	93.39	93.58
Dense	90.64	93.36	95.76	94.48	94.64	88.22	87.24	94.78	93.18	90.84
1000ROIs	Sparse	93.50	93.80	94.09	93.59	94.23	85.14	93.82	94.66	93.2	94.17
Medium	92.65	90.87	94.39	95.79	92.04	85.40	94.88	94.00	91.37	91.87
Dense	93.77	93.12	94.12	94.54	93.59	81.59	92.92	95.35	93.76	93.76
Table 4: Classification results in terms of accuracy on benchmark static datasets constructed with optimal setting. Blue indicates overall best results.
Dataset	NN	CNN	RF	k-GNN	GCN	SAGE	UniMP	ResGCN	GIN	Cheb	GAT	SGC	General	\( \mathbf{{GNN}}{}^{ * } \)
HCP-Task	97.78	95.88	88.98	93.23	94.21	94.78	94.72	94.61	89.79	94.45	95.2	94.17	93.62	98.20
HCP-Gender	86.67	76.39	69.9	82.13	75.46	77.69	76.67	78.33	75.56	59.07	76.20	76.48	78.89	89.07
HCP-Age	44.23	43.38	40.84	42.72	43.66	40.94	43.85	40.00	44.98	41.97	42.25	43.47	41.03	50.23
intriguing trend. For instance, based on the results from Table 3, the ratio of sparse:medium:dense on the gender classification dataset is  $6:0:4$ , while on the task dataset, it stands at  $4:2:4$  for 1000 number of ROIs. Furthermore, the differences in results, especially in cases where sparse graphs exhibit lower performance, are generally small. Recognizing the increased complexities stemming from memory usage, training demands, and the possibility of oversmoothing, we have chosen sparse graphs with the combination of large ROIs, and correlation features in our benchmarking setup.
5.3 Benchmarking with Optimal Settings
Considering the optimal setting obtained through exploring search space presented in the previous section, here we present the experimental setup and benchmarking results on the proposed 10 datasets.
The classification accuracy of all baseline models is detailed in Table 4. It is evident from the results that the GNN* stands out as the leading performer. However, the Neural Network's performance is also notably impressive. Similarly, the results pertaining to the regression problems have been outlined in Table 5. The leading performer on the regression problems is again GNN*. These results distinctly demonstrate that residual connections coupled with message passing play a pivotal role in enhancing performance in brain networks. This synergy arises from the capacity of message passing to glean meaningful representations from highly correlated features. Simultaneously, the inclusion of residual connections empowers the utilization of the input features with the learned representations obtained through message passing. This also underscores the effectiveness of correlation features in influencing the model's performance on brain graphs.
In Table 6, we lay out the classification and regression results obtained on the dynamic datasets. Given the consideration of a basic dynamic baseline and the construction of dynamic datasets using limited dynamic lengths and number of ROIs, the performance does not quite match up to the static datasets. Nonetheless, it's worth noting that UniMP, despite the constraints, consistently demonstrates competitive performance.
In this work, we introduce novel brain connectome benchmark datasets specifically tailored for graph machine learning, representing a promising avenue for addressing various challenges in neuroimaging. The inherent symmetries and complex higher-level patterns found in brain graphs make them well
Table 5: Results for HCP-FI and HCP-WM dataset using mean absolute error (MAE). Blue indicates overall best results.
Dataset	k-GNN	GCN	SAGE	UniMP	ResGCN	GIN	Cheb	GAT	SGC	General	GNN*
HCP-FI	0.284	0.288	0.283	0.287	0.281	6.548	0.278	0.290	0.282	0.283	0.264
HCP-WM	0.818	0.825	0.810	0.812	0.830	1.032	0.789	0.804	0.828	0.819	0.751
Table 6: Models' performance in terms of accuracy and MAE across five dynamic datasets
Dataset	Accuracy ↑	Dataset	MAE ↓
UniMP	k-GNN	GAT	SAGE	General	UniMP	k-GNN	GAT	SAGE	General
DynHCP-Task	89.66	73.03	89.67	90.93	68.84	DynHCP-FI	3.839	3.841	3.861	3.842	3.862
DynHCP-Gender	72.3	68.45	67.13	66.20	62.04	DynHCP-WM	10.589	10.596	10.592	10.597	10.571
DynHCP-Age	44.41	44.25	44.39	40.65	42.99
suited for graph machine learning techniques. To advance this vision, we present NeuroGraph, a comprehensive suite encompassing benchmark datasets and computational tools.
In our comprehensive exploratory study encompassing 35 datasets, we conducted a thorough analysis by running multiple machine learning models. Our key observations are as follows: (1) increasing the number of ROIs or employing large-scale brain graphs leads to improved performance compared to datasets with fewer ROIs, (2) employing a sparser graph setting enhances model performance and (3) while not intuitive, utilizing correlation as node features has significant potential to enhance model performance. Through a range of experiments across various learning objectives, we further highlight that GNNs exhibit superior performance compared to traditional NNs and 2D CNNs. These findings underscore the significant potential of GNNs in achieving improved performance across diverse tasks and underscore their suitability for graph-based Neuroimaging data analysis. Based on these insightful observations, we have developed NeuroGraph, a comprehensive benchmarks specifically designed for graph-based neuroimaging. Additionally, we provide computational tools to explore the design space of graph representation coming from Neuroimaging data, to facilitate the transformation of fMRI data into graph representations and showcase the potential of GNNs in this context. NeuroGraph serves as a valuable resource, offering a road map for researchers interested in leveraging graph-based approaches for fMRI analysis and demonstrating the effective utilization of GNNs in this domain.
This research incorporates contributions from the "Modeling and Model Integration" project of the Vanderbilt Institute for Software Integrated Systems, and by the National Science Foundation under Grant No. 2325417, 2321684, 2239881 and NIH Grant No. RF1MH125931. Work on "Modeling and Model Integration" is supported by Wellcome Leap as part of the Multi-Channel Psych Program. Moreover, the data were provided [in part] by the Human Connectome Project, WU-Minn Consortium (Principal Investigators: David Van Essen and Kamil Ugurbil; 1U54MH091657) funded by the 16 NIH Institutes and Centers that support the NIH Blueprint for Neuroscience Research; and by the McDonnell Center for Systems Neuroscience at Washington University.
[1] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI open, 1:57-81, 2020.
[2] Byung-Hoon Kim, Jong Chul Ye, and Jae-Jin Kim. Learning dynamic graph representation of brain connectome with spatio-temporal attention. Advances in Neural Information Processing Systems, 34:4314-4327, 2021.
[3] Xiaoxiao Li, Yuan Zhou, Nicha Dvornek, Muhan Zhang, Siyuan Gao, Juntang Zhuang, Dustin Scheinost, Lawrence H Staib, Pamela Ventola, and James S Duncan. Braingnn: Interpretable brain graph neural network for fmri analysis. Medical Image Analysis, 74:102233, 2021.
[4] Andrew T Drysdale, Logan Grosenick, Jonathan Downar, Katharine Dunlop, Farrokh Mansouri, Yue Meng, Robert N Fetcho, Benjamin Zebley, Desmond J Oathes, Amit Etkin, et al.
[54] Seyed Mehran Kazemi, Rishab Goel, Kshitij Jain, Ivan Kobyzev, Akshay Sethi, Peter Forsyth, and Pascal Poupart. Representation learning for dynamic graphs: A survey. The Journal of Machine Learning Research, 21(1):2648-2720, 2020.
[55] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning, pages 6861-6871. PMLR, 2019.
[56] Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. Advances in Neural Information Processing Systems, 33:17009-17021, 2020.
[57] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[58] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[59] Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
A Benchmarks Availability and Licensing
The fMRI data utilized in this research was sourced from the Human Connectome Project [1]. The proposed graph-based benchmark datasets can be accessed for download at https://anwar-said.github.io/anwartsaid/neurograph.html. These datasets are provided in PyG5 format, optimized for use with Graph Neural Networks (GNNs). However, they can also be conveniently incorporated into other platforms. Additionally, the associated code for downloading, preprocessing, and benchmarking is open to the public at https://github.com/Anwar-Said/NeuroGraph, complete with comprehensive documentation.
B NeuroGraph and Neuroimaging Data
Neuroimaging, a powerful field of study, enables researchers to delve into the complexities of the human brain by capturing detailed images and measurements. Recent advancements in technology have resulted in an abundance of neuroimaging data, particularly functional magnetic resonance imaging (fMRI), which offers invaluable insights into brain activity. However, understanding and analyzing fMRI data pose several challenges. Firstly, the high dimensionality of fMRI data presents a significant hurdle. Additionally, inherent noise and variability in fMRI signals can obscure underlying neural activity. Complex spatial and temporal dependencies further complicate fMRI data analysis, demanding advanced modeling techniques. Furthermore, the interpretation and analysis of fMRI data can be time-consuming and subjective. The graphical representation of fMRI data offers a plethora of opportunities to tackle these challenges. For instance, network science and graph theoretical approaches provide a diverse range of tools to explore brain regions and their connectivity patterns [2]. Furthermore, the application of graph machine learning techniques, such as GNNs are particularly well-suited for analyzing neuroimaging data and have the potential to provide valuable insights. The provision of graph-based neuroimaging benchmarks and computational tools play a crucial role to enhance the field, which is the main focus of this study.
B.1 fMRI Data Sources
Several initiatives have been undertaken in the past decade to assemble comprehensive fMRI datasets. One notable source is the Human Connectome Project (HCP) dataset [1]. The HCP dataset offers an extensive collection of multimodal neuroimaging data, including resting-state fMRI, task-based fMRI, and structural MRI scans, from a large cohort of healthy individuals. In addition to large neuroimaging datasets curated by institutions or projects, some notable resources are OpenNeuro, OpenfMRI and fcon_1000 $^6$  platforms, which host a diverse range of publicly available fMRI datasets contributed by researchers worldwide [3, 4]. These datasets cover various experimental paradigms (see Table 7), clinical populations, and research domains, providing researchers with a wealth of data for analysis and investigation.
We have chosen to utilize the HCP S1200 dataset from the Brain Connectome as a primary resource for our graph-based benchmarking [1]. This dataset is well-suited for graph-based benchmarking due to its extensive coverage of brain regions and their interconnections. Additionally, the HCP S1200 dataset provides valuable demographic and behavioral information, enabling comprehensive analyses that consider various factors influencing brain connectivity. Its wide availability and standardized processing pipelines further contribute to its suitability for graph-based benchmarking, ensuring consistency and comparability across studies. Thus, the HCP S1200 dataset from the Brain Connectome represents a robust choice for conducting graph-based benchmarking studies in the field of neuroimaging.
B.2 Reading HCP Dataset
Storing and reading fMRI datasets presents a formidable challenge due to their substantial storage requirements, necessitating significant disk space allocation, e.g., each subject of HCP S1200 requires 1.1 GB of space on disk. Moreover, the preprocessing of fMRI data calls for tools that are not only user-friendly but also highly efficient. Fortunately, the Human Connectome Database (HCP) offers
$^{5}$ https://pyg.org/  
 $^{6}$ http://fcon_1000.projects.nitrc.org/
Table 7: Description of fMRI paradigms in HCP Young Adult dataset.
Dataset	Volumes per run (TR)	Run duration (min)	Duration of task blocks (sec)	Description
Rest	1200	14 : 24	-	no stimuli
Working Memory	405	5 : 01	25	0-back, 2-back
Gambling	253	3 : 12	28	win, loss
Motor	284	3 : 34	12	various body parts
Language	316	3 : 57	approx. 30	story, math
Relational Processing	232	2 : 56	16	relational, control
Social Cognition	274	3 : 27	23	interaction, no interaction
Emotion Processing	176	2 : 16	18	face, shape
Table 8: fMRI scans required disk storage. The storage information is obtained from Human Connectome Project website.
Task	Storage (GB)
Rest	1260.95
Working Memory	527.70
Gambling	387.38
Motor	415.81
Language	426.72
Relational	343.40
Social	386.76
Emotion	295.91
an AWS instance (s3 bucket) that allows for seamless data crawling. NeuroGraph, with its implementation utilizing the boto3 Python package, provides an efficient solution for crawling the dataset. Boto3, a widely used Python package, enables seamless interaction with AWS services, facilitating efficient data retrieval and preprocessing in the NeuroGraph framework. Our implementation offers users the flexibility to either store the datasets or preprocess them on the fly if storage space is limited (see Table 8 for disk storage). To access the HCP data, users are required to obtain credentials from  $\mathrm{HCP}^7$  and provide them to NeuroGraph. Moreover, NeuroGraph also provides a Python class for preprocessing data from the local storage.