SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anwar Said
anwar.said@vanderbilt.edu
Roza G. Bayrak
roza.g.bayrak@vanderbilt.edu
Tyler Derr
tyler.derr@vanderbilt.edu
Mudassir Shabbir
Information Technology University
mudassir.shabbir@itu.edu.pk
Daniel Moyer
daniel.moyer@vanderbilt.edu
Catie Chang
catie.chang@vanderbilt.edu
Xenofon Koutsoukos
xenofon.koutsoukos@vanderbilt.edu
Machine learning provides a valuable tool for analyzing high-dimensional functional neuroimaging data, and is proving effective in predicting various neurological conditions, psychiatric disorders, and cognitive patterns. In functional magnetic resonance imaging (MRI) research, interactions between brain regions are commonly modeled using graph-based representations. The potency of graph machine learning methods has been established across myriad domains, marking a transformative step in data interpretation and predictive modeling. Yet, despite their promise, the transposition of these techniques to the neuroimaging domain has been challenging due to the expansive number of potential preprocessing pipelines and the large parameter search space for graph-based dataset construction. In this paper, we introduce NeuroGraph<sup>1</sup>, a collection of graph-based neuroimaging datasets, and demonstrated its utility for predicting multiple categories of behavioral and cognitive traits. We delve deeply into the dataset generation search space by crafting 35 datasets that encompass static and dynamic brain connectivity, running in excess of 15 baseline methods for benchmarking. Additionally, we provide generic frameworks for learning on both static and dynamic graphs. Our extensive experiments lead to several key observations. Notably, using correlation vectors as node features, incorporating larger numbers of regions of interest, and employing sparser graphs lead to improved performance. To foster further advancements in graph-based data driven neuroimaging analysis, we offer a comprehensive open-source Python package that includes the benchmark datasets, baseline implementations, model training, and standard evaluation.
<https://anwar-said.github.io/anwarsaid/neurograph.html
Table 1: Dataset statistics.  $|G|$  denotes the number of graphs,  $|N|_{avg}$  and  $|E|_{avg}$  denote the average number of nodes and edges,  $d$  indicates the degree, and  $K$  signifies the global clustering coefficient.
Dataset	Statistics	Node Feat. (dim)	#Classes	Prediction Task
|G|	|N|avg	|E|avg	dmax	davg	Kavg
Static	HCP-Task	7443	400	7029.18	153	17.572	0.410	400	7	Graph Classification
HCP-Gender	1078	1000	45578.61	413	45.579	0.466	1000	2	Graph Classification
HCP-Age	1065	1000	45588.40	413	45.588	0.466	1000	3	Graph Classification
HCP-FI	1071	1000	45573.67	413	45.574	0.466	1000	-	Graph Regression
HCP-WM	1078	1000	45578.61	413	45.579	0.466	1000	-	Graph Regression
Dynamic	DynHCP-Task	7443	100	843.04	57	8.430	0.427	100	7	Graph Classification
DynHCP-Gender	1080	100	874.88	53	8.749	0.439	100	2	Graph Classification
DynHCP-Age	1067	100	875.42	53	8.754	0.439	100	3	Graph Classification
DynHCP-FI	1073	100	874.82	53	8.748	0.438	100	-	Graph Regression
DynHCP-WM	1080	100	874.88	53	8.749	0.439	100	-	Graph Regression
Graph Neural Networks (GNNs) have demonstrated remarkable efficacy in a variety of domains including recommendations, forecasting, and biomedical data analysis [1, 2, 3]. In human neuroimaging research, GNNs have proven valuable in capturing the complex connectivity patterns within brain functional networks but also enhance the modeling and analysis with other relevant informative features [4, 3]. For instance, examining synchronized fluctuations of functional magnetic resonance imaging (fMRI) signals provides a useful means of measuring functional network connectivity [5].
Neuroimaging and Graph Machine Learning (GML) are two rapidly evolving fields with immense potential for mutual collaboration. However, a significant challenge lies in bridging the gap between these domains and enabling seamless integration of neuroimaging data into state-of-the-art GML approaches [6]. This gap is primarily attributed to the expansive number of potential fMRI data preprocessing workflows, the absence of an intuitive tool to generate fMRI graph representations for graph-based learning approaches, and the knowledge gap between the fields of neuroimaging and advanced graph machine learning [2]. To address these challenges, the main objectives of this study are, first, a careful exploration of graph-based dataset generation with the goal of formulating a roadmap for graph-based representations of fMRI data. Second, we conduct a rigorous evaluation of graph machine learning methodologies, with a special emphasis on GNNs, examining their efficacy when applied to diverse fMRI data configurations.
The human brain, a complex network of interconnected regions, can be represented as a graph, wherein nodes correspond to contiguous segments known as Regions of Interest (ROIs), and edges represent their relationships [7, 8]. Features of the functional connectome, such as correlations between the BOLD (Blood Oxygen Level Dependent) signals between different brain regions, typically employed for downstream machine learning tasks [6, 9], can be re-envisioned as node features within attributed graph representations. These representations pave the way for a rich assortment of graph-based data representations, wherein GNNs are exceptionally well-suited [10]. Yet, the vast potential offered by the intersection of fMRI datasets and GNNs remains untapped, due primarily to the expansive search space for data generation and the multifaceted nature of hyperparameters. In this study, we pioneer a rigorous exploration and benchmarking for GNNs, with the following primary contributions:
- We introduce NeuroGraph, a collection of static and dynamic brain connectome datasets tailored for benchmarking GNNs in classification and regression tasks including gender, age, task classification, and prediction of fluid intelligence and working memory scores. This enables an extensive exploration of brain connectivity and its associations with various cognitive, behavioral, and demographic variables. Details of the proposed datasets are provided in Table 1.
- We perform an extensive exploratory study in search of optimal graph-based data representations for neuroimaging data, implementing 15 baseline models on 35 different datasets. Additionally, we provide detailed benchmarking for the datasets we propose.
By offering NeuroGraph, we create an essential bridge between the neuroimaging and graph machine learning communities. Researchers in the neuroimaging field can more readily tap into the power of cutting-edge GNNs. Specifically, our datasets generation pipeline may guide researchers toward effectively transforming neuroimaging data into a unified graph representation suitable for graph machine learning. This integration facilitates the adoption of state-of-the-art graph-based techniques, unlocking new insights and accelerating discoveries in the neuroimaging field.
While functional brain connectomes have long been recognized as a rich source of information in neuroscience and neuroinformatics [11, 12], their value has become increasingly evident in recent years [13]. Propelled by growth in data availability and methodological breakthroughs, ML has shown remarkable efficacy on tasks such as decoding of cognitive processes [14, 15] and diagnosing mental health disorders [16, 17]. Simultaneously, there has been increased utilization of static and dynamic graph representation learning methods for brain analytics, which we briefly summarize in this section.
Static graph representations: GNNs have significantly evolved as a major field of exploration, offering an intuitive approach in learning from graph-structured data [18, 19, 20, 21, 22]. In a static setting, where individual data points are represented by single graphs, a variety of methods have been introduced [23, 21, 24, 22, 25, 26]. Recent studies have demonstrated the effectiveness of various approaches when applied to functional connectome data, which can be represented as different types of graphs, including weighted graphs [3, 27, 28], and attributed graphs [2, 29], among others. In benchmarking setup, BrainGB [27] stands out as a notable advancement, offering a unified framework for brain network analysis utilizing GNNs. Likewise, BrainGNN [3] introduces a specialized GNN architecture tailored for the discovery of neurological biomarkers from fMRI data. By leveraging the structured nature of the data and incorporating local information, GNNs not only enable learning from the functional connectivity patterns but also enhance modeling and analysis with other relevant informative features [9, 30, 22, 1, 31].
Dynamic graph representations: The field of learning dynamic graph representations in a graph machine learning setting remains relatively unexplored, especially in the realm of brain imaging [32]. In neuroimaging, dynamic graphs are constructed to capture the time-varying interactions and connectivity patterns in the brain [2, 33, 34]. While GML methods have not been commonly employed in this domain, recent years have seen the introduction of methods that yield impressive results on dynamic brain graphs[2, 35, 36, 37, 38]. Specifically, Kim et al. [2] have made significant strides by introducing dynamic GNNs tailored specifically for brain graphs. These methods have showcased the potential of effectively capturing and analyzing the dynamic nature of brain connectivity, opening up new avenues for advancements in our understanding of brain function and neurological processes.
3 NeuroGraph
The design space for graph construction from functional connectome is vast, since a variety of methods can be employed to generate various forms of graphs, such as simple undirected graphs [2], weighted graphs[3], attributed graphs [27], and minimum spanning trees [39, 40], among others. Thoroughly navigating this extensive design space and evaluating all potential parameter combinations is a challenging task. While recent efforts have been undertaken to leverage GNNs for predictive modeling on neuroimaging data, a consensus has yet to be reached regarding the preprocessing pipeline and hyper-parameter configurations best suited for generating expressive graph-based neuroimaging datasets [2, 3, 41, 28, 42]. In addition, although there are a multitude of GNNs models, no benchmark datasets have been created to evaluate GML approaches on brain connectome data. To fill this gap and provide a common ground between neuroimaging and GML communities, we use publicly available datasets and only minimally preprocess the data using standard fMRI preprocessing steps. We provide an illustration of the overall NeuroGraph preprocessing pipeline in Figure 1.
3.1 From fMRI to Graph Representations
fMRI data is typically represented in four dimensions, where the blood-oxygen level-dependent (BOLD) signal is captured over time in a series of 3-dimensional volumes. These volumes display the intensity of the BOLD signal for different spatial locations in the brain. However, since brain activity tends to exhibit strong spatial correlations, the BOLD signal is often summarized into a collection of special functional units, brain parcels. These units represent regions of interest (ROIs) whose constituent "voxels" (a smallest three dimensional resolution) exhibit temporally correlated activity.
The Human Connectome Project (HCP) [43] is a publicly available rich neuroimaging dataset containing not only imaging data but also a battery of behavioral and cognitive data. We select this dataset for benchmarking and utilize the established group level Schaefer [44] atlases to represent the
Figure 1: An illustration of the preprocessing pipeline, demonstrating the transition from fMRI data to the construction of both static and dynamic graphs.
measured BOLD signal. These atlases provide a parcellation of the cerebral cortex into hierarchically organized regions at multiple granularities (resolutions).
We use resting-state and seven task fMRI paradigms from the HCP 1200 dataset. All fMRI scans underwent the HCP minimal preprocessing pipeline [45]. We further regressed out six rigid-body head motion parameters and their derivatives, as well as the low-order trends, from the minimally preprocessed data. The mean fMRI time series was extracted from all voxels within each ROI for different parcellation schemes. Individual (subject-wise) ROI time-series signals were temporally normalized to zero mean and unit variance. In summary, our proposed fMRI preprocessing pipeline encompasses five steps: a) brain parcellation, removal of b) scanner drifts and motion artifacts, c) subject-level signal normalization, d) calculation of correlation matrices and finally, e) construction of static and dynamic brain graphs. For further in-depth details, we refer the reader to Appendix B.3.
Our study of these datasets encompasses two distinct modes of analysis: static and dynamic graph construction. We apply different GNNs to both types and perform benchmarking in five unique tasks. In the static graph construction, we investigate multiple parameters to build the graphs from the raw data, taking into consideration variations in node features, the number of nodes or regions of interest (ROIs), and the density of the graph. For node features, we take into account correlations, time-series signals, or a blend of both. For the number of nodes provided by [44] (i.e., ROIs), we examine three different resolutions: 100, 400, and 1000 nodes. As for graph density, we consider sparse, medium, and dense configurations. For the sparse setup, we choose the top  $5\%$  of values from the correlation matrix for edge selection, whereas for the medium and dense setups, we select the top  $10\%$  and  $20\%$  of values, respectively. We note that constructing brain graphs involves a number of parameters including the choice of parcellation methods, where various brain atlases can be employed to segment the brain into regions of interest. Equally crucial is defining the number of ROIs, as it yields a significant influence on exploring brain functions. The selection of ROIs count permits the creation of brain graphs of varying sizes, allowing the opportunity to focus on both the global and granular levels of the brain. We have opted for those more likely to yield superior performance [3, 2]. Additional details about the complexity of the search space in benchmark dataset construction and the rationale behind these parameters are presented in Appendix B.4. We performed extensive experiments in the exploratory analysis to find the suitable combination of parameters and use a total of 15 baseline methods for benchmarking. The baselines include 10 GNNs, 3 conventional machine learning methods and 2 new architectures. Using the optimal combination of parameters in the static setting, we generate benchmark datasets for corresponding tasks in the dynamic setting. In the subsequent sections, we first describe the generation of graph-based datasets, followed by the description of each task.
3.2 Graph Representation
The landscape of constructing brain graphs encompasses a variety of approaches. Previous research in this domain has explored various methodologies for constructing brain graphs and subsequent downstream tasks. For instance, in [2, 3, 46, 47], distinct measures such as mean activation, node index as coordinates, spatial one-hot encoding, and correlations have been employed as node features
with brain graphs. Additionally, a diverse range of measures including partial correlation, Pearson correlation, and geometric distances, among others, have found wide application in defining edges within brain graphs [48]. Our static graph representation encompasses the conventional methodology of generating a static functional connectome graph from an fMRI scan, see Appendix, B.3 for additional details. We define a connectome graph as  $G = (\mathcal{V}, \mathcal{E}, X)$ , wherein the node set  $\mathcal{V} = \{v_1, v_2, \dots, v_n\}$  represents ROIs, while the edge set  $\mathcal{E} \subseteq \mathcal{V} \times \mathcal{V}$  represents positive correlations between pairs of ROIs, determined via a defined threshold. The feature matrix is denoted by  $X \in \mathbb{R}^{n \times d}$ , where  $n$  signifies the total number of ROIs and  $d$  refers to the feature vector's dimension. In our benchmarking setup, we define correlation vectors as node features. Subsequently, we define a representation vector  $h_G$  for the graph  $G$ , obtained via a GNN with an objective to perform the desired downstream machine learning task.
fMRI data comprise numerous timepoints within a scan, permitting the construction of dynamic graphs and thereby emphasizing the temporal information encapsulated within the data. This strategy has been evidenced to be notably effective within the literature [2]. Within the dynamic context, we define a sequence of brain graphs over  $T$  timepoints, denoted as  $\mathcal{G} = \{G_1, G_2, \dots, G_T\}$ , wherein each graph  $G_t$  captured at index  $t$  to  $t + \Gamma$  from the fMRI scan. Here,  $\Gamma$  signifies the window length, set to 50 with a stride of 3 in our experiments. This setup allows us to capture functional connectivity within 36 seconds every 2.16 seconds, adhering to a common protocol for sliding-window analyses as outlined in [34]. Following the approach in [2], we opt to randomly crop the ROI-timeseries data to a length of 150 timepoints. This procedure results in a total of 34 frames per subject, mitigating the computational and memory overhead in training complex models.
The procedure for constructing a graph at each time-point similar to the one applied to the static graph. The initial preprocessing, including parcellation, noise removal, and addressing head motions, remains consistent in order to construct the timeseries object. For each window, individual normalization has been performed, and then correlation matrices and corresponding graphs are constructed. Subsequently,  $\mathcal{G}$  can be utilized to generate a dynamic graph representation  $h_{dyn}$  to execute the desired downstream ML task. We refer the reader to the Appendix B.3 for further details.
3.3 Benchmark Datasets
The benchmark datasets are primarily divided into three main categories: those constructed for classification of demographics and task states, and those constructed for estimation of cognitive traits. Each category encapsulates distinct aspects of the collected data and serves unique analytical purposes. A brief description is provided below for each of these categories with some basic statistics presented in Table 1.
Predicting Demographics: The category of demographic estimation in our dataset is comprised of gender and age estimation [41]. The gender attribute facilitates a binary classification with the categories being male and female. Age is categorized into three distinct groups as in [27]: 22-25, 26-30, and 31-35 years. A fourth category for ages 36 and above was eliminated as it contained only 14 subjects  $(0.09\%)$ , to maintain a reasonably balanced dataset. We introduce four datasets named: HCP-Gender, HCP-Age, DynHCP-Gender, and DynHCP-Age under this category. The first two are static graph datasets while the last two are the corresponding dynamic graph datasets.
Predicting Task States: The task-decoding involves seven tasks: Emotion Processing, Gambling, Language, Motor, Relational Processing, Social Cognition, and Working Memory. Each task is designed to help delineate a core set of functions relevant to different facets of the relation between human brain, cognition and behavior [49]. Under this category, we present two datasets: HCP-Task, a static representation, and DynHCP-Task, its dynamic counterpart.
Estimating Cognitive Traits: The cognitive traits category of our dataset comprises two significant traits: working memory (evaluated with List Sorting) [50] and fluid intelligence (evaluated with PMAT24) [51]. Working memory refers to an individual's capacity to temporarily hold and manipulate information, a crucial aspect that influences higher cognitive functions such as reasoning, comprehension, and learning [52]. Fluid intelligence represents the ability to solve novel problems, independent of any knowledge from the past. It demonstrates the capacity to analyze complex relationships, identify patterns, and derive solutions in dynamic situations [20, 30]. The prediction of both these traits, quantified as continuous variables in our dataset, are treated as regression problem. We aim to predict the performance or scores related to these cognitive traits based on the func
Figure 2: (a). Illustration of the architecture for learning dynamic graph representations. (b). Visualization of the GNN\* architecture featuring residual connections and concatenated features.
tional connectome graphs. We generate four datasets under cognitive traits: HCP Fluid Intelligence (HCP-FI), HCP Working Memory (HCP-WM), DynHCP-FI and DynHCP-WM.
3.4 Learning models
The functional connectome, which effectively captures the network structure of brain activity, has proven to be a valuable representation of fMRI data for machine learning, as demonstrated in numerous previous studies and our own experiments [4, 9]. Recognizing its significance in the learning process, we sought a suitable GNN framework that could effectively leverage the comprehensive functional connectome data through a combination of message passing and neural network. After thorough exploration, we implemented a GNN architecture, denoted as  $\mathrm{GNN^{*}}$  illustrated in Figure 2 (b), that incorporates residual connections and concatenates hidden representations obtained from message passing at each layer. To further enhance the model's performance, we employ batch normalization and a multi-layer perceptron (MLP) to effectively utilize the combined representations during training. While adaptive residual connections have been extensively explored in GNNs, we present this simple and unique architecture for brain graphs that effectively learns the representations for brain graphs [53].
Recently, a number of dynamic graph representation approaches in conjunction with recurrent neural networks (RNNs) such as GRU, LSTM, and transformers, have been introduced [33, 54]. However, assessing the effectiveness of GNN models in a unified dynamic setting using the existing approaches presents a significant challenge. Therefore, we implement a simple and generalized architecture tailored to process dynamic graphs for the graph classification problem, as illustrated in Figure 2 (a). Our architecture comprises two distinct modules. The first is a GNN-based learning module, responsible for deriving graph-level representations from each of the derived graph snapshot. Following this, a transformer module takes over, applying attention to the learned representations from the GNNs. Finally, the outputs are averaged into a single dynamic graph representation vector,  $h_{dyn}$ . This design offers a universally applicable method for evaluating multiple GNN methods within a dynamic graph setting for the downstream ML classification and regression problem.
4 Benchmarking Setup
In order to thoroughly evaluate the performance of brain graphs generated through different hyperparameters, we propose a series of questions, defined as hyperparameter probe. These questions seek to identify the optimal hyperparameter setting for our graph-based neuroimaging analysis and ultimately enhance the performance of the predictive models derived from it.
Question 4.1 What are the optimal node feature configurations?
The first question aims to identify the best configurations for node features. This involves an exploration and comparison of various feature representations to discern their effectiveness on the performance of the derived predictive models. In assessing node feature configurations, our analysis encompasses the correlation matrix, the time-series BOLD signals, as well as their combination. The
correlation matrix is generated by calculating the correlation values amongst all ROIs. On the other hand, the BOLD signals are derived post the preprocessing of the input fMRI image, adhering to the preprocessing pipeline outlined in Section 3.1.