SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hongting Ye  $^{1}$ , Yalu Zheng  $^{1}$ , Yueying Li  $^{1}$ , Ke Zhang  $^{1}$ , Youyong Kong  $^{1,2*}$ , Yonggui Yuan  $^{3}$
$^{1}$ Jiangsu Provincial Joint International Research Laboratory of Medical Information Processing School of Computer Science and Engineering, Southeast University
$^{2}$ Key Laboratory of New Generation Artificial Intelligence Technology and Its
$^{3}$ Department of Psychosomatics and Psychiatry, Zhongda Hospital
School of Medicine, Southeast University
{yehongting, 220212084, 230228504, kylenz, kongyouyong}@seu.edu.cn
yygylh2000@sina.com
Multimodal fusion has become an important research technique in neuroscience that completes downstream tasks by extracting complementary information from multiple modalities. Existing multimodal research on brain networks mainly focuses on two modalities, structural connectivity (SC) and functional connectivity (FC). Recently, extensive literature has shown that the relationship between SC and FC is complex and not a simple one-to-one mapping. The coupling of structure and function at the regional level is heterogeneous. However, all previous studies have neglected the modal regional heterogeneity between SC and FC and fused their representations via "simple patterns", which are inefficient ways of multimodal fusion and affect the overall performance of the model. In this paper, to alleviate the issue of regional heterogeneity of multimodal brain networks, we propose a novel Regional Heterogeneous multimodal Brain networks Fusion Strategy (RH-BrainFS). Briefly, we introduce a brain subgraph networks module to extract regional characteristics of brain networks, and further use a new transformer-based fusion bottleneck module to alleviate the issue of regional heterogeneity between SC and FC. To the best of our knowledge, this is the first paper to explicitly state the issue of structural-functional modal regional heterogeneity and to propose a solution. Extensive experiments demonstrate that the proposed method outperforms several state-of-the-art methods in a variety of neuroscience tasks.
Currently, a large number of neuroscience studies are based on unimodal imaging [2, 30, 47, 56, 61]. However, different brain imaging techniques, such as functional magnetic resonance imaging (fMRI) [54] and diffusion magnetic resonance imaging (dMRI) [44], reflect different aspects of the brain's internal characteristics. Therefore, it is often insufficient to use a single modality of data for neuroscience research and it is necessary to integrate multiple modalities of imaging data to achieve good performance in neuroscience tasks such as depression classification [3, 33] and gender classification [15, 53].
In multimodal brain networks fusion, existing researches are mainly focused on fusing structural and functional modalities (structural modality is constructed from dMRI and functional modality is
2The codes are available at https://github.com/Yedaxia1/RH-BrainFS.
constructed from fMRI) [6, 29, 32, 57, 58]. Most methods directly fuse the two modal representations via "simple patterns" (we define this as direct interaction, where two modal features/embeddings are directly combined to perform some computation, e.g. concatenation [32], weighted summation [57], or self-attention [58] techniques.) without considering the issue of regional heterogeneity [37] between this two modalities. However, extensive literature [22, 37] has shown that the relationship between structural connectivity (SC) and functional connectivity (FC) is complex and not a simple one-to-one mapping. Specifically, the coupling of structure and function at the regional level is heterogeneous and follows the molecular, cellular and functional hierarchical structure. In other words, structure may be more tightly coupled to function in some regions than in others. This shows that regional heterogeneity is a key factor in linking different modal brain networks.
Based on the above research gap, in this paper we propose a novel regional heterogeneous multimodal brain networks fusion strategy that aims to fully account for the regional heterogeneity among brain networks and achieve better multimodal brain networks fusion performance.
First, the brain network itself has strong regional characteristics [36, 41], and the combination of neighbouring brain regions can serve as an important criterion for neuroscience tasks [16, 24]. Specifically, regional characteristics behave as subnetwork (also called subgraph) features in the whole brain network [11, 28], yet other multimodal studies do not take this into account. In this paper, we focus on the characteristics of brain regions from the subgraph pattern. The subgraph pattern have become a relatively hot area of graph representation learning research in recent years [25, 39, 55]. Subgraph convolutional networks can obtain potential representations of each subgraph in the graph, that reflect the regional characteristics of the graph [7, 10]. Therefore, to effectively extract the regional characteristics of each brain region in the brain network, we introduce a Brain Subgraph Networks (BrainSubGNN) module in Sec. 3.2. Our BrainSubGNN is divided into two steps, subgraph sampling and subgraph embedding. The former is used to obtain the subgraph partition of the brain network, the latter to aggregate the subgraph characteristics of the brain network.
Next, since several previous studies have shown that the characteristics of different individual regions in the brain network have widely different influences on neuroscience tasks [1, 31, 40, 42], we need to pay attention to the influence of these brain region characteristics on neuroscience tasks and quantify them into accurate values. In this paper, we focus on the Transformer [45] to learn the accurate influence values of these brain region characteristics. Although originally proposed for NLP tasks, there has been recent interest in Transformers [45] as universal perceptual models [14]. Through the attention mechanism, Transformer can learn the accurate influence values of different tokens well for the classification result, which happens to meet our needs.
However, as mentioned above, there is the issue of modal regional heterogeneity between SC and FC, which is a key factor in linking these two modal brain networks [37]. Many previous multimodal fusion methods, via "direct interaction", have not considered this issue between SC and FC, which are inefficient ways of modality fusion and affect the overall performance of the model. Based on that, avoiding direct interaction between two modalities within the Transformer is the aim of our study. Inspired by MBT [26], we present a transformer-based fusion bottleneck (Trans-Bottleneck) module for fusing regional heterogeneous brain networks in Sec. 3.3. Specifically, the Trans-Bottleneck module contains two standard transformers [45] and a certain number of fusion bottlenecks. The standard transformers are used to learn the accurate influence values of each brain region on neuroscience tasks through the attention mechanism. The fusion bottlenecks, as intermediate media for modality fusion, establish connections between regional heterogeneous modalities and learn the key information of each modality in the latent space (we define this as indirect interaction, where two modal features/embeddings are not directly combined for the computation.).
- To alleviate the issue of regional heterogeneity of multimodal brain networks, we propose a novel Regional Heterogeneous multimodal Brain networks Fusion Strategy (RH-BrainFS), using BrainSubGNN module and Trans-Bottleneck module to fuse regional heterogeneous multimodal brain networks for neuroscience tasks.
- To the best of our knowledge, this is the first paper to explicitly state the issue of structural-functional modal regional heterogeneity and to propose a solution.
- Extensive experiments demonstrate the effectiveness of RH-BrainFS in multimodal brain networks fusion tasks on depression classification and gender classification datasets.
Figure 1: The overall framework of our proposed RH-BrainFS Model. The structural connectivity (SC) and functional connectivity (FC) are constructed from fMRI and dMRI respectively, the initial bottlenecks  $Z_{b}$  are a set of learnable tokens randomly sampled from standard normal distribution, and the subgraph partition  $(SP_{i})$  are obtained in the preprocessing stage. Function  $g(\cdot)$  denotes the BrainSubGNN module.
Brain Subgraph Networks: In recent years, subgraph techniques have gained popularity in brain network analysis due to their ability to accurately model certain aspects of brain organization with high consistency to established brain functional systems [27]. This has led to a focus on identifying informative and signaling subgraphs from the entire brain connectome that may be relevant to brain diseases [46]. Various subgraph-based methods have been proposed for brain network research, such as the adaptive dense subgraph discovery (ADSD) model [51] which uses a likelihood-based approach to extract disease-associated subgraphs from group-level whole-brain connectome data. Other methods, such as an earlier approach from [4], balance topological information of local and global graphs using subgraphs, while approach [5] utilizes subgraphs to represent local features in large-scale brain network. Recent research [19] has also classified brain network by extracting contrastive subgraphs, and the SBLR model [49] suggests that subgraphs have attractive neurological interpretations and may correspond to outcome-related anatomical circuits. In this paper, our model builds upon these approaches and employs a BrainSubGNN module to efficiently aggregate regional characteristics of brain network.
MultiModal Brain Networks Fusion: Over the past few years, several methods have been developed for multimodal fusion in neuroscience research [12, 8, 38, 23, 60]. One traditional approach, SNF [48], creates an initial similarity network for each feature and iteratively combines them with a non-linear graph fusion formula to generate a final fusion network. However, recent advancements in deep learning have led to the development of more sophisticated multimodal fusion techniques. For instance, the GBDM [57] model employs both structural and functional information from diffusion and functional magnetic resonance imaging (MRI), respectively, to effectively differentiate individuals with mild cognitive impairment (MCI) from age-matched controls. The MGCN [29] model uses manifold-based regularization terms to account for inter-modality and intra-modality relationships. One approach [13] involves deep collaborative learning to capture cross-modal associations and trait-related features. Another approach [32] combines representations to fuse multimodal data. Moreover, a method [62] applies a multimodal non-Euclidean brain network analysis technique based on community detection and convolutional autoencoder for epilepsy classification. Additionally, a new adversarial learning-based node-edge graph attention network (AL-NEGAT [6]) has been proposed for autism spectrum disorder (ASD) recognition based on multi-modal MRI data. However, none of these methods have addressed the issue of modal regional heterogeneity [37] in multimodal
brain networks. To fill this gap, we propose a new multimodal brain networks fusion strategy aimed at alleviating the modal regional heterogeneity issue and achieving better fusion performance.
In this section, we present our proposed regional heterogeneous multimodal brain networks fusion strategy RH-BrainFS (as shown in Fig. 1). We begin by discussing the definition of the multimodal brain networks fusion task in Sec. 3.1. We then explain how the BrainSubGNN module captures the regional characteristics of brain networks in Sec. 3.2. Finally, we describe how the Trans-Bottleneck module used in the RH-BrainFS model alleviate the issue of regional heterogeneity in multimodal brain networks in Sec. 3.3.
3.1 Perliminaries
Multimodal Brain Networks Fusion Task: Given a multimodal brain networks dataset  $D = \{Sample_0, Sample_1, \dots, Sample_{L-1}\}$ , where each sample represents a person, and  $L$  is the size of this dataset. Each  $Sample_i = (\mathcal{G}_{sc}, \mathcal{G}_{fc}, y)$  contains a structural connectivity graph (each brain region is viewed as a node in the graph), a functional connectivity graph (ditto) and a class label  $y \in \{0, 1\}$  (0, 1 represent different meanings on different tasks). Each kind of input graph  $\mathcal{G} = (A, X, V)$  is composed of a node set  $V$ , an adjacency matrix  $A \in \mathbb{R}^{N \times N}$  and node features  $X \in \mathbb{R}^{N \times d}$ , where  $N = |V|$  denotes the number of nodes and  $d$  denotes the input dimensional of node features. In the brain network graph, node set  $V$  denotes the collection of brain regions, adjacency matrix  $A$  denotes the connectivity between various brain regions and node features  $X$  denotes the original features of each brain region. In the multimodal brain networks classification task, the purpose is to find a mapping function  $g: (\mathcal{G}_{sc}, \mathcal{G}_{fc}) \rightarrow y$ .
3.2 Brain Subgraph Networks
We now explain how BrainSubGNN module captures the regional characteristics of brain networks. As shown in Fig. 2, the BrainSubGNN contains subgraph sampling step and subgraph embedding step.
3.2.1 Subgraph sampling
The purpose of subgraph sampling is to construct a receptive field for each brain region (also named node in graph), it represent the regional characteristics of this brain region. In our RH-BrainFS, rooted subgraph [59] is utilized to construct receptive field, as the rooted subgraph can exhibit even greater discriminative power than the first-order Weisfeiler-Leman (1-WL) test due to interconnectivity among neighboring nodes [50, 59].
Figure 2: Brain Subgraph Networks (BrainSubGNN). Including subgrah sampling process (1-hop) and subgraph embedding process.
Specifically, we define central subgraph for each node in the whole brain network. The central subgraph describes the surrounding region of the central node. For a input graph  $\mathcal{G}$  and the central node  $v_{i}$ , the central subgraph is denoted as  $\tilde{\mathcal{G}}_i^k = (\tilde{A}_i,\tilde{X}_i,\tilde{V}_i)\subset \mathcal{G}$  where  $\tilde{V}_{i} = \mathcal{N}^{k}(i)$  contains k-hop neighbor nodes of  $v_{i}$  and  $\tilde{A}_i$ ,  $\tilde{X}_i$  equal to the corresponding part in the original  $A$  and  $X$ . The central subgraph  $\tilde{\mathcal{G}}_i^k$  constructed above includes brain region  $i$  and its k-hops adjacent brain regions. Such a subgraph contains the characteristics of this local brain network, which is of great significance to the brain network. In this work, the subgraph sampling is a preprocessing process before network training, node and edge indices are saved as a binary file. In the training stage, the index are loaded directly without additional cost on training.
3.2.2 Subgraph embedding
The subgraph partitions of the brain network are obtained in the previous step, each subgraph contains a central brain region and several adjacent brain regions, so we expect to be able to use a method that integrates the characteristics of all the brain regions in the subgraph and extract an embedding to represent the entire subgraph. Inspired by the graph isomorphism network [52], we consider the subgraph representation task as a multi-set problem and use multi-layer perceptrons to learn an injective function for aggregating regional characteristics the brain subgraph network effectively and learn subgraph representation as:
$$
z _ {i} ^ {(l)} = \mathbf {M L P} ^ {(l)} \left(W _ {1} ^ {(l)} \tilde {h} _ {i} ^ {(l)} + \sum_ {j \in \tilde {V} _ {i} \backslash i} W _ {2} ^ {(l)} \tilde {h} _ {j} ^ {(l)}\right) \tag {1}
$$
where  $\tilde{h}_i$  denote the hidden representation of node  $i$ , and  $W_1^{(l)}$ ,  $W_2^{(l)} \in \mathbb{R}^{d^{(l)} \times d^{(l+1)}}$  are learnable weight matrices for the central node and other nodes in subgraph, respectively. By default, we set the MLP as a two-layer fully connected module. The obtained  $z_i$  represents the local characteristics of brain region  $i$ .
3.3 Transformer-Based Fusion Bottleneck
Through the BrainSubGNN described earlier, we obtain the regional representations of each modality of the brain network  $Z_{i} \in \mathbb{R}^{N \times d_{hid}}$ . Next, our goal is to apply an efficient fusion strategy to fuse critical and complementary information from  $Z_{sc}$  and  $Z_{fc}$ , extract a distinguishing embedding to represent the features of the whole brain, and serve as a criterion for downstream tasks. We now describe how Trans-Bottleneck module alleviate the issue of regional heterogeneity in multimodal brain networks.
3.3.1 Fusion Bottlenecks
Due to the issue of regional heterogeneity between SC and FC, we are committed to avoiding the direct interaction of two modalities, and we prefer to find an intermediate element as a bridge for the interaction between two modalities (means indirect interaction). Inspired by MBT [26], we introduce the fusion bottlenecks into neuroscience research.
Specifically, fusion bottlenecks are simply a set of learnable tokens  $Z_{b} \in \mathbb{R}^{N_{b} \times d_{hid}}$ , where  $N_{b}$  is a hyperparameter denoting the number of fusion bottlenecks. In this work, we utilize the fusion bottlenecks as intermediate medium to bridge contact of this two modalities (SC and FC). As shown in Fig. 3, the fusion bottlenecks allow information to flow between modalities and fusion bottlenecks (indirect interaction), but limit the flow of information between modalities (direct interaction). This procedure avoids direct interaction between regional heterogeneous modality data, effectively improving the model's performance. The initial bottlenecks are a set of learn
Figure 3: Interaction between three main tokens.
able tokens that are randomly sampled from standard normal distribution  $\mathcal{N}(0,1)$  and then passed to the first RH-BrainFS layer. Finally, the fusion bottlenecks output by the last RH-BrainFS layer are used as the classification basis for downstream tasks (gender classification, major depressive disorder diagnosis, etc.).
3.3.2 Transformer-Based Fusion
Although originally proposed for NLP tasks, there has been recent interest in Transformers as universal perceptual models due to their ability to model dense correlations between tokens. Meanwhile, it turns out that individual regional characteristics of brain networks have different influence values for neuroscience tasks [1, 31, 40, 42]. Based on these, our RH-BrainFS method utilizes the Transformer [45] as a baseline for the fusion strategy to capture key subgraph characteristics in brain networks.
Specifically, as shown in Fig. 1, we first concatenate the fusion bottlenecks  $Z_{b}$  and each modality representation  $Z_{i}$ , and then feed the concatenated tokens to the standard Transformer [45] model of the corresponding modality. In the Transformer model, the fusion bottlenecks learn important regional characteristics of the brain network for each modality through attention mechanism. The tokens output by the Transformer are re-split according to the previous splicing scheme to obtain new potential features  $Z_{i}^{l + 1}$  of the brain network and temporary fusion bottlenecks  $\hat{Z}_{b_i}^{l + 1}$  corresponding to each modality. After that, the temporary fusion bottlenecks  $\hat{Z}_{b_i}^{l + 1}$  of all modalities are matrix-averaged (also named average pooling) to obtain the fusion bottleneck latent representation  $Z_{b}^{l + 1}$ .
This procedure can be formulated as:
$$
\left[ Z _ {i} ^ {l + 1} \mid \mid \hat {Z} _ {b _ {i}} ^ {l + 1} \right] = \operatorname {T r a n s f o r m e r} \left(\left[ Z _ {i} ^ {l} \mid \mid Z _ {b} ^ {l} \right]; \theta_ {i}\right) \tag {2}
$$
$$
Z _ {b} ^ {l + 1} = \operatorname {A V E R A G E} \left(\hat {Z} _ {b _ {i}} ^ {l + 1}\right) \tag {3}
$$
where  $i \in \{SC, FC\}$ ,  $\theta_i$  denotes the modality specific Transformer,  $[\cdot ||\cdot ]$  denotes the concatenate operation,  $Z_b^l$  denotes the fusion bottlenecks in  $l^{th}$  layer, and  $\hat{Z}_{b_i}^{l + 1}$  denotes the temporary fusion bottlenecks in  $i^{th}$  modality specific Transformer.
In this procedure, each modality transfers importance regional brain network characteristics within its modality to the fusion bottlenecks, and the fusion bottlenecks utilize shared characteristics between modalities to guide the learning of each modality's brain network in the next layer.
3.3.3 Downstream Tasks
In the final stage, we complete classification basis for downstream tasks through the output  $Z_{b}^{l}$  in the last layer. Specifically, we first use global mean pooling as the readout function, and then input the result into a MLP to complete classification basis.
$$
\operatorname {L o g i t s} = \operatorname {S o f t m a x} \left(\operatorname {M L P} \left(\frac {1}{N _ {b}} \sum_ {N _ {b}} ^ {i} Z _ {i, b}\right)\right) \tag {4}
$$
where  $Z_{i,b}$  denotes the i-th row of  $Z_{b}$ , and  $Logits \in \{0,1\}$ . In this paper, on the depression classification task, 0 represents major depressive disorder, 1 represents normal control. And on the gender classification task, 0 represents male, 1 represents female.
In this section, we perform a series of experiments to evaluate the effectiveness of the proposed RH-BrainFS method. First, we provide the detailed experimental settings in Sec. 4.1. Then, we perform comparison experiments on all datasets to compare the performance of different methods in Sec. 4.2. Finally, we perform some ablation studies of the main modules and hyperparameters in the proposed RH-BrainFS method in Sec. 4.3.
Datasets. We evaluate our RH-BrainFS method on two different classification tasks investigating structure-function fusion. 1) The gender classification task on Human Connectome Project (HCP) dataset [43], which contains 560 female samples and 479 male samples. 2) The Major Depressive Disorder (MDD) diagnosis task on the hospital datasets [17, 18], including the Affiliated Zhongda Hospital of Southeast University (Zhongda hospital) and the Second Affiliated Hospital of Xinxiang Medical University (Xinxiang hospital). This study included 48 controls and 62 MDD patients from the Zhongda hospital and 46 controls and 31 MDD patients from the Xinxiang hospital. We also combine Zhongda and Xinxiang as Two-site dataset to construct more difficult and more realistic tasks.
Preprocessing. Here, we would briefly introduce how to construct the brain network of SC from dMRI and FC from fMRI.
- SC. The dMRI data is preprocessed using the brain's diffusion toolbox of FMRIB Software Library [34]. Next we construct  $\mathcal{G}_{sc} = (A_{sc},X_{sc},V_{sc})$  from preprocessed dMRI data. First, we obtain the brain regions  $(V_{sc})$  of the individual space by mapping the anatomical automatic labeling (AAL) template in the standard space to the individual space. Then, we using DSI Studio software [9] to implement Fiber tracking. Finally, we obtain the feature  $X_{sc}\in \mathbb{R}^{|V_{sc}|\times |V_{sc}|}$  by counting the number of structural connective fibres between the different regions of the AAL, and the adjacency matrix  $A_{sc}$  of the structural graph is obtained by thresholding  $X_{sc}$  with a threshold.
- FC. The fMRI data is preprocessed using the Data Processing Assistant for Resting-State Function (DPARSF) [35] MRI toolkit. Next we construct  $\mathcal{G}_{fc} = (A_{fc}, X_{fc}, V_{fc})$  from preprocessed fMRI data. First, averaged time series are first computed for each brain region with a predefined atlas. Then, the Pearson correlation is utilized to calculate the functional matrix. Finally, the functional matrix is thresholded by proportional quantization to obtain the adjacency matrix  $A_{fc}$ . The features  $X_{fc}$  are functional connectivity matrix obtained earlier.