Title: RH-BrainFS: Regional Heterogeneous Multimodal Brain Networks Fusion Strategy
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multimodal

Neuroscience

Subgraph

Transformer
Abstract: Multimodal fusion has become an important research technique in neuroscience that completes downstream tasks by extracting complementary information from multiple modalities. Existing multimodal research on brain networks mainly focuses on two modalities, structural connectivity (SC) and functional connectivity (FC). Recently, extensive literature has shown that the relationship between SC and FC is complex and not a simple one-to-one mapping. The coupling of structure and function at the regional level is heterogeneous. However, all previous studies have neglected the modal regional heterogeneity between SC and FC and fused their representations via "simple patterns", which are inefficient ways of multimodal fusion and affect the overall performance of the model. In this paper, to alleviate the issue of regional heterogeneity of multimodal brain networks, we propose a novel Regional Heterogeneous multimodal Brain networks Fusion Strategy (RH-BrainFS). Briefly, we introduce a brain subgraph networks module to extract regional characteristics of brain networks, and further use a new transformer-based fusion bottleneck module to alleviate the issue of regional heterogeneity between SC and FC. To the best of our knowledge, this is the first paper to explicitly state the issue of structural-functional modal regional heterogeneity and to propose a
solution. Extensive experiments demonstrate that the proposed method outperforms several state-of-the-art methods in a variety of neuroscience tasks.

RH-BrainFS: Regional Heterogeneous Multimodal Brain Networks Fusion Strategy

Abstract

Multimodal fusion has become an important research technique in neuroscience that completes downstream tasks by extracting complementary information from multiple modalities. Existing multimodal research on brain networks mainly focuses on two modalities, structural connectivity (SC) and functional connectivity (FC). Recently, extensive literature has shown that the relationship between SC and FC is complex and not a simple one-to-one mapping. The coupling of structure and function at the regional level is heterogeneous. However, all previous studies have neglected the modal regional heterogeneity between SC and FC and fused their representations via "simple patterns", which are inefficient ways of multimodal fusion and affect the overall performance of the model. In this paper, to alleviate the issue of regional heterogeneity of multimodal brain networks, we propose a novel R egional H eterogeneous multimodal Brain networks F usion S trategy (RHBrainFS). 2 Briefly, we introduce a brain subgraph networks module to extract regional characteristics of brain networks, and further use a new transformer-based fusion bottleneck module to alleviate the issue of regional heterogeneity between SC and FC. To the best of our knowledge, this is the first paper to explicitly state the issue of structural-functional modal regional heterogeneity and to propose a solution. Extensive experiments demonstrate that the proposed method outperforms several state-of-the-art methods in a variety of neuroscience tasks.

1 Introduction

Currently, a large number of neuroscience studies are based on unimodal imaging [2, 30, 47, 56, 61]. However, different brain imaging techniques, such as functional magnetic resonance imaging (fMRI) [54] and diffusion magnetic resonance imaging (dMRI) [44], reflect different aspects of the brain's internal characteristics. Therefore, it is often insufficient to use a single modality of data for neuroscience research and it is necessary to integrate multiple modalities of imaging data to achieve good performance in neuroscience tasks such as depression classification [3, 33] and gender classification [15, 53].

In multimodal brain networks fusion, existing researchs are mainly focused on fusing structural and functional modalities (structural modality is constructed from dMRI and functional modality is

constructed from fMRI) [6, 29, 32, 57, 58]. Most methods directly fuse the two modal representations via "simple patterns" (we define this as direct interaction , where two modal features/embeddings are directly combined to perform some computation, e.g. concatenation [32], weighted summation [57], or self-attention [58] techniques.) without considering the issue of regional heterogeneity [37] between this two modalities. However, extensive literature [22, 37] has shown that the relationship between structural connectivity (SC) and functional connectivity (FC) is complex and not a simple one-to-one mapping. Specifically, the coupling of structure and function at the regional level is heterogeneous and follows the molecular, cellular and functional hierarchical structure. In other words, structure may be more tightly coupled to function in some regions than in others. This shows that regional heterogeneity is a key factor in linking different modal brain networks.

Based on the above research gap, in this paper we propose a novel regional heterogeneous multimodal brain networks fusion strategy that aims to fully account for the regional heterogeneity among brain networks and achieve better multimodal brain networks fusion performance.

First, the brain network itself has strong regional characteristics [36, 41], and the combination of neighbouring brain regions can serve as an important criterion for neuroscience tasks [16, 24]. Specifically, regional characteristics behave as subnetwork (also called subgraph) features in the whole brain network [11, 28], yet other multimodal studies do not take this into account. In this paper, we focus on the characteristics of brain regions from the subgraph pattern. The subgraph pattern have become a relatively hot area of graph representation learning research in recent years [25, 39, 55]. Subgraph convolutional networks can obtain potential representations of each subgraph in the graph, that reflect the regional characteristics of the graph [7, 10]. Therefore, to effectively extract the regional characteristics of each brain region in the brain network, we introduce a Brain Subgraph Networks (BrainSubGNN) module in Sec. 3.2. Our BrainSubGNN is divided into two steps, subgraph sampling and subgraph embedding. The former is used to obtain the subgraph partition of the brain network, the latter to aggregate the subgraph characteristics of the brain network.

Next, since several previous studies have shown that the characteristics of different individual regions in the brain network have widely different influences on neuroscience tasks [1, 31, 40, 42], we need to pay attention to the influence of these brain region characteristics on neuroscience tasks and quantify them into accurate values. In this paper, we focus on the Transformer [45] to learn the accurate influence values of these brain region characteristics. Although originally proposed for NLP tasks, there has been recent interest in Transformers [45] as universal perceptual models [14]. Through the attention mechanism, Transformer can learn the accurate influence values of different tokens well for the classification result, which happens to meet our needs.

However, as mentioned above, there is the issue of modal regional heterogeneity between SC and FC, which is a key factor in linking these two modal brain networks [37]. Many previous multimodal fusion methods, via "direct interaction", have not considered this issue between SC and FC, which are inefficient ways of modality fusion and affect the overall performance of the model. Based on that, avoiding direct interaction between two modalities within the Transformer is the aim of our study. Inspired by MBT [26], we present a transformer-based fusion bottleneck (Trans-Bottleneck) module for fusing regional heterogeneous brain networks in Sec. 3.3. Specifically, the Trans-Bottleneck module contains two standard transformers [45] and a certain number of fusion bottlenecks. The standard transformers are used to learn the accurate influence values of each brain region on neuroscience tasks through the attention mechanism. The fusion bottlenecks, as intermediate media for modality fusion, establish connections between regional heterogeneous modalities and learn the key information of each modality in the latent space (we define this as indirect interaction , where two modal features/embeddings are not directly combined for the computation.).

The main contributions of this paper are summarized as follows:

To alleviate the issue of regional heterogeneity of multimodal brain networks, we propose a novel R egional H eterogeneous multimodal Brain networks F usion S trategy (RH-BrainFS), using BrainSubGNN module and Trans-Bottleneck module to fuse regional heterogeneous multimodal brain networks for neuroscience tasks.

To the best of our knowledge, this is the first paper to explicitly state the issue of structuralfunctional modal regional heterogeneity and to propose a solution.

Extensive experiments demonstrate the effectiveness of RH-BrainFS in multimodal brain networks fusion tasks on depression classification and gender classification datasets.

Modality FC

fMRI

ROI#2

ROI#90

. . .

. . .

Pearson

Correlation

thresholding

Modality SC

dMRI

Initial Bottlenecks

Final Bottlenecks

𝑸

𝑲

𝑽

MatMul (Scale)

Linear

Softmax

MatMul

Linear

Linear

Linear

FFN

+

𝑸

𝑲

𝑽

MatMul (Scale)

Linear

Softmax

MatMul

Linear

Linear

Linear

FFN

+

Avg

𝑸

𝑲

𝑽

MatMul (Scale)

Linear

Softmax

MatMul

Linear

Linear

Linear

FFN

+

𝑸

𝑲

𝑽

MatMul (Scale)

Linear

Softmax

MatMul

Linear

Linear

Linear

FFN

+

Avg

𝑸

𝑲

𝑽

MatMul (Scale)

Linear

Softmax

MatMul

Linear

Linear

Linear

FFN

+

𝑸

𝑲

𝑽

MatMul (Scale)

Linear

Softmax

MatMul

Linear

Linear

Linear

FFN

+

Avg

thresholding

RH-BrainFS Layer

MLP

GMP

Gender Classification

+

Avg

GMP

Matrix Concatenation

Matrix Average

Global Mean Pooling

Major Depressive

Disorder

Downstream Task

. . .

2 Related Work

Brain Subgraph Networks: In recent years, subgraph techniques have gained popularity in brain network analysis due to their ability to accurately model certain aspects of brain organization with high consistency to established brain functional systems [27]. This has led to a focus on identifying informative and signaling subgraphs from the entire brain connectome that may be relevant to brain diseases [46]. Various subgraph-based methods have been proposed for brain network research, such as the adaptive dense subgraph discovery (ADSD) model [51] which uses a likelihood-based approach to extract disease-associated subgraphs from group-level whole-brain connectome data. Other methods, such as an earlier approach from [4], balance topological information of local and global graphs using subgraphs, while approach [5] utilizes subgraphs to represent local features in large-scale brain network. Recent research [19] has also classified brain network by extracting contrastive subgraphs, and the SBLR model [49] suggests that subgraphs have attractive neurological interpretations and may correspond to outcome-related anatomical circuits. In this paper, our model builds upon these approaches and employs a BrainSubGNN module to efficiently aggregate regional characteristics of brain network.

MultiModal Brain Networks Fusion: Over the past few years, several methods have been developed for multimodal fusion in neuroscience research [12, 8, 38, 23, 60]. One traditional approach, SNF [48], creates an initial similarity network for each feature and iteratively combines them with a non-linear graph fusion formula to generate a final fusion network. However, recent advancements in deep learning have led to the development of more sophisticated multimodal fusion techniques. For instance, the GBDM [57] model employs both structural and functional information from diffusion and functional magnetic resonance imaging (MRI), respectively, to effectively differentiate individuals with mild cognitive impairment (MCI) from age-matched controls. The MGCN [29] model uses manifold-based regularization terms to account for inter-modality and intra-modality relationships. One approach [13] involves deep collaborative learning to capture cross-modal associations and trait-related features. Another approach [32] combines representations to fuse multimodal data. Moreover, a method [62] applies a multimodal non-Euclidean brain network analysis technique based on community detection and convolutional autoencoder for epilepsy classification. Additionally, a new adversarial learning-based node-edge graph attention network (AL-NEGAT [6]) has been proposed for autism spectrum disorder (ASD) recognition based on multi-modal MRI data. However, none of these methods have addressed the issue of modal regional heterogeneity [37] in multimodal

brain networks. To fill this gap, we propose a new multimodal brain networks fusion strategy aimed at alleviating the modal regional heterogeneity issue and achieving better fusion performance.

3 Method

In this section, we present our proposed regional heterogeneous multimodal brain networks fusion strategy RH-BrainFS (as shown in Fig. 1). We begin by discussing the definition of the multimodal brain networks fusion task in Sec. 3.1. We then explain how the BrainSubGNN module captures the regional characteristics of brain networks in Sec. 3.2. Finally, we describe how the Trans-Bottleneck module used in the RH-BrainFS model alleviate the issue of regional heterogeneity in multimodal brain networks in Sec. 3.3.

3.1 Perliminaries

Multimodal Brain Networks Fusion Task: Given a multimodal brain networks dataset D = { Sample 0 , Sample 1 , ..., Sample L -1 } , where each sample represents a person, and L is the size of this dataset. Each Sample i = ( G sc , G fc , y ) contains a structural connectivity graph (each brain region is viewed as a node in the graph), a functional connectivity graph (ditto) and a class label y ∈ { 0 , 1 } (0, 1 represent different meanings on different tasks). Each kind of input graph G = ( A,X,V ) is composed of a node set V , an adjacency matrix A ∈ R N × N and node features X ∈ R N × d , where N = | V | denotes the number of nodes and d denotes the input dimensional of node features. In the brain network graph, node set V denotes the collection of brain regions, adjacency matrix A denotes the connectivity between various brain regions and node features X denotes the original features of each brain region. In the multimodal brain networks classification task, the purpose is to find a mapping function g : ( G sc , G fc ) → y .

3.2 Brain Subgraph Networks

We now explain how BrainSubGNN module captures the regional characteristics of brain networks. As shown in Fig. 2, the BrainSubGNN contains subgraph sampling step and subgraph embedding step.

3.2.1 Subgraph sampling

The purpose of subgraph sampling is to construct a receptive field for each brain region (also named node in graph), it represent the regional characteristics of this brain region. In our RH-BrainFS, rooted subgraph [59] is utilized to construct receptive field, as the rooted subgraph can exhibit even greater discriminative power than the first-order Weisfeiler-Leman (1-WL) test due to interconnectivity among neighboring nodes [50, 59].

. . .

. . .

(Preprocess Stage)

(Training Stage)

. . .

Specifically, we define central subgraph for each node in the whole brain network. The central subgraph describes the surrounding region of the central node. For a input graph G and the central node v i , the central subgraph is denoted as ˜ G k i = ( ˜ A i , ˜ X i , ˜ V i ) ⊂ G where ˜ V i = N k ( i ) contains k-hop neighbor nodes of v i and ˜ A i , ˜ X i equal to the corresponding part in the original A and X . The central subgraph ˜ G k i constructed above includes brain region i and its k-hops adjacent brain regions. Such a subgraph contains the characteristics of this local brain network, which is of great significance to the brain network. In this work, the subgraph sampling is a preprocessing process before network training, node and edge indice are saved as a binary file. In the training stage, the indice are loaded directly without additional cost on training.

3.2.2 Subgraph embedding

The subgraph partitions of the brain network are obtained in the previous step, each subgraph contains a central brain region and several adjacent brain regions, so we expect to be able to use a method that integrates the characteristics of all the brain regions in the subgraph and extract an embedding to represent the entire subgraph. Inspired by the graph isomorphism network [52], we consider the subgraph representation task as a multi-set problem and use multi-layer perceptrons to learn an injective function for aggregating regional characteristics the brain subgraph network effectively and learn subgraph representation as:

where ˜ h i denote the hidden representation of node i , and W ( l ) 1 , W ( l ) 2 ∈ R d ( l ) × d ( l +1) are learnable weight matrices for the central node and other nodes in subgraph, respectively. By default, we set the MLP as a two-layer fully connected module. The obtained z i represents the local characteristics of brain region i .

3.3 Transformer-Based Fusion Bottleneck

Through the BrainSubGNN described earlier, we obtain the regional representations of each modality of the brain network Z i ∈ R N × d hid . Next, our goal is to apply an efficient fusion strategy to fuse critical and complementary information from Z sc and Z fc , extract a distinguishing embedding to represent the features of the whole brain, and serve as a criterion for downstream tasks. We now describe how Trans-Bottleneck module alleviate the issue of regional heterogeneity in multimodal brain networks.

3.3.1 Fusion Bottlenecks

Due to the issue of regional heterogeneity between SC and FC, we are committed to avoiding the direct interaction of two modalities, and we prefer to find an intermediate element as a bridge for the interaction between two modalities (means indirect interaction). Inspired by MBT [26], we introduce the fusion bottlenecks into neuroscience research.

Specifically, fusion bottlenecks are simply a set of learnable tokens Z b ∈ R N b × d hid , where N b is a hyperparameter denoting the number of fusion bottlenecks. In this work, we utilize the fusion bottlenecks as intermediate medium to bridge contact of this two modalities (SC and FC). As shown in Fig. 3, the fusion bottlenecks allow information to flow between modalities and fusion bottlenecks (indirect interaction), but limit the flow of information between modalities (direct interaction). This procedure avoid direct interaction between regional heterogeneous modality data, effectively improving the model's performance. The initial bottlenecks are a set of learn-

Indirect Interaction

Direct Interaction

Indirect Interaction

able tokens that are randomly sampled from standard normal distribution N (0 , 1) and then passed to the first RH-BrainFS layer. Finally, the fusion bottlenecks output by the last RH-BrainFS layer are used as the classification basis for downstream tasks (gender classification, major depressive disorder diagnosis, etc.).

3.3.2 Transformer-Based Fusion

Although originally proposed for NLP tasks, there has been recent interest in Transformers as universal perceptual models due to their ability to model dense correlations between tokens. Meanwhile, it turns out that individual regional characteristics of brain networks have different influence values for neuroscience tasks [1, 31, 40, 42]. Based on these, our RH-BrainFS method utilizes the Transformer [45] as a baseline for the fusion strategy to capture key subgraph characteristics in brain networks.

Specifically, as shown in Fig. 1, we first concatenate the fusion bottlenecks Z b and each modality representation Z i , and then feed the concatenated tokens to the standard Transformer [45] model of the corresponding modality. In the Transformer model, the fusion bottlenecks learn important regional characteristics of the brain network for each modality through attention mechanism. The tokens output by the Transformer are re-split according to the previous splicing scheme to obtain new potential features Z l +1 i of the brain network and temporary fusion bottlenecks ˆ Z l +1 b i corresponding to each modality. After that, the temporary fusion bottlenecks ˆ Z l +1 b i of all modalities are matrix-averaged (also named average pooling) to obtain the fusion bottleneck latent representation Z l +1 b .

This procedure can be formulated as:

where i ∈ { SC,FC } , θ i denotes the modality specific Transformer, [ · || · ] denotes the concatenate operation, Z l b denotes the fusion bottlenecks in l th layer, and ˆ Z l +1 b i denotes the temporary fusion bottlenecks in i th modality specific Transformer.

In this procedure, each modality transfers importance regional brain network characteristics within its modality to the fusion bottlenecks, and the fusion bottlenecks utilize shared characteristics between modalities to guide the learning of each modality's brain network in the next layer.

3.3.3 Downstream Tasks

In the final stage, we complete classification basis for downstream tasks through the output Z l b in the last layer. Specifically, we first use global mean pooling as the readout function, and then input the result into a MLP to complete classification basis.

where Z i,b denotes the i-th row of Z b , and Logits ∈ { 0 , 1 } . In this paper, on the depression classification task, 0 represents major depressive disorder, 1 represents normal control. And on the gender classification task, 0 represents male, 1 represents female.

4 Experiments

In this section, we perform a series of experiments to evaluate the effectiveness of the proposed RH-BrainFS method. First, we provide the detailed experimental settings in Sec. 4.1. Then, we perform comparison experiments on all datasets to compare the performance of different methods in Sec. 4.2. Finally, we perform some ablation studies of the main modules and hyperparameters in the proposed RH-BrainFS method in Sec. 4.3.

4.1 Experimental Settings

Datasets. We evaluate our RH-BrainFS method on two different classification tasks investigating structure-function fusion. 1) The gender classification task on Human Connectome Project (HCP) dataset [43], which contains 560 female samples and 479 male samples. 2) The Major Depressive Disorder (MDD) diagnosis task on the hospital datasets [17, 18], including the Affiliated Zhongda Hospital of Southeast University (Zhongda hospital) and the Second Affiliated Hospital of Xinxiang Medical University (Xinxiang hospital). This study included 48 controls and 62 MDD patients from the Zhongda hospital and 46 controls and 31 MDD patients from the Xinxiang hospital. We also combine Zhongda and Xinxiang as Two-site dataset to construct more difficult and more realistic tasks.

Preprocessing. Here, we would briefly introduce how to construct the brain network of SC from dMRI and FC from fMRI.

SC. The dMRI data is preprocessed using the brain's diffusion toolbox of FMRIB Software Library [34]. Next we construct G sc = ( A sc , X sc , V sc ) from preprocessed dMRI data. First, we obtain the brain regions ( V sc ) of the individual space by mapping the anatomical automatic labeling (AAL) template in the standard space to the individual space. Then, we using DSI Studio software [9] to implement Fiber tracking. Finally, we obtain the feature X sc ∈ R | V sc |×| V sc | by counting the number of structural connective fibres between the different regions of the AAL, and the adjacency matrix A sc of the structural graph is obtained by thresholding X sc with a threshold.

FC. The fMRI data is preprocessed using the Data Processing Assistant for Resting-State Function (DPARSF) [35] MRI toolkit. Next we construct G fc = ( A fc , X fc , V fc ) from preprocessed fMRI data. First, averaged time series are first computed for each brain region with a predefined atlas. Then, the Pearson correlation is utilized to calculate the functional matrix. Finally, the functional matrix is thresholded by proportional quantization to obtain the adjacency matrix A fc . The features X fc are functional connectivity matrix obtained earlier.

Metrics. In this study, we evaluate all the methods using 10-fold cross-validation with the same partition of training and testing splits. Our evaluation metrics include classification accuracy (ACC), sensitivity (SEN), specificity (SPE), f1 score (F1) and ROC-AUC (AUC). Higher values for all metrics indicate better performance. We record the mean and standard deviation on 10 random runs with 10-fold cross-validation.

Implementation Details. For all experiments, we adopt Adam as the optimizer and StepLR (step_size=50, gamma=0.8) as the scheduler. The initial learning rate is set to 5e-4 and the dropout rate is set to 0.3. Also we utilize a early stop mechanism that 300 epochs patience in total 500 epochs. In the RH-BrainFS model, we set the k-hop in the subgraph sampling to 1, the number of bottlenecks N b to 4, the number of attention heads in the Transformer to 4, and the total number of network layers to 2. All our experiments are implemented in PyTorch and trained on one NVIDIA 3090.

4.2 Comparison Experiments

In this section, we verify the performance of our RH-BrainFS against existing baselines on several datasets.

Baselines. We choose two categories of methods as comparison methods, both of which are methods for the direct study of SC and FC. The first category is unimodal methods, including FGDN [20] and BrainGNN [21], where FGDN uses a spectral graph convolution method to extract brain networks features, and BrainGNN proposes a ROI-aware graph convolution layer and uses pooling to extract brain networks features. In our experiments, SC and FC are used as inputs to the unimodal method, respectively. The second category is multimodal methods, including SVM, Random Forest, MGCN [29], GBDM [57], MMGNN [32] and AL-NEGAT [6], where SVM and Random Forest concatenate SC and FC as input, MGCN uses manifold-based regularization terms to consider inter-modality and intra-modality relationships, GBDM adopts weighted summation pattern to fuse multimodal brain networks, MMGNN utilises the concatenation method in multimodal tasks and AL-NEGAT combines multimodal information to construct edge feature maps and node feature maps. Code implementations of all baseline methods are taken from their respective original papers.

Results. As shown in Tab. 1, our model significantly outperforms the other comparison methods on the all selected datasets. The results show that multimodal methods generally have better performance than unimodal methods because they capture more complementary information. Among the multimodal methods, our RH-BrainFS method achieves the best performance on the all the selected datasets (improvement of 3.51% on HCP, 5.46% on Zhongda hospital dataset, 8.03% on Xinxiang hospital dataset and 5.50% on two-site dataset). The reason for the performance improvement is that our RH-BrainFS method fully considers the issue of regional heterogeneity between SC and FC and proposes an appropriate solution strategy for this issue.

MGCN

MMGNN

AL-NEGAT

Ours

female

male

Visualization. In order to intuitively display the performance of each multimodal method, we conduct visualization experiments on the HCP dataset. Specifically, we take the graph-level embedding output from the last layer of each multimodal method for t-SNE visualisation. As shown in Fig. 4, although the MGCN and GBDM all form two clusters, these two clusters do not distinguish the two types of samples well and there is still a lot of confusion. The MMGNN is loosely distributed and does not form good class boundary. It can be seen that the Al-NEGAT produces a similar distribution to our method, but there's still a lot of confusion at the class boundary. In contrast, our method eliminates the confusion at the class boundary, achieves a good classification effect, most of the samples can be accurately distinguished and only a small number of samples have errors.

4.3 Ablation Study

In this section, we further perform ablation studies on the main modules and hyperparameters in the proposed RH-BrainFS method. 3

2

4

6

8

# bottlenecks

70

72

74

76

78

80

accuracy

(a)

ours

AL-NEGAT

MMGNN

SVM

2

4

6

8

# bottlenecks

70

72

74

76

78

80

accuracy

(b)

ours

AL-NEGAT

MGCN

1

2

3

4

5

k

70

72

74

76

78

80

accuracy

(c)

ours

AL-NEGAT

MMGNN

SVM

1

2

3

4

5

k

70

72

74

76

78

80

accuracy

(d)

ours

AL-NEGAT

MGCN

Effectiveness of Main Modules. First, we perform an ablation study on HCP dataset to validate effectiveness of the main modules, BrainSubGNN and Trans-Bottleneck. Specifically, we replace BrainSubGNN with normal GIN [52] (process on the whole brain network), and replace TransBottleneck with a standard transformer (compute self-attention directly between two modalities, equal to direct interaction), respectively. As shown in Tab. 2, we can find that both BrainSubGNN and Trans-Bottleneck have a certain effect on the performance of the model (compare the first three rows in the table). Furthermore, we find that the two modules are compatible and complementary, as our RH-BrainFS method (combining the two modules) achieves the best performance among the ablation study.

Impact of Bottlenecks Number. We then investigate the impact of varying bottlenecks number. Specifically, we run experiments with # bottlenecks=2,4,6,8, respectively, with all other parameters unchanged. In order to ensure the credibility of the experiment, we perform experiments on two datasets (HCP dataset and Two-site dataset). As shown in Fig. 5, on both datasets, the performance of any number of bottlenecks exceeds the baselines. The main reason for this results is that the Trans-Bottleneck module takes into account the regional heterogeneity issue between SC and FC, and avoids the direct interaction of heterogeneous information, thus achieving good fusion performance. Such results demonstrate the importance of the regional heterogeneity issue in the modality fusion process of SC and FC. At the same time, we find that when # bottlenecks=4, the model performance reaches the best, which reflects that only a small number of bottlenecks is needed to achieve a good multi-modal fusion expression capability.

Impact of Subgraph Sampling Hops. Next, we investigate the impact of varying sampling hops of subgraph. In this experiment, we set the range of sampling hops of subgraph from 1 to 5. Likewise, we run experiments on HCP dataset and Two-site dataset. As shown in Fig. 5, it can clearly be seen that as the number of sampling hops increases, the performance of the model generally shows a downward trend, and when the sampling hops are too large, the model actually performs worse than the baselines. The reason for this results is that the brain network itself has strong regional characteristics, and too large sampling hops will cause the sampled subgraph to lose the local characteristics of the brain network and tend towards global characteristics.

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

threshold

70

72

74

76

78

80

accuracy

The results show that only 1 hop of subgraph sampling is needed to express the local characteristics of the brain network well.

Impact of Thresholding Values. Finally, in this paper, thresholding is used to obtain the adjacency matrix of the two modalities during data processing (thresholding X sc and X fc to obtain A sc and A fc , respectively). Thresholding value has been an important ablation study in neuroscience, and the same experiment was conducted in this paper. As shown in Fig. 6, we set the value range of thresholding to [0.02, 0.30], with 0.02 as a step, for a total of 15 thresholding experiments. From the experimental results, there is no clear trend between threshold values and model performance, but the model performs poorly when the threshold value is too high or too low, so in this paper we chose a threshold value of 0.12 as this is when the model performs best.

5 Discussions

Conclusion. In this paper, we introduce a brain subgraph networks and a transformer-based fusion bottleneck to alleviate the issue of regional heterogeneity between SC and FC, and propose a novel multimodal brain networks fusion strategy (RH-BrainFS). To the best of our knowledge, this is the first paper to explicitly state the issue of structural-functional modal regional heterogeneity and to propose a solution. We validate our method on a variety of downstream task datasets, achieving state-of-the-art performance.

Limitations and Future Work. Due to the scarcity and difficulty of collecting and processing data in neuroscience, the only datasets currently available are very limited, despite the fact that we have spent a great deal of manual effort in this area, and thus the research in this paper suffers from a number of possible data bias issues. In our future work, on the one hand, we will do more work on data to mitigate the problem of data bias. On the other hand, although this paper proposes the use of indirect rather than direct interaction, the two may not be mutually exclusive, and in the future we will investigate how to combine the two to achieve better research results.

Ethical Issues. With regard to possible ethical issues in data collection, the Human Connectome Project (HCP) dataset, as a publicly available dataset that has been used in numerous previous studies, is undoubtedly not ethically questionable. It is true that the hospital dataset is held in collaboration with our partner hospitals and is not yet publicly available, but the data is collected with the consent of the subjects who are clearly informed of the purpose of the sample collection, and all identifying information about the sample is hidden in the hospital dataset. Therefore, it does not adversely affect any individual, so there are no ethical or moral issues.

Possible Negative Social Impacts. As the research in this paper deals with the diagnosis of depression, it is necessary to elaborate here on the possible negative social impacts of this work, despite the fact that all the current work is at the stage of scientific research and has not been put to practical use. Including but not limited to:

Incorrect diagnosis. AI methods must have the possibility of error, which cannot be avoided, but an incorrect diagnosis will have a significant impact on individuals and society. Therefore, AI tools can only be used as a diagnostic aid, not as a decision maker, and the final decision should still be made by the doctor.

Leakage of privacy information. In depression dataset, the identity information of the subjects is highly private, and the leakage of identity information will also have unpredictable and significant impact on individuals and society. Therefore, in this work, we have completely hidden the subjects' identifying information (which is also not visible to the staff in the study group) as a way of preventing the leakage of private information.

Acknowledgments and Disclosure of Funding

This work is supported by 2022YFE0116700 National Key Research and Development Program of China, supported by 82271570, 31800825 and 31640028 National Natural Science Foundation of China, and also supported by the Big Data Computing Center of Southeast University. This work is partly supported by grant 2242023k30052 Central University Basic Research Fund of China. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

Kwangyeol Baek, Laurel S Morris, Prantik Kundu, and Valerie Voon. Disrupted resting-state brain network properties in obesity: decreased global and putaminal cortico-striatal network efficiency. Psychological medicine , 47(4):585-596, 2017.

Alaa Bessadok, Mohamed Ali Mahjoub, and Islem Rekik. Graph neural networks in network neuroscience. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Hanshu Cai, Zhidiao Qu, Zhe Li, Yi Zhang, Xiping Hu, and Bin Hu. Feature-level fusion approaches based on multimodal eeg data for depression recognition. Information Fusion , 59:127-138, 2020.

Bokai Cao, Liang Zhan, Xiangnan Kong, Philip S Yu, Nathalie Vizueta, Lori L Altshuler, and Alex D Leow. Identification of discriminative subgraph patterns in fmri brain networks in bipolar affective disorder. In Brain Informatics and Health: 8th International Conference, BIH 2015, London, UK, August 30-September 2, 2015. Proceedings 8 , pages 105-114. Springer, 2015.

Lucy R Chai, Ankit N Khambhati, Rastko Ciric, Tyler M Moore, Ruben C Gur, Raquel E Gur, Theodore D Satterthwaite, and Danielle S Bassett. Evolution of brain network dynamics in neurodevelopment. Network Neuroscience , 1(1):14-30, 2017.

Yuzhong Chen, Jiadong Yan, Mingxin Jiang, Tuo Zhang, Zhongbo Zhao, Weihua Zhao, Jian Zheng, Dezhong Yao, Rong Zhang, Keith M Kendrick, et al. Adversarial learning based nodeedge graph attention networks for autism spectrum disorder identification. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Zhiyi Chen, Xingwang Hu, Qi Chen, and Tingyong Feng. Altered structural and functional brain network overall organization predict human intertemporal decision-making. Human Brain Mapping , 40(1):306-328, 2019.

Brett W Fling, Youngbin Kwak, Scott J Peltier, and Rachael D Seidler. Differential relationships between transcallosal structural and functional connectivity in young and older adults. Neurobiology of aging , 33(10):2521-2526, 2012.

Aina Frau-Pascual, Jean Augustinack, Divya Varadarajan, Anastasia Yendiki, Bruce Fischl, and Iman Aganj. Detecting structural brain connectivity differences in dementia through a conductance model. In 2019 53rd Asilomar conference on signals, systems, and computers , pages 591-595. IEEE, 2019.

Hao Guo, Fan Zhang, Junjie Chen, Yong Xu, and Jie Xiang. Machine learning classification combining multiple features of a hyper-network of fmri data in alzheimer's disease. Frontiers in neuroscience , 11:615, 2017.

Man Guo, Tiancheng Wang, Zhe Zhang, Nan Chen, Yongchao Li, Yin Wang, Zhijun Yao, and Bin Hu. Diagnosis of major depressive disorder using whole-brain effective connectivity networks derived from resting-state functional mri. Journal of Neural Engineering , 17(5):056038, 2020.

Christopher J Honey, Olaf Sporns, Leila Cammoun, Xavier Gigandet, Jean-Philippe Thiran, Reto Meuli, and Patric Hagmann. Predicting human resting-state functional connectivity from structural connectivity. Proceedings of the National Academy of Sciences , 106(6):2035-2040, 2009.

Wenxing Hu, Xianghe Meng, Yuntong Bai, Aiying Zhang, Gang Qu, Biao Cai, Gemeng Zhang, Tony W Wilson, Julia M Stephen, Vince D Calhoun, et al. Interpretable multimodal fusion networks reveal mechanisms of brain cognition. IEEE transactions on medical imaging , 40(5):1474-1483, 2021.

Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning , pages 4651-4664. PMLR, 2021.

Mimansa Jaiswal and Emily Mower Provost. Privacy enhanced multimodal neural representations for emotion recognition. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 7985-7993, 2020.

Biao Jie, Mingxia Liu, Daoqiang Zhang, and Dinggang Shen. Sub-network kernels for measuring similarity of brain connectivity networks in disease diagnosis. IEEE Transactions on Image Processing , 27(5):2340-2353, 2018.

Youyong Kong, Shuwen Gao, Yingying Yue, Zhenhua Hou, Huazhong Shu, Chunming Xie, Zhijun Zhang, and Yonggui Yuan. Spatio-temporal graph convolutional network for diagnosis and treatment response prediction of major depressive disorder from functional connectivity. Human brain mapping , 42(12):3922-3933, 2021.

Youyong Kong, Wenhan Wang, Xiaoyun Liu, Shuwen Gao, Zhenghua Hou, Chunming Xie, Zhijun Zhang, and Yonggui Yuan. Multi-connectivity representation learning network for major depressive disorder diagnosis. IEEE Transactions on Medical Imaging , 2023.

Tommaso Lanciano, Francesco Bonchi, and Aristides Gionis. Explainable classification of brain networks via contrast subgraphs. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 3308-3318, 2020.

Jingcong Li, Fei Wang, Jiahui Pan, and Zhenfu Wen. Identification of autism spectrum disorder with functional graph discriminative network. Frontiers in Neuroscience , page 1282, 2021.

Xiaoxiao Li, Yuan Zhou, Nicha Dvornek, Muhan Zhang, Siyuan Gao, Juntang Zhuang, Dustin Scheinost, Lawrence H Staib, Pamela Ventola, and James S Duncan. Braingnn: Interpretable brain graph neural network for fmri analysis. Medical Image Analysis , 74:102233, 2021.

Sol Lim, Filippo Radicchi, Martijn P van den Heuvel, and Olaf Sporns. Discordant attributes of structural and functional brain connectivity in a two-layer multiplex network. Scientific Reports , 9(1):2885, 2019.

Manhua Liu, Danni Cheng, Kundong Wang, Yaping Wang, and Alzheimer's Disease Neuroimaging Initiative. Multi-modality cascaded convolutional neural networks for alzheimer's disease diagnosis. Neuroinformatics , 16:295-308, 2018.

Vasileios Mantzavinos and Athanasios Alexiou. Biomarkers for alzheimer's disease diagnosis. Current Alzheimer Research , 14(11):1149-1154, 2017.

Changping Meng, S Chandra Mouli, Bruno Ribeiro, and Jennifer Neville. Subgraph pattern neural networks for high-order graph evolution prediction. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems , 34:14200-14213, 2021.

Jonathan D Power, Alexander L Cohen, Steven M Nelson, Gagan S Wig, Kelly Anne Barnes, Jessica A Church, Alecia C Vogel, Timothy O Laumann, Fran M Miezin, Bradley L Schlaggar, et al. Functional network organization of the human brain. Neuron , 72(4):665-678, 2011.

Jonathan D Power and Steven E Petersen. Control-related systems in the human brain. Current opinion in neurobiology , 23(2):223-228, 2013.

Gang Qu, Li Xiao, Wenxing Hu, Junqi Wang, Kun Zhang, Vince D Calhoun, and Yu-Ping Wang. Ensemble manifold regularized multi-modal graph convolutional network for cognitive ability prediction. IEEE Transactions on Biomedical Engineering , 68(12):3564-3573, 2021.

Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience , 22(11):1761-1770, 2019.

Mikail Rubinov and Olaf Sporns. Complex network measures of brain connectivity: uses and interpretations. Neuroimage , 52(3):1059-1069, 2010.

Isaac Sebenius, Alexander Campbell, Sarah E Morgan, Edward T Bullmore, and Pietro Liò. Multimodal graph coarsening for interpretable, mri-based brain graph neural network. In 2021 IEEE 31st International Workshop on Machine Learning for Signal Processing (MLSP) , pages 1-6. IEEE, 2021.

Mohammed Senoussaoui, Milton Sarria-Paja, João F Santos, and Tiago H Falk. Model fusion for multimodal depression classification and level detection. In Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge , pages 57-63, 2014.

Preya Shah, Arian Ashourvan, Fadi Mikhail, Adam Pines, Lohith Kini, Kelly Oechsel, Sandhitsu R Das, Joel M Stein, Russell T Shinohara, Danielle S Bassett, et al. Characterizing the role of the structural connectome in seizure dynamics. Brain , 142(7):1955-1972, 2019.

Xiao-Wei Song, Zhang-Ye Dong, Xiang-Yu Long, Su-Fang Li, Xi-Nian Zuo, Chao-Zhe Zhu, Yong He, Chao-Gan Yan, and Yu-Feng Zang. Rest: a toolkit for resting-state functional magnetic resonance imaging data processing. PloS one , 6(9):e25031, 2011.

Olaf Sporns, Christopher J Honey, and Rolf Kötter. Identification and classification of hubs in brain networks. PloS one , 2(10):e1049, 2007.

Laura E Suárez, Ross D Markello, Richard F Betzel, and Bratislav Misic. Linking structure and function in macroscale brain networks. Trends in cognitive sciences , 24(4):302-315, 2020.

Heung-Il Suk, Seong-Whan Lee, Dinggang Shen, Alzheimer's Disease Neuroimaging Initiative, et al. Hierarchical feature representation and multimodal fusion with deep learning for ad/mci diagnosis. NeuroImage , 101:569-582, 2014.

Qingyun Sun, Hao Peng, Jianxin Li, Jia Wu, Yuanxing Ning, Philip S. Yu, and Lifang He. SUGAR: subgraph neural network with reinforcement pooling and self-supervised mutual information mechanism. CoRR , abs/2101.08170, 2021.

Jewell B Thomas, Matthew R Brier, Mario Ortega, Tammie L Benzinger, and Beau M Ances. Weighted brain networks in disease: centrality and entropy in human immunodeficiency virus and aging. Neurobiology of aging , 36(1):401-412, 2015.

Lixia Tian, Jinhui Wang, Chaogan Yan, and Yong He. Hemisphere-and gender-related differences in small-world brain networks: a resting-state functional mri study. Neuroimage , 54(1):191-202, 2011.

Martijn P van den Heuvel, René CW Mandl, Cornelis J Stam, René S Kahn, and Hilleke E Hulshoff Pol. Aberrant frontal and temporal complex network structure in schizophrenia: a graph theoretical analysis. Journal of Neuroscience , 30(47):15915-15926, 2010.

David C Van Essen, Stephen M Smith, Deanna M Barch, Timothy EJ Behrens, Essa Yacoub, Kamil Ugurbil, Wu-Minn HCP Consortium, et al. The wu-minn human connectome project: an overview. Neuroimage , 80:62-79, 2013.

Laura S Van Velzen, Sinead Kelly, Dmitry Isaev, Andre Aleman, Lyubomir I Aftanas, Jochen Bauer, Bernhard T Baune, Ivan V Brak, Angela Carballedo, Colm G Connolly, et al. White matter disturbances in major depressive disorder: a coordinated analysis across 20 international cohorts in the enigma mdd working group. Molecular psychiatry , 25(7):1511-1525, 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Joshua T Vogelstein, William Gray Roncal, R Jacob Vogelstein, and Carey E Priebe. Graph classification using signal-subgraphs: Applications in statistical connectomics. IEEE transactions on pattern analysis and machine intelligence , 35(7):1539-1551, 2012.

Nina Vogt. Machine learning in neuroscience. Nature Methods , 15(1):33-33, 2018.

Bo Wang, Aziz M Mezlini, Feyyaz Demir, Marc Fiume, Zhuowen Tu, Michael Brudno, Benjamin Haibe-Kains, and Anna Goldenberg. Similarity network fusion for aggregating data types on a genomic scale. Nature methods , 11(3):333-337, 2014.

Lu Wang, Feng Vankee Lin, Martin Cole, and Zhengwu Zhang. Learning clique subgraphs in structural brain network classification with application to crystallized cognition. NeuroImage , 225:117493, 2021.

Asiri Wijesinghe and Qing Wang. A new perspective on" how graph neural networks go beyond weisfeiler-lehman?". In International Conference on Learning Representations , 2022.

Qiong Wu, Xiaoqi Huang, Adam J Culbreth, James A Waltz, L Elliot Hong, and Shuo Chen. Extracting brain disease-related connectome subgraphs by adaptive dense subgraph discovery. Biometrics , 78(4):1566-1578, 2022.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

Dogucan Yaman, Fevziye Irem Eyiokur, and Hazim Kemal Ekenel. Multimodal age and gender classification using ear and profile face images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 0-0, 2019.

Chao-Gan Yan, Xiao Chen, Le Li, Francisco Xavier Castellanos, Tong-Jian Bai, Qi-Jing Bo, Jun Cao, Guan-Mao Chen, Ning-Xuan Chen, Wei Chen, et al. Reduced default mode network functional connectivity in patients with recurrent major depressive disorder. Proceedings of the National Academy of Sciences , 116(18):9078-9083, 2019.

Haoteng Yin, Muhan Zhang, Yanbang Wang, Jianguo Wang, and Pan Li. Algorithm and system co-design for efficient subgraph-based graph representation learning. arXiv preprint arXiv:2202.13538 , 2022.

Usman Zahid, Imran Ashraf, Muhammad Attique Khan, Majed Alhaisoni, Khawaja M Yahya, Hany S Hussein, and Hammam Alshazly. Brainnet: optimal deep learning feature fusion for brain tumor classification. Computational Intelligence and Neuroscience , 2022, 2022.

Lu Zhang, Li Wang, Jean Gao, Shannon L Risacher, Jingwen Yan, Gang Li, Tianming Liu, Dajiang Zhu, Alzheimer's Disease Neuroimaging Initiative, et al. Deep fusion of brain structurefunction in mild cognitive impairment. Medical image analysis , 72:102082, 2021.

Tao Zhang and Mingyang Shi. Multi-modal neuroimaging feature fusion for diagnosis of alzheimer's disease. Journal of Neuroscience Methods , 341:108795, 2020.

Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. arXiv preprint arXiv:2110.03753 , 2021.

Lulin Zheng, Ruidong Yang, Junbo Gao, Jun Chen, Jianzhong Liu, and Depeng Li. Quartz rb-sr isochron ages of two type orebodies from the nibao carlin-type gold deposit, guizhou, china. Minerals , 9(7):399, 2019.

Guangming Zhu, Bin Jiang, Liz Tong, Yuan Xie, Greg Zaharchuk, and Max Wintermark. Applications of deep learning to neuro-imaging techniques. Frontiers in neurology , 10:869, 2019.

Qi Zhu, Jing Yang, Shuihua Wang, Daoqiang Zhang, and Zheng Zhang. Multi-modal noneuclidean brain network analysis with community detection and convolutional autoencoder. IEEE Transactions on Emerging Topics in Computational Intelligence , 2022.

Appendix A: Full Results of Comparison Experiments

Due to the layout of the text and page limitations, only the experimental results for the ACC metric are presented in the main text. However, to ensure a comprehensive presentation of the results, the full results of the comparison experiments involving all four datasets, HCP, Zhongda Hospital, Xinxiang Hospital and Two-site, are presented in Tables 1 to 4. Four tables are presented detailing the mean and standard deviation values for each evaluation metric. The most exceptional results are shown in bold, while results below the optimal threshold are underlined for clarity and emphasis.

Analysis. Based on the results from the four tables, our RH-BrainFS method achieved excellent results in all four datasets. Specifically, we achieved the best results on three metrics in the HCP dataset, four metrics in both the Zhongda and Xinxiang Hospital datasets, and an impressive five metrics in the Two-site dataset. This demonstrates the strong performance of RH-BrainFS in effectively completing the fusion classification task for multimodal brain networks. Furthermore, we observe that on the hospital dataset, our RH-BrainFS method significantly outperforms most other methods in terms of bias in each metric, indicating higher stability. This improved stability contributes to the reliability and robustness of RH-BrainFS in fusion classification tasks.

Appendix B: Full Results of Hyperparameter Experiments

In the main text, our hyperparameter experiments are performed on the HCP dataset and the Two-site dataset. We explore two key hyperparameters in RH-BrainFS: the number of bottlenecks and the number of subgraph sampling hops k. However, in the main text we have only presented visualisations of the results of the hyperparameter experiments without giving specific numerical values. Hence, we present here the complete experimental results in Tables 5 to 8.

Peer reviews:
comments: SUMMARY:
The submission is not in my area, and it's difficult to give reasonable comments about this submission. My research interests focus on medical image reconstruction. please find another appropriate reviewer to review this paper.

STRENGTHS:
N/A

WEAKNESSES:
N/A

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach called the Regional Heterogeneous Multimodal Brain Networks Fusion Strategy (RH-BrainFS) to address the issue of regional heterogeneity between structural connectivity (SC) and functional connectivity (FC) in brain networks fusion. The proposed approach includes a brain subgraph networks module to extract regional characteristics of brain networks and a transformer-based fusion bottleneck module to alleviate the issue of regional heterogeneity between SC and FC. The authors claim that this is the first work to propose a solution to the issue of structural-functional modal regional heterogeneity. The paper presents extensive experiments that demonstrate that the proposed method outperforms several state-of-the-art methods in a variety of neuroscience tasks.\n\nSTRENGTHS:\n- The paper is well-structured and clearly written. The authors provide a clear introduction to the problem, a thorough review of related work, and a detailed description of their proposed method. They also clearly explain the experimental settings and datasets used in their research. The use of technical terms is appropriate for the intended audience, and the authors provide sufficient context and explanation to make the content understandable.\n- The issue of regional heterogeneity in multimodal brain networks is a significant challenge in the field, and the authors' proposed solution has the potential to improve the performance of multimodal fusion models. \n- The authors provided detailed preprocessingand implementation parameters.\n\nWEAKNESSES:\n- It is important for the authors to compare their proposed approach with existing methods in the field of multimodal brain network modeling. One such method is the joint embedding of structural and functional brain networks with graph neural networks proposed by Zhu et al.. The authors should compare their proposed approach with this method in terms of performance for modality fusion.\n- The transformer-based fusion is not original in terms of modality fusion. The authors are suggested to investigate and provide concrete interpretations on the fusion bottlenecks which utilize shared characteristics between modalities.\n- Lack of variant on GNN backbone. There are a bunch of GNN architectures for brain network modeling. While the paper acknowledges the use of a BrainSubGNN method, it would be valuable to consider other GNN architectures that have been developed specifically for brain network analysis. By incorporating a broader range of GNN backbones and comparing their performance, the authors can enhance the robustness and generalizability of their proposed approach for multimodal brain network fusion.\n- Lack of interpretation analysis. In the context of multimodal brain network modeling, interpretation analysis could involve identifying the brain regions or networks that are most relevant for the classification task, and investigating the functional and structural connectivity patterns that contribute to these regions.\n\nQUESTIONS:\n- How does the proposed approach compare to other state-of-the-art methods in terms of computational efficiency? Is the proposed approach computationally feasible for large-scale datasets?\n- The paper mentions that the proposed approach includes a brain subgraph networks module to extract regional characteristics of brain networks. Can the authors provide more details on the design motivation this module, as well as its neurological justification?\n- How would the subgraph sampling affect the effectiveness of the overall framework? \n- Would there be any potential modality bias in the experimented data? \n- Could you elaborate on how woyld the different brain imaging techniques mentioned in the text, such as functional magnetic resonance imaging (fMRI) and diffusion magnetic resonance imaging (dMRI)? reflect different aspects of the brain's internal characteristics?\n\nLIMITATIONS:\nThe authors have acknowledged some of the limitations of their work, such as the limited sample size and the difficulty of data collection in neuroscience. However, the paper does not provide a detailed discussion on the technical bias, e.g., modality bias during fusion of the proposed approach, nor does it address potential negative societal impact of the research. Adding those discussions would enhance the paper's credibility and relevance, and help readers better understand the potential implications of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper identifies a gap in the literature of multimodal brain networks fusion in which current methods are said to only use \"simple patterns\" to fuse modalities, ie, concatenation, weighted summation, and self-attention. To tackle this issue, the paper proposes RH-BrainFS, a new model fusing structural connectivity and functional connectivity data constructed from fMRI and dMRI, respectively. This model includes a BrainSubGNN model for subgraph sampling and subgraph embedding processes, as well as a fusion bottleneck based on transformers.\n\nSTRENGTHS:\nThe results of the experiments (as seen in table 1) seem to be particularly strong, and this work is very significant given the lack of varied work in the field of multimodal brain imaging fusion. It is very good that the authors use more traditional ML models beyond deep learning (ie, SVM and random forests) to evaluate the relevance of this work, and that several metrics are considered beyond the simple accuracy metric. Finally, it usually takes a lot of time and manual work to preprocess dMRI data to be in the same parcellation as the fMRI data, thus the fact that this data is not available out-of-the-box clearly adds to the significance of this applied work - I hope the authors will make this preprocessed dataset available at a certain point.\n\n\n---------------------------------\nEDIT AFTER REBUTTAL PERIOD:\n\nFrom the rebuttal period, the authors did a very good job in addressing most of my concerns, and therefore I'm increasing my decision from borderline accept to weak accept, the soundness score from 2 to 3, and the presentation score from 3 to 4.\n\nWEAKNESSES:\nI identify three main weaknesses that require clarification during the rebuttal process and are the reason for me to give a borderline accept despite the strengths. I will be happy to review my score as a result of the rebuttal process if my points are properly tackled. I number my comments for easier discussion.\n\n\n1. A key weakness of this work is that it seems to me that there's an overstatement when the authors write that this is the first work to propose a solution to the issue of strutural-functional model regional heterogeneity. Maybe this issue is ill-defined and need a more clear description, otherwise I don't see how some of the previous works mentioned do not try to create models that take into account different functional-structural interactions on different regions of the brain. For example, work [59] (mentioned in introduction as an example of \"self-attention\" technique) seems to use attention layers at different layers of the model and due to its (attention) nature, it seems too much to call it a \"simple pattern\". Similarly, for work [58], I believe the authors are oversimplifying it by classifying it as just a \"weighted summation\", as it seems to me to be a much more complex usage of graph neural networks (GNNs) to fuse in a non-linear fashion different modalities of the brain. Indeed, work [58] seems to follow other works that I've seen leveraging GNNs to fuse different brain modalities (eg, 10.48550/arXiv.2007.09777, 10.1016/j.media.2022.102471, 10.1007/978-3-030-32248-9_89) which the paper did not mention and for me they are clearly tackling the issue of funtional-structural interactions even if they did not use this exact name. Finally, even for the simplest case of just concatenating (embedded) features it doesn't mean a model is making a \"one-to-one mapping\" as defended in the paper.\n2. A second key weakness I see in this paper is how some decisions seem to be justified/made in section 3.3.1. Section 3.3.1 starts with \"Due to the issue of regional heterogeneity between SC and FC, it is obviously not feasible to pass messages directly between two modalities\". What do the authors mean by \"messages\" here and why is this so obvious? Wouldn't \"messages\" between the different modalities actually help in the issue of not just concatenate two modalities? This seems to be related to lines 187-189, when the paper says that it avoids direct interactions between modalities. Why is direct communication such a bad thing and even called a \"forbidden\" interaction as indicated in figure 3? It seems to me that direct communication, together with other means of \"communication\" could help any model achieve better performance. Could maybe the authors justify this point a bit better? A possible ablation analysis I see here to support this statement would be to remove Z_b altogether, and this would imply that just one transformer would be required to fuse functional/structural data, greatly reducing the complexity of the model for (maybe) a similar performance.\n3. The final key weakness I have to identify in this paper is that it's not clear to me whether the results are overly-optimistic, and I will appreciate any clarification the authors can provide on this if I misunderstood this point. The paper seems to use a single train/validation set (for each fold of the 10-fold CV procedure), therefore the evaluations in table 1 seem to be made on the validation set. It also seems the case that this same validation set is used to know when to stop the training procedure. If I understood this correctly, then the results in table 1 can be overly-optimistic, as the same data used in training (eg, to know when to stop) was also used for final evaluation.\n\n(Small typo, in Line 203, should not capitalise \"We\")\n\nQUESTIONS:\nBeyond what I wrote in the \"Weaknesses\" section, I have a few more questions:\n\n1. Am I correct to understand that both the functional and structural modality need to be parcellated with the same atlas? In figure 1's representation I got the impression that the two graphs could be different, but then from the text it doesn't seem that this is the case. Can the authors please clarify this?\n2. Have the authors considered analysing different choices for the readout function to check for possible result improvements?\n3. Have the authors considered other tasks beyond binary classification? For example a regression-based task like age prediction?\n4. Around line 249, if the feature matrix of the structural data is of shape |V_{SC}| x |V_{SC}|, doesn't it mean each node will have repeated features given the symmetric nature of the matrix? \n5. Around lines 259/260, I understood that the data is split in 50% train data and 50% validation data. If this is true, can the authors clarify why this was the case? In our field it is more common to have a higher percentage for the train data, and that's why I ask this.\n6. Although it is good that the paper contains traditional ML models as baselines (ie, SVM, random forest), why didn't the authors choose the traditional methods mentioned in the Related Work section?\n7. Work [59] seems to be the only previous work mentioned by the authors that use self attention. Given the model proposed by the authors highly relies on Transformers, shouldn't this work be included in the baselines comparison?\n8. Do the authors have any hypothesis on why the results of the two-site brings worse results than the two datasets separately? Could it be the way the two sites are separated when splitting the data in train/val sets? (I am assuming more data usually means better generability)\n\nLIMITATIONS:\nThe paper highlights some limitations, but no potential negative societal impact of work seems to have been mentioned. The fact that the model uses medical data, and that it can be used to predict depression, seems to me to be potentially be used in negative ways with direct impacts in someone's life?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposes a novel regional heterogeneous multimodal brain networks fusion strategy to alleviate the issue of regional heterogeneity of multimodal brain networks. This strategy uses a graph convolutional network for the extraction of initial features of nodes (brain region from AAL atlas) and a transformer-based fusion bottleneck module for the fusion of structural connectivity matrix and functional connectivity matrix. This fusion strategy achieved the best results on the tasks of gender classification and depression classification.\n\nSTRENGTHS:\n1. This work is well written and the figure is clear, making it easy for the reader to understand what has been done.\n    2. This work is of great significance. How the structural and functional networks of the brain fusion have been an open question because the understanding of how the anatomical constraint is related to the elaborate functions is nevertheless fragmentary. Addressing the fusion of brain structure and function from a deep learning perspective can help facilitate understanding of the relationship between structure and function.\n\nWEAKNESSES:\n1. The authors state \"alleviate the regional heterogeneity between multimodal brain networks\". This is a very ambitious question, and the high classification accuracy alone does not account for the alleviation of inter-regional heterogeneity. The authors should do detailed analysis experiments based on brain regions on the basis of high accuracy to explore which brain regions are non-heterogeneous between modalities, which brain regions are heterogeneous between modalities, and whether removing these heterogeneous brain regions can achieve better classification results.\n    2. The authors state \"they were inspired by MBT, so how does your work differ from MBT\"? It seems that the author's thinking is to not allow interaction of tokens between structure and function, why is that, shouldn't structure and function be meant to interact with each other? The authors say that it effectively improves the performance of the model (line 189), but I can't see from the experiments that there are results to prove this (Transformer with bottleneck token rather than w/o Trans-Bottleneck). The authors do not compare it with MBT's approach to prove the effectiveness of their proposed method.\n    3. The authors state in line 257 that the initial features of the function (Xfc) are averaged time series, but the Xfc shown in Figure 1 is the function connectivity matrix. These two points are contradictory. And, the initial features of the structure are the structural connectivity matrix, so why should the initial features of the function be the time series of fMRI instead of the functional connectivity matrix?\n    4. Are the Final Bottlenecks obtained based on Zb only? If so, why not cascade Zsc, Zb, Zfc as inputs to the MPL layer? The authors need to do ablation studies to prove that Zb-based classification is the most accurate. What does Nb in Equation 4 mean?\n    5. How is the threshold taken for the connectivity matrix to the adjacency matrix for the two modes? Firstly, the authors didn't write how much the threshold is taken. Secondly, the choice of the threshold for the structural and functional connectivity matrices has a big impact on the downstream task, it is suggested to do ablation studies to show how you take the threshold.\n\nQUESTIONS:\nSee Weaknesses for details.\n\nLIMITATIONS:\nSee Weaknesses for details", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article discusses the use of multimodal fusion as a research technique in neuroscience to extract complementary information from multiple modalities. Since previous research has neglected the regional heterogeneity between structural connectivity (SC) and functional connectivity (FC) and used inefficient ways of multimodal fusion. To address this issue, the authors proposed a novel Regional Heterogeneous multimodal Brain networks Fusion Strategy (RH-BrainFS), which uses a brain subgraph networks module to extract regional characteristics and a transformer-based fusion bottleneck module to alleviate the regional heterogeneity between SC and FC. The proposed method outperforms several state-of-the-art methods in various neuroscience tasks and is the first work to propose a solution to the issue of structural-functional modal regional heterogeneity.\n\nSTRENGTHS:\nFirst the research problem is significant. The structural-functional modal regional heterogeneity is a recent popular research topic. The authors first propose a method which could work well in this situation. The novelty of the paper is sufficient. To alleviate the issue of regional heterogeneity of multimodal brain networks, the authors propose a novel Regional Heterogeneous multimodal Brain networks Fusion Strategy (RH-BrainFS), using BrainSubGNN module and Trans-Bottleneck module to fuse regional heterogeneous multimodal brain networks for neuroscience tasks.The experiments are sufficient, the authors conduct the experiment on two main-stream MRI brain benchmarks. Extensive experiments demonstrate the effectiveness of RH-BrainFS in multimodal brain networks fusion tasks on depression classification and gender classification datasets.\n\nWEAKNESSES:\n1. authors have not explained in detail why using \"forbidden Interaction\" in the Fusion bottlenecks. Authors are suggested to provide more motivations and explanation of this part.\n2. authors have not provide the detailed speed and time-consumption comparison with other methods. The proposed method seems time-consuming.\n\nQUESTIONS:\n1. the authors use thresholding to get Asc and Afc from Xsc and Xfc, how the selection of thresholding effect the final results? how sensitive the results are.\n\nLIMITATIONS:\nsee the weakness part", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}