RH-BrainFS: Regional Heterogeneous Multimodal Brain Networks Fusion Strategy
Authors: Hongting Ye
Yalu Zheng
Yueying Li
Ke Zhang
Youyong Kong
Yonggui Yuan
Venue: NeurIPS 2023 poster (2023)
Keywords: Multimodal
Neuroscience
Subgraph
Transformer

Abstract: Multimodal fusion has become an important research technique in neuroscience that completes downstream tasks by extracting complementary information from multiple modalities. Existing multimodal research on brain networks mainly focuses on two modalities, structural connectivity (SC) and functional connectivity (FC). Recently, extensive literature has shown that the relationship between SC and FC is complex and not a simple one-to-one mapping. The coupling of structure and function at the regional level is heterogeneous. However, all previous studies have neglected the modal regional heterogeneity between SC and FC and fused their representations via "simple patterns", which are inefficient ways of multimodal fusion and affect the overall performance of the model. In this paper, to alleviate the issue of regional heterogeneity of multimodal brain networks, we propose a novel Regional Heterogeneous multimodal Brain networks Fusion Strategy (RH-BrainFS). Briefly, we introduce a brain subgraph networks module to extract regional characteristics of brain networks, and further use a new transformer-based fusion bottleneck module to alleviate the issue of regional heterogeneity between SC and FC. To the best of our knowledge, this is the first paper to explicitly state the issue of structural-functional modal regional heterogeneity and to propose a
solution. Extensive experiments demonstrate that the proposed method outperforms several state-of-the-art methods in a variety of neuroscience tasks.

RH-BrainFS: Regional Heterogeneous Multimodal Brain Networks Fusion Strategy
Hongting Ye Yalu Zheng ' Yueying Li Ke Zhang Youyong Kong 1.2 Yonggui Yuan"
Jiangsu Provincial Joint International Research Laboratory of Medical Information Processing
School of Computer Science and Engineering, Southeast University Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China Department of Psychosomatics and Psychiatry, Zhongda Hospital
School of Medicine, Southeast University
{yehongting, 220212084, 230228504, kylenz, kongyouyongkaseeu.com yygylh20000sina.com
Abstract
Multimodal fusion has become an important research technique neuroscience that completes downstream tasks by extracting complementary information from multiple modalities. Existing multimodal research on brain networks mainly focuses modalities, structural connectivity (SC) and functional connectivity (FC). Recently, extensive literature has shown that the relationship between SC and FC is complex and a simple one-to-one mapping. The coupling structure and function at the regional level is heterogeneous. However, all previous studies have neglected the modal regional heterogeneity between SC and FC and fused their representations via "simple patterns", which are inefficient ways of multimodal fusion and affect the overall performance of the model. In this paper, to alleviate the issue of regional heterogeneity of multimodal brain networks, we propose a novel Regional Heterogeneous multimodal Brain networks Fusion Strategy (RH- BrainFS) Briefly, we introduce brain subgraph networks module to extract regional characteristics of brain networks, and further use new transformer-base fusion bottleneck module to alleviate the issue of regional heterogeneity between SC and FC. To the best of our knowledge, this is the first paper to explicitly state the issue of structural -inucttoallunntiionallll modal regional heterogeneity and to propose solution. Extensive experiments demonstrate that the proposed method outperforms several state-of-fe of-te-of-the-att
the-art art methods in variety of neuroscience tasks.
1 Introduction
Currently, a large number of neuroscience studies are based on unimodal imaging 2 30 47
56
61]. However, different brain imaging techniques, such functional magnetic resonance imaging (IMRI) (54) and diffusion magnetic resonance imaging (dMRI) [44), reflect different aspects of the brain's internal characteristics. Therefore, it is often insufficient
a single modality of data for neuroscience research and it is necessary to integrate multiple modalities of imaging data to achieve good performance in neuroscience tasks such as depression classification classification 53).
and gender
In multimodal brain networks fusion, existing researchs mainly focused on fusing structural and functional modalities (structural modality is constructed from dMRI and functional modality is
"Corresponding author
"The codes are available at wttp://iwww...oeoeaaaaaaal///
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

constructed from fMRI) 6 29 32 57 58). Most methods directly fuse the two modal representations
via "simple patterns" (we define this as direct interaction, where two modal features/embecddngs
are directly combined to perform some computation, e.g. concatenation [32 weighted summation
[57], or self-attention (58) techniques.) without considering the issue of regional heterogeneity (37] between this two modalities. However, extensive literature (22 37 has shown that the relationship between structural connectivity (SC) and functional connectivity (FC) is complex and not a simple one-to-one mapping. Specifically, the coupling of structure and function at the regional level is heterogeneous and follows the molecular, cellular and functional hierarchical structure. In other words, structure may be more tightly coupled to function in some regions than in others. This shows that regional heterogeneity is a key factor in linking different modal brain networks.
Based on the above research gap, in this paper we propose a novel regional heterogeneous multimodal brain networks fusion strategy that aims to fully account for the regional heterogeneity among brain networks and achieve better multimodal brain networks fusion performance.
First, the brain network itself has strong regional characteristics [36] 41l, and the combination of neighbouring brain regions can serve as an important criterion for neuroscience tasks [16, 24). Specifically, regional characteristies behave subnetwork (also called subgraph) features in the whole brain network [2]. yet other multimodal studies do not take this into account. In this paper, we focus on the characteristics of brain regions from the subgraph pattern. The subgraph pattern have become a relatively hot area of graph representation learning research in recent years [25] 39 [55]. Subgraph convolutional networks can obtain potential representations of each subgraph in the graph. that reflect the regional characteristics of the graph 7 10] Therefore, to effectively extract the regional characteristics of each brain region in the brain network, we introduce a Brain Subgraph Networks (BrainSubGNN) module in Sec. 3.2 Our BrainSubGNN is divided into two steps, subgraph sampling and subgraph embedding. The former is used to obtain the subgraph partition of the brain network, the latter to aggregate the subgraph characteristics of the brain network.
Next, since several previous studies have shown that the characteristics of different individual regions in the brain network have widely different influences on neuroscience tasks [] 40 42], we need to pay attention to the influence of these brain region characteristics on neuroscience tasks and quantify them into accurate values. In this paper. we focus on the Transformer 45 to learn the accurate influence values of these brain region characteristics. Although originally proposed for NI.P tasks, there has been recent interest in Transformers [55 as universal perceptual models [14]. Through the attention mechanism, Transformer can learn the accurate influence values of different tokens well for the classification result, which happens to meet our needs.
However, as mentioned above, there is the issue of modal regional heterogeneity between SC and FC, which is key factor in linking these two modal brain networks (37). Many previous multimodal fusion methods, via "direct interaction", have not considered this issue between SC and FC. which are inefficient ways of modality fusion and affect the overall performance of the model. Based on that, avoiding direct interaction between two modalities within the Transformer is the aim of our study. Inspired by MBT (26). we present transformer-basee Trran-Botttennee module for fusing regional heterogeneous brain networks in Sec. B.3 Specifically, the
fusion bottleneck
45 and certain number of fusion bottlenecks. The standard transformers are used to learn the accurate influence values of each brain region on neuroscience tasks through the attention mechanism. The fusion bottlenecks, as intermediate media for modality fusion, establish connections between regional heterogeneous modalities and learn the key information of each modality in the latent space (we define this as indirect interaction, where two modal features/embeddings are not directly combined for the computation.).
module contains two standard transformers
The main contributions of this paper are summarized as follows:
To alleviate the issue of regional heterogeneity of multimodal brain networks, we propose novel Regional Heterogeneous multimodal Brain networks Fusion Strategy (RH--rainFSS) using BrainSubGNN module and Trans-Bottleneck module to fuse regional heterogeneous multimodal brain networks for neuroscience tasks.
To the best of our knowledge, this the first paper to explicitly state the issue of structural- functional modal regional heterogeneity and to propose a solution.
Extensive experiments demonstrate the effectiveness of RH-BrainFS in multimodal brain networks fusion tasks depression classification and gender classification datasets.
2

---

Madullty
Medally
Initill
oood
SP
SP
IMER
ONI
acen Pal
Xxc
renkelingg
DONO
oooo
0O00
0000
ML.P
212
0OOO
OODO
Aveosver Eat
Court Daayee
Ap
Figure 1: The overall framework our proposed RH-BrainF5s Model. The structural connectivity (SC) and functional connectivity (FC) are constructed from fMRI and dMRI respectively, the initial bottlenecks Zy are a set of learnable tokens randomly sampled from standard normal distribution, and the subgraph partition (SP.) are obtained in the preprocessing stage. Function g(-) denotes the BrainSubGNN module.
2
Related Work
Brain Subgraph Networks: In recent years, subgraph techniques have gained popularity in brain network analysis due to their ability to accurately model certain aspects of brain organization with high consistency to established brain functional systems (27). This has led to a focus on identifying informative and signaling subgraphs from the entire brain connectome that may be relevant to brain diseases [46]. Various subgraph-ban based methods have been proposed for brain network research, such as the adaptive dense subgraph discovery (ADSD) model [51] which uses a likelihood- based approach to extract disease-ant associated subgraphs from group-level whole-brain connectome data. Other methods, such as an earlier approach from 4 balance topological information of local and
global graphs using subgraphs, while approach [5]
utilizes subgraphs represent local features
19 has also classified brain network by extracting contrastive subgraphs, and the SBLR model 44 suggests that subgraphs have attractive neurological interpretations and may correspond to outcome-related anatomical circuits. In this paper, our model builds upon these approaches and employs BrainSubGNN module efficiently aggregate regional characteristics of brain network.
in large-scale brain network. Recent research
MultiModal Brain Networks Fusion: Over the past few years, several methods have been developed for multimodal fusion in neuroscience research [12 & 38
23 60]. One traditional approach, SNF 488. creates initial similarity network for each feature and iteratively combines them with non-linear graph fusion formula to generate a final fusion network. However, recent advancements in deep learning have led the development of more sophisticated multimodal fusion techniques. For instance, the GBDM [57] model employs both structural and functional information from diffusion and functional magnetic resonance imaging (MRI), respectively, effectively differentiate individuals with mild cognitive impairment (MCI) from age-matched controls. The MGCN (29) model uses manifold-based regularization terms account for inter-modality and intra-modality relationships. One approach [3] involves deep collaborative learning to capture cross- cross-modal associations and trait-related features. Another approach [32 combines representations
fuse multimodal data. Moreover, a method [62] applies a multimodal non-Euclidean brain network analysis technique based on community detection and convolutional autoencoder for epilepsy classification. Additionally, new adversarial learning-based node-edge graph attention network (AL-NEGAT (6) has been proposed for autism spectrum disorder (ASD) recognition based on multi-modal MRI data. However, none of these methods have addressed the issue of modal regional heterogeneity 33 in multimodal

---

brain networks. To fill this gap, we propose new multimodal brain networks fusion strategy aimed alleviating the modal regional heterogeneity issue and achieving better fusion performance.
3
Method
In this section, we present our proposed regional heterogeneous multimodal brain networks fusion strategy RH-BrainIFS (as shown in Fig. T.. We begin by discussing the definition of the multimodal brain networks fusion task in Sec. 3.1 We then explain how the BrainSubGNN module captures the regional characteristics of brain networks in Sec. [.2] Finally, we describe how the nuns-Bottleeeck module used in the RH-Bre RH-BrainF'S model alleviate the issue of regional heterogeneity in multimodal brain networks in Sec. 3.3
3.1 Perliminaries
Multimodal Brain Networks Fusion Task: Given a multimodal brain networks dataset D (Sampley, Sample1, Samplez-1), where each sample represents person, and L is the size of this dataset. Each Sample; = (g 9fcs contains a structural connectivity graph (each brain region viewed node in the graph), functional connectivity graph (ditto) and class label y {0. 1) (0, represent different meanings on different tasks). Each kind of input graph g = (A, X, V) is composed of a node set V, an adjacency matrix A E R N and node features X R Nxd where N = |V denotes the number of nodes and d denotes the input dimensional of node features. In the brain network graph, node set denotes the collection brain regions, adjacency matrix denotes the connectivity between various brain regions and node features X denotes the original features of each brain region. In the multimodal brain networks classification task, the purpose to find a mapping function g (Gse: Gfe) y-
3.2 Brain Subgraph Networks
We now explain how BrainSubGNN module captures the regional characteristics of brain networks. As shown in Fig. 2 the BrainSubGNN contains subgraph sampling step and subgraph embedding step.
3.2.1 Subgraph sampling
The purpose of subgraph sampling is to construct a receptive field for each brain region (also named node graph), it represent the regional characteristics this brain region. In our RH-Bas HH-BrainFS,, rooted subgraph [59] is utilized to construct receptive field, as the rooted subgraph can exhibit even greater discriminative power than the first-order Weisfeiler-Leman (1-WL) test due to interconnectivity
among neighboring nodes 50 59
Subgraph Sampling (Preprocess Stage)
Subgruph Embedding
(Training Stage)
Figure 2: Brain Subgraph Networks (BrainSubGNN), Including subgrah sampling process (1-hop) and subgraph embedding process.

---

Specifically, we define central subgraph for each node in the whole brain network. The central subgraph describes the surrounding region of the central node. For a input graph & and the central node vi, the central subgraph denoted as G = (A,, Xx V.) 9 where Vi = A (i) contains k-hop neighbor nodes V/ and A, X, equal to the corresponding part the original and X. The central subgraph G. constructed above includes brain region and its k-hops adjacent brain regions. Such subgraph contains the characteristics of this local brain network, which is of great significance to the brain network. In this work, the subgraph sampling is a preprocessing process before network training, node and edge indice are saved as binary file. In the training stage, the indice are loaded directly without additional cost on training.
3.2.2 Subgraph embedding
The subgraph partitions of the brain network are obtained in the previous step, each subgraph contains a central brain region and several adjacent brain regions, so we expect to be able to use a method that integrates the characteristies of all the brain regions in the subgraph and extract embedding to represent the entire subgraph. Inspired by the graph isomorphism network [52]. we consider the subgraph representation task multi-set problem and multi-layer perceptrons to learn an injective function for aggregating regional characteristics the brain subgraph network effectively and learn subgraph representation as:
( MLP
h + Σ w JEVIYA
where h₂ denote the hidden representation of node i, and ( lh E Id " xx(+1) are learnable weight matrices for the central node and other nodes in subgraph, respectively. By default, we set the MLP as a two-layer fully connected module. The obtained 2, represents the local characteristics of brain region i.
3.3 Transformer-Bant Based Fusion Bottleneck
Through the BrainSubGNN described earlier, obtain the regional representations of each modality of the brain network Z; E R Next, our goal is to apply an efficient fusion strategy to fuse critical and complementary information from Zec and Z extract distinguishing embedding to represent the features of the whole brain. and serve criterion for downstream tasks. We now describe how Trans-Bottleneck module alleviate the issue of regional heterogeneity in multimodal brain networks.
3.3.1 Fusion Bottlenecks
Due to the issue of regional heterogeneity between SC and FC, we are committed to avoiding the direct interaction of two modalities, and we prefer to find an intermediate element as bridge for the interaction between two modalities (means indirect interaction). Inspired by MBT 26 we introduce
the fusion bottlenecks into neuroscience research.
Specifically, fusion bottlenecks are simply a of learnable tokens Z, E N where N₂ is hyperparameter denoting the number of fusion bottlenecks. In this work, we utilize the fusion bottlenecks as intermediate medium bridge contact of this two modalities (SC and FC). As shown in Fig. 3 the fusion bottlenecks allow in- formation to flow between modalities and fusion bottlenecks (indirect interaction), but limit the flow of information between modalities (direct interaction). This procedure avoid direct interac- tion between regional heterogeneous modality data, effectively improving the model's 's perfor- mance. The initial bottlenecks are set of learn-
Indirect Interaction
Indirect Interaction
Z
Z
Z
O
0
Direct Interaction
Figure 3: Interaction between three main tokens.
5

---

able tokens that are randomly sampled from standard normal distribution ~ 1) and then passed to the first RH- HH--BrainFS layer. Finally, the fusion bottlenecks output by the last RH-B KH-BrainFS layer are used as the classification basis for downstream tasks (gender classification, major depressive disorder diagnosis, etc.).
3.3.2 Transformer-Base Fusion
Although originally proposed for NLP tasks, there has been recent interest in Transformers as universal perceptual models due to their ability to model dense correlations between tokens. Mcanwhile, turss out that individual regional characteristics of brain networks have different influence values for neuroscience tasks I 31 40 42 Based on these, our RH-BrainFS method utilizes the Transformer
45 as baseline for the fusion strategy to capture key subgraph characteristics in brain networks.
Specifically, as shown Fig. [ we first concatenate the fusion bottlenecks Zo and each modality representation Z, and then feed the concatenated tokens to the standard Transformer [45) model of the corresponding modality. In the Transformer model, the fusion bottlenecks learn important regional characteristics of the brain network for each modality through attention mechanism. The tokens output by the Transformer are re-split according to the previous splicing scheme to obtain new potential features Z!+1 of the brain network and temporary fusion bottlenecks 2/+1 corresponding to each modality. After that, the temporary fusion bottlenecks -/,1' of all modalities are matrix-averaged (also named average pooling) to obtain the fusion bottleneck latent representation Z/+1
This procedure be formulated
1 2 = Transformer( ( Z
(2)
2 AVERAGE (21,1)
where i E {SC, FC}, 0g denotes the modality specific Transformer, [-11 denotes the concatenate operation, Z denotes the fusion bottlenecks in 1th layer, and 11+ denotes the temporary fusion bottlenecks in 7th modality specific Transformer.
In this procedure, each modality transfers importance regional brain network characteristics within its modality to the fusion bottlenecks, and the fusion bottlenecks utilize shared characteristics between modalities to guide the learning of each modality's 's brain network the next layer.
3.3.3 Downstream Tasks
In the final stage, we complete classification basis for downstream tasks through the output Z/ in the last layer. Specifically, we first use global mean pooling as the readout function, and then input the result into MLP to complete classification basis.
Logits = Softmax (MLP(
Σ Zc,o))
(4)
No No
where Zi, denotes the i-th row of Zn, and Logits E {0, 1}. In this paper, on the depression classification task, 0 represents major depressive disorder, represents normal control. And on the gender classification task, ( represents male, represents female.
4
Experiments
In this section, we perform a series of experiments to evaluate the effectiveness of the proposed RH-BrainFS method. First, we provide the detailed experimental settings Sec. 4.1 Then, we perform comparison experiments on all datasets to compare the performance of different methods in Sec. 4.2 Finally, we perform some ablation studies of the main modules and hyperparameters the proposed RH-BrainFS method in Sec.

---

4.1 Experimental Settings
Datasets. We evaluate our RH-BrrainFS method on two different classification tasks investigating structure- functtion fusion. 1) The gender classification task on Human Connectome Project (HCP) dataset [43. which contains 560 female samples and 479 male samples. 2) The Major Depressive Disorder (MDD) diagnosis task on the hospital datasets [17 [8] including the Affiliated Zhongda Hospital of Southeast University (Zhongda hospital) and the Second Affiliated Hospital of Xinxiang Medical University (Xinxiang hospital). This study included 48 controls and 62 MDD patients from the Zhongda hospital and 46 controls and 31 MDD patients from the Xinxiang hospital. We also combine Zhongda and Xinxiang as Two-site dataset to construct more difficult and more realistic tasks.
Preprocessing. Here, we would briefly introduce how to construct the brain network of SC from dMRI and FC from fMRI.
SC. The dMRI data is preprocessed using the brain's diffusion toolbox of FMRIB Software Library [34]. Next we construct Gsc (Ase. Xec. Vse) from preprocessed dMRI data. First, we obtain the brain regions (Veg) of the individual space by mapping the anatomical automatic labeling (AAL) template in the standard space to the individual space. Then, we using DSI Studio software [9] to implement Fiber tracking. Finally, we obtain the feature Xac E |///l x Vael by counting the number of structural connective fibres between the different regions of the AAL, and the adjacency matrix Asc of the structural graph is obtained by thresholding Xsc with a threshold.
FC. The fMRI data is preprocessed using the Data Processing Assistant for Resting-Stau State Function (DPARSF) (35] MRI toolkit. Next we construct g5c (Afer X fes Vie) from preprocessed fMRI data. First, averaged time series are first computed for each brain region with a predefined atlas. Then, the Pearson correlation is utilized to calculate the functional matrix. Finally, the functional matrix is thresholded by proportional quantization to obtain the adjacency matrix A fcc The features fc functional connectivity matrix obtained earlier.
Metrics. In this study, evaluate all the methods using 10-fold cross- volidavali with the same partition of training and testing splits. Our evaluation metrics include classification accuracy (ACC), sensitivity (SEN), specificity (SPE), fl score (F1) and ROC-AUC (AUC). Higher values for all metrics indicate better performance. We record the mean and standard deviation on 10 random runs with 10-fold cross-validation.
Implementation Details. For all experiments, we adopt Adam as the optimizer and StepLR stee__size550, gamma-0.8) as the scheduler. The initial learning rate is set to 5e-4 and the dropout rate is set to 0.3. Also we utilize a early stop mechanism that 300 epochs patience in total 500 epochs. In the RH- Br-iBrranFF model, set the k-hop in the subgraph sampling I, the number of bottlenecks No to 4, the number of attention heads in the Transformer to 4, and the total number of network layers to 2. All our experiments are implemented in PyTocch and trained on one NVIDIA 3090.
4.2 Comparison Experiments
In this section, we verify the performance of our RH-BrainFS against existing baselines on several datasets.
Baselines. We choose two categories of methods comparison methods, both of which are methods for the direct study of SC and FC. The first category is unimodal methods, including FGDN (0) and BrainGNN (2I]. where FGDN uses spectral graph convolution method to extract brain networks features, and BrainGNN proposes a ROI-aware graph convolution layer and uses pooling to extract brain networks features. In our experiments, SC and FC are used as inputs the unimodal method, respectively. The second category is multimodal methods, including SVM, Random Forest, MGCN 22), GBDM [57], MMGNN 32 and AL-NEGAT (6). where SVM and Random Forest concatenate SC and FC as input, MGCN uses manifold- based regularization terms to consider inter- modaliyyy and intra-modality relationships, GBDM adopts weighted summation pattern to fuse multimodal brain networks, MMGNN utilises the concatenation method in multimodal tasks and AL-NEGAT combines multimodal information to construct edge feature maps and node feature maps. Code implementations of all baseline methods are taken from their respective original papers.
7

---

Table 1: Comparison experiments results (in percentage) on the all chosen datasets (only the accuracy shown, the full results can referred the Appendix Ap. The best results are marked in bold. The suboptimal results are marked underlined.
Datasets
Method
Modality
HCP
Zhongda
Xinxiang
Two-site
FGDN
FC
67.56+3.02
65.67+3.26
67.91
59,34+2.78
FGDN
SC
63,42+4.79
64.02=3.49
65.8945.15
68,91+2.53
BrainGNN
FC
66.4146.44
69.1843.39
73.4624.33
69.5543.23
BrainGNN
SC
67.3745.89
70.73+2.07
73.66#3.60
69.5142.58
SVM
SC,FC
74.49+2.97
63.2142.09
71.73+1.99
66.06+11.6
Random Forest
SC,FC
68.2442.94
61.45+2.80
62.78=1.63
62.4342.19
MGCN
SC,FC
67.9445.41
75.18+2.34
82.24+3.71
72.98+2, 17
GBDM
SC,FC
71.0244.39
74.81+2.44
80.71+2.83
72.48+1.91
MMGNN
SC,FC
73.33+2.82
60.6943.61
68.21+4.44
59.72+3, 18
AL-NEGAT
SC,FC
75.1243.66
73.95+3.45
75.7543.81
71.86+2.49
RH- Ii-BrrinFSS (ours)
SC,FC
78.6344.36
80.64#1.58
90.27=2.00
78.48:+.43
Results. As shown in Tab. our model significantly outperforms the other comparison methods on the all selected datasets. The results show that multimodal methods generally have better perfor- mance than unimodal methods because they capture more complementary information. Among the multimodal methods, our RH-BrainFS method achieves the best performance on the all the selected datasets (improvement of 3.51% % on HCP, 5.46% on Zhongda hospital dataset, 8.03% on Xinxiang hospital dataset and 5.50% on two-site dataset). The reason for the performance improvement is that our RH-BrriinS method fully considers the issue of regional heterogeneity between SC and FC and proposes an appropriate solution strategy for this issue
MGCN
GBDM
MMGNN
AL-NEGAT
Ours
female
male
Figure 4: t-SNE visualisation of multimodal method HCP dataset. Each dot denotes test sample.
Visualization. In order intuitively display the performance of each multimodal method, we conduct visualization experiments on the HCP dataset. Specifically, we take the graph-level embedding output from the last layer of each multimodal method for t-SNE visualisation. As shown in Fig. 4 although the MGCN and GBDM all form two clusters, these two clusters do not distinguish the two types of samples well and there is still lot of confusion. The MMGNN is loosely distributed and does not form good class boundary. It can be seen that the AI-NEGAT produces a similar distribution to our method, but there's still lot of confusion the class boundary. contrast, our method eliminates the confusion at the class boundary, achieves a good classification effect, most of the samples can be accurately distinguished and only a small number of samples have errors.
4.3 Ablation Study
In this section, we further perform ablation studies on the main modules and hyperparameters in the proposed CH-BrainFS method?
Full results of hyperparameter experiments refer Appendix

---

Table 2: Ablation study of the main modules in the RH-BrainFS method. The bold font indicates that the evaluation metric achieves the best performance in the ablation study.
Modules
ACC
SEN
AUC
w/o BrainSubGNN Trans-Bottlenee
76,2343.78
71.40+1
80.36+7.70
73.00=6.27
75.8814..12 75.65+4.19
BrainSubGNN
77.22=3.8
68.44+12.16
82.86+8.61
72.1646.34
w/o Trans-Botlleeck
76.80=2.93
71.14+10,50
81.61=7.10
73.50=5.13
76.3843.27
RH- BHH-BrainFS (uurs)
78.63s4.336
75.59
81.25+6.04
76.49s4.91
78.42+4.38
a
accuracy
Hoceuwey
accuracy
#
bottlenecks
# bottlenecks
k
Figure 5: The effect of varying hyperparameters. (a) and (c) are experiments on HCP datasets, (b)
and (d) are experiments on Two-site datasets.
Effectiveness of Main Modules. First, perform ablation study HCP dataset to validate effectiveness of the main modules, BrainSubGNN and Botts-oottleeck. Specifically, we replace BrainSubGNN with normal GIN [52 (process the whole brain network), and replace Trans- Bottleneck with a standard transformer (compute self-attention directly between two modalities, equal direct interaction), respectively. As shown in Tab. 2 we can find that both BrainSubGNN and
have certain effect on the performance of the model (compare the first three in the table). Furthermore, we find that the modules are compatible and complementary, as our RH-Brai BrainFS method (combining the two modules) achieves the best performance among the ablation study.
Impact of Bottlenecks Number. We then investigate the impact of varying bottlenecks number. Specifically, we experiments with # bottlenecks=22266688 respectively, with all other parameters unchanged. In order to ensure the credibility of the experiment, we perform experiments on two datasets (HCP dataset and Two-site dataset). As shown in Fig. [ both datasets, the performance of any number of bottlenecks exceeds the baselines. The main reason for this results is that the Trans-Bottlence module takes into account the regional heterogeneity issue between SC and FC, and avoids the direct interaction of heterogeneous information, thus achieving good fusion performance. Such results demonstrate the importance the regional heterogeneity issue in the modality fusion process of SC and FC. At the same time, we find that when # bottlenecks=4, the model performance reaches the best, which reflects that only a small number bottlenecks is needed achieve good multi-modal fusion expression capability.
Impact of Subgraph Sampling Hops. Next, we investigate the impact of varying sampling hops of subgraph. In this experiment, the range of sampling hops of subgraph from experiments HCP
1 to 5. Likewise, dataset and Two-site dataset. As shown in Fig.
accuracy
5) it can clearly be seen that as the number of sampling hops increases, the performance of the
model generally shows a downward trend, and when the sampling hops are too large, the model actually performs worse than the baselines. The reason for this results is that the brain network itself has strong regional characteristics, and too large sampling hops will cause the sampled sub- graph lose the local characteristics of the brain network and tend towards global characteristics.
threshold
Figure 6: The effect varying thresholding val-
ues.

---

The results show that only 1 hop of subgraph sampling is needed to express the local characteristics of the brain network well.
Impact of Thresholding Values. Finally, in this paper, thresholding is used to obtain the adjacency matrix of the two modalities during data processing (thresholding X and fc to obtain A,c and Afc. respectively). Thresholding value has been an important ablation study in neuroscience, and the same experiment was conducted in this paper. As shown in Fig. [ we set the value range of thresholding to [0.02, 0.30J. with 0.02 as a step, for a total of 15 thresholding experiments. From the experimental results, there no clear trend between threshold values and model performance, but the model performs poorly when the threshold value is too high or too low, so in this paper we chose a threshold value of 0. 12 as this is when the model performs best.
5 Discussions
Conclusion. In this paper, we introduce a brain subgraph networks and transformer- based fusion bottleneck to alleviate the issue of regional heterogeneity between SC and FC, and propose a novel multimodal brain networks fusion strategy first paper to explicitly state the issue of Atructuralffunctions modal regional heterogeneity and to propose solution. We validate our method on a variety of downstream task datasets, achieving state-of-the-ar performance.
To the best of our knowledge, this is the
Limitations and Future Work. Due to the scarcity and difficulty of collecting and processing data in neuroscience, the only datasets currently available are very limited, despite the fact that we have spent a great deal of manual effort in this area, and thus the research in this paper suffers from a number of possible data bias issues. In our future work, on the one hand, we will do more work on data to mitigate the problem of data bias. On the other hand, although this paper proposes the use of indirect rather than direct interaction, the two may not be mutually exclusive, and in the future we will investigate how to combine the two to achieve better research results.
Ethical Issues. With regard possible ethical issues in data collection, the Human Connectome Project (HCP) dataset, as publicly available dataset that has been used numerous previous studies, undoubtedly not ethically questionable. It true that the hospital dataset is held in collaboration with our partner hospitals and is not yet publicly available, but the data is collected with the consent of the subjects who are clearly informed of the purpose of the sample collection, and all identifying information about the sample is hidden in the hospital dataset. Therefore, it does not adversely affect any individual, so there are no ethical or moral issues.
Possible Negative Social Impacts. As the research in this paper deals with the diagnosis of depression, it is necessary elaborate here on the possible negative social impacts of this work, despite the fact that all the current work is at the stage of scientific research and has not been put to practical use. Including but not limited to:
Incorrect diagnosis. AI methods must have the possibility of error, which cannot be avoided, but an incorrect diagnosis will have significant impact on individuals and society. Therefore, Al tools can only be used as a diagnostic aid, not as a decision maker, and the final decision should still be made by the doctor.
Leakage of privacy information. In depression dataset, the identity information of the subjects is highly private, and the leakage of identity information will also have unpre- dictable and significant impact on individuals and society. Therefore, in this work, we have completely hidden the subjects' identifying information (which is also not visible to the staff in the study group) as a way of preventing the leakage of private information.
Acknowledgments and Disclosure of Funding
This work is supported by 2022 16700 National Key Research and Development Program of China, supported by 82271570, 31 31800825 and 31640028 National Natural Science Foundation of China, and also supported by the Big Data Computing Center of Southeast University. This work is partly supported by grant 2242023k30052 Central University Basic Research Fund of China. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
10

---

References
[1] Kwangycol Back, Laurel S Morris, Prantik Kundu, and Valerie Voon. Disrupted resting-state brain network properties in obesity: decreased global and putaminal cortico-striate network efficiency. Psychological medicine, 47(4):585-55) -96, 2017.
[2] Alaa Bessadok, Mohamed Ali Mahjoub, and Islem Rekik. Graph neural networks in network neuroscience. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[3] Hanshu Cai, Zhidiao Qu Zhe Li, Yi Zhang, Xiping Hu, and Bin Hu. Feature-level fusion approaches based multimodal eeg data for depression recognition. Information Fusion, 59:127-138, 2020.
[4] Bokai Cao, Liang Zhan, Xiangnan Kong, Philip S Yu, Nathalie Vizueta, Lori L Altshuler, and Alex D Leow. Identification of discriminative subgraph patterss in fmri brain networks in bipolar affective disorder. In Brain Informatics and Health: 8th International Conference, BIH 2015, London, UK, August 10-Seetember 2, 2015. Proceedings 8, pages 105-114. Springer, 2015.
[5] Lucy R Chai, Ankit N Khambhati, Rastko Ciric, Tyler M Moore, Ruben C Gur, Raquel E Gur, Theodore D) Satterthwaite, and Danielle S Bassett. Evolution of brain network dynamics in neurodevelopment. Network Neuroscience, 1(1):14-30, 2017.
[6] Yuzhong Chen, Jiadong Yan, Mingxin Jiang, Tuo Zhang, Zhongbo Zhao, Weihua Zhao, Jian Zheng, Dezhong Yao, Rong Zhang, Keith M Kendrick, et al. Adversarial learning based node- edge graph attention networks for autism spectrum disorder identification. IEFE Transactions on Neural Networks and Learning Systems, 2022.
[7] Zhiyi Chen, Xingwang Hu, Qi Chen, and Tingyong Feng. Altered structural and functional brain network overall organization predict human intertemporal decision-making. Human Brain Mapping, 40(1) (MM):336-325 2019.
[8] Brett W Fling, Youngbin Kwak, Scott Peltier, and Rachael Seidler. Differential relation- ships between transcallosal structural and functional connectivity in young and older adults. Neurobiology of aging, 33(10))2521-555 -5526, 2012.
[9] Aina Frau-Pascual, Jean Augustinack, Divya Varadarajan, Anastasia Yendiki, Bruce Fischl, and Iman Aganj. Detecting structural brain connectivity differences in dementia through conductance model. In 2019 53rd Asilomar conference on signals, systems, and computers, pages 591-595, IEEE, 2019.
[10] Hao Guo, Fan Zhang, Junjie Chen, Yong Xu, and Jie Xiang. Machine learning classification combining multiple features of a hyper-network of fmri data in alzheimer's disease. Frontiers neuroscience, 1 11:615, 2017.
[11] Man Guo, Tiancheng Wang, Zhe Zhang, Nan Chen, Yongchao Li, Yin Wang, Zhijun Yao, and Bin Hu. Diagnosis of major depressive disorder using whole-brain effective connectivity net- works derived from resting-state functional mri. Journal of Neural Engineering, 17(5):056038, 2020.
[12] Christopher Honey, Olaf Sporns, Leila Cammoun, Xavier Gigandet, Jean-Philipee Thiran, Reto Meuli, and Patric Hagmann. Predicting human resting-state functional connectivity from structural connectivity. Proceedings of the National Academy of Sciences, 106(6):20352200 2009.
[13] Wenxing Hu, Xianghe Meng. Yuntong Bai, Aiying Zhang, Gang Qu, Biao Cai, Gemeng Zhang, Tony W Wilson, Julia M Stephen, Vince D Calhoun, et al. Interpretable multimodal fusion networks reveal mechanisms of brain cognition. IEEE transactions medical imaging, 4065):1474-1483,, 2021.
[14] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention, In International conference on machine learning, pages 4651-4664, PMLR, 2021.

---

[15] Mimansa Jaiswal and Emily Mower Provost. Privacy enhanced multimodal neural representa- tions for emotion recognition. In Proceedings of the AAAI Conference on Artificial Intelligence volume 34, pages 7985-7993, 2020.
[16] Biao Jie, Mingxia Liu, Daoqiang Zhang, and Dinggang Shen. Sub-network kernels for measur- ing similarity of brain connectivity networks in disease diagnosis. IEEE Transactions on Image Processing, 27(5):2340-238 2018.
[17] Youyong Kong, Shuwen Gao, Yingying Yue, Zhenhua Hou, Huazhong Shu, Chumming Xie, Zhijun Zhang, and Yonggui Yuan. Spatio-temporal graph convolutional network for diagnosis and treatment response prediction of major depressive disorder from functional connectivity. Human brain mapping, 42(112):922-3392 2021.
[18] Youyong Kong, Wenhan Wang. Xiaoyun Liu, Shuwen Gao, Zhenghua Hou, Chunming Xie, Zhijun Zhang, and Yonggui Yuan. Multi-connectivtty representation learning network for major depressive disorder diagnosis. IEEE Transactions on Medical Imaging, 2023.
[19] Tommaso Lanciano, Francesco Bonchi, and Aristides Gionis. Explainable classification of brain networks via contrast subgraphs. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3308 3318, 2020.
[20] Jingcong Li, Fei Wang, Jiahui Pan, and Zhenfu Wen. Identification of autism spectrum disorder with functional graph discriminative network. Frontiers in Neuroscience, page 1282, 2021.
[21] Xiaoxiao Li, Yuan Zhou, Nicha Dvomek, Muhan Zhang, Siyuan Gao, Juntang Zhuang, Dustin Scheinost, Lawrence H Staib, Pamela Ventola, and James S Duncan. Braingnn: Interpretable brain graph neural network for finri analysis. Medical Image Analysis, 74:102233, 2021.
[22] Sol Lim, Filippo Radicchi, Martijn P van den Heuvel, and Olaf Sporns. Discordant attributes of structural and functional brain connectivity in two-layer multiplex network. Scienrific Reports, (1):2885, 2019.
[23] Manhua Liu, Danni Cheng, Kundong Wang, Yaping Wang. and Alzheimer's Disease Neu- roimaging Initiative. Multi-modality cascaded convolutional neural networks for alzheimer's disease diagnosis. Neuroinformaticc, 16:295-308, 2018.
[24] Vasileios Mantzavinos and Athanasios Alexiou. Biomarkers for alzheimer's disease diagnosis. Current Alzheimer Research, 14(11)1119-1154, 2017.
[25] Changping Meng. S Chandra Mouli, Bruno Ribeiro, and Jennifer Neville. Subgraph pattern neu- ral networks for high-order graph evolution prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[26] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems, 34:14200-14213, 2021.
[27] Jonathan D Power, Alexander L Cohen, Steven M Nelson, Gagan S Wig, Kelly Anne Barnes, Jessica A Church, Alecia C Vogel, Timothy O Laumann, Fran M Miezin, Bradley L Schlaggar, et al. Functional network organization of the human brain. Neuron, 72(4):665-6) 678, 2011.
[28] Jonathan D Power and Steven E Petersen. Control-related systems in the human brain. Current opinion in neurobiology, 23(2):223-228, 2013.
[29] Gang Qu, Li Xiao, Wenxing Hu, Junqi Wang, Kun Zhang, Vince D Calhoun, and Yu-Ping Wang. Ensemble manifold regularized multi-modal graph convolutional network for cognitive ability prediction. IEEE Transactions on Biomedical Engineering, 68( (8(12):3564-3573 2021.
[30] Blake A Richards, Timothy Lillierap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. deep learning framework for neuroscience. Nature neuroscience, 22(11))161111770 2019.
[31] Mikail Rubinov and Olaf Sporns. Complex network measures of brain connectivity: uses and interpretations. Neuroimage, 12(3):005-1006, 2010.
12

---

[32] Isaac Sebenius, Alexander Campbell, Sarah E Morgan, Edward T Bullmore, and Pietro Liò. Multimodal graph coarsening for interpretable, mri-based brain graph neural network. In 2021 IFFE 3 sst International Workshop on Machine Learning for Signal Processing (MISP), pages 1-6. IEFE, 2021.
[33] Mohammed Senoussaoui, Milton Sarria-Paja, João Santos, and Tiago H Falk. Model fusion for multimodal depression classification and level detection. In Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge. pages 57-63, 2014.
[34] Preya Shah, Arian Ashourvan, Fadi Mikhail, Adam Pines, Lohith Kini, Kelly Oechsel, Sand- hitsu R Das, Joel M Stein, Russell T Shinohara, Danielle S Bassett, et al. Characterizing the role of the structural connectome in seizure dynamics. Brain, (42(7):1955110)
2019.
[35] Xiao-Was Wei Song, Zhang- Ye Dong, Xiang- Yu Long, Su-Fang Li, Xi-Nian Zuo, Chao-Zhe Zhu, Yong He, Chao-Gan Yan, and Yu-Feng Zang. Rest: a toolkit for resting- functional magnetic resonance imaging data processing. PloS one, 6(9):e2531, 2011.
[36] Olaf Sporns, Christopher J Honey, and Rolf Kötter. Identification and classification of hubs in brain networks. PloS one, 2 2(10)e1 (499, 2007.
[37] Laura Suárez, Ross Markello, Richard F Betzel, and Bratislav Misic. Linking structure and function in macroscale brain networks. Trends in cognitive sciences, 24(4):302-315, 2020.
[38] Heung-Il Suk, Scong-Wiin Lee, Dinggang Shen, Alzheimer's Disease Neuroimaging Initiative, al. Hierarchical feature representation and multimodal fusion with deep learning for ad/mci diagnosis. Neurolmage, 101:569-582, 2014.
[39] Qingyun Sun, Hao Peng, Jianxin Li, Jia Wu, Yuanxing Ning, Philip S. Yu, and Lifang He. SUGAR: subgraph neural network with reinforcement pooling and self-supervised mutual information mechanism. CoRR, abs/2101.081 2021.
[40] Jewell Thomas, Matthew R Brier, Mario Ortega, Tammie Benzinger, and Beau M Ances. Weighted brain networks in disease: centrality and entropy in human immunodeficiency virus and aging. Neurobiology of aging, 36(1):401-412, 2015.
[41] Lixia Tian, Jinhui Wang, Chaogan Yan, and Yong He. Hemisphere-ann gender-related dif- ferences in small-world brain networks: a resting-state functional mri study. Neuroimage, 54(2):191200, 2011.
[42] Martijn van den Heuvel, René CW Mandl, Cornelis J Stam, René S Kahn, and Hilleke Hulshoff Pol. Aberrant frontal and temporal complex network structure in schizophrenia: a graph theoretical analysis. Journal of Neuroscience, 10(47):15115-159)9
2010.
[43] David C Van Essen, Stephen M Smith, Deanna M Barch, Timothy EJ Behrens, Essa Yacoub, Kamil Ugurbil, Wu-Minn HCP Consortium, et al. The wu-minn human connectome project: an overview. Neuroimage, 80:62-77 0900229977709 2013.
[44] Laura S Van Velzen, Sinead Kelly, Dmitry Isaev, Andre Aleman, Lyubomir Aftanas, Jochen Bauer, Bernhard T Baune, Ivan V Brak, Angela Carballedo, Colm G Connolly, et al. White matter disturbances in major depressive disorder: coordinated analysis across 20 international cohorts in the enigma mdd working group. Molecular psychiatry, 25(7):1511-152 2020.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[46] Joshua T Vogelstein, William Gray Roncal, R Jacob Vogelstein, and Carey E Priebe. Graph clas- sification using signal-suggraphs: Applications in statistical connectomics. IEEE transactions on pattern analysis and machine intelligence, 35(7):1539-1555 2012.
[47] Nina Vogt. Machine learning in neuroscience. Nature Methods, 15(1):33 2018.
[48] Bo Wang, Aziz M Mezlini, Feyyaz Demir, Mare Fiume, Zhuowen Tu, Michael Brudno, Ben- jamin Haibe-Kains, and Anna Goldenberg. Similarity network fusion for aggregating data types on genomic scale. Nature methods, 11(3):333-33, 2014.
13

---

[49] Lu Wang, Feng Vankee Lin, Martin Cole, and Zhengwu Zhang, Learning clique subgraphs in structural brain network classification with application to crystallized cognition. Neurolmage. 225:1 17493, 2021.
[50] Asiri Wijesinghe and Qing Wang. A new perspective on" how graph neural networks go beyond weisfeiler-lchman?.. International Conference on Learning Representations, 2022.
[51] Qiong Wu, Xiaoqi Huang, Adam J Culbreth, James A Waltz, L Elliot Hong, and Shuo Chen. Extracting brain disease-related connectome subgraphs by adaptive dense subgraph discovery. Biometrics, 78(4):1566-1577 2022.
[52] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 2018.
[53] Dogucan Yaman, Fevziye Irem Eyiokur, and Hazim Kemal Ekenel. Multimodal age and gender classification using ear and profile face images. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[54] Chao-Gan Yan, Xiao Chen, Le Li, Francisco Xavier Castellanos, Tong-Jian Bai, Qi-Jing Bo, Jun Cao, Guan-Mao Chen, Ning-Xuan Chen, Wei Chen, et al. Reduced default mode network functional connectivity in patients with recurrent major depressive disorder. Proceedings of the National Academy of Sciences, 116(18):9078990)) 9088,, 2019.
[55] Haoteng Yin, Muhan Zhang, Yanbang Wang, Jianguo Wang, and Pan Li. Algorithm and system co-design for efficient subgraph-based graph representation learning. arXiv preprint arXiv:2202.13538, 2022.
[56] Usman Zahid, Imran Ashraf, Muhammad Attique Khan, Majed Alhaisoni, Khawaja M Yahya, Hany S Hussein, and Hammam Alshazly. Brainnet: optimal deep learning feature fusion for brain tumor classification. Computational Intelligence and Neuroscience, 2022, 2022.
[57] Lu Zhang, Li Wang, Jean Gao, Shannon L Risacher, Jingwen Yan, Gang Li, Tianming Liu, Dajiang Zhu, Alzheimer's Disease Neuroimaging Initiative, et al. Deep fusion of brain structure- function in mild cognitive impairment. Medical image analysis. 72: 12:0022,, 2021.
[58] Tao Zhang and Mingyang Shi. Multi-modal neuroimaging feature fusion for diagnosis of alzheimer's disease. Journal of Neuroscience Methods, 341: 341:108795, 2020.
[59] Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. arXiv preprint arXiv:2110.03753 2021.
[60] Lulin Zheng, Ruidong Yang, Junbo Gao, Jun Chen, Jianzhong Liu, and Depeng Li. Quartz rb-sr isochron ages of two type orebodies from the nibao carlin-type gold deposit, guizhou, china. Minerals, 977):399, 2019.
[61] Guangming Zhu, Bin Jiang, Liz Tong, Yuan Xie, Greg Zaharchuk, and Max Wintermark. Applications of deep learning to neuro-imaging techniques. Frontiers in neurology, 10:869, 2019.
[62] Qi Zhu, Jing Yang, Shuihua Wang. Daoqiang Zhang, and Zheng Zhang. Multi-modal non- euclidean brain network analysis with community detection and convolutional autoencoder. IEEE Transactions on Emerging Topics in Computational Intelligence, 2022.
14

---

Appendix A: Full Results of Comparison Experiments
Due to the layout of the text and page limitations, only the experimental results for the ACC metric are presented in the main text. However, to ensure comprehensive presentation of the results, the full results of the comparison experiments involving all four datasets, HCP, Zhongda Hospital, Xinxiang Hospital and Two-site, are presented in Tables to 4. Four tables are presented detailing the mean and standard deviation values for each evaluation metric. The most exceptional results are shown in bold, while results below the optimal threshold are underlined for clarity and emphasis.
Analysis. Based the results from the four tables, our RH-BrainFS method achieved excellent results in all four datasets. Specifically, we achieved the best results on three metrics in the HCP dataset, four metrics in both the Zhongda and Xinxiang Hospital datasets, and an impressive five metrics in the Two-site dataset. This demonstrates the strong performance of RH-BrainFSS in effectively completing the fusion classification task for multimodal brain networks, Furthermore, we observe that on the hospital dataset, our RH-BrainFS method significantly outperforms most other methods in terms of bias in each metric, indicating higher stability. This improved stability contributes to the reliability and robustness of RH-BrainF'S in fusion classification tasks.
Table 3: Comparison experiments results on the HCP dataset.
HCP Datasel
Method
Modality
ACC
SEN
SPE
F1
AUC
FGDN
FC
67.5623.02
52.37=12.55
80.5448.24
59.0946.96
66.45=3.49
FGDN
SC
63.42+4.79
53.98+20.66 69
71.43#13.72
55.12=14.65
62.71#5.57
BrainGNN
FC
66.41=6.44
68 28.9226.99 92
63.66+12.30
65.4145.15
66.29#6.04
BrainGNN
SC
67.37=5.89
68.39+6.07
66.1748.55
65.88+4.45
67.28=5.80
SVM
SC,FC
74.49:2.97
71.18
77.3243.30
71.9613.49
74.25=3.05
Random Forest
SC.FC
68.24=2.94
54,8845.78
79.6424.16
61.3114.44
67.26#3
MGCN
SC.FC
67.9425.41
74.5348.20
62.3247.76
68.1045.59
68.42=5.44 .41
GBDM
SC.FC
71.02=4.39
61.95=12.37
79.1848.58
65.7626.35
70.5622432
MMGNN
SC,FC
73.33=2.82
71.17+4.52
75.1745.60 67
71.10+2.88
13.1722.72
AL-NEGAT
SC,FC
75.12=3.66
72.8647.74
84.4625.05
76.1344.70
78.66+3.81
RH-BrainFS (ours)
SC,FC
78.6324.36
75.5926.75
81.25+6.04
76.4914.91
78.42+4.38
Table 4: Comparison experiments results on the Zhongda hospital dataset.
Zhongda Dutaset
Method
Modality
ACC
SEN
SPE
AUC
FGDN
FC
65.67=3.26
78.31=8.26
49.25=12.11
70.1024.21
63.78+3.W8
FGDN
SC
64.02=3.49
65.67+14.37
61.50=16.65
60.67=8.24
63.58+3.55
BrainGNN
FC
69. 18.3393.39
73. 1..00.5.23
64.05 00
72.05#3.61
68. 68.57+33.4 46
BrainGNN
SC
70.73+2.07
75.93+3.95
64. 10+3.58
73.81+2.33
70.01+2.04
SVM
SC,FC
63.21+2.09
74.1742.06
49.00 09.00=3.22 .2
68.87+2.07
61.58+2. 20
Random Forest
SC,FC
61.45=2.80 80
86.69=2.78
29.00 99.0024.22
71.53+2.17
57. 8588552..0 96
MGCN
SC.FC
75.18=2.34
88.52+3.99
17.10=8.00 06
80.15=1.91
72.81+2.99 96
GBDM
SC.FC
74.81=2.44
81.77+5.21
18.1428.37
74.92+3.20
69. 69.96##..1
MMGNN
SC.FC
60.69=3.61
70.35=9.19
47.90=9.35
64.45=6.48
59.12+3.50
AL-NEGAT
SC,PC
73.95+3.45
90.71+7.24
52.05+7.0 .9
79.00=4.79
71.38+3.5
HH-BBrinFSS (ours)
SC.FC
80.64=1.58
90.0525.58
68.45:9.32
83,9621.13
79.25+2.24
15

---

Table 5: Comparison experiments results on the Xinxiang hospital dataset.
Xinxiang Dataset
Method
Modality
ACC
SEN
SPE
F1
AUC
FGDN
FC
67.91=3.27
43.58+8. 18
84,50+8.38
42.03+6.59
64,04#2. 88
FGDN
SC
65.89=5.15
61.83=14.81
68.85=15.06
53.0348.
65.34#3.82
BrainGNN
FC
73.46:4.33
10.92=10.57
86.2546. 10
55.7249.30
70.58a4.889
BrainGNN
SC
73.6623.60
56.83=10.76
85.3548.26
57.3948.58
71.09+3.76
SVM
SC,FC
71.73=1.99
49.92+4.35
86.55+3.42
54.5244.72
68,23=2. 19
Random Forest
SC,FC
62.78=1.63
12,08+4.8 81
97.20=1.96
17.0346.61
54.64#2. 11
MGCN
SC,FC
82.24=3.71
74. 16 62
87.45+3.45
76. 15+5.88.
80,80+4,
GBDM
SC,FC
80.71=2.83
60.20=10.03
88.43+5.88 86
63,4649.11 1
73,31 83
MMGNN
SC,FC
68.21#4.44
57.33s6..35
75.1048.1 12
55.48*6.03
66. 24.06.44.66
AL-NEGA
SC,FC
75.75=3.81
56.42=14.. 14.80
88.5046.35
61.19# 10.80
72.46 5.34
RH-BrainFS (ours)
SC,FC
90.27=2.00
80.75+5.86
96.65+2.34
85.43+3.90
88.70=2.48
Table 6: Comparison experiments results on the Two-site dataset.
Two-site Dataset
Method
Modality
ACC
SEN
SPE
F1
AUC
FGDN
FC
59.34+2.78
54.70+10.82
63.7948.88 83
49.9648.04
59.24+2.
FGDN
SC
68. 68.91422.3
70.02+12.42
67.8349.7
66.31
68.9342.52
BrainGNN
FC
69.5543.23
69.6245.22
68, 68.3646.61 67
67.5423.34
68, $..9433.22
BrainGNN
SC
69.51+2.58
68.1245.. .41
69.2416.66 66
66.76#3.42
68.6843.03
SVM
SC.FC
66.06+1.56
58.01#2.44
74.04+1.77
62.03=2.27
66.03+1.56
Random Forest
SC.FC
62.43+2.19
56.17+2.88
68. 18.60+3.48
59.04=2,73
62.38+2.20 20
MGCN
SC.FC
72.98+2.17
74.02+7.00
73.74+4.00
72.45=4.48
73.88+2.22
GBDM
SC.FC
72.48+1.91
71.5347.55
65.9745.61
68.74 74
68. 28.7542..0
MMGNN
SC,FC
59.7243.18
65. 10+4.38.
54.4244.42
59.9644.02
59.7643. 15
AL-EGGA
SC,FC
71.86+2.49
75.26+3.60
68. 08.1226.19
12.1222.24
71.6942.56
RH-Bram BrainFS (ours)
SC.FC
78.48:1.43
76.2024.06
80.72+3.60
77.35+1.97
78.46+1.43
16

---

Appendix B: Full Results of Hyperparameter Experiments
In the main text, our hyperparameter experiments are performed on the HCP dataset and the Two-site dataset. We explore two key hyperparameters in RH- WH--BrinFSS: the number of bottlenecks and the number of subgraph sampling hops However, in the main text we have only presented visualisations of the results of the hyperparameter experiments without giving specific numerical values. Hence, we present here the complete experimental results in Tables to
Table 7: Hyperparameter experiments of varying bottlenecks number HCP dataset.
HCP Datasel
# bottlenecks
ACC
SEN
SPE
F1
AUC
76.2643.58
69.70#11.51
80.00
71.7426.13
74.85+4.06
78.6324.36
75.5926.75
81.25+6.04
76.49+4.91
78.4244..8
76.85+3.41
69.72+8.32
80.54+7.38
72.29+4.26
75.13+3.46
76.51+3.05
71.16+9.82
81.25+8.11
73.45=4.42
76.20+3.18
Table 8: Hyperparameter experiments of varying bottlenecks number on Two-site dataset.
Two-site Dataset
# bottlenecks
ACC
SEN
SPE
F1
AUC
2
76.21
68.8844.98
83.61+4.56
73.2142.33
76.24+1.15
18.48
76.20+4.06
80.7243.60
77.35+1.97
78.46:1.43
76.35+1.43
70.3943.82
82.10
73.80+2.06
76.24
8
76.71#1.32
73.68+5.34
79.71
74.97+2.28
76.69+1.33
Table 9: Hyperparameter experiments of varying subgraph sampling hops HCP dataset.
HCP Dataset
k
ACC
SEN
SPE
F1
AUC
78.63+4.36
75.59*6.75
81.2556.04
76.49+4.91
78.42+4.38
2
77.0943.53
71.35+12.30
81.9619.40
73.7245.73
76.6643.81
3
77.4743.31
74.73+6.83
79.82+5.36
75.27+4.00
77.2843.39
4
76.6144.00
69.07411.44
83.04+4.01
72.63+6.79
76.0544.50
5
73.9242.94
64.50+6.03
81.96+6.41
69.4443.56
73.2342.88
17

---

Table 10: Hyperparameter experiments varying subgraph sampling hops on Two-site dataset.
Two-site Dataset
k
ACC
SEN
SPE
F1
AUC
1
78.48+1.43
76.20+4.06
80.72+3.60
77.35+1.97
78.46+1.43
2
13.99+1.32
69,4942.88
78.4243.04
71.53+2.04
73.96+1.32
3
7..37+1.81
72.4745.79
76.2643.42
72.66#3.37
74.36+1.79
4
74.55+1.64
73.13:6.88
75.9246.98
72.96+2.68
74.53+1.63
5
74.14
70.41+5.73
77.79+4.62
71.70#3.15
74.10+1.28
18


Reviews:
[{"review_id": "9mg3qKxJe5", "reviewer": "Reviewer_8rTt", "summary": "The article discusses the use of multimodal fusion as a research technique in neuroscience to extract complementary information from multiple modalities. Since previous research has neglected the regional heterogeneity between structural connectivity (SC) and functional connectivity (FC) and used inefficient ways of multimodal fusion. To address this issue, the authors proposed a novel Regional Heterogeneous multimodal Brain networks Fusion Strategy (RH-BrainFS), which uses a brain subgraph networks module to extract regional characteristics and a transformer-based fusion bottleneck module to alleviate the regional heterogeneity between SC and FC. The proposed method outperforms several state-of-the-art methods in various neuroscience tasks and is the first work to propose a solution to the issue of structural-functional modal regional heterogeneity.", "questions": "1. the authors use thresholding to get Asc and Afc from Xsc and Xfc, how the selection of thresholding effect the final results? how sensitive the results are.", "limitations": "see the weakness part", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "First the research problem is significant. The structural-functional modal regional heterogeneity is a recent popular research topic. The authors first propose a method which could work well in this situation. The novelty of the paper is sufficient. To alleviate the issue of regional heterogeneity of multimodal brain networks, the authors propose a novel Regional Heterogeneous multimodal Brain networks Fusion Strategy (RH-BrainFS), using BrainSubGNN module and Trans-Bottleneck module to fuse regional heterogeneous multimodal brain networks for neuroscience tasks.The experiments are sufficient, the authors conduct the experiment on two main-stream MRI brain benchmarks. Extensive experiments demonstrate the effectiveness of RH-BrainFS in multimodal brain networks fusion tasks on depression classification and gender classification datasets.", "weaknesses": "1. authors have not explained in detail why using \"forbidden Interaction\" in the Fusion bottlenecks. Authors are suggested to provide more motivations and explanation of this part.\n2. authors have not provide the detailed speed and time-consumption comparison with other methods. The proposed method seems time-consuming.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Answers to weaknesses：\n1. For the first weakness, \"forbidden interactions\" means that we remove direct interactions between SC and FC (just as in the standard Transformer model, a direct calculation of attention scores for different modal markers is a direct interaction). The initial motivation for this was that we believed that there was regional heterogeneity between SC and FC, and thus we believed that their features or potential embeddings would be misaligned, and thus direct interaction would lead to poorer results. Therefore, we proposed indirect interactions for fusion bottlenecks, and later also experimentally verified the effectiveness of indirect interactions (compare the ablation experiments in the third and fourth rows of Table 2). Therefore, we removed the direct interaction between SC and FC (i.e. direct interaction is forbidden) and adopted the indirect interaction of the fusion bottleneck.\n\n2. For the second weakness, we supplemented the time-consumption experiment (record the average training time for 100 epoches and the average time for 1 inference on the Two-site dataset), and as you would expect, our method incurred a greater time overhead. \n\n   To explain such results, we believe that firstly MGCN, GBDM and MMGNN are more focused on researching on processing of data features, so common GNN frameworks (which are simpler compared to Transformer or Attention) are used in the network structure, so they have better good computational efficiency. \n\n   Secondly, we consider AL-NEGAT to be an attentional-based approach that has similarities with our Transformer-based, so AL-NEGAT generates a time overhead that belongs to the same order of magnitude as Ours.\n\n   In response to such results, we admit that our method does underperform in terms of computational efficiency, but falls within an acceptable range, after all, our model performance is better improved.\n\nAnswers to questions：\n1. For the first question, regarding the value of the threshold, in fact, we have done an ablation experiment on the threshold before, through which we finally selected the optimal value as the threshold for the subsequent experiments. However, we did not put the results of this ablation experiment into the paper because we thought that this was not the focus of our research, and now we give the results in the \"global\" rebuttal pdf file (refer to Fig. 1 in \"global\" rebuttal pdf).\n\n| **Method**    | **Train Tine \\(s / 100 epoches\\)** | **Inference Time Cost \\(s / 1 inference\\)** |\n|:-------------:|:-----------------------------------:|:--------------------------------------------:|\n| **MGCN**      | 23\\.875                           | 0\\.017                                     |\n| **GBDM**      | 20\\.388                          | 0\\.013                                     |\n| **MMGNN**     | 23\\.905                          | 0\\.016                                     |\n| **AL\\-NEGAT** | 31\\.573                         | 0\\.023                                     |\n| **Ours**      | 33\\.875                          | 0\\.025                                    |\n"}, {"review_id": "luMsv6VSro", "reviewer": "Reviewer_A4mz", "summary": "The author proposes a novel regional heterogeneous multimodal brain networks fusion strategy to alleviate the issue of regional heterogeneity of multimodal brain networks. This strategy uses a graph convolutional network for the extraction of initial features of nodes (brain region from AAL atlas) and a transformer-based fusion bottleneck module for the fusion of structural connectivity matrix and functional connectivity matrix. This fusion strategy achieved the best results on the tasks of gender classification and depression classification. ", "questions": "See Weaknesses for details.", "limitations": "See Weaknesses for details", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "    1. This work is well written and the figure is clear, making it easy for the reader to understand what has been done.\n    2. This work is of great significance. How the structural and functional networks of the brain fusion have been an open question because the understanding of how the anatomical constraint is related to the elaborate functions is nevertheless fragmentary. Addressing the fusion of brain structure and function from a deep learning perspective can help facilitate understanding of the relationship between structure and function.", "weaknesses": "    1. The authors state \"alleviate the regional heterogeneity between multimodal brain networks\". This is a very ambitious question, and the high classification accuracy alone does not account for the alleviation of inter-regional heterogeneity. The authors should do detailed analysis experiments based on brain regions on the basis of high accuracy to explore which brain regions are non-heterogeneous between modalities, which brain regions are heterogeneous between modalities, and whether removing these heterogeneous brain regions can achieve better classification results.\n    2. The authors state \"they were inspired by MBT, so how does your work differ from MBT\"? It seems that the author's thinking is to not allow interaction of tokens between structure and function, why is that, shouldn't structure and function be meant to interact with each other? The authors say that it effectively improves the performance of the model (line 189), but I can't see from the experiments that there are results to prove this (Transformer with bottleneck token rather than w/o Trans-Bottleneck). The authors do not compare it with MBT's approach to prove the effectiveness of their proposed method.\n    3. The authors state in line 257 that the initial features of the function (Xfc) are averaged time series, but the Xfc shown in Figure 1 is the function connectivity matrix. These two points are contradictory. And, the initial features of the structure are the structural connectivity matrix, so why should the initial features of the function be the time series of fMRI instead of the functional connectivity matrix?\n    4. Are the Final Bottlenecks obtained based on Zb only? If so, why not cascade Zsc, Zb, Zfc as inputs to the MPL layer? The authors need to do ablation studies to prove that Zb-based classification is the most accurate. What does Nb in Equation 4 mean?\n    5. How is the threshold taken for the connectivity matrix to the adjacency matrix for the two modes? Firstly, the authors didn't write how much the threshold is taken. Secondly, the choice of the threshold for the structural and functional connectivity matrices has a big impact on the downstream task, it is suggested to do ablation studies to show how you take the threshold. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Answers to weaknesses：\n\n1. For the first weakness, we can understand your concern, and we have thought about it in the same way, but we have not come up with a good way to reflect the regional heterogeneity of brain networks at this time. Your suggestion of exploratory experiments based on brain regions was something we had considered, but since the number of brain regions present in each of our graphs is 90, it would require 2^90 experiments if we were to conduct exploratory experiments on brain regions, which is clearly unrealistic. So for now, we are still only able to measure the model in terms of classification performance on the downstream task.\n\n2. For the second weakness, since this work is investigating the regional heterogeneity between SC and FC, we believe that direct interaction will not achieve better results due to misaligned feature embeddings. So this indirect interaction of fusion bottlenecks is proposed.\n   The focus of our approach is that structure and function do not interact directly, but it does not mean that they do not interact. We adopt an indirect interaction approach, which is Figure 3 in the paper, where the structure and function interact indirectly with each other through fusion bottleneck. \n\n   Secondly, about the performance improvement of fusion bottleneck on the model. Actually, there is an experimental proof in the paper, maybe we didn't describe it clearly enough. The experimental setup about w/o Trans-Bottleneck is actually mentioned in the paper in line 308, which is to use the standard Transformer to directly compute the tokens of the two modalities, so for the third and fourth rows in Table 2, this ablation experiment actually shows the role of the Fusion bottleneck (Fig. 3) (i.e., the comparison of the difference between direct and indirect interactions). \n\n   Finally, about not comparing with MBT, it is because MBT itself is in the field of computer vision, doing audio-video modal fusion, so it can't be used directly in our scenarios, so I didn't include it in the comparison experiments to compare the methods (because it is mentioned in the paper in line 273-274 that our comparison methods are selected to be the research methods that are directly on SC and FC). \n\n3. For the third weakness, this one is indeed our writing error, what is shown in Figure 1 is the correct one, using the functional connectivity matrix as a feature.\n\n4. For the fourth weakness, thank you for this good suggestion, we are attaching the additional experiments to the rebuttal.\n\n   We apologize that Nb stands for the number of Bottlenecks, which is not specifically labelled here because it was mentioned earlier.\n\n5. For the fifth weakness, regarding the value of the threshold, in fact, we have done an ablation experiment on the threshold before, through which we finally selected the optimal value as the threshold for the subsequent experiments. However, we did not put the results of this ablation experiment into the paper because we thought that this was not the focus of our research, and now we give the results in the \"global\" rebuttal pdf file (refer to Fig. 1 in \"global\" rebuttal pdf).\n\n| Input                  | HCP Dataset  |              |              |              |              |\n|------------------------|--------------|--------------|--------------|--------------|--------------|\n|                        | Acc          | Sen          | Spe          | F1           | Auc          |\n| **Z\\_sc \\+ Z\\_fc**         | 77\\.62±3\\.69 | 74\\.96±7\\.70 | **81\\.79±7\\.40** | 76\\.30±4\\.33 | 78\\.38±3\\.68 |\n| **Z\\_sc \\+ Z\\_b \\+ Z\\_fc** | 77\\.57±4\\.06 | 74\\.77±9\\.26 | 80\\.00±9\\.03 | 75\\.34±4\\.50 | 77\\.38±4\\.03 |\n| **Only Z\\_b \\(ours\\)**     | **78\\.63±4\\.36** | **75\\.59±6\\.75** | 81\\.25±6\\.04 | **76\\.49±4\\.91** | **78\\.42±4\\.38** |\n\n\n| Input                  | Two\\-site Dataset |              |              |              |              |\n|------------------------|-------------------|--------------|--------------|--------------|--------------|\n|                        | Acc               | Sen          | Spe          | F1           | Auc          |\n| **Z\\_sc \\+ Z\\_fc**         | 75\\.73±1\\.83      | 69\\.01±4\\.87 | **82\\.39±3\\.01** | 72\\.55±3\\.04 | 75\\.70±1\\.82 |\n| **Z\\_sc \\+ Z\\_b \\+ Z\\_fc** | 75\\.29±1\\.89      | 69\\.09±3\\.96 | 81\\.50±4\\.89 | 72\\.34±2\\.58 | 75\\.29±1\\.89 |\n| **Only Z\\_b \\(ours\\)**      | **78\\.48±1\\.43**      | **76\\.20±4\\.06** | 80\\.72±3\\.60 | **77\\.35±1\\.97** | **78\\.46±1\\.43** |\n"}, {"review_id": "Ji58wfL6q5", "reviewer": "Reviewer_7Tgh", "summary": "The paper identifies a gap in the literature of multimodal brain networks fusion in which current methods are said to only use \"simple patterns\" to fuse modalities, ie, concatenation, weighted summation, and self-attention. To tackle this issue, the paper proposes RH-BrainFS, a new model fusing structural connectivity and functional connectivity data constructed from fMRI and dMRI, respectively. This model includes a BrainSubGNN model for subgraph sampling and subgraph embedding processes, as well as a fusion bottleneck based on transformers.", "questions": "Beyond what I wrote in the \"Weaknesses\" section, I have a few more questions:\n\n1. Am I correct to understand that both the functional and structural modality need to be parcellated with the same atlas? In figure 1's representation I got the impression that the two graphs could be different, but then from the text it doesn't seem that this is the case. Can the authors please clarify this?\n2. Have the authors considered analysing different choices for the readout function to check for possible result improvements?\n3. Have the authors considered other tasks beyond binary classification? For example a regression-based task like age prediction?\n4. Around line 249, if the feature matrix of the structural data is of shape |V_{SC}| x |V_{SC}|, doesn't it mean each node will have repeated features given the symmetric nature of the matrix? \n5. Around lines 259/260, I understood that the data is split in 50% train data and 50% validation data. If this is true, can the authors clarify why this was the case? In our field it is more common to have a higher percentage for the train data, and that's why I ask this.\n6. Although it is good that the paper contains traditional ML models as baselines (ie, SVM, random forest), why didn't the authors choose the traditional methods mentioned in the Related Work section?\n7. Work [59] seems to be the only previous work mentioned by the authors that use self attention. Given the model proposed by the authors highly relies on Transformers, shouldn't this work be included in the baselines comparison?\n8. Do the authors have any hypothesis on why the results of the two-site brings worse results than the two datasets separately? Could it be the way the two sites are separated when splitting the data in train/val sets? (I am assuming more data usually means better generability)", "limitations": "The paper highlights some limitations, but no potential negative societal impact of work seems to have been mentioned. The fact that the model uses medical data, and that it can be used to predict depression, seems to me to be potentially be used in negative ways with direct impacts in someone's life?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The results of the experiments (as seen in table 1) seem to be particularly strong, and this work is very significant given the lack of varied work in the field of multimodal brain imaging fusion. It is very good that the authors use more traditional ML models beyond deep learning (ie, SVM and random forests) to evaluate the relevance of this work, and that several metrics are considered beyond the simple accuracy metric. Finally, it usually takes a lot of time and manual work to preprocess dMRI data to be in the same parcellation as the fMRI data, thus the fact that this data is not available out-of-the-box clearly adds to the significance of this applied work - I hope the authors will make this preprocessed dataset available at a certain point.\n\n\n---------------------------------\nEDIT AFTER REBUTTAL PERIOD:\n\nFrom the rebuttal period, the authors did a very good job in addressing most of my concerns, and therefore I'm increasing my decision from borderline accept to weak accept, the soundness score from 2 to 3, and the presentation score from 3 to 4.", "weaknesses": "I identify three main weaknesses that require clarification during the rebuttal process and are the reason for me to give a borderline accept despite the strengths. I will be happy to review my score as a result of the rebuttal process if my points are properly tackled. I number my comments for easier discussion.\n\n\n1. A key weakness of this work is that it seems to me that there's an overstatement when the authors write that this is the first work to propose a solution to the issue of strutural-functional model regional heterogeneity. Maybe this issue is ill-defined and need a more clear description, otherwise I don't see how some of the previous works mentioned do not try to create models that take into account different functional-structural interactions on different regions of the brain. For example, work [59] (mentioned in introduction as an example of \"self-attention\" technique) seems to use attention layers at different layers of the model and due to its (attention) nature, it seems too much to call it a \"simple pattern\". Similarly, for work [58], I believe the authors are oversimplifying it by classifying it as just a \"weighted summation\", as it seems to me to be a much more complex usage of graph neural networks (GNNs) to fuse in a non-linear fashion different modalities of the brain. Indeed, work [58] seems to follow other works that I've seen leveraging GNNs to fuse different brain modalities (eg, 10.48550/arXiv.2007.09777, 10.1016/j.media.2022.102471, 10.1007/978-3-030-32248-9_89) which the paper did not mention and for me they are clearly tackling the issue of funtional-structural interactions even if they did not use this exact name. Finally, even for the simplest case of just concatenating (embedded) features it doesn't mean a model is making a \"one-to-one mapping\" as defended in the paper.\n2. A second key weakness I see in this paper is how some decisions seem to be justified/made in section 3.3.1. Section 3.3.1 starts with \"Due to the issue of regional heterogeneity between SC and FC, it is obviously not feasible to pass messages directly between two modalities\". What do the authors mean by \"messages\" here and why is this so obvious? Wouldn't \"messages\" between the different modalities actually help in the issue of not just concatenate two modalities? This seems to be related to lines 187-189, when the paper says that it avoids direct interactions between modalities. Why is direct communication such a bad thing and even called a \"forbidden\" interaction as indicated in figure 3? It seems to me that direct communication, together with other means of \"communication\" could help any model achieve better performance. Could maybe the authors justify this point a bit better? A possible ablation analysis I see here to support this statement would be to remove Z_b altogether, and this would imply that just one transformer would be required to fuse functional/structural data, greatly reducing the complexity of the model for (maybe) a similar performance.\n3. The final key weakness I have to identify in this paper is that it's not clear to me whether the results are overly-optimistic, and I will appreciate any clarification the authors can provide on this if I misunderstood this point. The paper seems to use a single train/validation set (for each fold of the 10-fold CV procedure), therefore the evaluations in table 1 seem to be made on the validation set. It also seems the case that this same validation set is used to know when to stop the training procedure. If I understood this correctly, then the results in table 1 can be overly-optimistic, as the same data used in training (eg, to know when to stop) was also used for final evaluation.\n\n(Small typo, in Line 203, should not capitalise \"We\")", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, thank you for your recognition of our work, but for the disclosure of data, it does need to be followed up by our discussion with multiple parties (including but not limited to the hospital side).\n\nAnswers to weaknesses：\n1. Maybe we did ill-define this issue, and the lack of clarity in the description. The reason why we say this is the first work is that our work focuses on indirect interactions of regional heterogeneous modalities rather than direct interactions, and we really did not find any work done in this field. As for the later examples of work [58, 59], they did do a lot of meaningful research on multimodal fusion, but they are still essentially direct fusion of multimodal information, so we define them as simple pattern (we define direct interaction as a simple pattern, not that these works are simple). \n\n   Secondly, \"one-to-one mapping\" just describes the fact that there is no \"one-to-one mapping\" between function and structure, not used to describe any previous work.\n\n2. The \"message\" is simply understood as the semantic information within the modality (like node's feature or the graph's structure information, etc.). \n\n   Secondly, we do not say that the information from different modalities is detrimental to the final task, but we emphasise that our research focuses on heterogeneity research, and the indirect interaction between different modalities, so the information from different modalities is still interacting in essence, but we have converted the direct to indirect. \n\n   Thirdly, to address this \"bad thing\" problem, because of the heterogeneity problem, we conceived the framework of indirect interaction, and finally verified the role of indirect interaction through experiments, so we believe that direct interaction is a bad thing compared to indirect interaction, so \"forbidden\" interactions mean that direct interactions between different modes are removed from our model. We did the corresponding ablation experiment in the paper, which may have been misunderstood because we did not describe it clearly. As in the ablation experiment, w/o Trans-Bottlenecks is to remove the fusion bottleneck, and use a standard Transformer to directly interact with the two modal information (which is what you want for ablation analysis).\n3. We can understand your concern. Our experimental setup is indeed using a 10-fold CV procedure with only the training and validation sets divided. The reason why we didn't divide the validation set and test set separately is that we researched a lot of articles and found that a lot of brain science related studies used cross-validation (refer to 10.1016/j.media.2021.102082, 10.1109/TNNLS.2022.3154755, 10.1016/j.media.2022.102550, 10.1109/TMI.2022.3222093), so we also adopted this approach. In our opinion, this 10-fold CV procedure has conducted experiments on multiple divisions and can better overcome the effects of division randomness.\n\nAnswers to questions:\n\n1. Structure and function are really two different graphs, they are two graphs with no connection at all, two modal information. The same atlas refers to the fact that we used the same atlas brain partitioning template, so that the two modal graphs have the same number of nodes and nothing else is the same.\n2. Thanks to your suggestion, we now add this experiment in the \"global\" rebuttal pdf file (refer to Tab. 1 and Tab. 2 in the \"global\" rebuttal pdf).\n3. It is true that we did not consider other types of tasks in the paper because our current study is based on current clinical needs, and in the future if there is a need for other types of tasks, we believe that our model can do a good job of migrating as well.\n4. Because SC essentially represents the number of fibre connections each brain region has to other brain regions, the feature matrix is an is of shape |V_{SC}| x |V_{SC}|. There may exist some of the same values, but they have different meanings.\n5. As we explained in detail in the answer to the third weakness above. In each fold, 90% of the data is used for training and 10% of the data is used for validation and testing.\n6. In this paper, we focus more on machine learning (including deep learning) methods, so we do not compare with SNF or ICA, which are traditional methods, in our comparison experiments.\n7. The fact that work [59] is based on brain imaging, whereas our study is based on brain network graphs, there is a fundamental difference between the image and graph. So we didn't include it in the comparison experiments (as it is mentioned in the paper in line 273-274 that our comparison methods are selected to be the research methods that are directly on SC and FC). The second thing is that work [59] does not disclose their code. We eventually added AL-NEGAT as a comparison method, which is an attentional-based approach.\n8. Since multi-site data are collected from different scanners with different acquisition parameters, non-neural inter-site variability may mask inter-group differences. Although multi-site increases the data but the overall data distribution is more complex, so it leads to performance degradation. In the experiments of two-site, we first divide the two datasets individually (90% train, 10% validation), and then combine them into a complete two-site dataset. Some multi-site work also leads to performance degradation, such as 10.1016/j.media.2021.102279, 10.1016/j.media.2020.101765.\n\nAnswers to limitations:\n\nRegarding the negative social impacts mentioned, firstly, our research is currently limited to scientific studies and has not been put to industrial use. Secondly, our data were collected with the consent of the subjects who were informed of the purpose for which the samples were collected. We also observe ethical and moral principles, and all of our samples were collected with personally identifiable information hidden. Overall, we can guarantee that the present exercise will not bring about any negative social impact.\n\n"}, {"review_id": "e8FwDxDjFw", "reviewer": "Reviewer_UEYi", "summary": "This paper proposes a novel approach called the Regional Heterogeneous Multimodal Brain Networks Fusion Strategy (RH-BrainFS) to address the issue of regional heterogeneity between structural connectivity (SC) and functional connectivity (FC) in brain networks fusion. The proposed approach includes a brain subgraph networks module to extract regional characteristics of brain networks and a transformer-based fusion bottleneck module to alleviate the issue of regional heterogeneity between SC and FC. The authors claim that this is the first work to propose a solution to the issue of structural-functional modal regional heterogeneity. The paper presents extensive experiments that demonstrate that the proposed method outperforms several state-of-the-art methods in a variety of neuroscience tasks.", "questions": "- How does the proposed approach compare to other state-of-the-art methods in terms of computational efficiency? Is the proposed approach computationally feasible for large-scale datasets?\n- The paper mentions that the proposed approach includes a brain subgraph networks module to extract regional characteristics of brain networks. Can the authors provide more details on the design motivation this module, as well as its neurological justification?\n- How would the subgraph sampling affect the effectiveness of the overall framework? \n- Would there be any potential modality bias in the experimented data? \n- Could you elaborate on how woyld the different brain imaging techniques mentioned in the text, such as functional magnetic resonance imaging (fMRI) and diffusion magnetic resonance imaging (dMRI)? reflect different aspects of the brain's internal characteristics?", "limitations": "The authors have acknowledged some of the limitations of their work, such as the limited sample size and the difficulty of data collection in neuroscience. However, the paper does not provide a detailed discussion on the technical bias, e.g., modality bias during fusion of the proposed approach, nor does it address potential negative societal impact of the research. Adding those discussions would enhance the paper's credibility and relevance, and help readers better understand the potential implications of the proposed approach.\n", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-structured and clearly written. The authors provide a clear introduction to the problem, a thorough review of related work, and a detailed description of their proposed method. They also clearly explain the experimental settings and datasets used in their research. The use of technical terms is appropriate for the intended audience, and the authors provide sufficient context and explanation to make the content understandable.\n- The issue of regional heterogeneity in multimodal brain networks is a significant challenge in the field, and the authors' proposed solution has the potential to improve the performance of multimodal fusion models. \n- The authors provided detailed preprocessingand implementation parameters.  ", "weaknesses": "- It is important for the authors to compare their proposed approach with existing methods in the field of multimodal brain network modeling. One such method is the joint embedding of structural and functional brain networks with graph neural networks proposed by Zhu et al.. The authors should compare their proposed approach with this method in terms of performance for modality fusion.\n- The transformer-based fusion is not original in terms of modality fusion. The authors are suggested to investigate and provide concrete interpretations on the fusion bottlenecks which utilize shared characteristics between modalities.\n- Lack of variant on GNN backbone. There are a bunch of GNN architectures for brain network modeling. While the paper acknowledges the use of a BrainSubGNN method, it would be valuable to consider other GNN architectures that have been developed specifically for brain network analysis. By incorporating a broader range of GNN backbones and comparing their performance, the authors can enhance the robustness and generalizability of their proposed approach for multimodal brain network fusion.\n- Lack of interpretation analysis. In the context of multimodal brain network modeling, interpretation analysis could involve identifying the brain regions or networks that are most relevant for the classification task, and investigating the functional and structural connectivity patterns that contribute to these regions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Answers to weaknesses：\n1. For the first weakness, in fact, we have followed this work (Zhu et al.) , but this work does not disclose their code, and we cannot compare them.\n2. For the second weakness, the initial motivation for fusion bottleneck was that we believed that there was regional heterogeneity between SC and FC, and thus we believed that their features or potential embeddings would be misaligned, and thus direct interaction would lead to poorer results. Therefore, we proposed indirect interactions for fusion bottlenecks, and later also experimentally verified the effectiveness of indirect interactions (compare the ablation experiments in the third and fourth rows of Table 2). Therefore, we removed the direct interaction between SC and FC (i.e. direct interaction is forbidden) and adopted the indirect interaction of the fusion bottleneck.\n3. For the third weakness, thanks to this suggestion, we have added this ablation experiment by replacing the BrainSubGNN module with four common GNN frameworks, and the results are shown in the table. (Note: Because the computational efficiency of our model is not as good as other methods, we only chose a simple GNN as the backbone, not a complex GNN).\n\n| Backbone             | Two\\-site Dataset |              |              |              |              |\n|:--------------------:|:-----------------:|:------------:|:------------:|:------------:|:------------:|\n|                      | Acc               | Sen          | Spe          | F1           | Auc          |\n| GCN                  | 70\\.04±1\\.16      | 65\\.40±5\\.26 | 74\\.34±4\\.39 | 66\\.25±3\\.23 | 69\\.87±1\\.13 |\n| GAT                  | 71\\.08±1\\.26      | 71\\.83±7\\.87 | 70\\.02±8\\.80 | 69\\.23±2\\.70 | 70\\.93±1\\.30 |\n| GIN                  | 74\\.65±0\\.99      | 70\\.61±7\\.49 | 78\\.50±5\\.99 | 72\\.07±3\\.19 | 74\\.56±1\\.18 |\n| GraphSAGE            | 70\\.03±1\\.63      | 66\\.40±5\\.94 | 73\\.16±6\\.34 | 67\\.12±2\\.51 | 69\\.78±1\\.63 |\n| BrainSubGNN \\(ours\\) | **78\\.48±1\\.43**      | **76\\.20±4\\.06** | **80\\.72±3\\.60** | **77\\.35±1\\.97** | **78\\.46±1\\.43** |\n\n4. For the fourth weakness, thanks for the good advice, we had thought about doing brain region importance analyses, but SC and FC are done separately, meaning that we can only know separately which brain regions are important for SC and which brain regions are important for FC, without an overall result. Secondly, we thought brain region importance analyses were not very relevant to our research motivation, and we were more focussed on solving the problem of regional heterogeneity than on studying brain region importance.\n\nAnswers to questions:\n\n1. For the first question, we supplemented the time-consumption experiment (record the average training time for 100 epoches and the average time for 1 inference on the Two-site dataset), and as we would expect, our method incurred a greater time overhead.\n\n   To explain such results, we believe that firstly MGCN, GBDM and MMGNN are more focussed on researching on processing of data features, so common GNN frameworks (which are simpler compared to Transformer or Attention) are used in the network structure, so they have better good computational efficiency.\n\n   Secondly, we consider AL-NEGAT to be an attentional-based approach that has similarities with our Transformer-based, so AL-NEGAT generates a time overhead that belongs to the same order of magnitude as Ours.\n\n   In response to such results, we admit that our method does underperform in terms of computational efficiency, but falls within an acceptable range, after all, our model performance is better improved.\n\n   Finally, our work can be adapted to large-scale data because each brain network graph is not large (only 90 nodes), so there is no problem of large-scale incomputability.\n\n| **Method**    | **Train Tine \\(s / 100 epoches\\)** | **Inference Time Cost \\(s / 1 inference\\)** |\n|:-------------:|:-----------------------------------:|:--------------------------------------------:|\n| **MGCN**      | 23\\.875                           | 0\\.017  |\n| **GBDM**      | 20\\.388                          | 0\\.013 |\n| **MMGNN**     | 23\\.905                          | 0\\.016  |\n| **AL\\-NEGAT** | 31\\.573                         | 0\\.023 |\n| **Ours**      | 33\\.875                          | 0\\.025  |\n\n2. For the second question, ss we mentioned inside introduction section (in line 42-43), the brain network itself has strong llocal characteristics, so we used a brain subgraph network to extract local characteristics efficiently.\n\n3. For the third question, different subgraph sampling strategies extract different brain local characteristics, and different hop counts reflect different brain local characteristics (refer to Fig. 5 of the ablation experiment), as stated in lines 329 to 331 of the paper, too large subgraph sampling hop counts will make the sampled subgraphs not have local characteristics\n\n4. For the fourth and fifth questions, structural neuroimaging datadiffusion magnetic resonance imaging (dMRI), reflecting voxel tissue density/volume or structural connectivity. The main purpose of structural studies is to reveal anatomical relationships in the brain, which can then be used for prediction.\n\n   Functional neuroimaging datafunctional magnetic resonance imaging (fMRI), reflecting changes in deoxyhemoglobin concentration caused by task-induced or spontaneously regulated neurometabolic. The main purpose of functional studies focuses on dynamic changes in brain activity or connectivity.\n\n   Because of the essential differences, there is necessarily a modal difference between structure and function.\n\nAnswers to limitations:\n   1. Thank you for your suggestion, we really should discuss our technical bias in terms of limitations. Regarding the mentioned negative social impacts, we have explained them in detail in the global rebuttal, and in short we can assure you that this work will not have any negative social impacts.\n"}, {"review_id": "2IeFOJ7BTv", "reviewer": "Reviewer_kgkZ", "summary": "The submission is not in my area, and it's difficult to give reasonable comments about this submission. My research interests focus on medical image reconstruction. please find another appropriate reviewer to review this paper.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "N/A", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:
In response to some reviewers and ethics reviewer's questions about the negative social impact, we would like to explain the following.

First, regarding the dataset collection process, the Human Connectome Project (HCP) dataset, as a publicly available dataset that has been used in numerous previous studies, is undoubtedly not ethically questionable. It is true that the hospital dataset is held in collaboration with our partner hospitals and is not yet publicly available, but the data is collected with the consent of the subjects who are clearly informed of the purpose of the sample collection, and all identifying information about the sample is hidden in the hospital dataset. Therefore, it does not adversely affect any individual, so there are no ethical or moral issues. 

Secondly, with regard to research on depression, the work we have done so far has been limited to the scientific research stage and has not been put to clinical application. We can guarantee that all the research conducted in this work will not have any negative social impact.

Thirdly, regarding the ethics reviewer's question about the dataset being too small and the use of the depression dataset. In fact, this is really the only dataset that available to us, because there are great difficulties in data collection in this field, and also, processing the data is very costly, so it is really small for the time being. For the question of using depression datasets, the previous two points have basically answered the question without any moral or ethical issues, and it is currently limited to scientific research without any negative social impacts.

(The explanation above will be appended to the full paper of the new version.)

In this global rebuttal we accompany the results of some supplementary experiments (a pdf file).

Meta-review: This paper received mixed ratings, and the authors submitted rebuttals. The reviewers read the rebuttals and further discussed with the authors. Some reviewers' concerns were addressed by the authors' rebuttals and responses. As a result, three reviewers raised their ratings. The AC read the authors' rebuttals and the discussions between authors and reviewers. During the AC-reviewer discussion period, the AC further gave the following summary, which has been approved by the reviewers. 

Strengths:

1) Addresses a significant research problem

2) A new spatially varying fusion strategy for multimodal Brain networks

3) Extensive experiments demonstrate the effectiveness of the proposed strategy in two multimodal brain networks fusion tasks

Weaknesses:

1) The proposed multimodal fusion framework and transformer-based fusion bottleneck module have limited novelty.

2) Improved performance at the expense of higher model complexity and computational cost

3) Lack of in-depth region-based interpretation analysis of why the proposed fusion strategy works well


At the end, the AC and reviewers feel reasons to accept outweigh reasons to reject.
Decision: Accept (poster)