Multi-Atlas Brain Network Classification through Consistency Distillation and Complementary Information Fusion

## Abstract

Abstract In the realm of neuroscience, identifying distinctive patterns associated with neurological disorders via brain networks is crucial. Resting-state functional magnetic resonance imaging (fMRI) serves as a primary tool for mapping these networks by correlating blood-oxygen-level-dependent (BOLD) signals across different brain regions, defined as regions of interest (ROIs). Constructing these brain networks involves using atlases to parcellate the brain into ROIs based on various hypotheses of brain division. However, there is no standard atlas for brain network classification, leading to limitations in detecting abnormalities in disorders. Some recent methods have proposed utilizing multiple atlases, but they neglect consistency across atlases and lack ROI-level information exchange. To tackle these limitations, we propose an Atlas-Integrated Distillation and Fusion network (AIDFusion) to improve brain network classification using fMRI data. AIDFusion addresses the challenge of utilizing multiple atlases by employing a disentangle Transformer to filter out inconsistent atlas-specific information and distill distinguishable connections across atlases. It also incorporates subject- and population-level consistency constraints to enhance cross-atlas consistency. Additionally, AIDFusion employs an inter-atlas message-passing mechanism to fuse complementary information across brain regions. Experimental results on four datasets of different diseases demonstrate the effectiveness and efficiency of AIDFusion compared to state-of-the-art methods. A case study illustrates AIDFusion extract patterns that are both interpretable and consistent with established neuroscience findings.

## 1 Introduction

In the field of neuroscience, a key objective is to identify distinctive patterns associated with neurological disorders (e.g., Alzheimer’s, Parkinson’s, and Autism) by the brain networks []. Resting-state functional magnetic resonance imaging (fMRI) is widely employed among various neuroimaging techniques to characterize the connectivities among brain regions [].
This results in brain networks where each node represents a specific brain region, referred to as a region of interest (ROI). Each edge indicates a pairwise correlation between the blood-oxygen-level-dependent (BOLD) signals of two ROIs [], revealing the connectivity between brain regions and indicating which areas tend to be activated synchronously or exhibit correlated activities.

Brain networks model neurological systems as graphs, allowing the use of graph-based techniques to understand their roles and interactions [, , ]. Constructing these brain networks involves using a specific atlas to parcellate the brain into ROIs. Various atlases based on different hypotheses of brain parcellation, such as anatomical and functional divisions, have been proposed to group similar fMRI regions and create ROIs [, , ]. Although proper brain parcellation is essential for detecting abnormalities in neurodegenerative disorders [], there is no golden standard atlas for brain network classification. Relying on a single atlas for brain network analysis has two main drawbacks. First, some voxels may not be assigned to any specific ROI, potentially leading to the loss of important information. Second, each atlas is based on a different parcellation hypothesis. The BOLD signal of an ROI is averaged from all voxels within it, possibly missing detailed information. To address these limitations, recent works have proposed using multiple atlases with different parcellation modes to enhance multi-atlas brain network analysis. Some methods [, ] independently encode brain networks from various atlases and then aggregate the graph representations as a late feature fusion scheme for the final prediction. Another approach [] incorporates early feature fusion by incorporating multi-atlas information from the raw data and using the fused feature for representation learning. However, these methods (1) neglect the need of consistency across atlases, potentially leading to the under-utilization of cross-atlas information; and (2) lack ROI-level information exchange throughout the entire representation learning process, which could hinder the models’ ability to discern complementary information across different atlases.

In this paper, we propose an Atlas-Integrated Distillation and Fusion network (AIDFusion) to address the aforementioned limitations by utilizing atlas-consistent information distillation and cross-atlas complementary information fusion. Specifically, AIDFusion introduces a disentangle Transformer to filter out inconsistent atlas-specific information and distill distinguishable connections across different atlases. Subject- and population-level consistency constraints are applied to enhance cross-atlas consistency. Furthermore, to facilitate the fusion of complementary information across ROIs in multi-atlas brain networks, AIDFusion employs an inter-atlas message-passing mechanism that leverages spatial information. In summary, our key contributions are:

- •
We propose a multi-atlas solution for brain network classification with fMRI data. AIDFusion takes full advantage of multi-atlas brain networks by enhanced atlas-consistent information distillation and intense fusion of cross-atlas complementary information.
- •
We evaluate AIDFusion on four resting-state fMRI brain network datasets for different neurological disorders. Our results demonstrate the superiority of AIDFusion over state-of-the-art baseline methods in terms of effectiveness and efficiency in brain network classification.
- •
We present a case study that underscores the intriguing, straightforward, and highly interpretable patterns extracted by our approach, aligning with domain knowledge found in neuroscience literature.

## 2 Related Work

### 2.1 Brain Network Analysis with Various Atlases

Multi-atlas methods introduce multiple brain atlases for each neuroimage, which can provide information that complements each other and offers ample details without being restricted by the parcellation mode. MGRL [] pioneered the construction of multi-atlas brain networks using various atlases. It applied graph convolutional networks (GCNs) to learn multi-atlas representations and perform graph-level fusion for disease classification. METAFormer [] proposed a multi-atlas enhanced transformer approach with self-supervised pre-training for Autism spectrum disorder (ASD) classification. A graph-level late fusion was utilized to aggregate the representations of different atlases. Lee et al. [] employed a multi-atlas fusion approach that integrates early fusion on the raw feature to capture complex brain network patterns. STW-MHGCN [] constructs a spatial and temporal weighted hyper-connectivity network to fuse multi-atlas information, and Huang et al. [] adopt a voting strategy to integrate the classification results of different classifiers (each corresponding to a different atlas) for ASD diagnosis. However, these studies did not consider the inherent consistency between atlases. Independently encoding multi-atlas brain networks without constraints might extract atlas-specific information, distracting from disease-related pattern modeling. Moreover, existing works only incorporate primitive early or late feature fusion between atlases. This absence of intermediate ROI-level interaction could hinder their models’ ability to discern complementary information in each atlas. To the best of our knowledge, our work is the first to introduce information distillation with consistency constraints and employ intermediate ROI-level interaction for complementary information fusion. Note that in our work, multiple atlases are applied to preprocessed images for parcellation, meaning our method is based on a single template. The difference between multi-atlas methods and multi-template methods are discussed in Appendix A.

Multi-modal and multi-resolution methods also explore brain networks using various atlases. Research about multi-modal brain networks [, , , , ] employed multiple modalities of neuroimaging data, including fMRI, Diffusion Tensor Imaging (DTI) and Positron Emission Tomography (PET), with various atlases to enhance brain network classification, as different modalities provide abundant information compared to a single modality.
However, these multi-modal methods focus on fusing structural and functional connectivity information instead of trying to capture the whole picture of the single modality data.
Another line of research [, , ] focuses on applying multi-resolution atlases to fMRI data to capture individual behavior across coarse-to-fine scales. However, the technical design of these approaches focuses on extracting information from both fine and coarse scales under the same parcellation mode. Although multi-modal and multi-resolution methods employ various atlases, they focus on different objectives from multi-atlas approaches, and the field of brain network analysis with multi-atlas is still in its infancy stage.

### 2.2 Graph Neural Networks (GNNs)

In recent years, there has been a surge of interest in employing Graph Neural Networks (GNNs) for the analysis of brain networks. Ktena et al. [] utilized graph convolutional networks to learn similarities between pairs of brain networks (subjects). BrainNetCNN [] introduced edge-to-edge, edge-to-node, and node-to-graph convolutional filters to harness the topological information within brain networks. MG2G [] utilizes a two-stage method where it initially learns node representations using an unsupervised stochastic graph embedding model based on latent distributions, which are then used to train a classifier, allowing for the identification of significant ROIs with Alzheimer’s disease (AD)-related effects. Zhang et al. [] incorporated both local ROI-GNN and global subject-GNN guided by non-imaging data. ContrastPool [] introduced a dual attention mechanism to extract discriminative features across ROIs for subjects within the same group.

An alternative method for graph representation learning involves Transformer-based models [], which adapt the attention mechanism to consider global information for each node and incorporate positional encoding to capture graph topological information. Graph Transformers have garnered significant attention due to their impressive performance in graph representation learning [, , ].
A series of brain network Transformer methods have emerged for brain network analysis. One such method [] applied Transformers to learn pairwise connection strengths among brain regions across individuals. THC [] introduced an interpretable Transformer-based model for joint hierarchical cluster identification and brain network classification. DART [] utilized segmenting BOLD signals to generate dynamic brain networks and then incorporated them with static networks for representation learning. Most GNN- and Transformer-based methods for brain network analysis are designed for single-atlas thus may lead to a dependency on specific parcellation mode.

## 3 Preliminaries

### 3.1 Brain Network Construction

We introduce our method using two atlases, $a$ and $b$, for simplicity. It can easily generalize to more atlases. In this work, we use the datasets preprocessed and released by Xu et al. []. Each subject is characterized by two brain networks, represented by connectivity matrices $\boldsymbol{X}^{a}\in\mathbb{R}^{n_{a}\times n_{a}}$ and $\boldsymbol{X}^{b}\in\mathbb{R}^{n_{b}\times n_{b}}$. These matrices are derived using different atlases, which divide the whole brain into $n_{a}$ and $n_{b}$ ROIs respectively. In these matrices, each entry represent an edge in the brain network, which is calculated by Pearson’s correlations between the region-averaged BOLD signals from pairs of ROIs. Thus these brain networks are weighed and fully-connected. Essentially, these brain networks capture functional relationships between different ROIs.

### 3.2 Problem Definition

Multi-atlas brain network classification aims to predict the distinct class of each subject by using various atlases for the same fMRI data. Given a dataset of labeled brain networks $\mathcal{D}=\{(\boldsymbol{X}^{a},\boldsymbol{X}^{b},y_{X})\}$, where $y_{X}$ is the class label of brain networks $\boldsymbol{X}^{a}$ and $\boldsymbol{X}^{b}$, the problem of brain network classification is to learn a predictive function $f$: $(\boldsymbol{X}^{a},\boldsymbol{X}^{b})\rightarrow y_{X}$, which maps input brain networks to the groups they belong to, expecting that $f$ also works well on unseen brain networks.
We summarize the notations used throughout the paper in Appendix B.

Figure: Figure 1: The framework of AIDFusion for multi-atlas brain network classification.
Refer to caption: x1.png

## 4 Methodology

In this section, we provide a detailed exposition of the design of our proposed Atlas-Integrated Distillation and Fusion network (AIDFusion), depicted in Figure 1. Two brain networks constructed with different atlases are separately processed in our model.
In the following, we first introduce the disentangle Transformer with identity embedding to remove inconsistent atlas-specific information (Section 4.1). We then describe the inter-atlas message-passing for spatial-based intense fusion of cross-atlas (Section 4.2). Finally, we discuss our design of the losses that enforce atlas-consistent information distillation with domain considerations (Section 4.3).

### 4.1 Disentangle Transformer with Identity Embedding

Identity Embedding. In graph Transformer models, positional embedding is commonly used to encode the topological information of the graph. However, designs like distance-based, centrality-based, and eigenvector-based positional embeddings [, , ] are impractical for brain networks due to their high density (always fully connected). Correlation-based brain networks already contain sufficient positional information for ROIs, making general positional embeddings both costly and redundant. Instead, we propose a learnable identity embedding that adaptively learns a unique identity for each ROI, aligning nodes in the same ROI across the same atlas. This embedding assigns the same identity to nodes within the same ROI. As shown in Eq. (1), we introduce a parameter matrix $\boldsymbol{W}_{ID}$ to encode node identities alongside original node features $\boldsymbol{X}$, with $\operatorname{MLP}(\cdot)$ denoting a multilayer perceptron (MLP).

$$ $\small\boldsymbol{H}_{ID}=\boldsymbol{X}+\operatorname{MLP}(\boldsymbol{X}+ \boldsymbol{W}_{ID}).$ (1) $$

Disentangle Transformer. Introducing learnable tokens in the input sequence of a Transformer has been a method used to capture global information. In natural language processing, Burtsev et al. [] first utilized a learnable $[CLS]$ token to improve machine translation tasks. In computer vision, Darcet et al. [] introduced register tokens to avoid recycling tokens from low-informative areas. Motivated by these prior works, we propose a disentangle Transformer to filter out inconsistent atlas-specific information by introducing incompatible nodes. We elaborate this module in Figure 1b. Specifically, given an identity-encoded graph feature matrix $\boldsymbol{H}_{ID}\in\mathbb{R}^{n\times d}$, where $n$ is the number of nodes and $d$ is the hidden dimension, we add $r$ learnable incompatible nodes $\boldsymbol{W}_{INC}\in\mathbb{R}^{r\times d}$ to the feature matrix:

$$ $\small\boldsymbol{H}^{\prime}=\left[\begin{array}[]{c}\boldsymbol{H}_{ID}\\ \boldsymbol{W}_{INC}\\ \end{array}\right],$ (2) $$

where $\left[\begin{array}[]{c}\cdot\\
\cdot\\
\end{array}\right]$ denotes the append operation. To enforce each incompatible node captures different information, we initialize them using the Gram-Schmidt process [] to ensure they are orthogonal to each other. Then the self-attention function [] is applied to $\boldsymbol{H}^{\prime}\in\mathbb{R}^{(n+r)\times d}$:

$$ $\small\operatorname{Attn}(\boldsymbol{H}^{\prime})=\operatorname{norm}\left( \boldsymbol{H}^{\prime}+\operatorname{softmax}\left(\frac{\boldsymbol{Q} \boldsymbol{K}^{\mathsf{T}}}{\sqrt{n+r}}\right)\boldsymbol{V}\right),$ (3) $$

$$ $\small\boldsymbol{Q}=\boldsymbol{H}^{\prime}\boldsymbol{W}_{Q},\boldsymbol{K}= \boldsymbol{H}^{\prime}\boldsymbol{W}_{K},\boldsymbol{V}=\boldsymbol{H}^{ \prime}\boldsymbol{W}_{V},$ (4) $$

where $\boldsymbol{W}_{Q},\boldsymbol{W}_{K},\boldsymbol{W}_{V}\in\mathbb{R}^{d\times
d}$ are parameter matrices and $\operatorname{norm}(\cdot)$ is a layer normalization.

In addition to the attention layer, a position-wise feed-forward network (FFN) with a layer normalization function is applied to each position to get the output node representations. The brain network of each atlas goes through a separate Disentangle Transformer. At the output of the disentangle Transformer, the incompatible nodes are discarded and only the ROI nodes are used.

Orthogonal Loss. As the brain networks derived from different atlases are based on the same fMRI data, we aim to ensure they contain similar information by filtering out inconsistent atlas-specific information. Therefore, we propose an orthogonal loss to enforce the representations of incompatible nodes to be orthogonal to each other across all atlases by minimizing their dot product:

$$ $\small\mathcal{L}_{orth}=\frac{1}{r}\sum\frac{||\boldsymbol{W}_{INC}^{a}\cdot \boldsymbol{W}_{INC}^{b}||}{||\boldsymbol{W}_{INC}^{a}||\cdot||\boldsymbol{W}_ {INC}^{b}||}.$ (5) $$

### 4.2 Inter-Atlas Message-Passing

The features at different atlases originate from totally different parcellation modes. Pulling those highly correlated features of two different atlases into a shared space allows their effective fusion. Existing literature on multi-atlas brain networks independently learns the representations of ROIs in each atlas without exchanging information across atlases [, ]. Additionally, the spatial relationship between ROIs in different atlases is neglected in these works. Our proposed AIDFusion enables inter-atlas message-passing between neighboring regions in different atlases by considering spatial information. Specifically, we use the spatial distance between the centroids of ROIs in different atlases to construct inter-atlas connections. As shown in Figure 1c, we utilize the $k$-nearest-neighbor ($k$NN) algorithm to connect each ROI to $k$ ROIs from the other atlas. Note that we only construct inter-atlas connections without considering intra-atlas connections since the information exchange within the same atlas has already proceeded in previous disentangle Transformer. Afterwards, an adjacency matrix $\boldsymbol{A}^{ab}\in\{0,1\}^{(n^{a}+n^{b})\times(n^{a}+n^{b})}$ is obtained and used for graph convolution []:

$$ $\small\operatorname{GCN}(\boldsymbol{A}^{ab},\boldsymbol{H}^{ab})=\sigma\left( \boldsymbol{D}^{-\frac{1}{2}}\boldsymbol{A}^{ab}\boldsymbol{D}^{-\frac{1}{2}} \boldsymbol{H}^{ab}\boldsymbol{W}_{GC}\right),$ (6) $$

where $\sigma$ is the activation function (e.g., ReLU), $\boldsymbol{D}$ is the degree matrix of $\boldsymbol{A}^{ab}$, $\boldsymbol{H}^{ab}\in\mathbb{R}^{(n^{a}+n^{b})\times d}$ is the combined node representation matrix for the two atlases, and $\boldsymbol{W}_{GC}$ is the learnable weight matrix of the GCN layer. An example of the adjacency matrix $\boldsymbol{A}^{ab}$ that used for inter-atlas message-passing is shown and discussed in Appendix C.

### 4.3 Subject- and Population-level Consistency

Subject-level Consistency. To ensure the high-level consistency for the two brain networks from different atlases, we introduce a contrastive loss on the subject level. First, we apply DiffPool [] to each atlas to capture higher-level patterns. The DiffPool contains two GCN layers. $\operatorname{GCN}_{pool}$ is used to learn a cluster assignment matrix $\boldsymbol{S}\in\mathbb{R}^{n\times n^{\prime}}$ as shown in Eq. (7). Herein, $n^{\prime}$ is a pre-defined number of clusters controlled by a hyperparameter named the pooling ratio. The other $\operatorname{GCN}_{emb}$ is used to obtain the embedded node feature matrix $\boldsymbol{Z}\in\mathbb{R}^{n\times d}$ as shown in Eq. (8). Both these two GCNs are defined similarly with Eq. (6). We sparsify the connectivity matrices $\boldsymbol{X}$ by keeping top 20% correlations and use them as the adjacency matrices $\boldsymbol{A}$ in these two GCNs, to avoid over-smoothing. The feature matrices of two atlas $\boldsymbol{H}$ are obtained from the output of inter-atlas message-passing.

$$ $\small\boldsymbol{S}=\operatorname{softmax}\left(\operatorname{GCN}_{pool}( \boldsymbol{A},\boldsymbol{H})\right).$ (7) $$

$$ $\small\boldsymbol{Z}=\operatorname{GCN}_{emb}(\boldsymbol{A},\boldsymbol{H}).$ (8) $$

Once we obtain the cluster assignment matrix $\boldsymbol{S}$ and the embedded node feature matrix $\boldsymbol{Z}$, we generate a new feature matrix $\hat{\boldsymbol{H}}\in\mathbb{R}^{n^{\prime}\times d}$ by $\hat{\boldsymbol{H}}=\boldsymbol{S}^{\mathsf{T}}\boldsymbol{Z}$. This coarsening process can reduce the number of nodes to get higher-level node representations. To avoid GNN treating each ROI and each node cluster equally, we adopt an entropy regularization to the assignment matrices of each atlas:

$$ $\small\mathcal{L}_{E}=\frac{1}{n^{\prime}}\sum^{n^{\prime}}_{i=1}\left( \operatorname{entropy}(\boldsymbol{S}^{a}[i,:])+\operatorname{entropy}( \boldsymbol{S}^{b}[i,:])\right),\operatorname{entropy}(\boldsymbol{p})=-\sum_{ j=1}^{n^{\prime}}\boldsymbol{p}_{j}\log(\boldsymbol{p}_{j}).$ (9) $$

We elaborate on the module of subject-level consistency in Figure 1d. Through two DiffPool layers, we produce high-quality representations for each atlas by extracting high-level node representations. Then we are able to apply a contrastive loss to them by considering representations from the same subject as positive pairs $\mathcal{P}^{pos}=\{(\hat{\boldsymbol{H}}^{a}_{i},\hat{\boldsymbol{H}}^{b}_{i}
)\}:i=1,\dots,bz\}$ and representations from different subjects as negative pairs $\mathcal{P}^{neg}=\{(\hat{\boldsymbol{H}}^{a}_{i},\hat{\boldsymbol{H}}^{b}_{
\neg i}):i=1,\dots,bz\}$:

$$ $\small\mathcal{L}_{SC}=-\operatorname{log}\frac{\sum\sum_{(\hat{\boldsymbol{H} }^{a}_{i},\hat{\boldsymbol{H}}^{b}_{i})\in\mathcal{P}^{pos}}\operatorname{exp} (\operatorname{sim}(\hat{\boldsymbol{H}}^{a}_{i},\hat{\boldsymbol{H}}^{b}_{i}) /\tau)}{\sum\sum_{(\hat{\boldsymbol{H}}^{a}_{i},\hat{\boldsymbol{H}}^{b}_{\neg i })\in\mathcal{P}^{neg}}\operatorname{exp}(\operatorname{sim}(\hat{\boldsymbol{ H}}^{a}_{i},\hat{\boldsymbol{H}}^{b}_{\neg i})/\tau)},$ (10) $$

where $\tau$ is a temperature hyper-parameter to control the smoothness of the probability distribution [], $bz$ is the batch size, and $\operatorname{sim}(\cdot)$ denotes the cosine similarity function that is applied to the same row in the two matrices.

Population-level Consistency. The readout function $\boldsymbol{m}=\operatorname{READOUT}\left(\boldsymbol{H}\right)$ is an essential component of learning the graph-level representations $\boldsymbol{m}\in\mathbb{R}^{d}$ for brain network analysis (e.g., classification), which maps a set of learned node-level embeddings to a graph-level embedding.
To further constrain the consistency for graph representations across different atlases, we introduce a mean squared error (MSE) loss on the population level. As shown in Figure 1e, a population graph $\boldsymbol{G}$ is constructed by computing the similarity of each two subjects’ graph representations in the same atlas. The intuition here is we aim to maintain the relationship of subjects across atlases, instead of directly enforcing graph representations of two atlases to be the same. Such loss is formulated as follows:

$$ $\small\mathcal{L}_{PC}=\frac{1}{bz}\sum(\boldsymbol{G}^{a}-\boldsymbol{G}^{b}) ^{2},\boldsymbol{G}[i,j]=\operatorname{sim}(\boldsymbol{m}_{i},\boldsymbol{m}_ {j}),\boldsymbol{m}_{i},\boldsymbol{m}_{j}\in\mathcal{M},$ (11) $$

where $\mathcal{M}$ is the set of graph representations in a batch.

Total Loss. The model is supervised by a commonly-used cross-entropy loss $\mathcal{L}_{cls}$ [] for graph classification. The total loss is computed by:

$$ $\small\mathcal{L}_{total}=\mathcal{L}_{cls}+\lambda_{1}*\mathcal{L}_{SC}+ \lambda_{2}*\mathcal{L}_{PC}+\lambda_{3}*\mathcal{L}_{E}+\lambda_{4}*\mathcal{ L}_{orth},$ (12) $$

where $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ and $\lambda_{4}$ are trade-off hyperparameters for balancing different losses.

## 5 Experimental Results

### 5.1 Brain Network Datasets

**Table 1: Statistics of Brain Network Datasets.**
| Dataset | Condition | Subject# | Class# | Class Name |
| --- | --- | --- | --- | --- |
| ABIDE | Autism Spectrum Disorder | 1025 | 2 | {TC, ASD} |
| ADNI | Alzheimer’s Disease | 1326 | 6 | {CN, SMC, MCI, EMCI, LMCI, AD} |
| PPMI | Parkinson’s Disease | 209 | 4 | {NC, SWEDD, prodromal, PD} |
| Matai | Mild Traumatic Brain Injury (mTBI) | 60 | 2 | {pre-season, post-season} |

We use four brain network datasets from different data sources for various disorders, which are ABIDE [] for Autism (ASD), ADNI [] for AD, PPMI [] for Parkinson’s disease (PD), and Mātai for mild traumatic brain injury (mTBI) []. Statistics of the brain network datasets are summarized in Table 1. The atlases we use are Schaefer [] and AAL [] with 100 and 116 ROIs, respectively. The detailed dataset description is provided in Appendix D while the implementation detail of our experiments is given in Appendix E.

### 5.2 Baseline Models

We use 7 single-atlas methods and 6 multi-atlas methods as baselines to evaluate our proposed AIDFusion, including: (1) Conventional machine learning (ML) models: Logistic Regression (LR) and Support Vector Machine Classifier (SVM) from scikit-learn []. These models take the flattened upper-triangle connectivity matrix as vector input, instead of using the brain network. (2) General-purposed GNNs: GCN [], a mean pooling baseline with a graph convolution network as a message-passing layer and Transformer [], a graph Transformer with mean pooling by taking the connectivity matrix as input. (3) Single-Atlas Models tailored for brain networks: BrainNetCNN [], the pioneering CNN regressor for connectome data; MG2G [], a two-stage method with an unsupervised stochastic graph embedding model; and ContrastPool [], a node clustering pooling using a dual-attention block for domain-specific information capturing. (4) Multi-atlas models: MultiLR, multi-atlas version of LR, concatenate the flatten feature of multiple atlases as input; MultiSVM, multi-atlas version of SVM, similar with MultiLR; MGRL [], a mean pooling baseline by individual GCN encoder with late fusion; MGT, a multi-atlas version of Transformer with the same fusion mechanism as MGRL; METAFormer [], a multi-atlas enhanced Transformer with self-supervised pre-training; and LeeNet [] a multi-atlas GCN approach with early-late fusion.

### 5.3 Main Results

**Table 2: Graph Classification Results (Average Accuracy ± Standard Deviation) over 10-fold-CV. The first and second best results on each dataset are highlighted in bold and underline.**
| atlas | model | ABIDE | ADNI | PPMI | Mātai |
| --- | --- | --- | --- | --- | --- |
| Schaefer100 | LR | 64.81 ± 3.70 | 61.97 ± 4.24 | 56.48 ± 6.76 | 60.00 ± 20.00 |
| SVM | 64.41 ± 5.09 | 61.52 ± 4.95 | 63.21 ± 8.62 | 56.67 ± 17.00 |  |
| GCN | 60.19 ± 2.96 | 60.40 ± 4.89 | 54.02 ± 9.06 | 56.67 ± 17.00 |  |
| Transformer | 59.90 ± 3.77 | 63.64 ± 2.61 | 59.33 ± 5.68 | 60.00 ± 20.00 |  |
| BrainNetCNN | 65.75 ± 3.24 | 60.48 ± 3.29 | 57.33 ± 10.32 | 61.67 ± 13.33 |  |
| MG2G | 64.41 ± 2.16 | 63.64 ± 5.10 | 55.45 ± 10.24 | 61.67 ± 19.79 |  |
| ContrastPool | 65.01 ± 3.84 | 65.67 ± 6.64 | 64.00 ± 6.63 | 61.67 ± 13.02 |  |
| AAL116 | LR | 63.80 ± 3.00 | 64.06 ± 1.80 | 56.00 ± 7.79 | 66.67 ± 21.08 |
| SVM | 65.72 ± 3.30 | 63.40 ± 1.90 | 64.12 ± 5.69 | 65.00 ± 20.34 |  |
| GCN | 60.10 ± 5.74 | 61.24 ± 2.47 | 53.14 ± 8.82 | 65.00 ± 21.67 |  |
| Transformer | 60.88 ± 4.39 | 63.27 ± 2.79 | 61.24 ± 7.22 | 63.33 ± 24.49 |  |
| BrainNetCNN | 64.58 ± 6.29 | 62.52 ± 2.91 | 51.19 ± 9.24 | 66.67 ± 18.33 |  |
| MG2G | 62.99 ± 4.01 | 64.41 ± 2.52 | 59.71 ± 9.11 | 70.00 ± 19.44 |  |
| ContrastPool | 64.70 ± 3.26 | 66.33 ± 4.10 | 63.56 ± 7.90 | 65.00 ± 20.82 |  |
| Schaefer100<br>+<br>AAL116 | MultiLR | 65.23 ± 5.13 | 64.99 ± 2.40 | 55.00 ± 6.25 | 56.67 ± 24.94 |
| MultiSVM | 64.31 ± 5.24 | 65.21 ± 2.74 | 63.60 ± 7.66 | 58.33 ± 17.08 |  |
| MGRL | 61.56 ± 4.90 | 62.74 ± 3.55 | 54.55 ± 10.67 | 68.33 ± 18.93 |  |
| MGT | 63.32 ± 3.90 | 63.99 ± 4.34 | 62.14 ± 9.90 | 65.00 ± 22.91 |  |
| METAFormer | 61.27 ± 4.05 | 66.52 ± 2.63 | 54.02 ± 8.81 | 61.67 ± 25.87 |  |
| LeeNet | 61.28 ± 3.12 | 64.63 ± 1.34 | 60.74 ± 4.39 | 58.33 ± 17.08 |  |
| AIDFusion (ours) | 66.35 ± 3.26 | 67.57 ± 2.04 | 66.00 ± 4.71 | 75.00 ± 13.44 |  |

We report the classification accuracy on 4 brain network datasets over 10-fold cross-validation in Table 2. For certain diseases, the effectiveness/informativeness of different atlases is different. On Matai, all the 7 baselines attain better performance with AAL116 than with Schaefer100. On ADNI, 6 out of 7 baselines also perform better with AAL116. In contrast on ABIDE, 5 out of 7 baselines achieve better results with Schaefer100 than with AAL116. It demonstrates the importance of using multi-atlas for brain network analysis instead of relying on one specific atlas. Moreover, it is evident that the multi-atlas baselines with a simple late fusion mechanism (MGRL and MGT) outperform their respective single-atlas models (GCN and Transformer). This highlights the effectiveness of multi-atlas approaches in enhancing the performance of base models. However, conventional ML models (MultiLR and MultiSVM) fail to outperform their single-atlas versions in some cases, possibly due to their inability to effectively utilize multi-atlas features with simple concatenate fusion.

We can also observe that our proposed AIDFusion consistently outperforms not only all single-atlas methods but also state-of-the-art multi-atlas methods across all datasets. Specifically, AIDFusion achieves improvements over all multi-atlas methods on these four datasets by up to 9.76% ((75.00% - 68.33%) / 68.33% = 9.76% on Mātai). Our model gains larger performance improvement on small datasets (PPMI and Mātai) than on large datasets (ABIDE and ADNI), which meets the intuition that information utilization tends to be more critical in applications with smaller sample sizes. Moreover, the results demonstrate that AIDFusion tends to have lower standard deviations compared to other multi-atlas models, indicating the robustness of AIDFusion. This robustness is particularly desirable in medical applications where consistency and reliability are crucial. Apart from accuracy, we also report other evaluation metrics on the ABIDE dataset in Appendix F. The results also reveal the superiority of AIDFusion. Besides Schaefer100 and AAL116, we also conduct experiments with 3 atlases on ADNI by using HO48 [], which is discussed in Appendix G. AIDFusion is a clear winner under all multi-atlas settings. The influence when using atlases with various resolutions is also discussed in Appendix G.

### 5.4 Model Interpretation

In neurodegenerative disorder diagnosing, identifying salient ROIs/connections associated with predictions as potential biomarkers is crucial. In this study, we utilize attention scores from the Transformer layer to generate heat maps for brain networks to interpret our model. We visualize these attention maps using the Nilearn toolbox []. Figure 2 presents attention maps for two atlases, where higher attention values mean better classification potential for AD (from the ADNI dataset). We utilized 7 networks [] to assess the connections between our highlighted ROIs and major networks potentially involved with disorders. ROIs from the AAL that do not overlap with these seven networks are excluded from the heat maps. The top 10 ROIs with the highest attention values are displayed in the brain view. As depicted in the attention maps, attention maps of both Schaefer and AAL atlases identify common connections between the visual network (VIS) and the dorsal attention network (DAN), recognized as key connectivities in AD research [, ]. Additionally, atlas-specific connections are highlighted.
For example, the attention map of Schaefer atlas emphasizes connections within the default mode network (DMN) corresponding with the observations of  Damoiseaux et al. [].
Findings on the attention map on AAL are consistent with Agosta et al. [], showing that AD is associated with connectivities in VIS, especially in frontal networks. These findings suggest that AIDFusion effectively captures complementary information from different atlases. We also find some highlighted ROIs that diverge from conventional neuroscientific understanding. For example, connections between VIS and somatomotor network (SMN) have a high attention weight in AIDFusion on Schaefer atlas, which may imply AD is related to the function of defining the targets of actions and providing feedback for visual activation. This insight has not been identified by existing literature. Discussion of ASD (from the ABIDE dataset) is included in Appendix H.

Figure: Figure 2: Visualization for attention maps on ADNI. VIS = visual network; SMN = somatomotor network; DAN = dorsal attention network; VAN = ventral attention network; LN = limbic network; FPCN = frontoparietal control network; DMN = default mode network.
Refer to caption: extracted/5886346/img/attention_brain_adni_schaefer100.png

### 5.5 Ablation Study

To inspect the effect of the key components in AIDFusion, we conduct experiments by disabling each of them without modifying other settings. The results on ADNI dataset are reported in Table 3. For inter-atlas message-passing (denoted as “IA-MP” in the table), subject-level consistency and population-level consistency, we disable them by simply removing these modules. When disabling “Disen TF”, we replace the disentangle Transformer and the identity embedding with a vanilla Transformer backbone (denoted as “TF” in the table). When disabling all key components (the first row in the table), our model will degenerate to MGT in Table 2. The results demonstrate that AIDFusion with all important modules enabled achieves the best performance. The component that affect the performance most is the population-level consistency. Besides, all variants of the proposed AIDFusion outperform the MGT baseline, demonstrating the effectiveness of our model design. We also conduct a case study of the role of the incompatible nodes in the disentangle Transformer in Appendix I. Further hyperparameter sensitivity analysis is provided in Appendix J.

**Table 3: Ablation study on the key components of AIDFusion on ADNI, with the best result bold.**
| Backbone | IA-MP | Subject-level Consistency | Population-level Consistency | acc ± std |
| --- | --- | --- | --- | --- |
| TF |  |  |  | 63.99 ± 4.34 |
| TF | ✓ | ✓ | ✓ | 66.82 ± 1.25 |
| Disen TF |  | ✓ | ✓ | 66.58 ± 1.72 |
| Disen TF | ✓ |  | ✓ | 66.37 ± 1.56 |
| Disen TF | ✓ | ✓ |  | 65.91 ± 2.08 |
| Disen TF | ✓ | ✓ | ✓ | 67.57 ± 2.04 |

### 5.6 Time Efficiency

We conducted an experiment to compare the total runtime cost of AIDFusion with other multi-atlas baselines. The results, reported in Table 4, demonstrate that AIDFusion requires dramatically fewer epochs to converge, resulting in significantly less time spent on ABIDE, ADNI, and PPMI datasets. For the Mātai dataset, AIDFusion’s time cost is still comparable with the other baselines. Besides, since AIDFusion does not contain any repeat layers as other baselines do, it has fewer parameters and thus results in higher efficiency. This showcases the efficiency of the proposed AIDFusion.

**Table 4: Time efficiency analysis. Total time (h) was recorded with a single run (including training, validation, and test) with 10-fold CV.**
|  | ABIDE | ADNI | PPMI | Mātai | #Param |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Time (h) | #Epoch | Time (h) | #Epoch | Time (h) | #Epoch | Time (h) | #Epoch |  |
| MGRL | 0.56 | 261.9 ± 0.7 | 0.91 | 262.7 ± 0.8 | 0.15 | 272.2 ± 8.0 | 0.09 | 291.2 ± 14.7 | 378k |
| MGT | 0.78 | 263.9 ± 2.4 | 0.89 | 108.2 ± 1.1 | 0.08 | 134.3 ± 10.9 | 0.18 | 266.4 ± 4.2 | 273k |
| METAFormer | 1.73 | 263.2 ± 1.5 | 1.47 | 268.3 ± 2.2 | 0.16 | 266.5 ± 4.7 | 0.12 | 270.4 ± 3.6 | 1886k |
| LeeNet | 1.56 | 200.0 ± 0.0 | 1.76 | 200.0 ± 0.0 | 0.19 | 200.0 ± 0.0 | 0.07 | 200.0 ± 0.0 | 526k |
| AIDFusion (ours) | 0.12 | 48.5 ± 19.0 | 0.26 | 64.6 ± 14.1 | 0.03 | 34.9 ± 12.0 | 0.10 | 119.5 ± 13.8 | 235k |

## 6 Conclusion

In this paper, we presented the Atlas-Integrated Distillation and Fusion network (AIDFusion), a novel approach to multi-atlas brain network classification. The disentangle Transformer mechanism, combined with inter-atlas message-passing and consistency constraints, effectively integrates complementary information across different atlases and ensures cross-atlas consistency at both the subject and population levels. Our extensive experiments on four fMRI datasets demonstrate that AIDFusion outperforms state-of-the-art methods in terms of classification accuracy and efficiency. Moreover, the patterns identified by AIDFusion align well with existing domain knowledge, showcasing the model’s potential for providing interpretable insights into neurological disorders.
Currently, our discussion of multi-atlas brain networks is restricted to 3 atlases, i.e., AAL, Schaefer, and HO. It is also worth extending the exploration to more atlases to find out which one benefits our model the most. Besides, experiments for atlases with various numbers of ROIs (Appendix F) indicate that AIDFusion performs better when the two atlases have a similar number of ROIs. How to utilize atlases with different resolutions remains a problem. Exploring datasets with more neuroimaging modalities and more atlases with different resolutions will provide insights into how atlas resolution affects the performance of multi-atlas methods. We plan to advance multi-scale brain network analysis by creating public benchmarks and leveraging inter-scale information to capture the brain’s hierarchical organization more effectively.

## Appendix A Difference between Multi-atlas and Multi-template Methods

In brain MRI analysis, the concepts of template and atlas can sometimes be confused. The atlas in our paper refers to a detailed map of brain structures [], often derived from anatomical, functional, and histological data, and includes labels for different brain regions based on specific criteria. A template, on the other hand, is a standard reference image that serves as a common coordinate system for comparing different brain images []. It is usually created by averaging brain images from a group of subjects, providing a standardized space for registering or aligning individual brain images. This allows for comparison and combination of data across different subjects or groups. Figures 3 and 4 illustrate the difference between atlas and template. Although some existing works [, , , ] are named multi-atlas, they are more akin to multi-template methods. Instead of registering brain images to different spaces in multi-template methods, the multi-atlas methods discussed in our paper segment brain images in the common space to define ROIs differently.

Figure: Figure 3: AAL116 and Schaefer100 atlases. Each atlas is based on a different parcellation hypothesis.
Refer to caption: x2.png

Figure: Figure 4: MNI template T1-w image.
Refer to caption: extracted/5886346/img/mni_t1.png

## Appendix B Notation Table

Notation-wise, we use calligraphic letters to denote sets (e.g., $\mathcal{X}$), bold capital letters to denote matrices (e.g., $\boldsymbol{X}$), and strings with bold lowercase letters to represent vectors (e.g., $\boldsymbol{x}$). Subscripts and superscripts are used to distinguish between different variables or parameters, and lowercase letters denote scalars. We use $\boldsymbol{S}[i,:]$ and $\boldsymbol{S}[:,j]$ to denote the $i$-th row and $j$-th column of a matrix $\boldsymbol{S}$, respectively. Table 5 summarizes the notations used throughout the paper.

**Table 5: Notation table**
| Notation | Description |
| --- | --- |
| $\mathcal{D}$ | Input dataset |
| $\boldsymbol{X}_{a},\boldsymbol{X}_{b}$ | A brain network of atlases $a$ and $b$ |
| $y_{X}$ | Label of $\boldsymbol{X}_{a}$ and $\boldsymbol{X}_{b}$ |
| $n_{a},n_{b}$ | Number of nodes in atlases $a$ and $b$ |
| $\boldsymbol{H}_{ID}$ | Identity-encoded feature matrix |
| $\boldsymbol{m}$ | Brain network representation |
| $d$ | Dimensionality of node representations |
| $\boldsymbol{W}_{ID},\boldsymbol{W}_{INC},\boldsymbol{W}_{GC},\boldsymbol{W}_{Q<br>},\boldsymbol{W}_{K},\boldsymbol{W}_{V}$ | Parameter matrices |
| $\boldsymbol{H}$ | Node representations after inter-atlas message-passing |
| $\boldsymbol{A}^{ab}$ | The cross-atlas adjacency matrix for atlases $a$ and $b$ |
| $\boldsymbol{H}^{ab}$ | The combined node representations for atlases $a$ and $b$ |
| $i,j$ | Index for matrix dimensions |
| $bz$ | Batch size |

## Appendix C An example of the Adjacency Matrix for Inter-Atlas Message-Passing

As shown in Figure 5, we generated an adjacency matrix for Schaefer100 and AAL116 by setting $k=5$. The connections between spatial neighborhoods across atlases are constructed. Note that node $v$ in atlas 1 is one of the nearest $k$ neighbors of node $u$ in atlas 2 does not mean $u$ is one of the nearest $k$ neighbors of $v$, thus the adjacency matrix is asymmetry.

Figure: Figure 5: The adjacency matrix for inter-atlas message-passing.
Refer to caption: extracted/5886346/img/ca_adj.png

## Appendix D Detailed Dataset Description

ABIDE The ABIDE initiative supports the research on ASD by aggregating functional brain imaging data from laboratories worldwide. ASD is characterized by stereotyped behaviors, including irritability, hyperactivity, depression, and anxiety. Subjects in the dataset are classified into two groups: TC and individuals diagnosed with ASD.

ADNI The ADNI raw images used in this paper were obtained from the ADNI database ([adni.loni.usc.edu](adni.loni.usc.edu)). The ADNI was launched in 2003 as a public-private partnership, led by Principal Investigator Michael W. Weiner, MD. The primary goal of ADNI has been to test whether serial magnetic resonance imaging (MRI), PET, other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of mild cognitive impairment (MCI) and early AD. For up-to-date information, see [www.adni-info.org](www.adni-info.org). We include subjects from 6 different stages of AD, from cognitive normal (CN), significant memory concern (SMC), mild cognitive impairment (MCI), early MCI (EMCI), late MCI (LMCI) to AD.

PPMI The PPMI is a comprehensive study aiming to identify biological markers associated with Parkinson’s risk, onset, and progression. PPMI comprises multimodal and multi-site MRI images. The dataset consists of subjects from 4 distinct classes: normal control (NC), scans without evidence of dopaminergic deficit (SWEDD), prodromal, and PD.

Mātai
Mātai is a longitudinal single site, single scanner study designed for detecting subtle changes in the brain due to a season of playing contact sports. This dataset consists of the brain networks preprocessed from the data collected from Gisborne-Tairāwhiti area, New Zealand, with 35 contact sport players imaged at pre-season (N=35) and post-season (N=25) with subtle brain changes confirmed using diffusion imaging study due to playing contact sports.

## Appendix E Implementation Details

The settings of our experiments mainly follow those in []. We split each dataset into 8:1:1 for training, validation and test, respectively. We evaluate each model with the same random seed under 10-fold cross-validation and report the average accuracy. The whole network is trained in an end-to-end manner using the Adam optimizer []. We use the early stopping criterion, i.e., we halve the learning rate when there is no further improvement on the validation loss during 25 epochs and stop the training once the learning rate is smaller than the minimum rate we set. All the codes were implemented using PyTorch [] and Deep Graph Library [] packages. All experiments were conducted on a Linux server with an AMD Ryzen Threadripper PRO 5995WX 64-Cores and an NVIDIA GeForce RTX 4090. The version for the software we used in AIDFusion is listed in Table 6.

**Table 6: The software dependency of AIDFusion.**
| Dependency | Version |
| --- | --- |
| Python | 3.10.13 |
| cudatoolkit | 12.2 |
| pytorch | 2.2.1+cu121 |
| DGL | 2.1.0+cu121 |
| scikit-learn | 1.4.1.post1 |
| numpy | 1.26.4 |
| matplotlib | 3.8.3 |
| nilearn | 0.10.4 |

All the baselines used in this paper are implemented by ourselves. In AIDFusion, we adopt a mean pooling layer as the readout function and a two-layer MLP with ReLU as the prediction head. The number of incompatible nodes $r$ is set to 4. The number of clusters $n^{\prime}$ for the cluster assignment matrix $\boldsymbol{S}$ is set to half of the average number of nodes for all atlases, e.g., for Schaefer100 and AAL116, $n^{\prime}=(100+116)/2/2=54$. The temperature hyper-parameter $\tau$ in Eq. (10) is set to 0.75. The hidden dimensions of all layers are set to 100. The batch size of each dataset is set to 10% of the subject number in the dataset. We tuned the other hyperparameters on validation set including $k$ in $k$NN for the inter-atlas message-passing, trade-off hyperparameters $\lambda_{1},\lambda_{2},\lambda_{4},\lambda_{4}$ for loss function Eq. (12), the initial learning rate $init_lr$, and the minimum learning rate $min_lr$ used for early stop. To be specific, we search $k$ from {3, 5, 10}, $\lambda_{1}$ from {1e-2, 1e-1, 1e0, 1e1, 1e2}, $\lambda_{2}$ from {1e-2, 1e-1, 1e0, 1e1, 1e2}, $\lambda_{3}$ from {1e-6, 1e-5, 1e-4, 1e-3, 1e-2}, $\lambda_{4}$ from {1e-1, 1e0, 1e1}, $inir_lr$ from {5e-5, 8e-5, 1e-4, 2e-4, 7e-4} and $inir_lr$ from {6e-5, 1e-5, 5e-6, 1e-6}. The optimized hyperparameters for AIDFusion is reported in Table 7.

**Table 7: The optimized hyperparameters for AIDFusion.**
|  | ABIDE | ADNI | PPMI | Mātai |
| --- | --- | --- | --- | --- |
| $k$ | 10 | 5 | 5 | 5 |
| $\lambda_{1}$ | 1e1 | 1e1 | 1e1 | 1e-2 |
| $\lambda_{2}$ | 1e-1 | 1e1 | 1e1 | 1e0 |
| $\lambda_{3}$ | 1e-2 | 1e-5 | 1e-4 | 1e-3 |
| $\lambda_{4}$ | 1e0 | 1e0 | 1e0 | 1e0 |
| $init_lr$ | 1e-4 | 8e-5 | 1e-4 | 7e-4 |
| $min_lr$ | 6e-5 | 6e-5 | 6e-5 | 6e-5 |

## Appendix F Results of More Evaluation Metrics

**Table 8: Results of more evaluation metrics on ABIDE dataset. The best result is highlighted in bold.**
|  | Precision | Recall | micro-F1 | ROC-AUC |
| --- | --- | --- | --- | --- |
| MGRL | 59.90 ± 6.42 | 60.03 ± 6.31 | 59.71 ± 5.10 | 61.39 ± 5.08 |
| MGT | 60.33 ± 4.78 | 69.29 ± 6.74 | 64.21 ± 3.62 | 63.60 ± 3.80 |
| METAFormer | 59.33 ± 4.05 | 61.91 ± 7.79 | 60.20 ± 4.26 | 61.31 ± 3.94 |
| LeeNet | 64.30 ± 5.24 | 43.04 ± 6.09 | 51.23 ± 4.74 | 60.44 ± 3.06 |
| AIDFusion (ours) | 62.25 ± 3.00 | 74.80 ± 4.38 | 67.90 ± 3.12 | 66.73 ± 3.25 |

In addition to accuracy, we also report other evaluation metrics, including precision, recall, micro-F1, and ROC-AUC, for all the multi-atlas deep models on the ABIDE dataset. As displayed in Table 8, AIDFusion performs the best across all these metrics except for precision. We observe that compared to other baselines, our AIDFusion can significantly improve recall without compromising precision. Moreover, in medical diagnostics, it is crucial to ensure that all individuals with a certain condition are correctly identified, even if it leads to some false positives. Missing a true positive (failing to diagnose a disease) can have severe consequences, while false positives can be further examined or retested. Therefore, models with higher recall rates, such as our AIDFusion, are more suitable for real-life medical auxiliary diagnosis.

**Table 9: Results of more atlases on ADNI dataset. The best results for each atlas setting are highlighted in bold.**
| Atlas | model | acc ± std |  |  |
| --- | --- | --- | --- | --- |
| Schaefer100 | AAL116 | HO48 |  |  |
| ✓ | ✓ |  | MGRL | 62.74 ± 3.55 |
| MGT | 63.99 ± 4.34 |  |  |  |
| METAFormer | 66.52 ± 2.63 |  |  |  |
| LeeNet | 64.63 ± 1.34 |  |  |  |
| AIDFusion | 67.57 ± 2.04 |  |  |  |
|  | ✓ | ✓ | MGRL | 64.48 ± 1.68 |
| MGT | 60.48 ± 1.91 |  |  |  |
| METAFormer | 64.48 ± 1.68 |  |  |  |
| LeeNet | 64.48 ± 1.58 |  |  |  |
| AIDFusion | 65.99 ± 2.62 |  |  |  |
| ✓ |  | ✓ | MGRL | 57.92 ± 2.82 |
| MGT | 60.64 ± 3.52 |  |  |  |
| METAFormer | 65.23 ± 3.25 |  |  |  |
| LeeNet | 63.42 ± 1.82 |  |  |  |
| AIDFusion | 65.91 ± 1.80 |  |  |  |
| ✓ | ✓ | ✓ | MGRL | 56.63 ± 4.66 |
| MGT | 62.89 ± 1.28 |  |  |  |
| METAFormer | 66.33 ± 2.80 |  |  |  |
| LeeNet | 64.40 ± 1.71 |  |  |  |
| AIDFusion | 66.59 ± 1.77 |  |  |  |

## Appendix G Results with More Atlases

To further evaluate the effectiveness of AIDFusion with other atlases, we conducted experiments using an additional atlas, HO48 [], on the ADNI dataset. The results, presented in Table 9 indicate that the proposed AIDFusion achieves the best performance across all four atlas settings. Notably, increasing the number of atlases does not necessarily enhance model performance. In some cases, using all three atlases yields lower accuracy compared to the combination of Schaefer100 and AAL116. Additionally, the choice of atlas combination is crucial for multi-atlas methods. In dual-atlas experiments, combining two atlases with a similar number of ROIs (Schaefer100 and AAL116) can mutually enhance and significantly improve model performance.

To further explore how the resolution of atlases will influence the performance of our model, we conduct experiments for atlases with various numbers of ROIs. The Schaefer atlas allows adjusting the resolution of ROIs (e.g., from 100 to 1000). We selected Schaefer100 for detailed study because a previous study [] found that using 100 ROIs with the Schaefer atlas usually performs better than using more ROIs. To verify this conclusion in multi-atlas brain network classification, we conducted experiments using AAL116 combined with Schaefer200, Schaefer500, and Schaefer1000. Results showed that AAL116 combined with Schaefer100 achieves the best results. Additionally, we provided experimental results using the HO48 atlas in Appendix F. The dual-atlas experiments demonstrate that combining two atlases with a similar number of ROIs (AAL116 and Schaefer100) can mutually enhance and significantly improve model performance. It is also interesting for us to explore using both atlases of around 200 nodes (or around 1000 nodes). We will leave such exploration about multi-scale brain networks in the future.

**Table 10: Results of more atlases with different resolutions on ABIDE dataset. The best results for each atlas setting are highlighted in bold.**
| Atlas 1 | Atlas 2 | acc ± std |
| --- | --- | --- |
| AAL116 | Schaefer100 | 66.35 ± 3.26 |
| AAL116 | Schaefer200 | 65.03 ± 5.10 |
| AAL116 | Schaefer500 | 64.72 ± 4.80 |
| AAL116 | Schaefer1000 | 63.15 ± 2.80 |

Figure: Figure 6: Visualization for attention maps on ABIDE. VIS = visual network; SMN = somatomotor network; DAN = dorsal attention network; VAN = ventral attention network; LN = limbic network; FPCN = frontoparietal control network; DMN = default mode network.
Refer to caption: extracted/5886346/img/attention_brain_abide_schaefer100.png

## Appendix H Brain Network Attention Map on ABIDE

In the ASD analysis using the ABIDE dataset (Figure 6), AIDFusion identifies common connections within the lingual gyrus of the VIS network in both Schaefer100 and AAL116. This aligns with existing ASD studies which suggest a greater reliance on visual perceptual processing and more effortful top-down control during semantic processing in ASD []. Besides, for Schaefer, AIDFusion emphasizes ROIs in the DAN, particularly the connection between the right posterior and VIS, consistent with Koshino et al. [] who found lower activation in ASD subjects in the inferior left prefrontal area (verbal processing and working memory) and the right posterior temporal area (theory of mind processing). In the AAL analysis, connections in the DMN and the frontoparietal control network (FPCN) are highlighted, supporting the understanding that (1) dysfunctions in DMN nodes and their interactions contribute to difficulties of ASD in attending to socially relevant stimuli [], and (2) the ASD group shows reduced lateral frontal activity and diminished hippocampal connectivity, especially between the hippocampus and FPCN regions []. These findings clarify why the features identified by AIDFusion are distinctive for ASD biomarker.

## Appendix I Ablation Study for Incompatible Nodes

In this section, we further explore the function of incompatible nodes by visualizing the attention map of ADFusion w/o incompatible nodes. The attention maps are shown in Figure 7. We can observe that, when not using incompatible nodes, the attentions of two atlases (in the right column) are remarkably imbalanced. Attentions on Schaefer are much higher than those on AAL. Besides, the attention map of AAL exhibits over-smoothing and no highlighted network is found, which indicates the model is not able to extract the distinguishable connections. This case study demonstrates that the incompatible nodes enable the model to filter out the inconsistent atlas-specific information.

Figure: Figure 7: Visualization for attention maps of AIDFusion w/ and w/o incompatible nodes.
Refer to caption: extracted/5886346/img/attention_map_adni_schaefer100.png

## Appendix J Hyperparameter Analysis

In this section, we study the sensitivity of four trade-off hyperparameters in Eq. (12). All experiments are conducted on the ADNI dataset. We tune the value of $\lambda_{1}$ from 1e0 to 1e2, $\lambda_{2}$ from 1e0 to 1e2, $\lambda_{3}$ from 1e-6 to 1e-4 and $\lambda_{4}$ from 1e-1 to 1e1. The results presented in Table 11 show that our model performs the best when $\lambda_{1}=$ 1e1, $\lambda_{2}=$ 1e1, $\lambda_{3}=$ 1e-5 and $\lambda_{4}=$ 1e0. We can exhibit that these trade-off hyperparameters in the loss function will marginally affect the model performance on ADNI (less than 1%), which demonstrates the stability of AIDFusion.

**Table 11: The hyperparameter sensitivity analysis for AIDFusion on ADNI dataset.**
| $\lambda_{1}$ | $\lambda_{2}$ | $\lambda_{3}$ | $\lambda_{4}$ | acc ± std |
| --- | --- | --- | --- | --- |
| 1e0 | 1e1 | 1e-5 | 1e0 | 66.97 ± 1.95 |
| 1e1 | 1e0 | 1e-5 | 1e0 | 66.44 ± 2.88 |
| 1e1 | 1e1 | 1e-6 | 1e0 | 67.04 ± 2.20 |
| 1e1 | 1e1 | 1e-5 | 1e-1 | 66.82 ± 1.98 |
| 1e1 | 1e1 | 1e-5 | 1e0 | 67.57 ± 2.04 |
| 1e1 | 1e1 | 1e-5 | 1e1 | 66.82 ± 2.64 |
| 1e1 | 1e1 | 1e-4 | 1e0 | 67.04 ± 2.21 |
| 1e1 | 1e2 | 1e-5 | 1e0 | 66.89 ± 2.10 |
| 1e2 | 1e1 | 1e-5 | 1e0 | 66.21 ± 2.34 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: These questions are adapted / copied from the weaknesses listed above:

1. It is unclear whether the parameters of the MLP in Eq. 1 are learnable. If so, what is the purpose behind W_ID?
2. The efficacy of the "disentangle Transformer" is not fully supported. How well are conflicting atlases disentangled? Was the orthogonal loss actually useful towards separating shared and conflicting information across atlases? An experiment to support the authors' intuition is missing.
3. The same is true for both subject- and population-level consistencies as well as message-passing. The explanation is sensible in text, but the actual efficacy of the proposed architecture and losses is not described in the experiments and results. The reader sees that the downstream classification is improved in light of the proposed components, but it is unknown whether these components accomplish what they are intended to. What evidence do the authors have towards these claims?
soundness: 3
strengths: This approach is well-motivated, as there is no consensus on the number of atlases to use. Using a single atlas conforms to the biases induced that particular atlas, and using multiple atlases involves the use of novel methods which do not regularize consistency across multiple atlases. The proposed approach aims to achieve both consistency across multiple atlases as well as providing their model ROI-level "information exchange." Accomplishing these two aims required the development and use of several techniques, including the proposed "disentangle transformer" and "identity embedding", the use of an orthogonality loss on a component of the representations, a nearly-linear learnable mapping for inter-atlas message passing, a bespoke contrastive loss for subject-level consistency, and a similarity loss for "maintain the relationship of subjects across atlases."

This paper is well-motivated and fairly comprehensive with respect to the classification experiments. The contribution of the approach as an improvement for classification of brain networks is clear. I recommend this paper be accepted.
summary: This paper aims to classify brain networks from fMRI BOLD ROIs using multiple (potentially conflicting) atlases. The authors resolve conflicts with a deep neural network (specifically a "disentangle transformer") alongside external prior information related to subject and population constraints.

To evaluate the efficacy of the proposed approach, the authors compare against several classical and current methods on four disparate datasets with a classification downstream task. Their method achieves the greatest quantitative results compared to all others in nearly all cases.
weaknesses: In addition to my recommendation for acceptance, I enumerate some concerns I have below:

1. The technical claims are not fully supported. The "identity embedding" is either mis-named or its explanation is unclear. The embedding is not identity; it seems instead to simply be a learnable embedding. It is unclear whether the parameters of the MLP in Eq. 1 are learnable. If so, what is the purpose behind W_ID?
2. The efficacy of the "disentangle Transformer" is not fully supported. How well are conflicting atlases disentangled? Was the orthogonal loss actually useful towards separating shared and conflicting information across atlases? An experiment to support the authors' intuition is missing.
3. The same is true for both subject- and population-level consistencies as well as message-passing. The explanation is sensible in text, but the actual efficacy of the proposed architecture and losses is not described in the experiments and results. The reader sees that the downstream classification is improved in light of the proposed components, but it is unknown whether these components accomplish what they are intended to.
4. Some figure and table captions are too terse. Figure1, table 1, table 3 could benefit from better description to help the reader understand their contents.
5. Table 5 experiment times include training, validation, and testing across multiple cross-validation folds. This is unusual; instead, reporting training+validation time and then testing time (for a single subject) separately would be more conventional. This guides future users of the proposed method towards how much time and compute should be budgeted for training the approach as well as what hardware is necessary to use the proposed method at scale in their own work.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - How is transformer training feasible on tiny datasets with sample sizes of N=60 and N=1300 without any self supervised pretraining?
- Why work with processed connectomes instead of raw data for transformer training?
- Please perform significance testing with multiple comparisons adjustments.
- Please describe how the conventional baselines were tuned and whether they were regularized
- Please add ablations for the key contributions in 4.1
soundness: 3
strengths: - The use of [transformer registers](https://arxiv.org/abs/2309.16588) (herein called “incompatible nodes”) for potentially filtering out incompatible information across atlases is a novel and interesting application.
- The use of spatial distances between regions of interest in the brain when constructing the message-passing framework is interesting and, to my limited knowledge, not commonly done.
- Generally clearly and straightforwardly presented.
summary: Resting-state functional MRI is often used to predict behavioral phenotypes by measuring correlations between disparate regions/parcels of the brain. There are many different parcellation conventions (“atlases” in this context) and downstream analyses are often quite dependent on the choice of atlas.

Submission 4706 presents a classification framework that integrates multiple different atlas conventions for a given subject. They propose a transformer that extracts complementary information and gain modest accuracy increases across four different datasets.
weaknesses: I do not work in this specific subfield and would be happy to revisit my score and look forward to the discussion phase. I also did not read the appendix so please correct me if I missed something.

### 1. Unclear motivation for transformer-based approach on processed connectomes

The datasets in fMRI connectomics are (understandably) limited in sample size, ranging here from N=60 to N=1300. 

However, it is unclear how the submission can adequately train transformers on 60 data samples. Without inductive bias (beyond permutation equivariance), transformers require several orders of magnitude higher training sets across the literature. I do not see any mention of pretraining either on larger datasets (e.g. UKBiobank) that would potentially enable few-shot finetuning.

Further, could the authors please elaborate on why they chose to work only with the processed connectomes instead of the high-dimensional raw data where there may be more potential for self-supervised pretraining?

### 2. Using only two atlases in experiments

The submission motivates itself by claiming that multiple atlases provide complementary information (which is a reasonable hypothesis) but its experiments only use two atlases (Schaefer100 and AAL116). Given that there is a wide range of potential atlasing procedures and many other atlases are often used, could the authors please clarify why two atlases were used in the experiments?

### 3. Unclear significance

The main comparative results (presented in Table 2) are largely well within each others’ error bars without clear significance.

Is there high inter-subject or inter-site variability s.t. standard deviations are inflated? Plotting per-subject performance as a supplemental figure should clarify this.

Further, could the authors please perform significance tests with corrections for multiple comparisons s.t. readers can assess whether these gains are meaningful?

### 4. Baseline choices and clarifications

#### 4.1. Iterative/conventional baselines

The only iterative baselines included for comparison are logistic regression and SVMs. It is not mentioned whether these methods were regularized in any form (e.g. LASSO) and, if so, whether the regularization hyperparameters were tuned at all. As far as I’m aware, practitioners [largely use LASSO-style methods for behavior prediction](https://db.humanconnectome.org/megatrawl/HCP820_MegaTrawl_April2016.pdf) and find that the regularization weight strongly affects the results.

Further, to my knowledge, deep nets and methods such as kernel regression are largely equivalent on this task when tested on much larger sample sizes ( [reference](https://pubmed.ncbi.nlm.nih.gov/31610298/) ).

As all results are well within each other's error bars (see point 3 above), could the authors please detail why only two non-DL methods were benchmarked and whether these methods were regularized and tuned?

#### 4.2 Transformer baselines

Several transformer-based approaches to fMRI classification are neither cited nor benchmarked against. For example,
- https://link.springer.com/chapter/10.1007/978-3-031-43993-3_28 (MICCAI’23)
- https://link.springer.com/chapter/10.1007/978-3-031-72390-2_14 (MICCAI’24)

Additionally, while not a one-to-one comparison to these papers use of static splits vs. the submission using 10-fold CV, these papers report much higher performance on ABIDE (high-70s vs the paper’s mid-60s accuracy). 

Could the authors please address differences relative to these works, whether they could be used as baselines, and what could explain the major performance differences?

### 5. Missing ablations for key contributions

The paper has several moving parts, all of which are claimed as novel contributions. However, the core parts of Section 4.1 (the identity embedding, the “disentangle Transformer”, and the orthogonal loss) are not ablated. Please do so in the rebuttal or future versions.

### 6. Minor
These aspects do not affect my score.

- The paper claims that not all regions of the brain are covered by all atlases and thus integrating multiple atlases may have benefits. Please correct me if I’m wrong but don’t atlases such as Shen and Craddock cover most of the brain? 
- Somewhat orthogonally, multiple atlas _harmonization_ can also be performed as with iterative methods like [CAROT](https://www.sciencedirect.com/science/article/am/pii/S136184152300124X). Could the authors please clarify the differences in motivation for their approach vs. atlas harmonization? I ask as much of the paper is motivated by the goal of integrating information across atlases and there are existing methods to do so.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. For the identity embedding in the disentangle transformer. It is not clear how the identity embedding was implemented. "... a parameter matrix W_ID to encode nodes within the same ROI", but it looks like one node corresponds to one brain ROI in the proposed method.
2. The brain networks from different atlases will contain atlas-consistent information shared across atlases and atlas-specific information, both types of information may be informative for classification. While the proposed method adopts several techniques to get enhanced atlas-consistent information, the atlas-specific information may not be effectively captured. Not sure if the proposed disentangle transformer may discard atlas-specific information that are useful for classification.
3. The Eq.(9) is not correct. I think i refers to ROI in atlas a or b, not cluster.
4. The AAL atlas contains both cerebral cortex, subcortical structures, and cerebellum regions, while the Schaefer atlas only covers cerebral cortex regions. Not sure if it is proper to learn consistent representations from brain networks with different brain coverages.
5. As demonstrate in the experimental results, the fusion of multiple arbitrary atlases may not improve classification performance. A principled way to identify what atlases should be fused is needed.
6. When applied to more than 2 atlases, will the orthogonal loss, inter-atlas message passing, and subject/population-level consistency be computed for each pair of atlases?
soundness: 3
strengths: 1. The idea of learning unified representation from multiple brain networks is promising to improve classification performance.
2. Multiple techniques are proposed to facilitate the unified representation learning, e.g., disentangle transformer, inter-atlas message-passing, and multi-level consistency.
summary: In this paper, a novel method named AIDFusion is proposed to learn unified and informative representations of brain functional networks derived from multiple brain atlases, which are expected to facilitate the brain network based classification. The method consists of several modules including disentangle transformer, inter-atlas message-passing, subject-level and population-level consistency to learn atlas-consistent information. The proposed method has been evaluated on multiple datasets to demonstrate its effectiveness.
weaknesses: 1. The brain networks from different atlases will contain atlas-consistent information and atlas-specific information, both types of information may be informative for classification. While the proposed method adopts several techniques to get enhanced atlas-consistent information, the atlas-specific information may not be effectively captured. Not sure if the proposed disentangle transformer may discard atlas-specific information that are useful for classification.
2. As demonstrate in the experimental results, the fusion of multiple arbitrary atlases may not improve classification performance. It is not clear and not verified what atlases should be fused.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Given the small sample sizes of rs-fMRI datasets, as stated in lines 325-330, how did you overcome the risk of overfitting during the training process and determine hyperparameters?
soundness: 2
strengths: - Originality: This paper demonstrates some novelty. For instance, the concept of inter-atlas message-passing is interesting.
- Quality: The methodology is validated across four datasets representing different diseases, effectively showcasing its effectiveness and efficiency.
- Clarity: The paper is well-structured and easy to follow, enhancing comprehension for readers.
- Significance: This research offers new insights into brain network analysis within the field of neuroscience.
summary: This paper proposes a method called AIDFusion, designed to improve brain network classification using rs-fMRI data. 

The authors note that existing methods often rely on single atlases for classification, while approaches that utilize multiple atlases tend to overlook cross-atlas consistency and fail to facilitate information exchange at the ROI level. To address these issues, the authors first introduce a disentangled Transformer to learn atlas-level embeddings. They then propose an inter-atlas message-passing mechanism to fuse complementary information across brain regions. Additionally, subject-level and population-level consistency losses are employed to enhance cross-atlas coherence.
weaknesses: **1. There are too many hyperparameters to tune, which undermines the credibility of the results.**

For example, the training loss comprises five components, with four parameters requiring tuning. Additionally, hyperparameters from other modules of the method, such as the k value in KNN (lines 258-259), the keeping ratio (lines 278-279), and the temperature parameter (lines 302-303), also need to be set appropriately.

**2. The presence of numerous networks to learn complicates the training process, potentially leading to instability.**

This includes the transformers in lines 216-240 and three GCNs in lines 263-267 and 280-285.

**3. The motivations for some modules are unconvincing.**

For example, in lines 216-221 and 242-245, the authors mention introducing incompatible nodes and using orthogonal loss to filter out inconsistent atlas-specific information, citing the [CLS] token in NLP as motivation. However, the relationship is not adequately established, and the effect of these incompatible nodes is not clearly illustrated or justified.

**4. The comparisons are insufficient.**

Given that the atlas embedding learning is a transformer-based method, comparisons with additional transformer-based methods, such as [1], should be included. Additionally, other multi-atlas methods, like [2], should be considered. the manuscript may lack ablation studies that directly concatenate embeddings from single-atlas methods across different atlases similar in [2][3].

*[1] Kan, X., Dai, W., Cui, H., Zhang, Z., Guo, Y., & Yang, C. (2022). Brain network transformer. Advances in Neural Information Processing Systems, 35, 25586-25599.*

*[2] Wang, W., Xiao, L., Qu, G., Calhoun, V. D., Wang, Y. P., & Sun, X. (2024). Multiview hyperedge-aware hypergraph embedding learning for multisite, multiatlas fMRI based functional connectivity network analysis. Medical Image Analysis, 94, 103144.*

*[3] Zhao, B. W., You, Z. H., Wong, L., Zhang, P., Li, H. Y., & Wang, L. (2021). MGRL: Predicting Drug-Disease Associations Based on Multi-Graph Representation Learning. Frontiers in genetics, 12, 657182. https://doi.org/10.3389/fgene.2021.657182*

Meta-review: This paper introduces a transformer for fMRI data (presumably only resting state correlation matrices). They present classification task accuracy on four datasets (ABIDE, ADNI, PPMI, and Matai).

Reviewer responses are mixed, though discussion has improved scores. There remain unanswered questions however. Among these, specifically concerning are the high standard error; after rebuttal, there are now reportedly low p-values, but these are paired. This appears unanswered, as raised by `CMqs` and `3zUh`, and inconsistent with literature results, as raised by `9YF5`. Reviewer `9YF5` also raises questions about the number of hyper-parameters that may be tuned. Even if all other claims are believed at face value, this is problematic.

There are positive aspects to this submission and innovations in their work. However, it is below acceptance quality, and experimental concerns are unacceptable in general.

Beyond this, I find that the motivation is unclear. Is atlas fusion, embedding, or disambiguation the science in question? If so, we should see experiments showing that this model consistently identifies "correct" atlases. This is both a question at the identity embedding stage and at the "disentangle transformer" stage, both of which remain untested experimentally (with respect to identification properties). If the purpose is not better atlases (or identifying a "true" atlas), then pure classification remains the motivating factor, which is weak.

I encourage the authors to incorporate the concerns of Reviewer `CMqs` in particular. I find that the contributions of this manuscript have merit, but remain unpolished and leave questions unexplored in the science of brains and fMRI. Seeing as the transformer contributions are low (which is admittedly not the purpose of the paper; it is an "application" paper), for the above reasons I recommend rejection of this submission.

Decision: Reject