Brain Network Transformer
Authors: Xuan Kan
Wei Dai
Hejie Cui
Zilong Zhang
Ying Guo
Carl Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Brain Network
Graph Transformer
Graph Neural Network

Abstract: Human brains are commonly modeled as networks of Regions of Interest (ROIs) and their connections for the understanding of brain functions and mental disorders. Recently, Transformer-based models have been studied over different types of data, including graphs, shown to bring performance gains widely. In this work, we study Transformer-based models for brain network analysis. Driven by the unique properties of data, we model brain networks as graphs with nodes of fixed size and order, which allows us to (1) use connection profiles as node features to provide natural and low-cost positional information and (2) learn pair-wise connection strengths among ROIs with efficient attention weights across individuals that are predictive towards downstream analysis tasks. Moreover, we propose an Orthonormal Clustering Readout operation based on self-supervised soft clustering and orthonormal projection. This design accounts for the underlying functional modules that determine similar behaviors among groups of ROIs, leading to distinguishable cluster-aware node embeddings and informative graph embeddings. Finally, we re-standardize the evaluation pipeline on the only one publicly available large-scale brain network dataset of ABIDE, to enable meaningful comparison of different models. Experiment results show clear improvements of our proposed Brain Network Transformer on both the public ABIDE and our restricted ABCD datasets. The implementation is available at https://github.com/Wayfear/BrainNetworkTransformer.

BRAIN NETWORK TRANSFORMER
Xuan Kan' Emoryy University
Wei Dai
Hejie Cui
Zilong Zhang Ying Guo' Carl Yang'
University of International Business and Economics {xuan. kan, kan,e cui, yguo2. carlyang) Semory. edu dvd. ai@stanf ord. edu 2019570200uibe edu. cn
Stanford University
Abstract
Human brains are commonly modeled as networks of Regions of Interest (ROIs) and their connections for the understanding of brain functions and mental dis- orders. Recently, Transformer-based models have been studied over different types of data, including graphs, shown to bring performance gains widely. In this work, we study Transformer-based models for brain network analysis. Driven by the unique properties of data, we model brain networks as graphs with nodes of fixed size and order, which allows us to (1) use connection profiles as node features to provide natural and low-cost positional information and (2) learn pair- wise connection strengths among ROIs with efficient attention weights across individuals that are predictive towards downstream analysis tasks. Moreover, we propose an ORTHIONORMAL CLUSTERING READOUT operation based on self- supervised soft clustering and orthonormal projection. This design accounts for the underlying functional modules that determine similar behaviors among groups of ROIs, leading to distinguishable cluster- node embeddings and informa- tive graph embeddings. Finally, we re-standardize the evaluation pipeline on the only one publicly available large-scale brain network dataset of ABIDE, to en- able meaningful comparison of different models. Experiment results show clear improvements of our proposed BRAIN NETWORK TRANSFORMER both the public ABIDE and our restricted ABCD datasets. The implementation available at hhttps://gtt tht.g www.tttt.....teeeteeeeeeeeeeeeee whtp://wwwgeeeeeaaaaaaeettttaaaa
1 Introduction
Brain network analysis has been an intriguing pursuit for neuroscientists to understand human brain organizations and predict clinical outcomes [50 59
58 5
18 27
52
29 58
28
41 44 311. Among
various neuroimaging modalities, functional Magnetic Resonance Imaging (fMRI) is one of the most commonly used for brain network construction, where the nodes are defined Regions of Interest (ROIs) given an atlas, and the edges are calculated as pairwise correlations between the blood-oxygen-
level-dependent (BOLD) signal series extracted from each region [54] 53 59 [6]. Researchers
observe that some regions can co-activate or co-deactivate simultaneously when performing cognitive- related tasks such action, language, and vision. Based on this pattern, brain regions can be classified into diverse functional modules to analyze diseases towards their diagnosis, progress understanding and treatment.
Nowadays Transformer-basee models have led a tremendous success various downstream tasks across fields including natural language processing [56] 17] and computer vision 20 10 55 Recent
efforts have also emerged to apply Transformer-basee designs to graph representation learning. GAT (57) firstly adapts the attention mechanism to graph neural networks (GNNs) but only considers the local structures of neighboring nodes. Graph Transformer 21 injects edge information into the attention mechanism and leverages the eigenvectors of each node as positional embeddings. SAN [40] further enhances the positional embeddings by considering both eigenvalues and eigenvec- tors and improves the attention mechanism by extending the attention from local to global structures.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Graphomer [64]. which achieves the first place on the quantum prediction track of OGB Large-Scale Challenge 30.. designs unique mechanisms for molecule graphs such as centrality encoding to enhance node features and spatial/edge encoding to adapt attention scores.
However, brain networks have several unique traits that make directly applying existing graph Transformer models impractical. First, one of the simplest and most frequently used methods to construct brain network in the neuroimaging community is via pairwise correlations between This impedes the designs like centrality,
BOLD time courses from two ROIs 44 spatial, and edge encoding because each node the brain network has the same degree and connects to every other node by a single hop. Second, in previous graph transformer models, eigenvalues and eigenvectors are commonly used as positional embeddings because they can provide identity and positional information for each node [15] 26]. Nevertheless, in brain networks, the connection profile, which defined as each node's 's corresponding row the brain network adjacency matrix, is recognized as the most effective node feature [3] This node feature naturally encodes both structural and positional information, making the aforementioned positional embedding design based on eigenvalues and eigenvectors redundant. The third challenge is scalability. Typically, the numbers of nodes and edges molecule graphs are less than and 2500, respectively. However, for brain networks, the node number is generally around 100 to 400, while the edge number can be up to 160,000. Therefore, operations like the generation of all edge features in existing graph transformer models can be cine-consu,,ng, if not infeasible.
63 69]
Non-orthonormal Bases
Orthonormal Bases
Somatomotor
Auditory
[a] Node features projected to 3D space PCA. Colors indicate functional modules.
(b) Orthonormal baves can make indistingushabble nodes in non-- orthonormal easily distinguishable
Figure 1: Illustration of the motivations behind ORTHONORMAI. CLUSTERING READOUT.
In this work, we propose to develop BRAIN NETWORK TRANSFORMER (BRAINNETTF), which leverages the unique properties of brain network data to fully unleash the power of Transformer-base models for brain network analysis. Specifically, motivated by previous findings on effective GNN designs for brain networks [13,, we propose to use the effective initial node features of connection profiles. Empirical analysis shows that connection profiles naturally provide positional features for Transformer-basee models and avoid the costly computations of eigenvalues or eigenvectors. Moreover, recent work demonstrates that GNNs trained on learnable graph structures can achieve superior effectiveness and explainability (35], Inspired by this insight, we propose learn fully pairwise attention weights with Transformer-basee models, which resembles the process of learning predictive brain network structures towards downstream tasks.
One step further, when GNNs are used for brain network analysis, a graph-level embedding needs be generated through readout function based the learned node embeddings (37 43 [K]. As is shown in Figure [). a property of brain networks is that brain regions (nodes) belonging to the same funetional modules often share similar behaviors regarding activations and deactivations in response to various stimulations [7]. Unfortunately, the current labeling of functional modules is rather empirical and far from accurate. For example, [ provides more than 100 different functional module organizations based on hicrarchical clustering. In order to leverage the natural functions of brain regions without the limitation of inaccurate functional module labels, we design new global pooling operator, ORTHONORMAL CLUSTERING READOUT, where the graph-level embeddings are pooled from clusters of functionally similar nodes through soft clustering with orthonormal projection. Specifically, we first devise a self-supervised mechanism based on [60] to jointly assign soft clusters brain regions while learning their individual embeddings. To further facilitate the learning of clusters and embeddings, we design an orthonormal projection and theoretically prove its effectiveness in distinguishing embeddings across clusters, thus obtaining expressive graph-level embeddings after the global pooling, as illustrated in Figure l[b..

---

Finally, the lack of open- datasets has been a non-negligible challenge for brain network analysis. The strict access restrictions and complicated extraction/rrerooessingg of brain networks from fMRI data limit the development of machine learning models for brain network analysis. Specifically, among all the large-scale publicly available fMRI datasets in literature, ABIDE [6] is the only one provided with extracted brain networks fully accessible without permission requirements. However, ABIDE is aggregated from 17 international sites with different scanners and acquisition parameters. This inter-site variability conceals inter-group differences that are really meaningful, which is reflected the unstable training performance and the significant gap between validation and testing performance in practice. To address these limitations, we propose apply stratified sampling method in the dataset splitting process and standardize a fair evaluation pipeline for meaningful model comparison on the ABIDE dataset. Our extensive experiments on this public ABIDE dataset and a restricted ABCD dataset NETWORK TR TRANSFORMER.
S show significant improvements brought by our proposed BRAIN
2 Background and Related Work
2.1 GNNs for Brain Network Analysis
Recently, emerging attention has been devoted to the generalization of GNN-based models to brain network analysis [4 [2 GroupINN (62 utilizes a grouping- layer to provide explainability
and reduce the model size. BrainGNN [43 designs the ROI-aware GNNs to leverage the functional information in brain networks and uses a special pooling operator to select these crucial nodes, IBGNN [4] proposes interpretable framework to analyze disorder-specific ROIs and prominent connections. In addition, FBNetGen [35 considers the learnable generation of brain networks and explores the explainability of the generated brain networks towards downstream tasks. Another benchmark paper [13] systematically studies the effectiveness of various GNN designs over brain network data. Different from other work focusing static brain networks, STAGIN [Р) utilizes GNNs with spatio-temporal attention to model dynamic brain networks extracted from fMRI data.
2.2 Graph Transformer
Graph Transformer raises many researchers' interest currently due its outstanding performance in graph representation learning. Graph Transformer [21] firstly injects edge information into the attention mechanism and leverages the eigenvectors as positional embeddings. SAN (40) enhances the positional embeddings and improves the attention mechanism by emphasizing neighbor nodes while incorporating the global information. Graphomer [64] designs unique mechanisms for molecule graphs and achieves the SOTA performance. Besides, a fine-grained attention mechanism is developed for node classification (8). Also, the Transformer is extended larger-scale heterogeneous graphs with particular sampling algorithm in HGT 322 EGT (33] further employs edge augmentation to
assist global self-attention. addition, LSPE 22 leverages the learnable structural and positional
encoding to improve GNNs' representation power, and GRPE [49] enhances the design of encoding node relative position information in Transformer.
3 BRAIN NETWORK TRANSFORMER
3.1 Problem Definition
In brain network analysis, given a brain network X e RVVV, where V is the number of nodes (ROIs), the model aims to make prediction indicating biological sex, presence of a disease or other properties of the brain subject. The overall framework of our proposed BRAIN NETWORK TRANSFORMER is shown in Figure 2 which is mainly composed of two components, an L-layer attention module MHSA and a graph pooling operator OCREAD. Specifically, in the first component of MHSA, the model learns attention-enile enhaance node features Z" through a non-linear mapping X z E VXV. Then the second component of OCREAD compresses the enhanced node embeddings - to graph-level embeddings Zg E R where K is hyperparameter representing the number of clusters. ZG is then flattened and passed to multi-layer perceptron for graph-level predictions. The whole training process is supervised with the cross-entropy loss.
3

---

FMRRI
MHSA
OCRead
Linear
Project
XL Layers
Concal
ROW1 Mwwppphyattee
Scaled Dot Product
ROW2 Aortttpp
7
Man ROI83
Linear
Linear
Lincar
X
Bedigiall my, Assion scectrum doardlor..
Figure 2: The overall framework of our proposed BRAIN NETWORK TRANSFORMER.
3.2 Multi-Head Self-Attention Module (MHSA)
To develop a powerful Transformer-based model suitable for brain networks, two fundamental designs, the positional embedding and attention mechanism, need be reconsidered to fit the natural properties of brain network data. existing graph transformer models, the positional information usually encoded via cigendecomposttion, while the attention mechanism often combines node positions with existing edges to calculate the attention scores. However, for the dense (often fully connected) graphs of brain networks, eigendecomposition is rather costly, and the existence of edges hardly informative.
ROI node features on brain networks naturally contain sufficient positional information, making the positional embeddings based on eigendecompoosiionn redundant. Previous work on brain network analysis has shown that the connection profile X,, for node i, defined as the corresponding row for each node in the edge weight matrix X, always achieves superior performance over others such node identities, degrees eigenvector-basee initialization, the colfeconnettion weight H; on the diagonal is always equal to one, which encodes sufficient information to determine the position of each node in fully connected graph based on the given brain atlas. To verify this insight, we also empirically compare the performance of the original connection profile with Le., connection profile w/ identity feature and connection profile w/ eigen feature. The results indeed show benefit brought by the additional computations (c.f. Appendix BB. As for the attention mechanism, previous work 113 has empirically demonstrated that integrating edge weights into the attention score calculation can significantly degrade the effectiveness of attention on complete graphs, while the generation of edge-wise embedding can be unaffordable given a large number of edges in brain networks. On the other hand, the existence of edges provides no useful information for the computation of attention scores as well because all edges simply exist in complete graphs.
embeddings 43 35 13].
With this node feature
variants concatenated with additional positional information,
Based the observations above, (1) adopting the connection profile as initial node features and eliminating any extra positional embeddings and (2) adopting the vanilla pair-wise attention mechanism without using edge weights or relative position information to learn singular attention score for each edge in the complete graph.
design the basic BRAIN NETWORK TRANSFORMER by
Formally, we leverage L-layer non-linear mapping module, namely Multi-Head Self-Attentin (MHSA), to generate more expressive node features z MHSA(X) E xV For each layer I, the output Z is obtained by
wh.m (1-1(Whing 21-1)T
z h
(m W: h Softmax
W Wl. z
(1)
dx
where n = X, is the concatenation operator, M is the number of heads, is the layer index, I. wht are learnable model parameters, and de is the first dimension of w
W!. W/m w

---

3.3 ORTHONORMAL CLUSTERING READOUT (OCREAD)
The readout function is an essential component learn the graph-level representations for brain network analysis (e.g., classification), which maps set of learned node-level embeddings to graph- level embedding. Mean(.), Sum(-) and Max(-) are the most commonly used readout functions for GNNs. Xu et al. [6] show that GNNs equipped with Sum(-) readout have the same discriminative power as the Weisfeiler-Lehmanl Test. Zhang et al. [66] propose sort pooling to generate the graph-level representation by sorting the final node representations. Ju et al. readout by extending the node information aggregated from the last layer of GNNs to all layers. However, none of the existing readout functions leverages the properties of brain networks that nodes in the same functional modules tend to have similar behaviors and clustered representations, as shown in Figure a To address this deficiency, we design novel readout function to take advantage of the modular-level similarities between ROIs in brain networks, where nodes are assigned softly to well-chosen clusters with unsupervised process.
[44] present a layer-wise
Formally, given K cluster centers, each center has V dimensions, E E RR Softmax projection operator is used as the function to calculate the probability Pa of assigning node to cluster k.
eZtt Ex.)
P.ke ΣN clzt
(2)
where ( denotes the inner product and Th is the learned set of node embeddings from the last layer of MHSA module. With this computed soft assignment P E RV xxxk the original learned node representation Z can be aggregated under the guidance of the soft cluster information, where the graph-level embedding Zg obtained by ZG P z
However, jointly learning node embeddings and clusters without ground-truth cluster labels difficult. To obtain representative soft assignment P. the initialization of K cluster centers E is critical and should be designed delicately. To this end, we leverage the observation illustrated in Figure Ib), where orthonormal embeddings can improve the clustering of nodes in brain networks w.r.t the functional modules underlying brain regions.
Orthonormal Initialization. To initialize group of orthonormal bases as cluster centers, we first adopt the Xavier uniform initialization [25] to initialize K random centers and each center contains V dimensions C' E R&XV. Then, we apply the Gram-Schmidr process to obtain the orthonormal bases E, where
Σ (uj Ci.) (u,, Un)
21 C
Ujj
Eg.
(3)
In the next section, we theoretically prove the advantage of this orthonormal initialization.
3.3.1 Theoretical Justifications
In OCREAD, proper cluster centers can generate higher-quality soft assignments and enlarge the difference between P from different classes. [51 46) showed the advantages of orthogonal initial- ization in DNN model parameters. However, none of them proves whether it is an ideal strategy to obtain the cluster centers. We propose two methods from the perspective of statistics as follows.
Firstly, to discern features of different nodes, we would expect larger discrepancy among their similarity probabilities indicated from the readout. One way to measure the discrepancy is using the variance of P for each feature. Let P = 1/ K denote the mean of any discrete probabilities with K values. Variance of P measures the difference between and P. We average over the feature vector space: if the result is small, then there is a large tendency that different P approaches P and hence cannot be discerned easily. Specifically, the following theorem holds for our function Eq. (
Theorem 3.1. For arbitrary > 0, let B, = {2 E RVV 2|| <r) denote the round ball centered at origin of radius with 2 being fracture vectors. Let V, be the volume of B The variance of Softmax projection averaged over B,
1
1 Σ
K
el (
1 1
dz,
(4)
JR. R. J
EK Enr)
K
attains maximum when E is orthonormal.
5

---

Despite the concise form, it is unclear whether the above integral has an elementary antiderivative. Even though, we can circumvent this problem and rigorous proof given in Appendix C
The second statistical method shows that for general readout functions without a known analytical form, initializing with orthonormal cluster centers has larger probability of gaining better perfor- mance. To set up the proper statistical scenario, we assume that the unknown readout is obtained by regression of some samples (Z()) P(.t)). This formally converts the exact functional relationship between Z Eg. and P. to statistical relationship:
Pr(Z,. Ex.) P(Z, (Z Ex. + Ei, E ~0,,00) E(e:) -0 D(e,) = o²,
(5)
with Pr being the probability truly reflecting similarities between nodes and clusters and e, denoting the stochastic error. It is almost impossible to find Pr, but by computing the so-called variation inflation factor 47. we show that regression in orthonormal case has higher accuracy than that in non-orthonormal case. Combining with a hypothesis testing, we obtain the following
Theorem 3.2. The significance level Ex which reveals the probability of rejecting well-estimated pooling is lower when sampling from orthonormal centers than that from non-orthonormal centers.
More details can be seen in Appendix C
3.4 Generalizing OCREAD to Other Graph Tasks and Domains
In this work, we tested the proposed OCREAD on functional connectivity (FC) based brain networks. Other popular modalities of brain networks include structural connectivities (SC), which describe the anatomical organization of the brain by measuring the fiber tracts between brain regions 44. In SC-based brain networks, ROIs that positionally close each other the structural connectivity networks tend to share similar connection profiles. This means the idea of OCREAD is also naturally applicable SC networks, where the orthonormal clustering is based the physical distances instead of the functional modules on FC.
At a higher level, the idea of our proposed OCREAD is not confined to graph-level prediction tasks on brain networks but can also be generalized to other graph learning tasks and domains. Preciscly, there is growing tendency in node/edge level prediction tasks enhance the node/edge representation learning by utilizing the subgraph embeddings around each target node/edge [67 665. In this process, substructure learning needs be performed on the subgraphs, where our proposed OCREAD be adapted for compressing a set of node embeddings to subgraph embeddings. Besides, OCREAD is also potentially useful for other types of graphs in the biomedical domains. For example, for protein- protein interaction networks, proteins can be implicitly grouped by families that share common evolutionary origins (88. whereas for gene expression networks, genes can be grouped based on the latent pathway information (36). Both of them are potential directions for the future application of OCREAD, among many others driven by biological or other types of prior knowledge regarding underlying node/edge groups
4 Experiments
This section evaluates the effectiveness of our proposed BRAIN NETWORK TRANSFORMER (BRAINNETTF) with extensive experiments. We aim address the following research questions:
RQ1. How does BRAINNETTF perform compared with state-of-the-art models of various types?
RQ2. How does proposed OCREAD module perform with different model choices?
RQ3. Does the learned model of BRAINNETTF exhibit consistency with existing neuroscience knowledge and suggest reasonable explainability?
4.1 Experimental Settings
Datasets. We conduct experiments on two real-world fMRI datasets. (a) Autism Brain Imaging Data Exchange (ABIDE): This dataset collects resting-state functional magnetic resonance imaging (rs-fMRI) data from 17 international sites, and all data are anonymous (6]. The used dataset contains brain networks from 1009 subjects, with 516 (51.14%) being Autism spectrum disorder

---

(ASD) patients (positives). The region definition is based on Craddock 200 atlas [1].. As the most convenient open-source large-scale dataset, provides generated brain networks and can be downloaded directly without permission request. Despite the ease of acquisition, the heterogeneity of the data collection process hinders its use. Since multi-site data are collected from different scanners with different acquisition parameters, non-neural inter-site variability may mask inter-group differences. In practice, we find the training unstable, and there is a significant gap between validation and testing performances. However, we discover that most models can achieve stable performance if we follow an appropriate stratified sampling strategy by considering collection sites during the training- iedidttioo-testinn splitting process for ABIDE. Training curves in Appendix A alsso show how different models achieve a stabler performance on our designed new splitting settings than the random splitting. Therefore, use ABIDE as one of the benchmark datasets in this work, and we share our re-standardized data splitting to provide a fair evaluation pipeline for various future methods. (b) Adolescent Brain Cognitive Development Study (ABCD): This is one of the largest publicly available fMRI datasets with restricted access (a strict data requesting process needs to be followed to obtain the data) [8) The data we use in the experiments are fully anonymized brain networks with only biological sex labels. After the quality control process, 7901 subjects are included in the analysis, with 3961 (50.1%) among them being female. The region definition is based on the HCP 360 ROI atlas 24
Metrics. The diagnosis of ASD is the prediction target ABIDE, while biological sex prediction is used as the evaluation task for ABCD. Both prediction tasks are binary classification problems, and both datasets are balanced between classes. Hence, AUROC is proper performance metric adopted for fair comparison at various threshold settings, and accuracy is applied to reflect the prediction performance when the threshold is 0.5. Besides, since the model is mainly for medical applications, we add two critical metrics for diagnostic tests, Sensitivity and Specificity, which respectively refer to true positive rate and true negative rate. All reported performances are the average of 5 random runs on the test set with the standard deviation.
Implementation details. For experiments, we use a two-layer Multi-Head and set the number of heads M to for each layer. We randomly split 70% of the datasets for training. 10% for validation, and the remaining are utilized the test set. In the training process of BRAINNETTF. we use an Adam optimizer with an initial learning rate of 10 4 and a weight decay of 10-4 The batch size is set 64. All models are trained for 200 epochs, and the epoch with the highest AUROC performance on the validation set is used for performance comparison on the test set. The model is trained Appendix G for the full implementation of BRAINNETTF.
Module
NVIDIA Quadro RTX 8000. Please refer to the repository and
Computation complexity. In BRAINNETTF, the computation complexity of Multi-Head Self- Attention Module and OCREAD are (LLV)2 and O(KV) respectively, where the layer number of Multi-Head Self-Attenton Module, V is the number of nodes, M is the number of heads, and K is the number of clusters in OCREAD. The overall computation complexity of BRAINNETTF is thus OVV2), which is the same scale as common GNNs on brain networks such as BrainGNN 43 and BrainGB 13
4.2 Performance Analysis (RQ1)
We compare BRAINNETTF with baselines of three types. The details about how tune hyperparam- eters of various baselines can be found in Appendix [F] Besides, Appendix shows the comparison of the number of parameters between our model and other baseline models, which shows that the parameter size of BRAINNETTE is larger than GNN and CNN models but smaller than other trans- (a) BRAINNETTF vs. other graph transformers. We compare BRAINNETTF with two popular graph Transformers. SAN [40] and Graphormer [64). In addition, we also include a basic version of BRAINNETTF without OCREAD, composed of a Transformer with 2-layer Multi-Head Self Attf-Attention and a CONCAT--asse readout named VanillaTF. Our BRAINNETTF outperforms SAN and Graphormer by significant margins, with up 6% absolute improvements on both datasets. VanillaTF also surpasses SAN and Graphormer. We believe this downgraded performance of existing graph transformers results from their design flaws facing the natures of brain networks. Specifically, both the preprocessing and the training stages of the Graphormer model accepts only discrete, categorical data. A bin operator has to be applied on the adjacency matrix, coarsening the node feature from connection profiles and dramatically hurting the performance.
former models.
7

---

Furthermore, since brain networks are complete graphs. key designs like centrality encoding and spatial encoding of Graphormer cannot be appropriately applied. Similarly, for SAN, experiments in Appendix B show that adding eigen node features to connection profiles cannot improve the model's performance. Besides, the benchmark paper [13] reveals that injecting edge weights into the attention mechanism can significantly reduce the prediction power. Furthermore, Appendix [D] shows our BRAINNETTF is much faster than other graph transformers due to special optimizations towards brain networks. (b) BRAINNETTH vs. neural network models fixed brain networks. We further introduce another three neural network baselines on fixed brain networks. BrainGNN [ designs ROI-aware GNNs for brain network analysis. BrainGB [3] is systematic study of how to design effective GNNs for brain network analysis. We adopt their best design as the BrainGB baseline. BrainnetCNN (37 represents state-of oftte-of-thh-ar art of specialized GNNs for brain network analysis, which models the adjacency matrix of a brain network similarly as a 2D image. As is shown in Table T BRAINNETTF consistently outperforms BrainGNN, BrainGB and BrainnetCNN. (C) BRAINNETTE vs. neural network models on learnable brain networks. Unlike classical GNNs, FBNETGEN 35), DGM (38 and BrainNetGNN 445 hold similar idea, which is apply GNNs based on learnable graph. FBNETGEN achieves SOTA performance on the ABCD dataset for biological sex prediction, and the learnable graphs can be seen as a type of attention score. Experiment results show that our proposed BRAINNETTF beats all three of them on both datasets.
Table I: Performance comparison with different baselines (%). The performance gains of BRAIN- NETTF over the baselines have passed the t-test with p-value-c0.03.
Dataset:
ABIDE
Dataset: ABCD
Type
Method
AUROC
Accuracy
Sensitivity
Specificity
AUROC
Accuracy
Sensitivity
Specificity
Transformer
Graph
Graphormer
VanillaTF
SAN
71,3+2.1
63.5+3.7
76.421.2
65.3+2.9
65.2a1.2
60.82.77
78.7=22.3
66.4x11.4
55.4+9.2
71.1412.0
36.7423.5
68.3+7.5
89.0+1.4
90.1+1.2
94.320.7
81.0+13
K5.921.4
80.2+1.3
87.742.4
84.9+3.5
77.5+4.1
82.623.9
82.4=7.4 44
Fixed
BrainGNN
62.4+3.5
59.442.3
367424.0
70.7419.3
OOM
00M
00M
Network
BrainGB
69.7+3.3
63.6+1.9
63.7+8.3
60.4+10.1
91.920.3
83.5
84.664.3
81.5=399
BrainNeCCNN
74.922.4
67.812.7
63.829.7
71.0110.2
93.520.3
85.720.8
87.913.4
83.024.4
Leamable Network
FRNETGNN
78.6+1.2
68.0+1
64.7+8.7
62.4+9.2
94,5+0.7
87.2+1.2
87.042.5
86.7=2.8
NetiNN
55.3+1.9
51.245.4
67.7+37.5
13.9434,2
75.3+5.2
67.5e4.7
67.745.7
68.0
DGM
52.743.8
60.7212.6
53.8241.2
51.12
76.8=19.0
68.628.1
40.5a29.7
95.624.2
Ours
BRAINNETTE
80.2:1.0
71.011.2
72.545.2
69.316.5
96.220.3
88.4:0.4
89.412.6
88.421.5
4.3
Ablation Studies on the OCREAD Module (RQ2)
4.3.1 OCREAD with varying readout functions
We vary the readout function for various Transformer architectures, including SAN, Graphormer and VanillaTI, to observe the performance of each ablated model variant. The results shown in Table 2 demonstrate that our OCREAD is the most effective readout function for brain networks and improves the prediction power across various Transformer architectures.
Table 2: Performance comparison AUROC (%) with different readout functions.
Dataset: ABIDE
Dataset: ABCD
Readout
SAN
Graphormer
Vanilla
SAN
Graphormer
VanillaTF
MEAN
63.7=2.4
10.1
73.4
88.5=0.9
87.6
91.3+0.7
MAX
61.9=2.5
54.5+3.6
75.6=1.4
17.4
81.6#0.8
94,4=0.6
SUM
62,0=2.3
54.1=1.3
70.3=1.6
84.2=0.8
71.5=0.9
91.620.6
SonPooling
68.7+2.3
51.3+2.2
72.4+1.3
84.6+1.1
86.7+1.0
89,910.6
DifiPool
57.4=5.2
50.5+4.7
62.9=7.3
78.1
70.0+1.9
83,9=1.3
CONCAT 71.3+2.1
63.5+3.7
76.4+1.2
90.1
89.0+1.4
94.3=0.7
OCREAD
70.6=2.4
64.9=2.7
80.2+1.0
91.2=0.7
90.2+0.7
96.2=0.4
4.3.2 OCREAD with varying cluster initializations
To further demonstrate how the design of OCRFAD influences the performance of BRAINNETTF, we investigate two key model selections, the initialization method for cluster centers and the cluster
8

---

number K. For the initialization, three different kinds of initialization procedures are compared, namely (a) Random: the Xavier uniform [25] leveraged to randomly generate group of centers, which are then normalized into unit vectors; (b) Learnable: the same initial process as Random, but the generated centers are further updated with gradient descent; (c) Orthonormal: our proposed
process as described in Eq.
Specifically, we test each initialization method with the cluster number K equals to 3, 4, 10, 50, 100. The results of adjusting these two hyper-parameters on ABIDE and ABCD datasets are shown in Figure a). We observe that: (1) When cluster centers are orthonormal, the model's performance increases with the number of clusters ranging from 2 to 10, and then drops with the cluster number rising from 10 to 100, suggesting the optimal cluster number to be relatively small, which leads to less computation and is consistent with the fact that the typical number of functional modules are smaller than 25; (2) With a sufficiently large cluster number, all three initialization methods, Random, I Learnable and Orthonormal, tend to reach similar performance, but orthonormal performs stably better when the number of clusters is smaller; (3) It also notable that our OCREAD consistently achieves the best performance over other initialization methods regarding smaller standard deviations.
97
96
AURD013)
94
Random Leumable
Randoca
Leaenable
Orthoncemal
Orthenorzaal
5 10
100
10
100
Chokt susber N en ABIDE
Clatc: manber M ABCD
[a] Influence of two key hyper-parameters for model performance.
(b) Attention heatmap on
Figure 3: The hyper-parameter influence and the heatmap from self-attention.
4.4 In-depth Analysis of Attention Scores and Cluster Assignments (RQ3)
Figure (b) displays the self-attention score from the first layer of Multi-Head Self-Attention. The attention scores are the average across all subjects in the ABCD test set. This figure shows that the learned attention scores well match the divisions of functional modules based on available labels. demonstrating the effectiveness and explainability of our Transformer model. Note that since there exists no available functional module labels for the atlas of the ABIDE dataset, we cannot visualize the correlations between attention scores and functional modules.
Figure 4 shows the cluster soft assignment results P nodes in OCREAD with two initialization methods. The cluster number K is set to 4. The visualized numerical values are the average P of all subjects in each dataset's test set. From the visualization, we observe that (a) Base on Appendix H. orthonormal initialization produces more discriminative P between classes than random initialization: (b) Within each class, orthonormal initialization encourages the nodes to form groups. These observations demonstrate that our OCREAD with orthonormal initialization can leverage potential clusters underlying node embeddings, thus automatically grouping brain regions into potential functional modules.
5 Discussion and Conclusion
Neuroimaging technologies, including functional magnetic resonance imaging (IMRI) are powerful noninvasive tools for examining the brain functioning. There is an emerging nation-wide interest in conducting neuroimaging studies for investigating the connection between the biology of the brain, and demographic variables and clinical outcomes such as mental disorders. Such studies provide an unprecedented opportunity for cross-cutting investigations that may offer new insights to the differences in brain function and organization across subpopulations in the society (such as biological sex and age groups) as well as reveal neurophysiologica mechanisms underlying brain disorders (such as psychiatric illnesses and neurodegenerative diseases). These studies have tremendous impact in social studies and biomedical sciences. For example, mental disorders are the leading cause of disability in the USA and roughly I in 17 have a seriously debilitating mental illness. To

---

Control
Nale
1.0
Clasber
Custedt
Orthonormal
Disgnosis
Femele
Canteer
Clatery
Control
Hale
Choters CLatert
Random
Clognosis
Female
Choters
ABIDE
ABCD
Figure 4:
Visualization of cluster (module-leved embeddings learned with Orthonormal vs. Random cluster center initializations two datascts. Each group in the dotted box contains two heatmaps (one for each prediction class) with the same node ordering on the x-axis.
address this burden, national institutions such as NIH have included wrain-behavior research as one of their strategic objectives and stated that sound efforts must be made to redefine mental disorders into dimensions or components of observable behaviors that are more closely aligned with the biology of the brain. Using brain imaging data to predict diagnosis has great potential to result in mechanisms that target for more effective preemption and treatment.
In this paper, we present BRAIN NETWORK TRANSFORMER, a specialized graph Transformer model with ORTHONORMAL CLUSTERING READOUT for brain network analysis. Extensive experiments on two large-scale brain network datasets demonstrate that our BRAINNETTF achieves superior performance over SOTA baselines various types. Specifically, to model the potential node feature similarity in brain networks, we design OCREAD and prove its effectiveness both theoretically and empirically. Lastly, the re-standardized dataset split for ABIDE can provide a fair evaluation for new methods in the community. For future work. BRAINNETTF can be improved with explicit explanation modules and used as the backbone for further brain network analysis, such as digging essential neural circuits for mental disorders and understanding cognitive development in adolescents.
6
Acknowledgments
This research was supported in part by the University Research Committee of Emory University, and the internal funding and GPU servers provided by the Computer Science Department of Emory Uni- versity. The authors gratefully acknowledge support from NIH ynder award number RO1MH 11505561 and R01MH1 18771. The content is solely the responssaayyl AorAuoos and does not necessarily represent the official views of the National Institutes of Health.
Data used in the preparation of this article were obtained from the Adolescent Brain Cognitive Development (ABCD) Study (https: : //abddstddy.yyy.. org). held in the NIMH Data Archive (NDA). This is multisite, longitudinal study designed to recruit more than 10,000 children age 9-10 and follow them over 10 years into early adulthood. The ABCD Study @ is supported by the National Institutes of Health and additional federal partners under award numbers U01DA041048, UOIDA050989, UOIDA05 1016, U01DA041022 UOIDA051018, UOIDA05 1037, UOIDA050987, U01DA041174 U01DA041 106, U01DA041117, U01DA041028. U01DA041134 U01DA050988, UOIDA05 1039, UO1DA04115 156, U01DA041025, U01DA041120 UO1DA05 1038, UO1DA041 148, U0IDA041093, U0IDA041089, U24DA041122 123, U24DA041 147. A full list of supporters is available -ffederal-partner- partners. html A listing of participating sites and a
at hhttp:://abcdstut org/federal-peeeeeeeeeeeeeeeeeep complete listing of the study investigators can be found at https:://// //tp:://bbbla abcdstudy............/.a..a....a parg/consortium members/ ABCD consortium investigators designed and implemented the study and/or provided data but did not necessarily participate in the analysis or writing of this report. This manuscript reflects the views of the authors and may not reflect the opinions or views of the NIH or ABCD consortium investigators. The ABCD data repository grows and changes over time. The ABCD data used in this report came from NIMH Data Archive Release 4.0 (DOI 10.15154/1523441 DOIs can be found at httttss: ptatts://ndaa. //nda, nih, gov/abed
10

---

References
[1] G. and C. J. Wild. Nonlinear Regression: Seber/Nonlinear Regression. 1989.
[2] David Ahmedt-Aristizabee Mohammad Ali Armin, Simon Denman, Clinton Fookes, and Lars Petersson. Graph-based deep learning for medical diagnosis and analysis: past, present and future. Sensors, 21:4758, 2021.
[3] Teddy J Akiki and Chadi G Abdallah. Determining the hierarchical architecture of the human brain using subject-level clustering of functional networks. Scientific Reports, 9:1-15, 2019.
[4] Parinaz Babaeeghazvini, Laura M. Rueda-Delgado, Jolien Gooijers, Stephan P. Swinnen, and Andreas Daffertshofer. Brain structural and functional connectivity: A review of combined works of diffusion magnetic resonance imaging and eeeetllenccephllrrapph.. Frontiers in Human Neuroscience, 15, 2021.
[5] Ed Bullmore and Olaf Sporss Complex brain networks: graph theoretical analysis of structural and functional systems. Nature Reviews Neuroxcience, 10: 1eo:186-198, 2009.
[6] Craddock Cameron, Benhajali Yassine, Chu Carlton, Chouinard Francois, E. Aykan Alan, Jakab András, Khundrakpam Budhachandra, Lewis John, Liub Qingyang. Milham Michael, Yan Chaogan, and Bellee Pierre. The neuro bureau preprocessing initiative: open sharing of preprocessed neuroimaging data and derivatives. Frontiers in Neuroinformatics, 2013.
[7] Alfonso Caramazza and Max Coltheart. Cognitive neuropsychology twenty years on. Cognitive Neuropsychology, 2006.
[8] B.J. Casey, Tariq Cannonier, and May I. Conley et The adolescent brain cognitive develop- ment (abed) study: Imaging acquisition across 21 sites. Developmental Cognitive Neuroscience, 32:43-54, 2018.
[9] Gemai Chen and N. Balakrishnan. A General Purpose Approximate Goodness-of-Fis Test. Journal of Quality Technology, 27:154-161, 1995.
[10] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NeurIPS, 2021.
[11] A. Colin Cameron and Frank A.G. Windmeijer. An R-squared measure of goodness of fit for some common nonlinear regression models. Journal of Econometrics, 77:329-342. -322, 1997.
[12] R Cameron Craddock, G Andrew James, Paul E Holtzheimer III, Xiaoping P Hu, and Helen S Mayberg. whole brain fmri atlas generated via spatially constrained spectral clustering. Human Brain Mapping, 33:1914-1928, 2012.
[13] Hejie Cui, Wei Dai, Yanqiao Zhu, Xuan Kan, Antonio Aodong Chen Gu, Joshua Lukemire, Liang Zhan, Lifang He, Ying Guo, and Carl Yang. Braingb: A benchmark for brain network analysis with graph neural networks, 2022.
[14] Hejie Cui, Wei Dai, Yanqiao Zhu, Xiaoxiao Li, Lifang He, and Carl Yang. Interpretable graph neural networks for connectome-based brain disorder analysis. In MICCAI, 2022.
[15] Hejie Cui, Zijie Lu, Pan Li, and Carl Yang. On positional and structural node features for graph neural networks on non-attributed graphs. CIKM, 2022.
[16] Tian Dai, Ying Guo, Alzheimer's Disease Neuroimaging Initiative, al. Predicting individual brain functional connectivity using a bayesian hierarchical model. Neurolmage, 147:772-7877 2017.
[17] Zihang Dai, Zhilin Yang, Yiming Yang. Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhut- dinov. Transformer-xl: Attentive language models beyond fixed-length context. ACI, 2019.

---

[18] Gustavo Deco, Viktor K. Jirsa, and Anthony R. McIntosh. Emerging concepts for the dynamical organization of resting-state state activity the brain. Nature Reviews Neuroscience, 12:43-56, 2011.
[19] Morris H. DeGroot and Mark J. Schervish. Probability and Statistics. 4th ed edition, 2012.
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[21] Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. AAA/ Workshop on Deep Learning Gruphs: Methods and Applications, 2021.
[22] Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. In ICLR, 2022.
[23] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
[24] Matthew F. Glasser, Stamatios N. Sotiropoulos, J. Anthony Wilson, Timothy S. Coalson, Bruce Fischl, Jesper L. Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan R. Polimeni, David C. Van Essen, and Mark Jenkinson. The minimal preprocessing pipelines for the human connectome project. Neurolmage, 80: 80:105-124 2013.
[25] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
[26] Shupeng Gui, Xiangliang Zhang, Pan Zhong, Shuang Qiu, Mingrui Wu, Jieping Ye, Zhengdao Wang, and Ji Liu. Pine: Universal deep embedding for graph nodes via partial permutation invariant set functions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44:770-782, 2022.
[27] Ying Guo and Giuseppe Pagnoni. A unified framework for group independent component analysis for multi-subject fmni data. Neurolmage, 42(3):1078-1095: 2008
[28] Ixavier A Higgins, Suprateek Kundu, Ki Sueng Choi, Helen Mayberg, and Ying Guo. A difference degree test for comparing brain networks. Human brain mapping, pages 4518-4536, 2019.
[29] Ixavier A Higgins, Suprateek Kundu, and Ying Guo. Integrative bayesian analysis of brain functional networks incorporating anatomical knowledge. Neuroimage, 181 11:23-22788 278, 2018.
[30] Weihua Hu, Matthias Fey, Marinka Zitiik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In NeurIPS, 2020.
[31] Yingtian Hu, Mahmoud Zeydabadinezhad, Longchuan Li, and Ying Guo. A multimodal multilevel neuroimaging model for investigating brain connectome development. Journal of the American Statistical Association, pages 1-15, 2022.
[32] Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In www, pages 2704-271 2710, 2020.
[33] Md Shamim Hussain, Mohammed Zaki, and Dharmashankar Subramanian. Edge-augmented graph transformers: Global self-attention is enough for graphs. arXiv, 2021.
[34] Mingxuan Ju, Shifu Hou, Yujie Fan, Jianan Zhao, Liang Zhao, and Yanfang Ye. Adaptive kernel graph neural network. AAAI, 2022.
[35] Xuan Kan, Hejie Cui, Joshua Lukemire, Ying Guo, and Carl Yang. FBNETGEN: Task-aware GNN-based fMRI analysis via functional brain network generation. Medical Imaging with Deep Learning, 2022.
12

---

[36] Minoru Kanehisa and Susumu Goto. Kegg: kyoto encyclopedia of genes and genomes. Nucleic Acids Research, 28:27-30, 2000.
[37] Jeremy Kawahara, Colin J. Brown, Steven P. Miller, Brian G. Booth, Vann Chau, Ruth E. Grunau,
Jill G. Zwicker, and Ghassan Hamarneh. BrainNeiCNN: Convolutional neural networks for brain networks; towards predicting neurodevelopment. Neurolmage, 146:1038- 1038-1049, 2017.
[38] Anees Kazi, Luca Cosmo, Seyed-Ahmad Ahmadi, Nassir Navab, and Michael Bronstein. Differentiable graph module (dgm) for graph convolutional networks. IEEF Transactions on Puttern Analysis and Machine Intelligence, pages 1-1, 2022.
[39] Byung-Hoon Kim, Jong Chul Ye, and Jae-Jin Kim. Learning dynamic graph representation of brain connectome with spatio-temporal attention. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, NeurlPS, 2021.
[40] Devin Kreuzer, Dominique Beaini, William L. Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. In A. Beygelzimer, Y. Dauphin, Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[41] Suprateek Kundu, Joshua Lukemire, Yikai Wang, and Ying Guo. A novel joint brain network analysis using longitudinal alzheimer's disease data. Scientific reports, 9(1):1-18, 2019.
[42] Xiaoxiao Li, Nicha C. Dvornek, Yuan Zhou, Juntang Zhuang, Pamela Ventola, and James S. Dunean. Graph neural network for interpreting task-fmri biomarkers. In MICCAI, 2019.
[43] Xiaoxiao Li, Yuan Zhou, Siyuan Gao, Nicha Dvornek, Muhan Zhang, Juntang Zhuang, Shi Gu, Dustin Scheinost, Lawrence Staib, Pamela Ventola, et al. Braingnn: Interpretable brain graph neural network for fmri analysis. Medical Image Analysis, 2021.
[44] Joshua Lukemire, Suprateek Kundu, Giuseppe Pagnoni, and Ying Guo. Bayesian joint modeling of multiple brain functional networks. Journal of the American Statistical Association, 116:518- 530, 2021.
[45] Usman Mahmood, Zening Fu, Vince D. Calhoun, and Sergey Plis. A deep learning model for data-driven discovery of functional connectivity, Algorithms, 14, 2021.
[46] Haitao Mao, Xu Chen, Qiang Fu, Lun Du, Shi Han, and Dongmei Zhang. Neuron Campaign for Initialization Guided by Information Boutleneck Theory. 2021.
[47] F. H. C. Marriott, J. Neter, W. Wasserman, and M. H. Kutner. Applied Linear Regression Models. Biometrics, 1985.
[48] Jaina Mistry, Sara Chuguransky, Lowri Williams, Matloob Qureshi, Gustavo A Salazar, Erik L L Sonnhammer, Silvio C E Tosatto, Lisanna Paladin, Shriya Raj, Lorna Richardson, Robert D Finn, and Alex Bateman. Pfam: The protein families database in 2021. Nucleic Acids Research, 49:D412-DD -4119, 2020.
[49] Wonpyo Park, Woonggi Chang, Donggeon Lee, Juntae Kim, and Seung-won Hwang. Grpe: Relative positional encoding for graph transformer, 2022.
[50] Theodore D. Satterthwaite, Daniel H. Wolf, David R. Roalf, Kosha Ruparel, Guray Erus, Simon Vandekar, Efstathios D. Gennatas, Mark A. Elliott, Alex Smith, Hakon Hakonarson, Ragini Verma, Christos Davatzikos, Raquel E. Gur, and Ruben C. Gur. Linked Sex Differences in Cognition and Functional Connectivity in Youth. Cerebral Cortex, 25:2383-2394, 2015.
[51] Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. 2014.
[52] Ran Shi and Ying Guo. Investigating differences in brain functional networks using hierarchical awvariate-ajjssee independent component analysis. The annals of applied statistics, page 1930, 2016.
13

---

[53] Sean L Simpson, F DuBois Bowman, and Paul J Laurienti. Analyzing complex functional brain networks: fusing statistics and network science to understand the brain. Statistics Surveys, 7:1, 2013.
[54] Stephen M. Smith, Karla L. Miller, Gholamreza Beckmann, Thomas E. Nichols, Joseph D. Ramsey, and Mark W. Woolrich. Network modelling methods for FMRL. Neurolmage, 54, 2011.
Matthew Webster, Christian F.
[55] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. ECCV, 2022.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention all you need. In Guyon, U. Von Luxburg,
S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, NeurlPS, 2017.
[57] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. ICLRR 2018.
[58] Yikai Wang and Ying Guo. A hierarchical independent component analysis model for longitudi- nal neuroimaging studies. Neurolmage, 189:380-4000 400, 2019.
[59] Yikai Wang, Jian Kang, Phebe B. Kemmer, and Ying Guo. An Efficient and Reliable Statistical Method for Estimating Functional Connectivity in Large Scale Brain Networks Using Partial Correlation. Frontiers in Neuroscience, 10:123, 2016.
[60] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICML, 2016.
[61] Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
[62] Yujun Yan, Jiong Zhu, Marlena Duda, Eric Solarz, Chandra Sripada, and Danai Koutra. Groupinn: Grouping-based interpretable neural network-based classification of limited, noisy brain data. In KDD, 2019.
[63] Yi Yang, Yanqiao Zhu, Hejie Cui, Xuan Kan, Lifang He, Ying Guo, and Carl Yang. Data- efficient brain connectome analysis via multi-task meta-learning. KDD, 2022.
[64] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? In NeurIPS, 2021.
[65] Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. NeurlPS, 2018.
[66] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In AAAI, 2018
[67] Muhan Zhang and Pan Li. Nested graph neural networks. In NewrlPS, 2021.
[68] Jianan Zhao, Chaozhuo Li, Qianlong Wen, Yiqi Wang, Yuming Liu, Hao Sun, Xing Xie, and Yanfang Ye. Gophormer: Ego-graph transformer for node classification, 2021.
[69] Yanqiao Zhu, Hejie Cui, Lifang He, Lichao Sun, and Carl Yang. Joint embedding of structural and functional brain networks with graph neural networks for mental illness diagnosis. In 2022 44th Annual International Conference of the IEEE Engineering Medicine Biology Society (EMBC), pages 272-276, 2022.
14


Reviews:
[{"review_id": "uMl_mWVOQT9", "reviewer": "Reviewer_CYN4", "summary": "In this work, the authors have proposed a self-attention based graph model to classify a) ASD and b) gender using functional neuroimaging fMRI datasets. The authors also proposed a new readout method based on clustering for graph classification. The proposed method seems to perform a bit better in the classification of both tasks in comparison to other baseline methods.\n", "questions": "- Figure 2 shows multiple $X$ matrices and it is unclear how they were acquired. My guess is it's the same matrix, rather the self-attention module receiving rows of the matrix.\n- How is the matrix $X$ computed? \n- If classification performance is the goal then why not use simple ML models like SVM, LR on FC matrices computed via PCC. With hand-crafted features (FC matrices), ML models give better classification performance.\n- How were the results for comparing studies calculated? \n- Why not use HCP data for gender classification as it is publically available?", "limitations": "- Results were not compared against many other existing relevant studies. Mentioned in weaknesses.\n- The readout method compared against basic methods such as sum, max, etc. but not against serious and widely available contenders.\n- Marginal improvement in classification accuracy ($\\approx1$%). \n- No cross-validation, no multiple test folds. \n- Missing details about input data ($X$). \n", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The biggest strength is the classification performance of the model for ASD and the gender classification tasks. The classification performance is ~1% better in terms of accuracy when comparing with next best (FBNETGNN)\n\n## Weaknesses\n- There are some comments in the paper which are either incorrect or not properly justified. For example, lines 43-44 mention \"First, a brain network is a correlation matrix defined on a complete graph.\"  This is not correct as correlation matric computed using methods like Pearson’s Correlation Cofficient is a way to measure brain connectivity. There can be many other ways to represent brain network e.g. Effective Connectivity, Directed connectivity, Entropy based, causality based (Granger causality), etc. Lines 49-51: it would be better to explain how a 1D time-series of a voxel has positional information.\n- The authors do not mention how they compute the $X \\in RV *V$ matrix.\n- The authors cite multiple self-attention and graph based methods but do not compare with them, for example BrainGNN, STAGIN. It would be better to compare with these methods and others like these on the same task.\n- There are others self-attention and graph based methods developed for neuroimaging datasets.  \n   1. A Deep Learning Model for Data-Driven Discovery of Functional Connectivity https://doi.org/10.3390/a14030075\n   2. Deep Dynamic Effective Connectivity Estimation from Multivariate Time Series https://arxiv.org/abs/2202.02393\n- It is unclear how the results used for comparison of the methods are acquired. Are they taken from other papers? Was the code open source? How the hyper-parameters were tuned for these methods?\n- There are many readout methods developed already which perform much better than the simple ones (add, max, avg, min) such as attention based, topkPooling, Edgepooling, SAG pooling and many more. These methods are also acquire a smaller graph as the readout method proposed by the authors. Other global pooling methods also exist such as GlobalAttention. Comparison with them is lacking.\n- Similar concerns arise regarding the results in Table 2 as well. The manuscript compares against basic methods while many sophisticated and more powerful methods are already proposed. Table 2 also shows that the gains by the readout method proposed by the authors are minute when comparing against CONCAT method which is still not the most sophisticated among the available.\n- The authors mention that they randomly selected $70$% data for training, $10$% for val and $20$% for testing. As I understand, there is no cross-validation or testing on multiple folds. ABIDE dataset has a high variation of performance across subjects and the single-point results cannot be fully trusted. This can be a reason that in Fig 5, it is seen that the proposed model performs better on test data than val data. It is very much possible that the authors considered test subjects that generally provide high classification performance.\n- The paper provide very limited interpretation of the results.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cCtSN5Y0Qoc", "reviewer": "Reviewer_wuUg", "summary": "The paper presents a graph transformer for deriving effective representations of brain networks. In particular, this handles several caveats of brain networks: 1) use of connection profiles as positional embeddings and 2) deriving per-graph representation by projecting the node embeddings to orthonormal centroids for initialization. The downstream tasks using these embeddings on two public fMRI datasets show superior performance on autism spectrum disorder and gender prediction.", "questions": "The paper is overall solid. There are some questions related to the weaknesses section:\n1. Structural connectivities are often not fully connected via heuristics (e.g., thresholding) and/or biology (e.g., no fiber bundles between certain regions). Would the proposed method also properly leverage the connectivity profile?\n2. Was BrainNetTF tested with other readout functions?\n3. Why was gender chosen to be predicted using the ABCD dataset? Were there no other meaningful response variables to study? I am not sure how meaningful gender prediction is in general.\n\n[Update after the rebuttal]\nI am keeping my score (6). Most of my concerns were addressed, so I appreciate the authors' effort. I still think the stratified sampling of the sites is still just a simple heuristic.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper is overall well-written and easy to understand.\n+ The experiments are thorough with numerous baselines and ablation studies.\n+ The theoretical justifications for the benefits of orthonormality in a statistical manner are interesting.\n\nWeaknesses:\n- Predicting the gender with the ABCD dataset is not particularly interesting or useful.\n- If the soft clustering based on the proposed orthonormal bases shows empirical benefits, it would have been interesting to also see how the clusters imply additional neuroscientific findings based on their regions or functional modules.\n- The stratified splitting based on the site information is a common procedure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "b3wq_xuveLf", "reviewer": "Reviewer_baE1", "summary": "This paper proposed brain network transformer that learns representation from brain network with a transformer-based design. More specifically, the two main designs of the proposed brain network transformer are the multi-head self-attention module and the orthonormal clustering readout. In experiments, the proposed method showed performance improvements over multiple baseline methods.", "questions": "Please address the above W1 and W2 if possible.", "limitations": "n/a", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1. This paper is overall clearly written and easy to follow.\n\nS2. Brain network analysis is an important field of study and the proposed method is interesting and reasonably designed.\n\nS3. Both the experimental results in the tables and the case study features in Section 4.4 validates the effectiveness of the proposed method.\n\nW1. In Introduction, the authors mentioned the advantage on efficiency of the proposed method over existing graph transformer models. Hence, runtime experiments would be helpful to validate this claim.\n\nW2. Other than the transformer-base GNN methods, I wonder how does normal GNN perform on the brain networks. I would suggest the authors to also include vanilla GNN baselines for comparison. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FWwuPcaQYj", "reviewer": "Reviewer_xfsP", "summary": "The paper introduces a new transformer-based model for brain network analysis bringing better improvements on the ABIDE and ABCD datasets when compared to other transformer-based models. The model's architecture is motivated on insights from two previous papers [12, 25] to decide on the initial node features and to learn graph structures to achieve superior effectiveness. The paper makes a further contribution of providing a re-standardisation of an evaluation pipeline for the ABIDE dataset, given findings on unstable training.\n\n\n--------------------------------------\n[UPDATED AFTER REBUTTAL DISCUSSIONS]\nIn my opinion, the authors tackled my question in a satisfactory way, and I am making these changes to my review:\n\nSoundness: 2 > 3\nContribution: 2 > 3\nRating: 5 > 7", "questions": "The five points in \"Major weaknesses\" are the ones that made me not give a higher score, therefore I believe those are the points which could change my opinion the most on the scores given.", "limitations": "The paper contains predictions using human brain data, with some labels being sex and diagnosis. Therefore, I find it incorrect that the paper does not follow the recommendations on writing about societal impact when the prediction of these labels in specific can have so many different impacts in society.  I believe the acknowledgement of these variables would be the bare minimum needed for a NeurIPS paper (maybe starting from the last sentence of conclusion).\n\nWith regards to limitations, I believe they were tackled properly in Conclusion and in other sections of the paper. However, as a consequence of the answers to my questions this will be be better after the rebuttal process.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I think this is a good, well-written paper. Despite being yet another paper exploring transformer-based models, the results seem to be sound and the ablation analysis goes one step further and shows that the OCRead module can be useful in some other architectures. The proofs in supplementary material are a good addition and I thank the inclusion of a link for me to analyse the code during this review process. All these strengths are enough for me to recommend (borderline) acceptance, but I have some important concerns that make me not give a higher score at the moment, which I hope the authors can answer during the rebuttal period. I will first highlight what are my major concerns/doubts, and then leave a list of some other less important comments/suggestions.\n\nBefore going into it, **I must ask the authors to please double-check their terms to anonymise in the code shared** because while I was analysing the code, I found at least one `.yaml` file under the `setting` folder with an absolute path. It was quite obvious to me that it contained a username resembling quite closely some important papers referenced so I stopped looking more into it, but still would ask the authors to correct this in case we need to open the anonymised repo during the rebuttal process.\n\n## Major weaknesses\n\n1. The paper cites Kan et al. [25] to justify the idea of having the network to learn the graph structure, and uses the model presented in this reference (FBNETGEN) as another baseline for comparison. Although I know that choosing baselines can be a very subjective topic, I must say that selecting only one baseline for a learnable brain network is not enough. I have seen at least two possible papers learning a graph structure (arXiv:2002.04999, 10.1016/j.media.2020.101709) which were never evaluated on ABIDE/ABCD and could therefore be another interesting contribution of this paper to use this previous literature on unexplored datasets, even though the 2 papers I leave here had some analysis on medical images and were not used in the original FBNETGEN paper. From Table 1, I noticed that BrainGB and FBNETGNN seem to come from the same group as they come from the two references used to motivate the some choices on the model (refs 12 and 25), which is another non-subjective reason to include another baseline from another paper. Finally, the \"learnable brain network\" type in Table 1 is the one with the closest performance to BrainNetTF and therefore another baseline of the same type would improve the significance of this model. For all these reasons, I believe the authors need to add at least another meaningful baseline on the \"learnable brain network\" category in order for their results to be more sound.\n2. I believe the authors are mistakenly using \"gender\" when they mean biological sex. As far as I know, these datasets do not provide information on the gender of the people in the datasets, but rather the biological binary sex. I therefore request the authors to correct this for example in figure 2 and in the other occurrences of \"gender\".\n3. On lines 170-171 the authors claim that \"none of the existing readout functions leverages the properties of brain networks that nodes in the same functional modules tend to have similar behaviors and clustered representations\". After reading this sentence I immediately thought about the DiffPool paper (arXiv:1806.08804) and so many other papers adapted from this differentiable graph pooling module which, in practice, forms different clusters as a way to pool the nodes representations. I know of at least one recent paper (10.1016/j.media.2022.102471) using it in the medical context as readout function, so I believe the authors should probably downtown this sentence, and find a more complex (and who knows fairer) readout/pooling operator to compare OCRead with. From what I see in the code, the authors used DGL with pytorch, which I believe has implementations available for more complex pooling layers (https://docs.dgl.ai/en/0.8.x/api/python/nn-pytorch.html#global-pooling-layers) or DiffPool (https://github.com/dmlc/dgl/blob/master/examples/pytorch/diffpool/model/tensorized_layers/diffpool.py). \n4. I do not agree with the paper's justification in lines 270-272 that AUROC is the most proper performance metric, given for example that in the medical context a model needs to use a single threshold to decide on the positive/negative output, and AUROC tests all thresholds in practice. Besides, the model is trained such that the highest AUROC performance on the validation set is attained, which could artificially inflate these results and we do not know how they affect other very important metrics in the medical context like sensitivity, specificity, or even model calibration. I request the authors to at least add values on sensitivity and specificity (or equivalent) to show the strengths of this work. \n5. In Table 1 I find it difficult to understand how fair the comparisons are. Are these models similar in terms of number of layers? Or number of parameters? Given BrainNetTF seems to have had some sort of hyperparameter search, it is probably unfair to compare with out-of-the-box models?\n\n\n## Other small weaknesses/suggestions\n\nI order my comments in this section approximately by the order they appear in the paper. \n\n1. In lines 34 and 37, references [41] and [29] appear at the beginning of the line, which the authors can easily avoid in latex by using the tilde character (\"\\~\"): `text~\\cite{...}`.\n2. Figure 1, pane B, indicates that orthonormal bases are able to distinguish nodes that were indistinguishable in non-orthonormal bases. However, it seems to me that those points were already distinguishable before? What am I missing here? Also, do these points correspond to points in the dataset (like pane A), or are these just for illustration purposes?\n3. I like the highlight in lines 84-85 (and in other places in the paper) that ABIDE is the only dataset \"fully\" accessible online. However, I do not see this as a big issue in practice: most of the brain datasets like ADNI, PPMI, HCP, and NACC are quite easy to be accessed, and the logistics in place are more as a general control given we are talking about sensitive human data. Most of the times there's preprocessed data available as well.\n4. In section 2.1 maybe the authors will like to know about this (possibly) interesting review on GNNs: arXiv:2105.13137\n5. Line 111: Typo in \"STOA\".\n6. From figure 2/section 3.1, I can see that the authors used a symmetric weighted matrix as input. Given the authors are using the Transformer model, have the authors thought of using only the upper-half of the matrix as input to reduce model complexity? Or maybe skip this step altogether with the models I mentioned in my point (1) from major weaknesses?\n7. In line 145 the \"Connection Profile\" is capitalised. Authors probably want it non-capitalised (as in line 139)\n8.  Typo in line 187: missing \"initialization\" after \"Xavier uniform\".\n9.  Line 207 has a paragraph which starts as \"The second method\". I believe readibility would be improved if the method is actually specified here. \n10. Section 3.3.2 seems misplaced. It mentions possible generalisations of OCRead to other domains, but this is not explored in the paper. Wouldn't this be better fit in a discussion section or in conclusion?\n11. RQ3, in line 244, uses the term \"interpretability\", but it seems to me the authors are actually looking to the term \"explainability\" as (for example) defined in 10.1016/j.inffus.2019.12.012. I am aware these terms might be used in different ways so I'd ask the authors to include a reference to which framework they are using.\n12. In section 4.1, I think it is important to mention which tools were used, specifically DGL and pytorch from what I saw in the code (and reference accordingly).\n13. Typo line 249: \"Rs-fMRI\" is not capitalised.\n14. In line 263 and forward, is rs-fMRI also used from the ABCD dataset, or some task-based one?\n15. In Table 1, why was a t-test performed (e.g., under what normality assumptions?)?\n16. Given a statistical test was used in Table 1, why is there no statistical test in table 2?\n17. The paper seems to provide an overly optimistic interpretation of figure 3. Sometimes variation in plot is very small (in the order of 1-2%), but the authors seem to ignore this fact? Maybe the authors could caution about this interpretation as, from what I see, a Random initialisation could be preferred for situations where model complexity can be a problem and a non-significant drop in performance is ok.\n18. Typo in line 339: it is RQ3, not RQ4?\n19. In Figure 4, I find it difficult to agree with the statement that orthonormal initialisations produce more discriminative values. The way I see it, I think the authors mean that visually (to the human eye) we can see some divisions on the *ordered* set of features, but that doesn't mean necessarily that the variability seen in the Random part of ABIDE is less discriminative (maybe the groups are there, not just ordered). I think this section needs to do more to be able to prove what is said.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept