Learning Dynamic Graph Representation of Brain Connectome with Spatio-Temporal Attention
Authors: Byung-Hoon Kim
Jong Chul Ye
Jae-Jin Kim
Venue: NeurIPS 2021 Poster (2021)
Keywords: neuroimaging
connectome
fmri
graph neural network
attention
explainability

Abstract: Functional connectivity (FC) between regions of the brain can be assessed by the degree of temporal correlation measured with functional neuroimaging modalities. Based on the fact that these connectivities build a network, graph-based approaches for analyzing the brain connectome have provided insights into the functions of the human brain. The development of graph neural networks (GNNs) capable of learning representation from graph structured data has led to increased interest in learning the graph representation of the brain connectome. Although recent attempts to apply GNN to the FC network have shown promising results, there is still a common limitation that they usually do not incorporate the dynamic characteristics of the FC network which fluctuates over time. In addition, a few studies that have attempted to use dynamic FC as an input for the GNN reported a reduction in performance compared to static FC methods, and did not provide temporal explainability. Here, we propose STAGIN, a method for learning dynamic graph representation of the brain connectome with spatio-temporal attention. Specifically, a temporal sequence of brain graphs is input to the STAGIN to obtain the dynamic graph representation, while novel READOUT functions and the Transformer encoder provide spatial and temporal explainability with attention, respectively. Experiments on the HCP-Rest and the HCP-Task datasets demonstrate exceptional performance of our proposed method. Analysis of the spatio-temporal attention also provide concurrent interpretation with the neuroscientific knowledge, which further validates our method. Code is available at https://github.com/egyptdj/stagin

Learning Dynamic Graph Representation of Brain Connectome with Spatio-Temporay Attention
Byung-Hoon Kim
Jong Chul Ye Department of Bio/Brain Engineering Kim Jaechul Graduate School of Al KAIST
Department of Psychiatry Institute of Behavioral Sciences in Medicine College of Medicine, Yonsei University egyptdj@yonsei.ee ac.kr
jong yeekyiskkait...m ac
Jae-Jin Kim
Department of Psychiatry Institute Behavioral Sciences in Medicine College of Medicine, Yonsei University jaejkin@yonsei. ac kr
Abstract
Functional connectivity (FC) between regions of the brain can be assessed by the degree of temporal correlation measured with functional neuroimaging modal- ities. Based on the fact that these connectivities build network, graph-based approaches for analyzing the brain connectome have provided insights into the functions of the human brain. The development of graph neural networks (GNNs) capable of learning representation from graph structured data has led increased interest in learning the graph representation of the brain connectome. Although recent attempts to apply GNN to the FC network have shown promising results, there is still a common limitation that they usually do not incorporate the dy- namic characteristics of the FC network which fluctuates over time. In addition, few studies that have attempted to use dynamic FC reported reduction in performance compared to static FC methods, and did not provide temporal explainability. Here, we propose STAGIN, method for learn- ing dynamic graph representation of the brain connectome with spatio-tempores attention. Specifically, temporal sequence of brain graphs is input to the STAGIN to obtain the dynamic graph representation, while novel READOUT functions and the Transformer encoder provide spatial and temporal explainability with tention, respectively. Experiments on the HCP-Rest and the HCP-Task datasets demonstrate exceptional performance of proposed method. Analysis of the spatio-temporal attention also provide concurrent interpretation with the neuro- scientific knowledge, which further validates our method. Code available https: //::::/giitti /gtggttt......////////g wttpe/oott///sttteeeeeeeeeeeoe
input for the GNN
Introduction
Neuroimaging modalities provide measurements of brain activity by capturing the signals of neural activity. Functional magnetic resonance imaging (fMRI) non-invasive imaging method that measures the blood-oxygen level dependence (BOLD) in order to estimate the neural activity of the whole brain over time [20]. Functional connectivity (FC) defined the degree of temporal
This work was conducted while the first author was at KAIST
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

correlation between regions of the brain. Based on the fact that these connectivities form networks that change over time, graph- network analysis of brain connectome has been one of the key approaches to understanding how the brain works [7, 4, 433.
Graph neural networks (GNNs) are type of deep neural networks that have recently been successful in learning the representation of graph-structured data [50]. The graph-structured nature of the brain has led to an increased interest in learning the reperesentation of the brain FC network with the GNNs. Learning the representation of the brain connectome can be linked to decoding trait or state from human brain signal measurements. Accordingly, the current trend in studies attempting to apply GNN to the brain connectome is to input the FC graph from either resting-state [25, 26, 2, 33, 23, 48, 49] or task fMRI data [29, 30, 32] and predict a particular phenotype of the subjects, such as gender [26, 2, 23, 22] or presence of specific disease [26, 33, 29, 30, 32, 22]. While these studies have shown potential strengths and opportunities for learning the network representation of the brain, they also suggest limitations of current GNN-based methods.
One of the most common limitations with previous GNN-based FC network analysis methods is that most of them fail to take advantage of the dynamic properties of the FC network, which fluctuates over time. Incorporating the dynamic features of the FC network into the neuroimaging analysis has been an important direction in the field of functional neuroimaging [21, 37J. A work by [15] tried to address this issue by using the Spatial Temporal Graph Convolutional Network (ST-GCN) [54] model to incorporate dynamic features of the FC network. However, [15] reported lower accuracy than other non-dynamic GNN-based FC methods [23, 2] in the gender classification experiment, leaving question about the effectiveness of the dynamic FC method. In addition, another limitation of the method is that no temporal explainability is provided from the model. This is a major drawback considering that the goal of applying GNNs to functional neuroimaging methods is not only to achieve high classification accuracy, but also to uncover the functional basis of the brain [23, 32]. Another recent work by [3], using GraphNets [5] and DiffPool [58] for the dynamic FC analysis, also suffers from the same limitations in terms of poor classification accuracy and lack of temporal explainability.
Here, we propose Temporal Attention Graph Isomorphism Network (STAGIN) for learning the dynamic graph representation of the brain connectome with spatio-temporal attention. The proposed method exploits the temporal features of the dynamic FC network graphs to improve the classification accuracy of the model. particular, we address the issue that the node features of the input dynamic graph should contain temporal information and concatenate encoded timestamp with the node features (Section 4.1). In addition, the proposed method includes novel iitention-based READOUT modules (Section 4.2) and the Transformer encoder [46] (Section 4.3) in order to further improve the classification performance and provide spatial-temporal explainability at the same time. STAGIN achieves state-of-the-an performance with the Human Connectome Project (HCP) dataset [45] in gender classification for resting-stat fMRI and task decoding for task fMRI. We inherit k-means clustering analysis of the resting-state dynamic FC [1] and general linear model (GLM) statistical mapping of task fMRI [14] for interpreting the spatio-temporee attention learned from STAGIN, which are widely accepted analysis methods for the fMRI data. The interpretation of the attention replicates neuroscientific findings from previous large-scale fMRI
learned spatio-tempon studies in both resting-state and task fMRI, which further validates our proposed method.
Our work holds potential societal impact in that brain decoding methods can be linked to finding neural biomarkers of important phenotypes or diseases. However, potential negative impact related to privacy concerns that arise from abuse or misuse accurate decoding methods should also be noted, Although our method is yet behind the decoding capability that can be abused or misused, our research cannot still be free from these ethical considerations.
2 Related works
2.1 Graph Neural Network on Dynamic Graphs
Many networks that arise around us are inherently dynamic, with changes in the existence of nodes and edges over time. Learning the representation of dynamic graphs has piqued the interest of researchers and has led to development of methods that can embed dynamic graphs using their time information [35]. Methods that incorporate attention for learning the representation of dynamic graphs have also been proposed [51, 40J. However, it is not easy to apply these techniques directly to the dynamic brain graphs because of the different inherent properties of the dynamic brain graphs that

---

do not include any addition or deletion of nodes and are sampled uniformly over time. Nonetheless, our work is inspired by these earlier studies, particularly for the encoding of temporal information and their concatenation to the node features, proposed in Section 4.1 [51, 40].
2.2 Attention in Graph Neural Networks
Bringing attention to the GNNs is topic that is being actively studied in the field of geometric deep learning [27]. One of the most successful uses of attention is to compute the attention at edges of the graph and scale the importance of the links when the features of the neighborhood node are aggregated [47, 6]. often providing performance gain in learning the representation of input graphs. Another stream of applying attention to the GNNs comes with the motivation to define a pooling function on the graph domain. Since it is not straightforward to decide on what basis the coarsening should be carried out for graph structured data, works such as [16, 28, 38] have addressed this problem by selecting the nodes with top scores computed from projecting the node feature vectors into a learnable parameter vector, or from GNN layer aggregated local graph features. Although the motivation may have been different, these graph pooling methods are closely related to the spatial attention modules that we propose in Section 4.2 in that they exploit learned relative scores across the vertices of the graph. While some works have already been aware that the appropriate use of mode attention can improve performance of downstream tasks [55, 11], we note that previous methods tend to score attention based on randomly initialized parameters or local graph structures which may be suboptimal for graph classification tasks that require taking the whole graph feature into account.
3 Theory
3.1 Problem definition
The goal of our study is to train neural network
f Gayn + hG
where Gayn = (G(I), G(T)) is the sequence of brain graphs with T timepoints and IG E R is the vector representation of the dynamic graph G(t) with length D. The graph G(t) (V),), E(t)) at time is a pair of vertex set (t) {or (t), ..., (t) of N nodes and edge set E(t)
{ar, (0)) j ((1), (I,...NV} where (ii denotes the neighborhood of the vertex i. If f learns to extract disentangled representation of the dynamic brain graph Gdyn. then the classification of certain phenotypic characteristic (e.g. gender) from hG.p can be performed with a linear mapping as downstream task. Another important consideration in this work is to ensure the explainability of the model f, being able to inform us which part of the brain at which timepoint was considered important when extracting the meaningful representation hG Specifically, we formulate f = q o g as a composition of the GNN g and the Transformer encoder g, where g outputs the set of graph representations ho()) from each timepoint and q exploits self-attention to integrate ho())) into the final representation hG...
g : Gayn + (ba()))...aa(m)
q: (ha())..., + hG
(2)
We will omit timepoint notation (t) for brevity, whenever it is not of contextual importance.
3.2 Graph Isomorphism Network
The GNNs are generally composed of functions that (i) integrate the node features from its neighbors, and (ii) embed the integrated information with a nonlinear transformation to obtain the next layer node features. These functions are called AGGREGATE, and COMBINE functions, respectively, and the choice of these functions define many variants of the GNN,
(k) ACGREGATE(A) ({ ((A--') :u (EN))))))
a
(3)
h (k) COMBINE (1( .a.))
(4)
3

---

where h. denotes the feature vector of node v at layer & and hep := Z.
The Graph Isomrrphism Network (GIN) is a variant of the GNN suitable for graph classification tasks, which is known to be as powerful as the WL-test under certain assumptions of injectivity [52]. The GIN typically defines sum the AGGREGATE and a multi-layer perceptron (MLP) with two
layers as the COMBINE updating the node representation he (k) at layer [52] by:
h (k) MLP(K)
(1++(() 1((-1)) + Σ
h (1(-1))
(5)
(EN(()
where is a learnable parameter initialized with zero. Equation (5) can be easily reformulated into the matrix form [23] by:
H (b) ((c( I+
where
H (k)
h
(R)
(1))
E R DxN
is the stack of node feature vectors, I is the identity matrix, A is the adjacency matrix between the node features, W is the network weights of the MLP, and is the nonlinearity function.
The READOUT function takes the updated node features h to compute the representation of the whole graph:
(k) G
READOUT
(h! (k) 1 (uEG)).
In general, the READOUT function defined simply as computing the sum or average of the input node features. This is equivalent multiplication with the length N pooling vectors o
or db mean [1/,, 1/N] for the matrix form:
h
(k)
H (k)
Omcan.
(8)
3.3 Encoder-decoder understanding of GNNS
Although formulating the GIN (5) as a combination of AGGREGATE and COMBINE function might suggest its close relationship with convolutional neural networks (CNNs) at first glance, previous works by [23, 8] show that the matrix formulation of the GIN operation (6) can be thought of a CNN layer with shift operation of the convolution as the adjacency matrix A. We extend the understanding of encoder-decoder CNN as framelet expansion [57, 56] to the GIN to formulate node feature vectors (A) at layer with respect to the input node feature E; as:
Vec (H)(
(k) E()T (2) E x, where a == Vec ([#1, ... ,
where Vec(.) denotes the vectorization operation, and the k-th layer encoder matrix E(k) is defined
(A) (A) @ (c( I A")
where & refers to the Kronecker product, and (k)) is the diagonal matrix with values or 0 depending on the activation pattern of the nonlinearity. Now, Φmean of equation (8) can be thought as the decoder the k-thh layer which yields the whole graph feature vector from the encoded node feature vectors. Proposition 1. The READOUT function Φmean in (8) generates decoder with fixed constant bases.
Proof. From the READOUT function (8), have
hg
Vec
(k) (2 = Vec (A) Dmccn)) = (ØT menn & 1) Vec (K))))
(oT & 1) (k) E (K)T.....
(1) E
Now let by and b, denote the i-th column of the encoder matrix E() decoder matrix TT @ I), respectively. Then, it is straight to obtain the following representation:
E ( and the
(k) h
(b.,,
Therefore, we can see that although the encoder basis b, function of æe, the decoder basis b, is constant.

---

Time
grath paghtegpuua rephoaaaaiioo rechrrrrtti
Key
Quarry
Scalee
Sigmaid
Transformer
hg
GARO
Ruza
Horr
attention
GARO SERO
GIN
Bigmokk
Suquation
AC33
AQ23
ALF)
XC23
Spatial atterdion
SERO
Ripace
Bi
Figure 1: Schematic illustration of the proposed method. (a) Overall framework of the STAGIN. A sequence dynamic graph is first input to the GIN followed by GARO or SERO which produces a sequence of spatially attended graph representation vectors hG())- Temporal attention computed over ho(t) and the temporally attended graph representations averaged to generate the final representation iuG... (b) Attention based READOUT modules. Both GARO and SERO compute spatial attention Sspace with global average-pooled graph feature he prior.
We address the issue that the decoder being a constant function restrict the expressivity of the neural network, and explore adaptive READOUT functions with attention in Section 4.2.
4 STAGIN: Spatio-Temporal Attention Graph Isomorphism Network
In this section, discuss the details of main contribution. Specifically, we propose STAGIN with two novel attention-based READOUT modules for learning the dynamic graph representation of the brain connectome (Figure 1).
4.1 Dynamic graph definition
The sequence of input dynamic FC graphs constructed from 4D (MRI data with 3D voxels across time. The ROI-timeseries matrix P E IR x is extracted by taking the mean values within pre- defined 3D atlas which consists of N ROIs at each timepoint. Values of each ROI are standardized across time. Constructing dynamic FC matrix follows the sliding-window approach, where the temporal window of length I is shifted across time with stride S to generate windowed matrices (tt) Nx (Figure 2 (a)). The FC time t is defined as the correlation coefficient matrix R(t) of the windowed timeseries between Pt(t) and (t):
[Tmax r/S]
Cov(p:(t),,,()) ERNXN.
(t) pp. (t)op, (t)
where the subscript i and] the and column indices of P(t), Cov denotes the covariance, and op denotes the standard deviation of p. The final binary adjacency matrix A(t) E {0, NX obtained from the FC matrix R(t) by thresholding the top 30- 50-percentile values of the correlation matrix as connected, and otherwise unconnected following [23]. Other thresholds for binarizing the correlation matrix also experimented and the results provided in the Appendix Section C.2.
Unlike the adjacency matrix A(t), conventional definition of node feature vectors (,(t) at node index coordinates [29], incan-activation [29, 15], one-hot encoding [23], do not change over disregarding any temporal variation. To address this issue, we concatenate encoded timestamp (tt) IR' D to the spatial one-hot encoding e., followed by linear mapping with learnable parameter matrix W R (N++D) to define the input node feature,
(10) Here, the learnable timestamp encoder 7) Gated Recurrent Unit (GRU) [9] which takes ROI- as the input. Both the vertex set (tt) and the
x W (l)())
timeseries upto the endpoint of the
5

---

8
(b)
One-hot ROI 6
Linear W + xx(
ROI
Encoded Timestamp m(0)
Tyer
Timestamp encoder
Dynamic A(t)
Sliding
Window
Defining the dynamic graph. (a) Scheme of extracting dynamic graph from ROI-timeseries matrix P. (b) Example constructed dynamic graph.
Figure 2:
edge set E(t) of graph (it) now incorporates temporal information at time t. See Figure 2 for an illustration of the dynamic graph definition.
4.2 Spatial attention with attention- READOUT
As suggested from Proposition I, conventional READOUT function GNN can be thought as fixed decoder that decodes whole-graph feature from the node features with no learnable parameters. We address this issue by incorporating attention to the READOUT function, which the attention here refers to the scaling coefficient actoss the nodes learned by the model. Specifically, the spatial attention vector zzucee (t) E [0, 1] is computed by taking the as a prior:
Sspace s(H),
(11)
hg
H Zquaes
(12)
where : PXN + [0, 1 the attention function and HG denotes spatially attended graph representation lec. We propose two types of attention function s(.) for the attention -based READOUT, named Graph-Attention READOUT (GARO) and Squeeze-Excitation READOUT (SERO) inspired by the attention mechanisms of [46] and [19]. respectively.
4.2.1 GARO: Graph-Attention READOUT
The GARO follows key-query embedding based attention of the Transformer [46]. However, the key embedding is computed from the matrix of node features H, while the query embedding is computed from the vector unattended graph representation Hmaan'
K W key H,
q W query H Ameans
K
space = sigmoid
(13)
VD
where Wkey DDD Wauery& E DDD learnable key-query parameter matrices, K 18 DxN is the embedded key matrix, and q E R is the embedded query vector.
4.2.2 SERO: Squeeze-Excitation READOUT
The SERO follows MLP based attention of the Beneeze-anddExxittaiiin Networks [19]. However, attention from the squeezed graph representation does scale the channel dimension, but the node dimension in SERO:
Zypace sigmoid W20 W.HHo @mean)).
(14)
where the nonlinearity function and Wi DDD W, NDD are learnable parameter matrices. This type of spatial dimension squeeze-excitation module has been shown to improve

---

performance of the CNN models [41], but was not easily applicable to general graphs which may vary in number of nodes for each graph. We exploit the fact that the brain graphs have fixed number of nodes N across participants based on the chosen atlas.
4.2.3 Orthogonal regularization
we take closer look at (8) and (12), computation of graph feature vector hG from the node feature matrix H can also be viewed as reconstructing signal hG from the basis frames H with vectors Φmean and Sspce. respectively. While space provides further expressivity of the model with adaptive coefficients when compared to @maan, find it desirable to encourage the orthogonality of H as elaborated in the Appendix Section A. The orthogonal regularization Lothh is defined as:
Locho
1/m H H - 111
(15)
where m max(H
( H). The scaling term 1 1/m ensures the columns of the matrix H become orthogonal to each other with the same length, while not restricting the specific length that the column vectors should follow.
4.3 Temporal attention with Transformer encoder
For attention across time, we employ a single-headed Transformer encoder [46] upon the sequence of graph features (hott)s..,,hary)) The temporal attention can be measured by the self seff-attention weights Zime [0, 1|TXT after the softmax function of the Transformer encoder. Per-layer dynamic graph representation Gaa
(k) is computed by summing the temporally attended feature output from the
Transformer encoder across time at each layers, where the final representation:
(k)) hG. concatenate Geh
{L,..., K}})
(16)
is the concatenation of dynamic graph representation of all layers following [53].
5 Experiment
5.1 Dataset
Publicly available² fMRI data from the HCP S1200 release [45] was used for our experiments. The data was collected from voluntary participants with informed consent and was fully anonymized. We constructed two datasets, the HCP-Rest and the HCP-Task, depending on whether the subject was resting or performing specific tasks during the acquisition of the image. The HCP-Rest dataset consisted of pre-processed and ICA denoised resting-state fMRI data [17], which the subjects were instructed to rest for 15 minutes during the data acquisition. We used first run data the four sessions, and excluded data with short acquisition time with Tmax 1200. There were 1093 images finally included in the dataset, which consisted of 594 female and 499 male subjects. The gender of each subject served as the labels of the HCP-Rest dataset letting the number of classes C = 2. The HCP-Task consisted of pre- proccesceeeed task fMRI data [17], which the subjects were instructed to perform specific tasks during data acquisition. For example in the "Motor" task fMRI, participants were told to perform one of the subtasks during the acquisition to make motor movements on one's left hand, left foot, right hand, right foot, or tongue. There were seven types of tasks including working memory, social, relational, motor, language, gambling, and emotion. After excluding the IMRI data with short acquisition time, there were 7450 images included in the dataset. The task type during the data acquisition served as the labels of the HCP-Task dataset, letting C' = 7. A more detailed description of the experiment datasets with note on the twin subjects of HCP can be found in the Appendix Section B.
5.2 Experimental settings
Experiments were performed on a workstation with two NVIDIA GeForce GTX 1080 Ti GPUs. The STAGIN model f is trained end-to-end in a supervised manner with the loss L = Exeat + À Lottho
httpps:/ :/tbps://db... hww.acccnnnncccomm..... org

---

Table 1: Comparative study on HCP-Rest and HCP-Task dataset.
HCP-Rest
HCP-Task
Model
Type of FC
# Params
Accuracy (%)
AUROC
Accuracy (%)
SIAGIN-SERO
88.20 + 1.33
0.9296 + 0.0187
99.19 = 0.20
Dynamic
1,209k
STAGIN-GARO
87.01 = 3.00
0.9151 0.0258
99.02 = 0.17
Dynamic
1,068k
ST-GCN [15]
76.95 = 3.00
0.8545 = 0.0316
98.92 = 0.27
Dynamic
355k
MS-G3D [10]
79.16 + 2.53
0.8912 + 0.0329
Dynamic
3,045k
BAnD++ [36]
97.20 + 0.57
None
2,010k
BAnD [36]
95.10 + 0.62
None
2,010k
r-BAnD
98.90 0.27
Dynamic
664k
GIN [23]
81.34 + 2.40
0.8955 + 0.0237
93.87 0.66
Static
169k
GCN [24]
80.79 2.00
0.8741 = 0.0174
45.07 1.63
Static
101k
GraphSAGE [31]
75.48 + 1.97
0.8237
+ 0.0228
54.52
Static
202k
ChebGCN [2]
77.76 2.09
0.8582 0.0233
73.06 + 0.68
Static
704k
where Lxent is the cross entropy loss and À is the scaling coefficient of the orthogonal regularization. We set the number of layers K = 4, embedding dimension D 128, window length I = 50, window stride S 3, and regularization coefficient À = 1.0 10-5 The window length and stride correspond to capturing the FC within 36 seconds every 2. 2616 seconds, which follows the standard setting of the sliding-window representation hGar and rate 0.1 is applied to the attention vectors Sspnce and Ztime during training. For nonlinearity o in (6) and (14), GELU [18] is used instead of ReLU with batch normalization before each One-cycle learning rate policy is employed, which the learning rate is gradually increased from 0.0005 to 0.001 during the early 20% of the training. and gradually decreased to 5.0 10 afterwise. Thirty training epochs were run for the HCP-Rest dataset with minibatch size 3, while ten epochs were run with minibatch size 16 for the HCP-Task dataset. We performed 5-fold stratified cross-validation of the dynamic graphs from the dataset, and report mean and standard deviation across the folds. To extract the ROI-timeseries, the Schaefer atlas [42] with 400 regions (N 400) labelled with 7 intrinsic connectivity networks (ICNs) of ROI-timeseiies matrix P was randomly sliced with a fixed length (600 for HCP-Rest, 150 for HCP-Task) each steps during training for (i) relieving computational overload, (ii) stochastic augmentation of the training dataset, (iii) mitigating unwanted memorization of the specific timing of subtask onset, and (iv) matching the number of timepoints T across different task labels for the HCP-Task dataset. Unsliced full matrix P was used for inference at test time. The end-to- end-to-end inference from the construction of the dynamic graph to the acquisition of the final prediction required 1.68 seconds per sample with given experimental settings.
-widow dFC analyses [59, 37]. Dropout rate 0.5 is applied to the final dynamic graph
used. The time dimension
5.3 HCP-Rest: Gender classification
We first validate our proposed method by gender classification on the HCP-Rest dataset. The two proposed methods, named STAGIN-GARE and STAGIN-SERI based on the type of the spatial attention module, resulted in 57.01% and 88. 88.20% mean accuracy the 5-fold cross validation, respectively (Table 1). The mean area under receiver operator characteristic curve (AUROC) were 0.9151 and 0.9296. Classification performance of STAGIN is compared with other GNN methods for reprensentation learning of dynamic/static FC network, including ST-GCN [15], MS-G3D [10], GIN [23], GCN [24], GraphSAGE [31], and ChebGCN [2]. We used the code by the authors of [15] and [10]* but modified the cross validation scheme to avoid early stopping based on the test dataset for fair comparison. It can be seen from Table that our proposed method outperforms other GNN based methods. The results of the ablation study are shown in Table 3 in the Appendix.
We use STAGIN-SERO, which showed the best accuracy, for analyzing temporal and spatial attention
of the dynamic FC networks. We define the temporal attention vector (kk) E [0, 1]7 at layer
(A:) as the average of row elements in the self-attention weight matrix z [i] $ _T= Zij where
www.h/www...oooeeiieelllllaaeeea fmri
8

---

Cluster 6
Cluster
Cluster?
Cluster 4
Cluster
Cluster
Cluster
DMN
SMN
Female
1.299
1.208
1.228
0.807
0.690
0.642
0.576
Male
0.927
0.950
0.967
1.036
1.079
1.041
1.128
F/M
1.401
1.272
1.269
0.778
0.640
0.617
0.510
Figure 3: Analysis of temporal attention of the gender classification experiment with k-means clustering. The DMN and SMN of the 7 cluster centroids are plotted and the relative proportion of temporally attended clusters for female and male subjects are written below. The clusters are sorted in descending order of the female/male attended cluster ratio.
(k) [i] and Zij are j-th element of = and (i.j)-th element _ for the resting-state data, "ime
(k)
respectively. To employ k-means clustering to the resting-state dynamic FC analysis [l]. we first define a set of attended timepoints T {e Prime [4] > Ziins } where is the cutoff coefficient, (k)
and o (0))) denotes the standard deviation of - time' Defining the threshold based on standard deviation
inherits the practice of the point-process analysis for dynamic FC, so set 1.0 following [44]. Pattern of the FC matrices at attended timepoints A {A(t) E T} for each subject can now be analyzed with the k-means clustering. Specifically, we fit 7 template cluster centroids from the dynamic FC matrices (t) over all subjects, and assign elements of A into one of the template clusters. The ratio of each clusters from A with respect to A can then be analyzed with the subset of A including only the female or male subjects.
Evidences from large scale studies suggest that female subjects show hyperconnectivity of the DMN [34, 39] and hypoconnectivity of the SMN when compared to male subjects [39, 13]. We accordingly hypothesized that the FC at attended timepoints will show higher values for the DMN and lower values for the SMN in female participants. Figure 3 demonstrates that the clusters mainly attended by female participants show trend of hyperconnectivity of the DMN and hypoconnectivity of the SMN. This can be interpreted to mean that the STAGIN is properly trained to take the dynamic state of the FC networks into account for predicting the phenotype of the subject.
The spatial attention across regions of the brain is analyzed with the 2space averaged across time =(k) Espace :- + ΣT =quace (t). The regions with top 5 percentile attention values of Siquce are plotted with respect to the seven iCNs Figure 9 in the Appendix. can be seen that the majority of the top attended regions are from the SMN, which further suggests gender difference of resting-state FC within the SMN. A notable limitation here is that the threshold for determining the top attended region is heuristically set. Statistically determining the spatially attended regions from the resting-state data would further provide validity of the method, which is left as a future work.
(k)
5.4 HCP-Task: Task decoding
Task decoding refers classifying which of the seven tasks the subject was performing during the acquisition of the brain IMRL. The STAGIN-GARO and STAGIN-SERG showed 99.02% and 99.19% mean accuracy for the task decoding experiment, respectively (Table 1). It can be seen that the proposed methods outperform the previous offte--o-the-aat art model BAND and BAnD++ [36], which applied self-attention of the Transformer encoder directly to 3D ResNet extracted representation vectors of the fMRI without considering the network property of the brain. To account for the possible statistical disadvantage of voxel-based feature extraction, we further implemented new region-based BAnD (r-BAnD) by using GIN without attention-base bbaed READOUT instaed of the 3D ResNet. Accuracy of r-BAnD resulted in an accuracy of 98.90% suggesting that our method shows superior performance even when the statistical disadvantages are matched. Experiment

---

Layer 2
Layer
Layer 2
04
Layer
Layer 4
D6N
SVN
OGN
DAN
IN
20N VN
DMN
AVN
CGN
DAN
UN
3MN
VN
Layer
Layer 4
0.4
2
DNN DAN LN
DVN
D9RV 1994
DGN
1MRV
Figure 4: Analysis of spatio-temporal attention for working memory task of the task decoding experiment. (a) Plot of average temporal attention matrix z(K) across subjects. (b) Proportion of statistically significant regions within the 7 ICNs from the spatial attention GLM.
other models including ST-GCN [15], GIN [23], GCN [24], GraphSAGE [31], and ChebGCN [2] demonstrate exceptional performance of our proposed method for HCP-Task (Table 1). The fact that subtask timing information is completely lost may reflect the reason behind poor performance of static FC methods, which can be a critical disadvantage in task classification.
We interpret the result from the working memory task for spatio-temporal attention analysis, where the subtask consists of either performing an n-back memory task or rest. Our key expectation the temporal attention analysis dynamic FC graphs. then Zime should represent which subtask the subject was upto. Surprisingly, that the Transformer encoder of STAGIN learns to attend to the timing of subtasks from Figure 4 (a), which demonstrates mean temporal attention Zzime across all subjects. Notice that no supervision is provided the STAGIN model regarding the subtask timing during training.
that if STAGIN learns accurately attend temporal features of the
can be clearly
To analyze the spatially attended regions Espace of STAGIN, we construct a GLM [14] to statistically evaluate how much each region is responsible for performing the subtasks. The parameter vectors Brssk E R. and Brest E R. are estimated with the sequence of spatial attention vectors and the subtask timing design matrix M {0, 1 by solving the following with least-squares estimation:
Expace (0), Sspace (T)) =M BBask Brest] + E,
where e denotes residual error. The contrast of the estimated parameters Busk and Breat was set to [1.-1] so the rejection of null hypothesis indicates Bams [V] > Bres [i] the i-th ROI. Multiple
comparisons of the N ROIs are family-wise error (FWE) corrected.
Figure 4 (b) shows the proportion of statistically significant regions within the ICNs for each layers. Interestingly, the layer and 2 share a similar trend that the regions from SMN, visual network (VN), and salience/ventral attention network (SVN) dominance of the regions from DMN and cognitive control network (CCN). We denote the layer and 2 as the low-order layers (LoL) and the layer 3 and 4 as the high-order layers (HoL). The dominance of SMN and VN at LoL can be understood as the low-level sensorimotor function for perceiving the task being processed within the short-range 1- or 2-hop connection of the networks. On the other hand, the dominance of DMN and CCN at HoL reflects the high-level cognitive integration for executing and controlling the given task being processed within the long-range 3- or 4-hop connection of the networks. Considering that the SVN is a network for integrating the low-level sensorimotor networks and the high-level executive networks to provide dynamic balancing between the two functions, the significant regions of SVN being present at both Lol. and HoL. is not surprising. Temporal and spatial attention plot of other six tasks are further provided in the Appendix Section D.2.
dominant. In contrast, layer 3 and 4 suggest
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-202 IM3ESD9025019, NRF-2020R AABSB003020B000000000 This work was also supported by Institute Information communications Technology Planning & Evaluation (IITP) grant funded by the Korea goVernmen((MSIT) (No.2019-0-0005) Artificial Intelli- gence Graduate School Program(KAIST)) and the KAIST Key Research Institute (Interdisciplinary
Research Group) Project.
References
[1] Elena A Allen, Eswar Damaraju, Sergey M Plis. Erik B Erhardt, Tom Eichele, and Vince D Calhoun. Tracking whole-brain connectivity dynamics in the resting state. Cerebral cortex, 24(3):63-6676 2014.
[2] Salim Arslan, Sofia Ira Ktena, Ben Glocker, and Daniel Rueckert. Graph saliency maps through spectral convolutional networks: Application sex classification with brain connectivity, In Graphs in Biomedical Image Analysis and Integrating Medical Imaging and Non-Imaging Modalities, pages 3-13. Springer, 2018.
[3] Tiago Azevedo, Alexander Campbell, Rafacl Romero-Garcia, Luca Passamonti, Richard AI Bethlehem, Pietro Lio, and Nicola Toschi. deep graph neural network architecture for modelling teatio-temporal dynamics in sesting-statingg functional mri data. bioRxiv, 2020.
[4] Danielle S Bassett and Olaf Sporns. Network neuroscience. Nature neuroscience, 20(3):353, 2017.
[5] Peter W Battaglia, Jessica Hamrick, Victor Bapst, Alvaro Gonzalz, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01200 2018.
[6] Shaked Brody, Uri Alon, and Eran Yahav, How attentive are graph attention networks? arXiv preprint 2rXiv:2105.14491, 2021.
[7] Ed Bullmore and Olaf Sporns. Complex brain networks: graph theoretical analysis of structural and functional systems. Nature reviews neuroscience, 10(3):186 198, 2009.
[8] Mark Cheung, John Shi, Oren Wright, Lavendar Y Jiang, Xujin Liu, and José MF Moura. Graph signal processing and deep learning: Convolution, pooling, and topology. IEEE Signal Processing Magazine, 37(6):139 149, 2020.
[9] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412.3555 2014.
[10] Simon Dahan, Logan ZJ Williams, Daniel Rueckert, and Emma Robinson. Improving phenotype prediction using long-range spatio-temporal dynamics of functional connectivity. In International Workshop on Machine Leurning in Clinical Neuroimaging, pages 145-154. Springer, 2021.
[11] Xiaolong Fan, Maoguo Gong, Yu Xie, Fenlong Jiang, and Hao Li. Structured self-attention architecture for graph-level representation learning. Puttern Recognition, 100:107084, 2020.
[12] Matthias Fey and Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv: 1903.02488, 2019.
[13] Massimo Filippi, Paola Valsasina, Paolo Misci, Andrea Falini, Giancarlo Comi, and Maria Rocca. The organization of intrinsic brain activity differs between genders: A resting-state fmri study large cohort of young healthy subjects. Human brain mapping, 1436)1333--333,, 2013.
11

---

[14] Karl Friston, Andrew P Holmes, Keith J Worsley, J-P Poline, Chris D Frith, and Richard SJ Frackowiak. Statistical parametric maps in functional imaging: a general linear approach. Human brain mapping, 2(4):189-210, 1994.
[15] Soham Gadgil, Qingyu Zhao, Adolf Pfefferbaum, Edith V Sullivan, Ehsan Adeli, and Kilian M Pohl. Spatio-temporal graph convolution for resting-state fmri analysis. International Conference on Medical Image Computing and Compuler-Assisted Intervention, pages 528-538. Springer, 2020.
[16] Hongyang Gao and Shuiwang Ji. Graph u-nets. In international conference on machine learning. pages 2083-2092. PMLR, 2019.
[17] Matthew F Glasser, Stamatios N Sotiropoulos, J Anthony Wilson, Timothy S Coalson, Bruce Fischl, Jesper L. Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan R Polimeni, al. The minimal preprocessing pipelines for the human connectome project. Neuroimage, 80:105-124, 2013.
[18] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 160.::66660044155 2016.
the [19] Jie Hu, Li Shen, and Gang Sun. Squeeze-ande aatezze--nd--eeeee and-excitation networks. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[20] Scott A Huettel, Allen W Song, and Gregory McCarthy. Functional magnetic resonance imaging, volume Sinauer Associates Sunderland, MA, 2004.
[21] Matthew Hutchison, Thilo Womelsdorf, Elena A Allen, Peter Bandettini, Vince D Calhoun, Maurizio Corbetta, Stefania Della Penna, Jeff H Duyn, Gary H Glover, Javier Gonzalez-Castillo et al. Dynamic functional connectivity: promise, issues, and interpretations. Neuroimage, 80:360-331 378, 2013.
[22] Anees Kazi, Soroush Farghadani, and Nassir Navab. Ia-gen: Interpretable attention based graph convolutional network for disease prediction. arXiv preprint 20t.:2/033//58877 2021.
[23] Syung-Hoon Kim and Jong Chul Ye. Understanding graph isomorphism network for rs-fmri functional connectivity analysis. Frontiers in neuroscience, 14:630, 2020.
[24] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint 1rXiv:160.002907 2016.
[25] Soñia Ira Ktena, Sarah Parisot, Enzo Ferrante, Martin Rajchl, Matthew Lee, Ben Glocker, and Daniel Rueckert. Distance metric learning using graph convolutional networks: Application to functional brain networks. In International Conference Medical Image Computing and Computer-Assisted Intervention, pages 469-477. Springer, 2017.
[26] Sofia Ira Ktena, Sarah Parisot, Enzo Ferrante, Martin Rajchl, Matthew Lee, Ben Glocker, and Daniel Rueckert. Metric learning with spectral graph convolutions brain connectivity networks. Neurolmage, 169:431-442, 2018.
[27] John Boaz Lee, Ryan A Rossi, Sungchul Kim, Nesreen K Ahmed, and Eunyee Koh. Attention models in graphs: A survey. ACM Transactions Knowledge Discovery from Data (TKDD), 13(6):1-25, 2019.
[28] Junhyun Lee, Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling, In International Conference on Machine Learning, pages 3734-3743. PMLR, 2019.
[29] Xiaoxiao Li, Nicha C Dvornek, Yuan Zhou, Juntang Zhuang, Pamela Ventola, and James S Dun- can. Graph neural network for interpreting fusk-fmri biomarkers. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 485-493. Springer, 2019.
[30] Xiaoxiao Li, Nicha C Dvornek, Juntang Zhuang, Pamela Ventola, and James Duncana. Graph embedding using infomax for asd classification and brain functional difference detection, arXiv preprint arXiv: 04769, 2019.
12

---

[31] Xiaoxiao Li, Yuan Zhou, Nicha C Dvornek, Muhan Zhang, Juntang Zhuang, Pamela Ventola, and James S Duncan. Pooling regularized graph neural network for fmri biomarker analysis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2020.
pages
[32] Xiaoxiao Li, Yuan Zhou, Siyuan Gao, Nicha Dvornek, Muhan Zhang, Juntang Zhuang, Shi Gu, Dustin Scheinost, Lawrence Staib, Pamela Ventola, et al. Braingnn: Interpretable brain graph neural network for fmri analysis. bioRxiv, 2020.
[33] Guixiang Ma, Nesreen K Ahmed, Ted Willke, Dipanjan Sengupta, Michael W Cole, Nick Turk- Browne, and Philip S Yu. Similarity learning with higher-order proximity for brain network analysis. arXiv preprint arXiv: 1.0662, 2018.
[34] Lauren Mak, Luciano Minuzzi, Glenda MacQueen, Geoffrey Hall, Sidney H Kennedy, and Roumen Milev. The default mode network in healthy individuals: systematic review and meta-analysis. Brain connectivity, 7(1):25-33, 2017.
[35] Giang Hoang Nguyen, John Boaz Lee, Ryan A Rossi, Nesreen K Ahmed, Eunyee Koh, and Sungchul Kim. Continuous-time dynamic network embeddings. In Companion Proceedings of the The Web Conference 2018, pages 969-976, 2018.
[36] Sam Nguyen, Brenda Ng, Alan D Kaplan, and Priyadip Ray. Attend and decode: 4d fmni task state decoding using attention models. In Machine Learning for Health, pages 267-279. PMLR, 2020.
[37] Maria Giulia Preti, Thomas AW Bolton, and Dimitri Van De Ville. The dynamic functional connectome: State-of-the oorthe--rtttttttee and perspectives. Neuroimage, 160:41-544 2017.
[38] Ekagra Ranjan, Soumya Sanyal, and Partha Talukdar. Asap: Adaptive structure aware pooling for learning hierarchical graph representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5470-5477, 2020.
[39] Stuart J Ritchie, Simon R Cox, Xueyi Shen, Michael V Lombardo, Lianne M Reus, Clara Alloza, Mathew A Harris, Helen L. Alderson, Stuart Hunter, Emma Neilson, et al. Sex differences in the adult human brain: evidence from 5216 uk biobank participants. Cerebral Cortex, 28(8):2959-2975, 2018.
[40] Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint 10X6:22006.10637 2020.
[41] Abhijit Guha Roy, Nassir Navab, and Christian Wachinger. Recalibrating fully convolutional networks with spatial and channel "squeeze and excitation" blocks. IEEE transactions on medical imaging, 38(2):540-549, 2018.
[42] Alexander Schaefer, Ru Kong, Evan M Gordon, Timothy O Laumann, Xi-Nian Zuo, Avram J Holmes, Simon B Eickhoff, and BT Thomas Yeo. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri. Cerebral Cortex. 28(9):3095-3144 2017.
[43] Olaf Sporns. Graph theory methods: applications in brain networks. Dialogues in clinical neuroscience, 20(2):1 11, 2018.
[44] Enzo Tagliazuechi, Pablo Balenzuela, Daniel Fraiman, and Dante R Chialvo. Criticality in large- scale brain fmri dynamics unveiled by a novel point process analysis. Frontiers in physiology. 3:15, 2012.
[45] David C Van Essen, Stephen M Smith, Deanna M Barch, Timothy EJ Behrens, Essa Yacoub, Kamil Ugurbil, Wu-Minn HCP Consortium, et al. The wu-minn human connectome project: an overview. Neuroimage, 80:62-79, 2013.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008. 2017.
13

---

[47] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 1xXiv:1710.10003. 2017
[48] Simon Wein, WM Malloni, Ana Maria Tomé, Sebastian M Frank, G-I Henze, Stefan Wüst, Mark W Greenlee, and Elmar W Lang. A graph neural network framework for causal inference in brain networks. Scientific reports, 11(1):1-18, 2021.
[49] Dongya Wu, Xin Li, and Jun Feng. Connectome- bassd individual prediction of cognitive behaviors via the graph propagation network reveals directed brain network topology. bioRxiv, 2021.
[50] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks, IEEE transactions on neural networks and learning systems, 2020.
[51] Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Inductive representation learning on temporal graphs. arXiv preprint arXiv:2002.07662, 2020.
[52] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint
2018.
[53] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. arXiv preprint arXiv: 1806.03536 2018.
[54] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[55] Yichao Yan, Jie Qin, Bingbing Ni, Jiaxin Chen, Li Liu, Fan Zhu, Wei-Shi Zheng, Xiaokang Yang, and Ling Shao. Learning multi-attention context graph for group-based re-identification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[56] Jong Chul Ye, Yoseob Han, and Eunju Cha. Deep convolutional framelets: general deep learning framework for inverse problems. SIAM Journal Imaging Sciences, 11(2)9911-1048, 2018.
[57] Jong Chul Ye and Woon Kyoung Sung. Understanding geometry of encoder-decoder CNNs. In International Conference on Machine Learning, pages 7064-7073, 2019.
[58] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovee. Hierarchical graph representation learning with differentiable pooling. In Advances in neural information processing systems, pages 4800-4810, 2018
[59] Andrew Zalesky and Michael Breakspear. Towards a statistical test for functional connectivity dynamics. Neuroimage, 114:466-477 470, 2015.
14


Reviews:
[{"review_id": "fpOmmZrnX2p", "reviewer": "Reviewer_RkTU", "summary": "This paper proposes a GNN model (STAGIN) for learning from dynamic, spatio-temporal fMRI data. The model, based on GIN, takes a temporal sequence of brain graphs as input and includes a time encoding in the node features. The standard readout function is modified to incorporate spatial attention in two possible ways (scaled dot product attention and squeeze-excitation model). The series of graph features is then passed to a transformer-style self-attention encoder to incorporate temporal attention. Orthogonality of node feature vectors is also encouraged in the loss function. The method was tested on the public HPC datasets (~1100 subjects) to classify gender from resting state fMRI and decode task from task-based fMRI, and analysis of the temporal and spatial attention results was performed.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:  \nThe authors present some new methods for learning dynamic graph representations in the context of brain connectomics. While prior work on methods for learning from temporally changing brain graphs are presented, I believe there is some discussion missing of prior work on the proposed methods of including attention in GNN networks, specifically for including attention in the readout layer (eg., [1,2]). Please discuss how the proposed methods differs from such prior work. \n\n[1] Fan et al., Structured self-attention architecture for graph-level representation learning, 2020.  \n[2] Yan et al., Learning Multi-Attention Context Graph for Group-Based Re-Identification, 2020.\n\nQuality:  \nThis is a nice paper that brings in some often used modules (scaled dot attention / squeeze excitation, self attention) to the GNN model to analyze dynamic graph data, specifically 4D fMRI.  The methods seem appropriate and is a well thought-out piece of work, with some nice posthoc analysis of the attention results with respect to connectome understanding (e.g., visualizations showing the changes in spatial attention following the changes in task, visualizations of network differences for males/females). My primary critiques lie in the experimental methods. Some detailed comments:\n\n1. In Sec. 3.2.3, the authors discuss the use of orthogonal regularization for the node features, and say they \"find it desirable to encourage the orthogonality of H\". I am not sure I understand the motivation behind the orthogonal constraint, compared to some other form of regularization - it would be nice for authors to elaborate on that.\n2. My biggest concern lies in the experimental portion of the paper. The authors do include results of ablation experiments in the supplementary materials, but in the proposed method is compared to only 1 other approach for the gender classification task and 2 variations of 1 approach for the decoding task.  Furthermore, while it seems the result for the gender task was produced by the authors, the decoding results were taken from the literature, and it is unlikely that the exact same subjects and preprocessing step/experimental methods were used. Thus, I would say the numbers are informative but not directly comparable. Also, given that the ST-GCN result was produced by the authors, I don't know why this method wasn't also tested on the decoding task.  \n3. Authors appropriately list all the hyperparameter settings, and it seems that the same settings were used for all folds (no tuning). However, I am wondering about the setting for the time window length $\\Gamma$. While authors give references to support their choice, in general window length in standard dynamic connectivity analysis is highly debated (actually as mentioned in one of their references) and the choice is generally arbitrary, with analysis results potentially greatly depending on the windowing. While the authors do provide a fairly complete study overall, I think it would be great to see the effect of different window sizes on the proposed method's results.\n4. I am wondering for the k-means clustering of attended graphs, how/why the authors chose 7 clusters? \n\nClarity:  \nThis is a well-written paper - it is well-organized with appropriate amount of background and the writing is very clear. Just a couple notes:\n1. p.2, line 82: Authors wrote $f = g \\circ q$ where $g$ is the GNN and $q$ is the transformer encoder - should be $q \\circ g$\n2. Fig. 3 caption: \"attened\" --> \"attended\"\n\nSignificance:  \nImproved dynamic analysis of fMRI data could be very significant, allowing for better understand of underlying neurological processes and disease, and I think this work would definitely be of interest to readers in this field. I think the proposed method for including spatial and temporal attention to GNN analysis could lead to others building off of such ideas. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations are addressed (threshold for determining attended regions), and they mention that the understanding brain connectome has both positive and negative societal impact, as linked to search for new biomarkers, e.g., for disease. However, it is not really clear what negative impact they mean by this, and I think some elaboration is needed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ag_NLsqhOl", "reviewer": "Reviewer_EUFS", "summary": "The paper proposes an attention based GNN method, which it names STAGIN. The new method handles dynamic FC matrices for gender and task classification using fMRI data and learns to focus on important time points and brain regions based on attention. The method achieves better accuracy and AUC for classification compared to the relevant competing methods that STAGIN is compared against. The temporal attention matches with the task-relevant time points in the input data.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of the paper is in handling dynamic FC matrices and finding important time points and brain regions relative to the downstream task of classification. The paper uses GNN to model brain data into nodes representing brain regions and edges representing functional connectivity. The paper shows higher accuracy and AUC scores for the task of gender and task classification compared to the methods mentioned in the paper. The paper has the following major and minor drawbacks.\n\n## Major Drawbacks:\n\n1.  The idea of applying attention to graph networks is not novel as there have been many published studies for this e.g. GAT, GATV2 etc. The paper applies attention to the published method GIN but does not compare similarities or differences with existing such approaches e.g. “Understanding Attention and Generalization in Graph Neural Networks” ChebyGIN.\n2.  The author(s) propose a novel attention based Readout method and add it in GIN to create a single embedding for the complete graph. The authors propose two variations of the method. Again, such methods are already well established e.g. topKpooling, SAG pooling, ASA pooling. The author(s) do not discuss or compare against such methods and rather mention the original method of GIN. The author(s) uses temporal data with a sliding window approach to create dynamic FC matrices but do not compare against methods which handle temporal data and dynamic graphs e.g. TGAT, TGN-attention, CTDNE etc. These methods may not have been applied for the same downstream classification task but still a comparison with them would reveal if the proposed method is improving over existing analogous approaches. The paper is well written and experiments are thorough and well structured, but it is unclear whether a technical novelty is introduced or largely existing methods are applyed to fMRI data for gender/task classification. (Also se item 2 in the minor drawbacks)\n3.  The paper needs to report results from much more published studies, especially those which handle dynamic/temporal data as mentioned above. The author(s) also need to show how their attention mechanism is better than existing methods of attention in GNNs.\n4.  The author(s) need to mention how they got the result of BAND and BaND++ and whether the presented comparison is made on the same data with the same processing. BAND and BaND++ are applied to 4D data of voxels and not on region based data and thus would be at a statistical disadvantage if compared with the method that uses region based aggregation.\n\n## Minor Drawbacks:\n\n1.  In Fig 3, the left three clusters namely \"6,2,7\" have more females than male but DMN does not have hyper connectivity compared to SMN. Whereas the last three clusters with more male do show hyperconnectivity in SMN. This should be explained in more detail.\n2.  Since the FC is not learned rather is computed using covariance, the author(s) need to show how the connectivity of DMN and SMN looks in time points which are not attended by the model. It is quite possible that the DMN and SMN connectivity is different for males and females per the referenced literature in all time points and not just the attended ones. If this is true, then the value provided by the presented model is even less clear.\n3.  The author(s) mention at line 237 that it's average of columns but looks like average of rows. Also the index j is not specified.\n4.  The author(s) should explain why the 30th percentile was chosen when creating the adjacency matrix using FC.\n5.  The author(s) should be more clear with what they mean by dynamics in this work. Notably, 1) The regions/nodes do not change through time 2) The number of edges also remain fixed. Please be specific and explicitly explain what does change?\n6.  The ablation study in the appendix shows that attention doesn't improve AUC results by much. Also, the last row for the main atlas shows very high accuracy and AUC without any of the four components. This needs more explanation as it looks like the underlying classification task is not difficult.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Addressed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "UJwsZQXk-CK", "reviewer": "Reviewer_i7Br", "summary": "This paper proposes a method, STAGIN, to model dynamic connectivity. The architecture combines graph isomorphic network (GIN) with spatial and temporal attention. STAGIN is applied to HCP data to classify gender as well as tasks with higher accuracy than state-of-the-art methods shown.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\nWhile each component of STAGIN is based on existing works, the proposed combination shows improved performance over state-of-the-art.\n\nQuality\nThe method seems sounded, but some claims and results raise concerns.\n1. It is mentioned that “we address the issue that the node features of the input dynamic graph do not contain any temporal information and concatenate encoded timestamp with the node features”, but [11] uses windowed sub-sequences as node features of ST-GCN, which is analogous to the timestamp used in STAGIN, with the difference being not all sub-sequences are fed in simultaneously. Also, [3] uses temporal convolution network at the input, which would capture temporal dynamics. This latter method should be quantitatively compared and not brush off with one sentence.\n2. Temporal explainability is claimed to be a limitation of existing methods. For resting state fMRI, why would someone care which time points are most relevant for classifying gender or other phenotypes. For task fMRI, while it is reassuring to see attention is paid at the time of stimulus, are we gaining any new information? Please provide examples of when temporal explainabiltiy is useful.\n3. For interpretation, “k-means clustering analysis of the resting-state dynamic FC [1] and general linear model (GLM) statistical mapping of task fMRI [10] for interpreting the spatio-temporal attention learned from STAGIN” were used. The question is why can’t we apply these to latent representations or other models? This is not a particular property of STAGIN.\n4. How sensitive are results to thresholding the top 30-percentile values of the correlation matrix? How were other hyper-parameters chosen?\n5. How were subjects divided into folds? HCP is a twin study hence subjects are correlated. Were one of the twins removed for each pair?\n6. How was temporal correlation accounted for when dividing the timeseries in the case of task classification? Or was classification performed at subject level with entire session timeseries taken as one sample, in which case, how was correlation between sessions from the same subject accounted for?\n7. Would like to see STAGIN applied to harder tasks e.g. ASD classification.\n8. The current experiments and the way results are visually presented (e.g. Figures 3, 6, and 7) do not provide much neuroscience insights. Would be nice to see less superficial neuroscience findings to show the gain in interpretability by STAGIN.\n\nClarity\nMost parts of the paper are clearly written but technical details/contributions are deferred to references, partly since the proposed architecture is mainly based on existing works. A few minor points that might be worth considering below.\n1. While it might be true that this is the “first study to demonstrate the capability to learn the dynamic graph representation of both resting-state and task fMRI data with a single framework”, why is this difficult? Any general DL architecture can be applied to both rest and task fMRI data.\n2. The clear description of GIN is appreciated and the “encoder/decoder understanding of GIN” does provide some motivations to use attention instead of sum/average. However, that section seems to be an overkill to say attention is better than just sum/average.\n3. Figure 3 is hard to follow. For example, cluster 7 is supposedly attended more by females but SMN has bright red blocks, i.e. not hypoconnectivity. The qualitative evaluation needs more work. Is the ratio of female/male attentiveness significantly different from 1 statistically?\n\nSignificance\nIt is a reasonable empirical study showing that graph convolution followed by spatial and temporal attention can improve fMRI classification. The current classification task might not be difficult enough to show the true gain of STAGIN. Could potentially be of broad interest if the result interpretation part is further developed.\n\nPost Author Reponses\nI have read through the reviews and responses, and will keep my original score since that score assumed correct statistical setup for the classification evaluation, which the authors have clarified and \"promised\" to acknowledge the issue with twin data. A point worth noting is that just finding which time points drive classification does not help interpretation for resting state data. What might help is the relative number of time points with \"aberrant signals\" between subject groups for example.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Description of limitations and negative societal impact is sparse. Might be helpful to mention some of the limitations stated above especially on result interpretation and the chosen experiments.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns in this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "6tC9eo-xS1j", "reviewer": "Reviewer_qvpg", "summary": "The authors propose the Spatio-Temporal Attention Graph Isomorphism Network (STAGIN),  which can learn the dynamic graph representation of the brain connectome with spatio-temporal attention. The paper uses visualization for the interpretable display to a certain extent.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a new dynamic FC graph model and explores the interpretability of spatio-temporal attention. This is an interesting research work. But there are still some issues that need to be clarified.\n\n1、The innovation of the paper is limited. Both the attention mechanism and transformer are existing technologies. Authors do not highlight their proposed technology details and technical novelty. In addition, the paper lacks theoretical innovation.\n\n2、The authors claim that the proposed method achieves SOTA performance. The authors introduced a lot of related work in the Introduction, but only used 2 baseline methods. The authors should increase the baseline methods and report the method parameters and running time, etc.\n\n3、The effect of hyperparameters on the results is not discussed. For example, λ and edge threshold (30%).\n\n4、There are many methods to score attention. Why did the authors only choose SERO and GARO？\n\n5、The author did not perform an interpretable analysis of the constructed dynamic graph (the graph before spatio-temporal attention). This is also important for verifying the authors' research motivation or interpretability.\n\n6、More description should be added to the caption in Figure 1.\n\n7、Figure 2 does not seem to be a complete display. For example, there is no node feature $\\boldsymbol{x}_{v}(t)$.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations\n-Simple combination of existing technologies\n-The performance of the proposed model is affected by hyperparameters.\n-Interpretability", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a method for learning dynamic graph representation of brain connectome that
achieves state-of-the art performance while providing spatio-temporal explainability.
The work was appreciated by reviewers, as it put together novel tools (scaled dot attention / squeeze
excitation, self attention) with a graph neural network model to analyze dynamic graph data, specifically
4D fMRI. The application to the brain imaging data analysis was thought to be convincing. The reviewers
also consider that the paper is well-written.
However, it was found that
• not enough baselines had been used in applications, questioning the practical usefulness of the
method.
• some technical details were still somewhat arbitrary, questioning the generalizability of the method.
• comparisons against alternatives, in particular, attention-based methods, were not sufficient
Overall, the authors did a good work addressing the comments used, in particular regarding
comparisons/benchmarks against state of the art. However, identifying more difficult benchmarks would
indeed make the paper more convincing overall.
The final consensus is thus a weak accept.
Decision: Accept (Poster)