Deep Representations for Time-varying Brain Datasets
Authors: Sikun Lin
Shuyun Tang
Ambuj Singh
Venue: ICLR 2022 Submitted (2022)
Keywords: fMRI
graph neural networks
feature attribution

Abstract: Finding an appropriate representation of dynamic activities in the brain is crucial for many downstream applications. Due to its highly dynamic nature, temporally averaged fMRI (functional magnetic resonance imaging) cannot capture the whole picture of underlying brain activities, and previous works lack the ability to learn and interpret the latent dynamics in brain architectures. In this paper, we build an efficient graph neural network model that incorporates both region-mapped fMRI sequences and structural connectivities obtained from DWI (diffusion-weighted imaging) as inputs. Through novel sample-level adaptive adjacency matrix learning and multi-resolution inner cluster smoothing, we find good representations of the latent brain dynamics. We also attribute inputs with integrated gradients, which enables us to infer (1) highly involved brain connections and subnetworks for each task (2) keyframes of imaging sequences along the temporal axis, and (3) subnetworks that discriminate between individual subjects. This ability to identify critical subnetworks that characterize brain states across heterogeneous tasks and individuals is of great importance to neuroscience research. Extensive experiments and ablation studies demonstrate our proposed method's superiority and efficiency in spatial-temporal graph signal modeling with insightful interpretations of brain dynamics.

Under review as a conference paper at ICLR 2022
DEEP REPRESENTATIONS FOR TIME-VARYING BRAIN DATASETS
Anonymous authors Paper under double-blind review
ABSTRACT
Finding an appropriate representation of dynamic activities in the brain is crucial for many downstream applications. Due to its highly dynamic nature, temporally averaged fMRI (functional magnetic resonance imaging) cannot capture the whole picture of underlying brain activities, and previous works lack the ability to learn and interpret the latent dynamics in brain architectures. In this paper, we build an efficient graph neural network model that incorporates both region-mapped IMRI sequences and structural connectivities obtained from DWI (diffusion-weiig)) imaging) as inputs. Through novel sample-level adaptive adjacency matrix learn- ing and multi-resolutie inner cluster smoothing, find good representations the latent brain dynamics. We also attribute inputs with integrated gradients, infer (1) highly involved brain connections and subnetworks for each task (2) keyframes of imaging sequences along the temporal axis, and (3) subnetworks that discriminate between individual subjects. This ability to iden- tify critical subnetworks that characterize brain states across heterogeneous tasks and individuals of great importance neuroscience research. Extensive experi- ments and ablation studies demonstrate our proposed method's ' superiority and ef- ficiency in spatial-temporal graph signal modeling with insightful interpretations of brain dynamics.
which enables
INTRODUCTION
Neuroimaging techniques such fMRI (functional magnetic resonance imaging) and DWT (diffusion-weighteed imaging) provide a window into complex brain processes. Yet, modeling and understanding these signals has always been a challenge. Network neuroscience (Bassett & Sporns) 2017) views the brain as a multiscale networked system and models these signals in their graph rep- resentations: nodes represent brain ROIs (regions of interest), and edges represent either structural or functional connections between pairs of regions.
With larger imaging datasets and developments in Graph Neural Networks (Scarselli et al., 2009), recent works leverage variants of the graph deep learning, modeling brain signals with data- data-driven models and getting rid of Gaussian assumptions typically existed in linear models (Zhang et al.) 2019 Li et al. 2019). These methods are making progress on identifying physiological characteris- ties and brain disorders: In Kim & Ye (2020), authors combine grad-CAM (Selvaraju et 2017) and GIN (Xu et al.) 2018) to highlight brain regions that are responsible for gender classification with resting-state IMRI data. Li al. (2020) utilizes the regularized pooling with GNN iden- tify fMRI biomarkers. Noman et al. (2021) embeds both topological structures and node signals of fMRI networks into dimensional latent representations for a better identification of depression. However, the first two works use time-averaged fMRI, losing rich dynamics in the temporal domain. The third combines nodes' temporal and feature dimensions instead of handling them separately, leading to suboptimal representation (as discussed in section 3.2). To overcome these issues, propose ReBraiD (Deep Representations for Time-varying Brain Datasets), a graph neural network model that jointly models dynamic functional signals and structural connectivities, leading more comprehensive deep representation of brain dynamics.
To simultaneously encode signals along spatial and temporal dimensions, some notable works in traffic prediction and activity recognition domains such as Graph WaveNet (Wu et al., 2019b) alter- nate TCN (temporal convolution network) (Lea et al. 2016) and GCN (graph convolutional network)

---

Under review as a conference paper at ICLR 2022
(MRI signals
explicit
Batech stzz
predicted class
distribution
batch sire
2.2 ())
learnable Out
aap olt
Az
MILP
pooling layers on node axis
2.2 (III)
TCNeea
Fanh
GNNNara
inner-cluster smooting
lincar linear
2.2 (1I)
1xc
TCNyue
GNNippo
Conv
2.2 (IV)
residual
linear
Figure 1: The proposed ReBraiD model for integrating brain structure and dynamics (the architec- ture shown is for classification). For each batch with batch size B, input X has dimension of N,, r and A, Audp both have the dimension NB, N).. The encoder (green part) encodes temporal and spatial information alternatively, producing latent representation in (B, duem, N, 1)- These embeddings are followed by linear layers for pooling and classification. The final output has dimension of (B, C).
(Kipf & Welling, 2017). Others (Song et al. 2020 Liu et al. 2020) use localized spatial- tempoaal
graph to embed both domains' information this extended graph. There are also works incorporat- ing gated recurrent networks for the temporal domain such as (Seo et al. 201 Ruiz et al.) (020). We choose the first option for ReBraiD, as is more memory and time efficient, and can support much longer inputs. We also explore the best option when alternating spatial and temporal layers for encoding brain activities with extensive ablation studies. Upon this structure, we propose novel sample-level adaptive adjacency matrix learning and multi-resolution inner cluster smoothing, both of which learn and refine latent dynamic structures. We also make the model more efficient while being effective with the choice of the temporal layer.
Equally important as finding a good representation of brain dynamics is interpreting them. We utilize integrated gradients (Sundararajan various processes. This can lead to better behavioral understanding, biomarker discoveries, and characterization of individuals or groups with their brain imagings. We also make the novel contri- bution of identifying temporally important frames with graph attribution techniques; this can enable more fine- fine-graing reee--rrindd temporal analysis around keyframes when combined with other imaging modali- ties such as EEG (elecrrenccehallogran) In addition, our subject-level and group-level attribution studies unveil heterogeneities among ROIs, tasks, and individuals.
al.) (017)) to identify how brain ROIs participate in
2
METHOD
2.1 PRELIMINARIES
We utilize two brain imaging modalities mapped onto same coordinate: SC (structural connec- tivity) from DWI scans, and vary-ng fMRI scans. We represent them set of L graphs G. (A,, X.) with E L]. which A, E NxN represents normalized adjacency matrix with an added self-loop: A, D&c sc, SC, SC, + Ix and Dsc, Σ.. (SC,)vw is the diagonal node degree matrix. Graph signal matrix obtained from fMRI scans the 7th sample represented as X; NxT Here N is the number of nodes, and each node represents brain region; T is the input signal length on each node. Our objective focuses on classifying brain signals G; into one of task classes through learning latent graph structures.
2.2 MODEL.
ReBraiD takes (A, as inputs, and outputs task class predictions. The overall model structure is shown in fig. For the ęth sample X; E R NxIXT the initial 1 x convolution layer increases
Axis order follows PyTorch conventions. Dimension the second index is the expanded feature dimension.
2

---

Under review as a conference paper at ICLR 2022
diaaion
distion
causal paddings
diadon 1
sorided TON
dialoted TON
Figure 2: Comparison of strided non-causal TCN (left) and dilated causal TCN (right), The causal aspect is achieved through padding (kernelsize 1) x dilation zeros to the layer's input. The resulting y always has the same length as input x, which yr only depends on inputs Xigr- We can view strided non-causal TCN the rightmost node of dilated causal TCN.
its hidden feature dimension to dı1, outputting (N, The encoder then encodes temporal and spatial information alternately, reducing information to feature axis and generating hidden representation of size (N, dxa, 1). The encoder is followed by two linear layers perform node embedding pooling and two MLP layers for classification. Cross entropy is used as the loss function:
L Ee zi log S.. where y E RC is the one-hot vector of ground truth task label and ÿ. E RC model's s predicted distribution. We now explain different components of the model.
(I) Learning sample-level latent graph structures. Structural scans serve as our graph adjacency matrices. However, they are static not only across temporal frames but also across different tasks. In contrast, functional connectivities (FC) are highly dynamic, as shown in appendix [A.1.]] To better capture dynamic graph structures, signal. Unlike other works such as Wu al. (2019b) that use a universal latent graph structure, our model do not assume all samples share the same latent graph. Instead, in ReBraiD, each sample has a unique latent structure reflecting its signal status. This implies that the latent adjacency matrix cannot be treated
assign adaptive adjacency matrix to each input graph
learnable parameter as a part of the model. To solve this, minimize the assumption down shared projection addp that projects each input sequence into embedding space and use this embedding to generate the latent graph structure. Ondp can be learned an end- to-end manner. The generated adaptive adjacency matrix for the 2thh sample can be written as, where Softmax is applied column-wise:
A Softmax (ReLU ((X,00)) (Xx:wx)))
Ouup
(II) Gated TCN (Temporal Convolutional Network). To encode temporal information, we use the gating mechanism as in Oord et al. (0166) in our temporal layers: (1+1) tanh (TCN (H ())
(TCN (11 where (0)) E R. Nxxt is one sample's activation matrix of the 1 layer, @ de- notes the Hadamard product, and is the Sigmoid function. Different from TCNs generally used in sequence to sequence models that consist of dilated Convld and causal padding along the temporal dimension (van den Oord et al. (016))), we simply apply Convld 1d with kemne 2 and stride 2 as TCNemb and TCN gate to embed temporal information. The reason is twofold: first, for a sequence to sequence model with length-T output, y, should only depend on Zigr to avoid information leakage, and causal convolution can ensure this. In contrast, our model's task is classification, and the goal of our encoder along the temporal dimension axis while reducing temporal dimension to 1. The receptive field of this single temporal point (with multiple feature channels) meant be the entire input sequence. Essentially, our TCN is the same as the last output node of a kernel 2 causal TCN whose dilation increases by 2 at each layer (fig. 2 Second, from practical perspective, directly using strided non-causal TCN works the same as us- ing dilated causal TCNs and taking the last node, while simplifies the model structure and reduces training time to less than a quarter.
embed signal information into the feature
(M)) Graph Network layer. In our model, every / temporal layers (appendix A.2.3 studies the best 7 choose) are followed by a spatial layer to encode signals with the graph structure. Building tem- poral and spatial layers alternatively helps spatial modules to learn embeddings at different temporal scales, and this generates better results than putting spatial layers after all temporal ones.
To encode spatial information, Kipf& & Welling (2017) uses first-order approximation of spectral fil- ters to form the luser-wise propagation rule of a GCN layer: ((++)) = (C)(H = f(AH (w(0) w.O). It can be understood as spatially aggregating information among neighboring nodes to form new node embeddings. In the original setting without temporal signals, H e R Nxd is the activation matrix of tth layer, A N'N denotes the normalized adjacency matrix with self-connections
3

---

Under review as a conference paper at ICLR 2022
discussed in section 2.1 with Raxd' is learnable model parameters, and f is nonlinear activation function of choice. Parameters d and d' are the number of feature channels.
We view a GCN layer as a local smoothing operation followed by an MLP, and simplify stacking K layers A as Wu et al. (2019a). ReBraiD, every graph network layer aggregates infor- mation from each node's s K K-hop neighborhoods based on both brain structural connectivity and the latent adaptive adjacency matrix: namely we have both A, A W and A 17( aadd for input H We also gather different levels (from to K) of neighbor information with concatenation. In other words, one graph convolution layer here corresponds to small module that is equivalent to simple GCN layers with residual connections. We can write our layer as;
H (1+1) GNN@ (H²) MLP [Concat (H R..LUAAA" * ( ReLU )
With the additional temporal dimension, H E RNxxxx in eq. 2 and A; E RNNN applies on ' 's first two dimensions while multiplying. Outputs of different GNN(") layers are parametrized and then skip connected with summation. Since the temporal lengths of these outputs are different because of TCNs, max-pooling is used before each summation to make the lengths identical.
(IV) Multi-resolution inner cluster smoothing. While GNN layers can effectively passing in- formation between neighboring nodes, long-range relationships among brain regions that neither appear in SC nor learned by latent Aupp can be better captured using soft assignments similar to DIFFPOOL (Ying et al. (2018). To generate the soft assignment tensor assigning N nodes into c
clusters (e chosen manually), we use GNN(() poot that obeys the same propagation rule as in eq. ?
followed by Softmax along This assignment is applied to the output of GNN!!) out the spatial embedding for the 7th layer input H(:;
emb which carries
50h) Softmax
(GNN())
pool (IT()).1)
R Nxexs
-0 GNN(O
(T(1)
Nxdxe
(3)
T SUU z Rexxxx
The extra temporal dimension allows nodes be assigned to heterogeneous clusters at different frames. We find that using coarsened A(++1) S g()) Rexe as the graph adjacency matrix leads to worse performance compared to using SC-generated A; and learned Ai (comparison in section (.I). In addition, the number of nodes is changed, residual connections coming from the beginning of temporal-spatial blocks can not be used and this impacts overall performance. To continue use A; and A; sdp as graph adjacency matrices and allow residual connections, reverse- assign (3) with assignment tensor obtained from applying Softmax on S(I) along N, so that the number nodes is kept unchanged:
5th Softmax (SUDT, ReXNNI
(4)
H (++ = 50) A E R Nxdxa
perform signal smoothing on nodes within each soft-assigned cluster (ap- pendix A.1 shows a toy example). With the bottleneck N, the model is forced to pick up latent community structures. This inner-cluster smoothing is carried out at different spatial resolutions: as the spatial receptive field increases with more graph layers, we decrease cluster number e for the assignment operation.
In fact, eqs. 33 and
2.3 ATTRIBUTION WITH IG (INTEGRATED GRADIENTS).
As one approach to model interpretability, autribation assigns credits to each part of the input, as- sessing how important they are to the final predictions. Wiltschko et al. (2020) gives an extensive comparison between different graph attribution approaches, which IG (Sundararajan et al. (2017)) top-performing and can be applied to trained models without extra alterations of the model struc- ture. IG also has other desirable properties such as implementation invariance that other gradient methods are lacking. It is also more rigorous and accurate than obtaining explanations from atten- tion weights or pooling matrices that span multiple feature channels. Intuitively, IG calculates how real inputs contribute differently compared to selected baseline; it does so by aggregating model gradients linearly interpolated inputs between the real and baseline inputs. For each sample,
4

---

Under review as a conference paper at ICLR 2022
80-
70
Weighted
60
8
16
64
128
256
2-d Namber CNN layee TCN lgen
Input Length (frames)
(a)
Figure 3: 3aj Ablation studies on different input length (please see table 3 in appendix for numerical values of weighted Fl under each setting): 3b; Choosing number of GNN to TCN layer ratio.
calculate attributions at each point of both input A E RNNN and X E RNT,
M OF (Auapi X)
ATTRA (Amm Now)
Anmup
A'
m M
(A A')
M
m
ATTRX,, = (X X²O
OF (A, Xmmpi)
Xtmup
X' m
(x - xx)
O.
M
F(A,X) here represents our signal classification model, M is the step number when doing Riemann approximation of the path integral, and A', X' are baselines of A, X (see section 3.3 for more details). Note that eq. (5) calculates the attribution of one point one sample. The process is repeated for every point of the input, so attributions have identical dimensions as inputs. To obtain brain region importance of task, we aggregate attributions across multiple samples of that task.
EXPERIMENTS
We use fMRI signals from CRASH dataset (Laubaratanahirun al. (020 for experiments. The model classifies input fMRI into 6 tasks: resting state, visual working memory task (VWM), dynamic attention task (DYN), math task (MOD), dot probe task (DOT), and psychomotor vigilance task (PVT) (appendix A.2.1 has detailed task descriptions). We preprocess 4D voxel-level fMRI images into graph signals G (A,X) by averaging voxel activities into regional signals with the 200-ROI cortical parcellation (voxel to region mapping) specified by Schhafeer et al. (2018). We also standardize signals for each region and discard scan sessions with obvious abnormal spikes that may be caused by head movement, etc.. DWI scans are mapped into the same MNI152 coordinate and processed into adjacency matrices with the same parcellation fMRI uses. Valid data contains 1940 scan sessions from 56 subjects, session length varies from 265 frames to 828 frames and TR (Repetition Time) is 0.91s. These 1940 scan sessions are separated into training, validation, and test sets with ratio of 0.7-0. weight decay are chosen with grid search based on validation loss, and all results reported this section obtained from the test set. For each scan session,
Hyperparameters including dropout rate, learning rate, and
sliding window to generate input sequences (in the following experiments E (8, 16, 32, 64, 128, 256}) and feed them to the model. To encode temporal and spatial information alternatively, we find stacking two TCN layers per one GNN layer leads to better performance most times (fig. 3bb see more on appendix [A.2.3] (I)). Models are written in PyTorch, trained with Google Colab GPU runtimes, and 60 epochs run for each experiment setting. Codes and data will be released upon acceptance.
3.1 MODEL COMPONENTS
Graph adjacency matrices. For each input sample Ge we test different options to provide graph adjacency matrices to the GNN layer. They include (i) proposed method: using both adaptive adjacency matrix A; adp and SC-induced A;; (ii) only using A;: (iii) only using A; adp: (iv) replacing Ag in setting i with A derived from functional connectivity; (v) only using random graph
5

---

Under review as a conference paper at ICLR 2022
adjacency matrices with the same level of sparsity real As. We set hadp to be 5 in eq. which works better for our data than larger hadp choices. K is set to 2 for eq. ② meaning each GNN ag- gregates information from 2-hop neighbors based on the provided adjacency matrices. We evaluate our model with weighted Fl as the metric because of the imbalance among tasks. The results under different settings are reported in fig. 3a (and table 3 in appendix for numerical values).
From the results of setting (ii) plotted in fig. 3a we see that removing the adaptive adjacency matrix impacts the performance differently at different input lengths: the gap peaks for signals of length 64 128, and becomes smaller for either shorter or longer sequences. This could suggest the existence of more distinct latent states of brain signals of this length that cannot be captured by structural connectivities. On the other hand, removing SC (setting (iii) seems to have a more constant impact on the model performance, with shorter inputs more likely to see a slightly larger drop. In general, only using Aspp leads to smaller performance drop than only using SC, indicating the effectiveness of Aadp in capturing useful latent graph structures. More detailed studies in appendix A.2.3 shows Aadp learns distinct representations not captured by A.
As mentioned in section 2 our motivation behind creating sample-level adaptive adjacency matrices is FC's highly dynamic nature. Therefore, for setting (iv), we test directly using adjacency matri-
ces Fe obtained from FC instead of the learned adp- In particular, A = D $ DC, PC, E R 20000000 where (FC)) corr((X.).., (X.)..). FC, = FC, + IN and Drc, = (FC.) Fig.
3a shows Ar constantly underperforms Ai except for being really close with length-8 inputs. Larger performance gaps are observed for longer inputs, where Corr((X,)o- (X,) struggles to capture the changing dynamics in the inputs. This demonstrates that our input-based latent Ai has better representation power than input-based FC. We also notice batch correlation coefficients calculation for A results in a slower training speed than computing Asadp.
An interesting result comes from setting (v), where with the edge creation probability the same as averaged edge existence probability of As. Its perfor- mance is at similar level or even better than settings (ii) and (iii). Our hypothesis is the model can learn the latent graph structure out of randomness, and we will verify this hypothesis in section 333
randomly generated Frdōs-Rééní graphs
Huulti-reoouutin inner cluster smoothing. To verify the capability of inner-cluster smoothing operation in capturing latent graph dynamics, we test the following settings: (vi) using our proposed model and inputs, except removing paralleled GNN poot and inner-cluster smoothing module; (vii) previous setting (v) but remove GNN poot and inner-cluster smoothing module; (viii) keep GNN but using coarsened graph instead of performing smoothing and increasing the node number back (essentially performing DUFFPOLL with an added temporal dimension). In this last setting, we hierarchically pool graph nodes until node number reaches 1, and we keep the total number of GNN layers the same as our other settings. Values of soft-assigned cluster number e are chosen to be halved per smoothing module, starting from half of the graph nodes number, namely: 100, 50, 25, etc. for our experiments. Different choices of affect model converging rate, but only have minor impacts on the final performance (see appendix A...3 (III). Results are reported in fig. Ba (and table in in appendix). Apart from these three settings, we also test adding pooling regularization terms (described in appendix A.1.33) into the loss function but this does not lead to much of a difference.
The above results demonstrate that both setting (vi) and (vii) outperforms (vii) by large margin, indicating the importance of keeping the original node number when representing brain signals. In addition, all three settings underperform our proposed method, and they mostly worse than changing graph adjacency matrices as in settings (ii)-(v:: this shows inner-cluster smoothing module has a large impact in learning latent graph dynamics. We also find using adaptive adjacency matrices and inner cluster smoothing can stabilize training, making the model less prone to over-fitting and achieving close-to-best performance over a larger range of hyperparameters (see fig. ITp.
3.2 MODEL COMPARISONS
In this section, we compare our model with the vanilla GCN from Kipf & Welling 2017). Chebyshev Graph Convolutional Gated Recurrent Unit (GConvGRU) from Sco et al. (2018). GraphSAGE from Hamilton et al.] (017). GAT V2 from Brody et al. (20T)) and Graph Transformer as in (Shi et al. (2021). To use them fMRI classificaiton problem, we directly take corresponding layers from

---

Under review as a conference paper at ICLR 2022
Table 1: Model comparisons under the 256 input length setting.
Accuracy (%)
Training time
Model
Weighted F1
(s epoch)
GCN (Kipf & Welling (2017))
41.53
42.84
713
GAT V2 (Brody et al. 2021
50.44
50.36
1142
GConv (Seo et al. (2018)
52.26
56.05
9886
GraphSAGE (Hamilton et al. (217))
61.84
61.87
1048
Graph Transformer (Shi et al.
(2021)
66.51
66.11
1890
ReBraiD (proposed method: TCN GNN)
85,56
90.85
298
ReBraiD (TCN only)
72.44
71.98
119
ReBraiD (TCN CNN)
75.89
75.79
124
PyTorch Geometric and PyTorch Geometric Temporal aand construct the models similar ours: four graph layers taking in both signals and adjacency matrices, followed by two linear layers along node axis and two linear layers for the final classification. We train baseline models with the same setting as our model: 256-frame inputs, Adam optimizer, cross entropy loss, and 60 epochs (all models well-converged). Grid search used optimize the rest of hyperparameters. We compare accuracy, weighted F1 score, and training time per epoch in table [ we also plot our model and Graph Transformer's confusion matrices appendix fig. 12
We observe that our proposed method significantly outperforms the baseline graph models by a mar- gin of 20 to percent and has much less training time. This demonstrates our proposed model's effectiveness in capturing latent brain dynamics. For these baseline models, temporal content is used as features; the comparison shows separating them into different axis is more advantageous. This is further confirmed with models only having TCN layers: test both removing GNN layers all to- gether and replacing them with x CNN layers. Both outperform graph models that focuses on the spatial modeling aspect. Although temporal modeling is crucial, including the spatial information in its graph format as our proposed model can improve the performance much further.
3.3 INTERPRETATIONS WITH IG
In this section, we study the contributions of different brain ROIs and subnetworks defined by their functionalities. For the subnetwork definition, we choose to use 17 networks specified in Thomas Yeo et al. (20IT) which has mapping from previous 200-ROI parcellation. See table appendix for all subnetwork names. We compute IG of a model trained on length-256 in- put signals because the model has higher performance with longer inputs, leading to accurate attributions. To select baseline inputs, we follow the general baseline selection principle for attribu- tion methods: when the model takes baseline input, it should produce near-zero prediction and Softmax(outpun
should give each class about the same probability in a classification model. All- zero A' and X' can roughly achieve this for our model, so we choose them as our baseline inputs. For each task, the IG computation is done on 900 inputs to get an overall distribution.
Temporal importance. On the single input level, we can attribute which parts of the inputs in G; are more important in predicting the target class by looking into (ATTRx)) x)TRR)). This attribution map not only shows which brain regions contribute more but also reveals the important signal frames. One critical drawback of fMRI imaging is its low temporal resolution, but if we can know which part more important, we can turn to more temporally fine-grained signals such as EEG to see there are any special activities during that time. To confirm that the attributions we get are valid and consistent, we perform a samity check of IG results on two overlapped inputs with an offset the first input is obtained from window [to, tu T] and the second is obtained from window [to + la + + 7). Offset aligned results are shown in fig. 4a and we can see the attributions agree with each other quite well.
Spatial importance. We examine the connection importance between brain ROIs by looking at A a (lask-averaged ATTR are plotted in fig. [4 in appendix). particular, columns in ATTRA with higher average values are sender ROIS of high-contributing connections, which is what matters in the GNN operation. We also explore why using random graph adjacency matrices (setting (v) in section 3.1 can produce a similar result for length-256 inputs compared to using both SC-induced

---

Under review as a conference paper at ICLR 2022
CC m0
+
(a)
(b)
Figure 4: 4a Temporal importance sanity check of IG results two pieces of inputs with large overlap period. Attribution maps are offset aligned: 4b] ATTRx distributions across brain regions for VWM task, where the upper one separates left and right hemispheres and the lower one combines them (e.g. LH. VisCent and RH VisCent are combined to region VisCent). Refer to table in appendix brain region names that the x-axis numbers represent.
www M
Figure 5: Column averages of task-averaged ATTR (mapped into 34 subnetworks defined by the 17-network parcellation with left, right hemispheres). Top row is obtained from real SC induced A and bottom rows is obtained from random SC induced Arand Attributions are normalized to [0, 1].
A; and ad (setting (i)). By examining ATTRA under both settings (fig. 5) we see that the column averages of A under these two settings are similar for almost all tasks, meaning the model can learn where to pay attention to even using random adjacency matrix inputs. We credit this ability partially to multi-resolution inner cluster smoothing, as the performance would drop notably without (setting (vii)). However, using ground truth SC not only gives us higher performance for shorter inputs but also provides the opportunity to better interpret brain region connections. We can directly use task-averaged ATTR as the weighted adjacency matrix to plot edges between brain ROIs, just as in fig. 6 Important brain regions obtained from ATTRA mostly comply with the previous literature (see appendix A.2.5 for details).
ATTRx can also give us insights on spatial importance when the attribution maps are averaged or summed up along the temporal dimension. But does so from another perspective: instead of showing important structural connections that support information passing, it reveals regions or subnetworks that are sources of the important signals. In fig. 4b we plot the distribution of t- averaged and subnetwork-averrgee (mapping 200 ROIs into 17 subnetworks) ATTRx during VWM task. We can see the clear dominance of VisCent, DorsAttnA. and ContA subnetworks (numbered as 5, 11), indicating signals from these regions are useful for model to decide if the input is from VWM task. For the boxplots of other tasks and subnetwork rankings, please see fig. [6 table 5 and table in appendix. More informative than the rankings is the distribution itself: even though VisCent. Dors/AttnA, and ContA ranked top 3 for both resting state and VWM task for signal attributions. their relative importance and attribution distribution variances are totally different. In sense, the distribution can act task fingerprint based on brain signal states.
We notice that signal-important ROIs are not necessarily the same as connection-importan ROIs: top-ranked subnetworks for resting state are DefaultA and DefaultB by ATTRA, and VisCent and DorsAttnA by ATTRx; although they do coincide with each other for tasks like VMN. This disparity is reflected in fig. 6 edge and node differences.
Group, session, and region heterogeneity. Average variances of attributions are very different across tasks, especially those of ATTRx: VWM and DYN have much smaller attribution variances compared to other tasks. This can be caused by either task dynamics when certain tasks have more phase transitions and brain status changes, or/and group heterogeneity when individuals carry out
8

---

Under review as a conference paper at ICLR 2022
Figure 6: ROI attributions from ATTR A and ATTRx. x. Tasks are: Resting, VWM, DYN, DOT, MOD, PVT from left to right. Edge color and width are based on task-averaged ATTR, E 2000 and nodes color and size are based on task and remporal-averagee purpose, only edges with highest attributions are kept to ensure sparsity being 0.009 (down from around 0. 0.6)... For ROI attributions based only A where important sender ROIs are reflected by node sizes, please refer to fig. 13 in appendix.
ATTRx E 22000 For the visualization
SUBJACT SuRjerT? 2
subject 2upjer? SuBject
SUBJECT 3
0962
.
0.00
4000
Figure 7: ATTRx x distributions of subjects doing VWM task (left) and MOD task (right). Outliers that go beyond [Q1 1.5 IQR, Q3 1.5 IQR] are omitted. VWM has much smaller average attribution variance than MOD.
certain tasks more differently than the others. We investigate the cause by looking into 3 subjects; each them has multiple scan sessions for every task.
We report the following findings: (1) Even only aggregating attributions over single subject's s sessions, attribution variances the other four tasks are still larger than VWM and DYN. The variance values comparable to that of aggregating
many subjects. This means the large variances are not mainly due to group heterogeneity, rather some tasks having more states than others. (2) Apart from different task dynamics, there is still group heterogeneity. For tasks with more dynamics (high attribution variances), the group heterogeneity is also more obvious. We can from fig. that attributions for VMM are much more concentrated and universal across subjects than that of MOD. (3) Flexibility of different subnetworks varies: subnetworks that have small distribution IQR (Interquartile Range) of the same subject's different sessions will also be more consistent across subjects. One example is subnetwork 18 during MOD task has both higher within- subject IQR and larger surosst differences compared to subnetwork 19. This indicates for certain task, some subnetworks are more individual and flexible (may activate differently across (), while others are more universal and fixed. In summary, we can reveal both critical regions that particular task must rely on, and regions that can characterize individual differences during tasks.
4 CONCLUSIONS
In this paper, we propose ReBraiD, high-performing and efficient graph neural network model that embeds both structural and dynamic functional brain signals for task classifications. To better capture latent dynamics, we propose input-dependent adjacency matrix learning and inner-cluster smoothing at multiple resolutions. Apart from quantitative results showing ReBraiD's superiority in representing brain actívities, we also leverage integrated gradients to attribute and interpret the im- portance of both spatial brain regions and temporal keyframes, as well as presenting heterogeneities among subnetworks, tasks, and individuals. These findings potentially reveal new neural basis or biomarkers of tasks and brain disorders when combined with behavioral metrics, and enable more fine-grained temporal analysis around keyframes when combined with other imaging techniques.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Danielle S Bassett and Olaf Sporns. Network neuroscience. Nature neuroscience, 20(3):353-364,
2017.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks?, 2021.
Joshua M Carlson, Felix Beacher, Karen S Reinke, Reza Habib, Eddie Harmon-Jones, Lilianne R Mujica-Parodi, and Greg Hajcak. Nonconscious attention bias to threat is correlated with an- terior cingulate cortex gray matter volume: voxel-based morphometry result and replication. Neuroinage, 59(2):1713 1718, 2012.
Joshua M Carlson, Jiook Cha, and Lilianne Mujica-Parodia Functional and structural amygdala- anterior cingulate connectivity correlates with attentional bias to masked fearful faces. Cortex, 49 (9):2595-2600, 2013.
Sean PA Drummond, Amanda Bischoff-Grethe, David F Dinges, Liat Ayalon, Sara C Mednick, and MJ Meloy. The neural basis of the psychomotor vigilance task. Sleep, 28(9):1059-106) 2005.
Roland M Friedrich and Angela Friederici. Mathematical logic in the human brain: semantics.
PLoS One, 8(1):e53699, 2013.
Roland H Grabner, Gernot Reishofer, Karl Koschutnig, and Franz Ebner. Brain correlates of math- ematical competence in processing mathematical representations. Frontiers human neuro- science, 5:130, 2011.
William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference Neural Information Processing Systems, pp. (025-1035, 2017.
Sayan Kahali, Marcus E Raichle, and Dmitriy Yablonskiy. The role of the human brain neuron- glia-synaptic composition in forming resting state functional connectivity networks. bioRxiv, 2021.
Byung-Hoon Kim and Jong Chul Ye. Understanding graph isomorphism network for rs-fmri func- tional connectivity analysis. Frontiers in Neuroscience, 14:630, 2020. ISSN 1662-453X. doi: 23338//tnnns2222000666
Jangjin Kim, Edward A Wasserman, Leyre Castro, and John H Freeman. Anterior cingulate cor- tex inactivation impairs rodent visual selective attention and prospective memory. Behavioral neuroscience, 1 130(1):75, 2016.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations (ICLR). 2017.
Nina Lauharatanahirun, Kanika Bansal, Steven M Thurman, Jean M Vettel, Barry Giesbrecht, Scott Grafton, James C Elliott, Erin Flynn-Eyans, Emily Falk, and Javier O Garcia. Flexibility of brain regions during working memory curtails cognitive consequences lack of sleep. arXiv preprint 2rXiv.2009.07233. 2020.
Colin Lea, René Vidal, Austin Reiter, and Gregory Hager. Temporal convolutional networks: A unified approach action segmentation. In Gang Hua and Hervé Jégou (eds.), Computer Vision ECCV 2016 Workshops, pp. 47-54, Cham, 2016. Springer International Publishing. ISBN 978- 3-319.49900-8.
Robert Leech and David J Sharp. The role of the posterior cingulate cortex in cognition and disease. Brain, 137(1):12-33) 2014.
Lingge Li. Dustin Pluta, Babak Shahbaba, Norbert Fortin, Hernando Ombao, and Pierre Baldi. Modeling dynamic functional connectivity with latent factor gaussian processes. Advances in neural information processing systems, 32:8263-822 2019.
10

---

Under review as a conference paper at ICLR 2022
Xiaoxiao Li, Yuan Zhou, Nicha C. Dvorek,, Muhan Zhang, Juntang Zhuang, Pamela Ventola, and James Duncan. Pooling regularized graph neural network for fmri biomarker analysis. Medical image computing and computer-assisted assisted intervention MICCAI... ... International Conference Medical Image Computing and Computer-Assisted Intervention, 12267:625 635, 2020.
Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang. Disentangling and unifying graph convolutions for skeleton-basel action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 143-152, 2020.
Sylvia Loh, Nicole Lamond, Jill Dorrian, Gregory Roach, and Drew Dawson. The validity of psy- chomotor vigilance tasks of less than 10-minute duration. Behavior Research Methods, Instru- ments, & Computers, 36(2):339 2004.
Steven J Luck and Edward K Vogel. The capacity of visual working memory for features and conjunctions. Nature, 390(6657):279-255 2811, 1997.
Andrew Mattarella-Micke, Jill Mateo, Megan N Kozak, Katherine Foster, and Sian Beilock. Choke thrive? the relation between salivary cortisol and math performance depends on in- dividual differences in working memory and math-anxiety. Emotion, 11(4):1000, 2011.
Fuad Noman, Chee-Ming Ting, Hakmook Kang, Raphael C. W. Phan, Brian D. Boyd, Warren D. Taylor, and Hernando Ombao. Graph autoencoders for embedding learning in brain networks and major depressive disorder identification, 2021.
Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelcnn decoders. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 4797-4805, 2016.
Marcus E Raichle. The brain's default mode network. Annual review of neuroscience, 38:433-447, 2015.
Luana Ruiz, Fernando Gama, and Alejandro Ribeiro. Gated graph recurrent neural networks. IEEE Transactions on Signal Processing, 68:6303-615 (3188 2020.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80, 2009. doi:
Alexander Schaefer, Ru Kong, Evan M Gordon, Timothy O Laumann, Xi-Nian Zuo, Avram J Holmes, Simon B Eickhoff, and BT Thomas Yeo. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri. Cerebral cortex, 28(9):3095-3144 2018.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Youngjoo Seo, Michaël Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured sequence modeling with graph convolutional recurrent networks. In International Conference on Neural Information Processing, pp. 362-373. Springer, 2018.
Yunsheng Shi, Zhengjie Huang, Shikun Feng. Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for sun-suueeeeeee classification, 2021.
Maurice Sipos, Yair Bar-Haim, Rany Abend, Amy B Adler, and Paul Bliese. Postdeploy- ment threat-related attention bias interacts with combat exposure to account for ptsd and anxicty symptoms in soldiers. Depression and Anxiety, 31 2014.
Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-tempora network data forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 914-921, 2020.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference Machine Learning, pp. 3319-3328, PMLR, 2017.

---

Under review as a conference paper at ICLR 2022
BT Thomas Yeo, Fenna M Krienen, Jorge Sepulcre, Mert R Sabuneu, Danial Lashkari, Marisa Hollinshead, Joshua Roffman, Jordan W Smoller, Lilla Zöllei, Jonathan R Polimeni, et al. The organization of the human cerebral cortex estimated by intrinsic functional connectivity. Journal of neurophysiology. 106(3): 1 125-1 163, 2011.
J Jay Todd and René Marois. Capacity limit of visual short-term memory in human posterior parietal cortex. Nature, 428669844 2004.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A Generative Model for Raw Audio. Proc. ISCA Workshop on Speech Synthexis Workshop (SSW 9), pp. 125, 2016.
Alexander Wiltschko, Benjamin Sanchez-Lengeling. Brian Lee, Emily Reif, Jennifer Wei, Kevin James McCloskey, Lucy Colwell, Wesley Qian, and Yiliu Wang. Evaluating attribution for graph neural networks. Google Research, 2020.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International conference on machine learning. pp- 6861-6871. PMLR, 2019a.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. International Joint Conferences on Artificial Intelligence (IJCAI), 2019b.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful graph neural networks? In International Conference on Learning Representations, 2018.
Steven Yantis, Jens Schwarzbach, John T Serences, Robert L Carlson, Michael A Steinmetz, James J Pekar, and Susan M Courtney. Transient neural activity in human parietal cortex during spatial attention shifts. Nature neuroscience, 5(10):995-1022 2002.
Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. Proceedings of the 32nd International Conference Neural Information Processing Systems, pp. 4805-4815 15, 2018.
Gemeng Zhang, Biao Cai, Aiying Zhang, Julia M Stephen, Tony W Wilson, Vince D Calhoun, and Yu-Ping Wang. Estimating dynamic functional brain connectivity with sparse hidden markov model. IEEE transactions on medical imaging, 39(2):488 498, 2019.
12

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1 MODELS
A.1.1 DYNAMIC FUNCTIONAL CONNECTIVITIES.
Fig. 8 shows functional connectivities (FCs) among N brain regions, where each FC E NNN, The value FC,; is calculated as the Pearson correlation coefficient between signals of brain region and region j. The figure shows 6 FCs calculated from 6 consecutive sliding windows within same [MRI session, with signal window length being 30 and sliding stride being 30. From the figure, we clearly tell that FCs are highly dynamic.
Figure 8: Dynamic functional connectivities.
A.1.2 SINER-CLUSTER SMOOTHING TOY EXAMPLE.
Here we show a toy example demonstrating the inner-cluster smoothing module described in eqs. (3) and (4). Note that will only show one time slice, and the same operation is done along every t: particular 4, we have Z E R xd S E Nx. We will use N = 3, 2 and node values a, b, e E Rdd for this toy example. In addition, this example is just to illustrate the concept behind the smoothing operation, and Softmax along the axis is simplified as normalization for a clearer presentation.
cluster assignments
0
a 1
Z = S
1
2
1
=
H
ST
+
1/2
5 = row-normalized
(5T)
0
2
1/2
2 a
+
H Haew = 5TH
3
cluster
nodes
Figure 9: Inner-cluster smoothing toy example.
In this example, 1 and 2nd nodes are assigned the first cluster, and 2nd and 3rd node are assigned to the second cluster. The final H Hnew after our smoothing module will mingle the first two nodes' values, and the last two nodes" values (based on assignment weights) while keeping their original node number unchanged.
A. 1.3 REGULARIZATION TERMS FOR THE SOFT-ASSIGNMENT.
For each soft assignment matrix S R Nxxt in eq.
3 test three regularization terms:
Similar to DIFTPOLL, for ensuring more clearly defined node assignment, namely each node only assigned to few clusters (the closer to one the better), we can minimize the entropy of single node assignments: Lg, Ei= (S,), where S; is along
To ensure representation separation among nodes, meaning the assignment shouldn' 't sign all the nodes same way, we maximize the entropy of node assignment patterns across all nodes: TE,, -! Σ H T Siÿ), where j along and along
13

---

Under review as a conference paper at ICLR 2022
To make assignment along temporal axis smoother, we penalize assignment variances
within small window [ł, r): Lr - Ei (( where represents standard deviation.
Together with cross entropy classification loss LeF, the final loss function of the model will be:
Lrog ai LoE + a2LF, + LgLF, + aALT: Σa, =1
(6)
A.2 EXPERIMENTS
A.2.1 TASK DESCRIPTIONS.
The following are task descriptions of CRASH (Cognitive Resilience and Sleep History) dataset:
Resting state: The subject simply lays in the scanner awake, with eyes open for 5 minutes.
Visual working memory task (VWM)) The subject is presented with a pattern of colored squares on computer screen for very brief period (100ms). After 1000ms, they are presented with single square and must determine it is the same or different color as the previously presented square at that location. Responses are made with button press (Luck & Vogel (1997))
Dynamic Attention Task (DYN): Two streams of orientation gratings are presented to the left and right of fixation. Subjects monitor specified stream for target (about degree shift in orientation, clockwise or counter clockwise) that indicates whether the subject should continue to monitor the current stream (hold) monitor the other stream (shift) and respond with button press Yantis et al. 2002)).
Dot Probe Task (Faces) (DOT): On each trial, two faces presented, one neutral and the other happy or angry for 500ms. Then, either of two simple symbols is presented at the position of either of the faces. The subject must make a forced choice discrimination of the symbol. Reaction time differences as a function of the valance for the preceding facial expression are calculated. There is increased variability of the bias with PTSD and fatigue (Sipos et (2014)).
Math task (MOD): Subjects perform a modular math computation every 8 seconds and respond with yes or button press. The object of modular arithmetic is judge the validity of problems such as 51=19(mod 4). One way to solve it is to subtract the middle number from the first number (i.e.. 51-19) and then divide this difference by the last number (32/4). the dividend is a whole number, the answer is "true." Otherwise the answer is false MMattarella-Mick
al. (2011)..
Psychomotor vigilance task (PVT): The subject monitors the outline of red circle on a computer screen for 10 minutes, and whenever counter clockwise red sweep begins, they press button as fast as possible. Subjects are provided with response time feedback. The experimenter records response latencies (Loh et al. (2004)).
A.2.2 DATA DETAILS
After discarding scan sessions with abnormal spikes that may be caused by head movements, the valid session details for different tasks are listed in table
Table 2: fMRI scan details for six tasks.
Tasks
Resting
VWM
DYN
DOT
MOD
PVT
(Total)
Valid scan sessions
209
514
767
155
138
157
1940
Frames/ Scan
321
300
265
798
828
680
A.2.3 ABLATION STUDIES
Numerical values of fig. 3a are reported in table Training time ranges from 51 seconds epoch for length-8 inputs to 298 seconds / epoch for length-256 inputs. Although the model is trained for 60 epochs in all experiments, it converges relatively stable loss level within 20 epochs.
14

---

Under review as a conference paper at ICLR 2022
Table 3: Weighted Fl of ablation study settings.
Input length (frames)
8
16
32
64
128
256
(i): SC + adp
66.19
70.18
75.87
76.14
82.91
90.85
(ii:: SC only
64.54
65.58
71.79
70.31
73.63
89.79
(iti): adp only
64.32
65.20
74.01
71.42
80.63
89.46
(iv): SC + FC
66.10
67.58
70.26
75.02
76.91
84.68
(v): random adj
62.17
66.25
72.30
73.72
76.58
89.22
(vi): (i) without smoothing
63.57
62.82
70.19
65.82
72.91
79.65
(vii): (v) without smoothing
56.88
64.08
72.27
62.72
75.16
83.75
(viii): coarsened graph
37.92
42.23
46.18
52.12
57.17
64.25
(a) A. E 12200x 200 of 6 randomly sampled inputs during DOT task.
(b) Ad E R x 200 of consecutive inputs from session during DOT task.
(c) Aupp E R 200 for resting state, VWM, DYN, DOT, MOD, PVT.
(d) 200 ROIS' column averages of task- averaged Audp-
Task order the same as (c).
04134
70
(e) 34 (17 with LR) subnetworks' column averages of task-averaged Astp Task order is the same (c).
Figure 10: Learned latent adaptive adjacency matrices Andp
(I) Latent adaptive adjacency matrix Aadp- As we mentioned in section 3.1 latent Audp can complement the task- and temporal-fixed A. We will now show that the learned Ag sparse for each sample, has evident task- patterns, and differs from what A; can provide: fig. T0 shows visualizations latent Andp- which we can tell is quite sparse as fig. [Oa each input only gets few important columns (information providing nodes in GNN), and they vary from one sample to another, indicating Asdp ability to adapt to changing inputs within a same task. However, when we look into inputs (not shuffled) generated by consecutive sliding windows from same session as in fig. [0b] we can see the latent structures appear to be in a smooth transition. In addition, when averaged across many samples for each task, undeniable task-based patterns emerge, in figs. 10c
15

---

Under review as a conference paper at ICLR 2022
to 10e But the task-average patterss are also different from what we saw from Attra in fig. 5 suggesting Aadp is embedding dynamics that are not captured by A. One interesting phenomenon in fig. [0e] is that across all tasks, LimbicB OFC, Default B, and LimbicA TempPole are always among the most important subnetworks appeared in the latent Audp- Further exploration is needed for explaining the case.
MII) Number of GNN layers. The total number of temporal layers depends on the input signal length since each strided TCN layer reduces the temporal length by a factor of 2: if the input length is 2', there need to be i temporal layers. But is alternating every TCN with GNN the best strategy, or do we only need to follow one GNN after a few TCNs? We study this question with different input lengths.
Model weighted FI are plotted in fig. 3b for all possible GNN to total TCN ratios (e.g. length-256 1
inputs requires 8 TCN layers. The possible ratios are 8: 4: 2 since we can insert one GNN per 8, 4, 2, TCN layers). The figure shows alternating every layer rarely yields the highest perfor- mance and the best ratio lines around one GNN per two TCN layers for our dataset. We repeat the experiment for K = 1,3 (in eq. (2) to rule out the possibility that this result is related to how many neighbors one GNN layer can reach; we find they have roughly the same pattern as the K = 2 case. Our hypothesis is that lower GNN to TCN ratio does not capture enough spatial context, while higher ones might be overfitting. We leave exploring the relationship between this ratio and nodes number N to a future study.
The best GNN TCN ratio also depends on whether model incorporates latent adjacency matrices or not: without Audp. length- 128 signals achieves its relative best (among all ratios) when having one GNN per two TCNs, but only needs one GNN per three TCNs if using Audp- This shows learning latent structures Asdp not only improves overall model accuracy but can also reduce model parameters, thus complexity, in achieving the relative best results.
(III) Effects of soft-assignment cluster numbers. During our experiments, we find as long as the smoothing module is used, the final performance will be close each other, only the convergence rates are different. Fig. shows how validation loss converges with different e settings and when there is no smoothing module used. From it, we can see halving numbers (100-50-25-12) is most helpful and we use it for our other experiments; decreasing numbers (160- 10--120-80440 or all larger numbers (all 100) works better than increasing numbers (12-25-50- 100) or all smaller numbers (all 12). Using the inner-cluster smoothing module, all cluster number settings converges to around 0.23 at their smallest when trained for 60 epochs and have test weighted F1 from 89.47 (model with 12- 25-50- 100) to 90.85 (model with 100-50-25-12). On the contrary, if no smoothing module is used, the model overfits easily and the validation loss can only reach about 0.4 before going up (with the best set of learning rate and weight decay parameters found with grid search). It is understandable that the model is prone to overfitting given the complexity of GNN and the relatively small dataset size. Our added inner-cluster smoothing module seem effectively countering the effect and brings the loss down further and stabler.
A.2.4 MODEL COMPARISONS.
We plot confusion matrices of our model, model from ablation study setting (viii), and the best performing baseline in fig. 12 Misclassification pairs clustered as the first three tasks (resting, VWM, DYN) and the latter three (DOT, MOD, PVT). Shown confusion matrices are from models trained on 256-frame inputs. We note that these misclassification pairs may be different for models trained on other input lengths (like 128-frame, etc.).
A.2.5 IG Attr AND Attrx
See fig. 13 for the visualization of important ROIs based on Attr A, and fig. 14 f4 task-averaged Attr under real and random SC settings. Many discriminatory regions obtained from Attr A com- plies with the literature:
For resting state: the top attributed ROIs belong to the default mode network, which is regarded salient during the resting state (Raichle (001)).
For VWM: the dominant attributions are from visual regions and posterior parietal regions, which complies with Todd & Marois (2004).
For DYN: attributions from our model suggest regions along cingulate gyrus (defaultA-
16

---

Under review as a conference paper at ICLR 2022
2.00
38030 20-30-55-22
1.75
Weigitted F1
08-
1.50
1.25
61
1.00
0.75
60
0.50
0.25
5
10
30
weight decay (10-1)
(a)
(b)
Figure 11: 11a adding inner cluster smoothing or input-dependent adaptive adjacency matrix makes the model more stable across various learning rates. [1b Validation loss training epochs. Input length is 256 and four smoothing modules are used. Legends are the soft- assignment cluster numbers of the four smoothing modules. Our other experiments using decreasing cluster numbers that halved each module, corresponding to the 100-50-25-12 here.
600
400
800
700
A00
400
600
S00
200
200
100
100
Figure 12: Confusion matrices of: 12a ReBraiD (our proposed model), 12b) model with coarsened graph (setting (viii), [2c) Graph Transformer peest-performing baseline). Tasks are I-Resting, 2- VWM, 4-DOT, 4-DOD, 6-MOD,6.PVT. 6-PVT.
Figure 13: Important ROIs based on Attr Tasks are: Resting, VWM. DYN, DOT, MOD, PVT from left to right. Node sizes are based on column sums of Aulra 2000x200 and edge width are direcly based on ALtr For the visualization purpose, only edges with highest attributions are kept to ensure sparsity being 0.009 (down from around 0.196).
Cww.Atttt-Coo---CCC ContCC-effulltC,, as well as peripheral visual and somatomotor regions. Liter- ature suggests anterior cingulate cortex (ACC) to be active (Kim et al. 2016) and posterior cingulate cortex (PCC) to inactive (Leech & Sharp) 2014) during visual attention tasks. This means both regions provide discriminative information about classifying DYN states, which is what our attribu- tion method is voting for.
For DOT: important ROIs from analysis are located in control networks, in particular both ACC and PCC, as well as in the peripheral visual system. In the literature, dorsal and rostral regions of the ACC proved to be involved with dot-probe performance (Carlson et al. 2012 2013).
17

---

Under review as a conference paper at ICLR 2022
Figure 14: Task-averaged ATTRA E 2,00x The top row is obtained from real SC induced A, and the bottom row is obtained from random SC induced Arand-
For MOD: our important ROIs mostly in femporal-parictar regions and default mode network (anatomically fronto-part panto-parietal), and literature suggests similar regions: parietal (Grabner et al. 2011 and prefrontal (Friedrich & Friederici 2013).
For PVT: our top attributed ROIs belong to control networks, attention networks, and somatomotor regions. Similar stated in Drummond et al. (2005), where both attention and motor systems considered important.
To view brain regions in the 17-networks setting instead of 200-ROI parcellations, table has the subnetwork rankings based on column-average Attr and tablesojas the subnetwork rankings based on temporal-averaged Attrx. For visualization of the 17-network parcellation, please refer to fig. 15
See fig. [6] for the complete attribution distributions for every task based on lemporal-averagee Attrx. Corresponding brain regions of certain number are listed in table 4 17 regions, we combine L.H and RH for their common network.
For boxplots showing
Table 4: Brain subnetworks the 17-netoork definition.
L.H_isCem
18
RH_Vi VisCettent
LH VisPeri
RH VisPeri
LH SomMotA
RH SomMotA
LH SomMotB
21
RH SomMotB
DorsAAAtA
22
RH AtrsAtttA
LH_DorsAttaB
23
RH.DDrsAttaB
LH_ Sa.etAAeeAAA
24
RH. AentAAAA
LH. SalVentAutB
25
RH. SeIVeAntt
LH. LimbicB -OFC
26
RH LimbicB COC
10 LH LimbicA,,Temae TeTepPole
27
RH LimbicA,,emmin TempPole
11 LH_ComLA
28 RH_CootA
LH. HoCCom
RH. HonCo
13
LH. ContC
30
RH ContC
14 Le_DafaultA
31
RH Ce_aafaultA
15
LefaulB
32
RH DefaultB
16
C._DefauliC
33
RH. CH_DefaultC
L._TempPar
RH_TempPar
18

---

Under review as a conference paper at ICLR 2022
R13
N11
N5
H14
HB
M13
M1G
N11
M4
N12
N17
M11
H1
N:7
N7
N12
M14
N1S
F10
R3
FC
N7
M24
H14
NB
416
NI: VitCont
N2: VicPeri
SomMMetA
WS: DarsAAtnA
N6: DersAttnt
MZ: SufVentAtrAA
NR: Salvenhhiii
N9:Limbics
POD: LimbicA
M31 Connt
N12: Corii
NL4: DefautsA
Delauui
#16 Defaulic
7 Touppfrrr
Figure 15: Network parcellation of Yeo's 17-networks. Figure is from Kahali et al. (2021)
Table 5: At Top 10 brain subnetworks initiate important connections during different tasks
(fig. 13 provides ROI-based visualization).
rest
VWM
DYN
MOD
PVT
ReLaalallAA
VisPer
LH. SomMoIB
RH. VisPeri
RH.DefallAA
RE.CoalA
HeDeefaltA
VisPer
LJI_DefaultA
Rei_SalNenAAttnA
3
RH DefaulB
VisCent
Lis
RH. BH.CCoCC
LHLDefauliC
RELConiC
RH_CrniB
RH_ VisCett
LH AmnAAAAA
Dar_AAAA
2H_DeffulCC
L.H. Ann
RII LimbicB OFC
LH DefaulIC
RII ConiC
RII SalVentAAnnA
RII DefaultB
RII DefaulIA
RH THLSSa/emmAmmm
RH SomMotB
THLContA
I.HCConCC
RH LiiiicB__FFFFF OFC
H_CComC
DefaulCC
DefaultA
MotB
8
RH_TempPar
1.H.S MomMoMo
RH VeVeAAtenA
I.H. SomMotR
RHJ Jimhic Temppole
HHII Limicc,
LH DefaultB
LH Limiiii,, LimhicA,TempPolee
LH. DefauliC
RH ContA
LH DefaultB
LH. SomMolB
10
NeL.AANeatAA
RH Hil.LLimicA.Tenneee
RH. VisPeri
RIL VisCent
RemMoB
RH Jel_DefulIC
Table 6: Attrx: Top 10 brain subnetworks that are sources of the important signals during different tasks (fig. proovides a ROI-based visualization).
rest
VWM
DYN
DOT
MOD
PVT
RH. VisCent
RH. VisCent
LHI Templar
RH LimbicB OFC
LBI Templar
REI VisCent
Vis.eeeent
VisCent
SomMoeBB
TempPole
LEl.LimbicA Temppole
1.H_VisPeri
RH_Defaltt
RH KHH_ooMMeR
I.H. Vis_ ent
RH DorsAlLA
RH DorsAttnA
LH_DefallIA
TempPar
LH. DefauliB
AlnnAA
J.H.CoIA
LHDDefuliCC
JHHJiibicA_Lee Pomp
RH.LimbicH
LH DeesAiinB
LH DonsAliBB
DefauliC
RE TempPar
RH LimbicA TempPole
LH CoatA
LH, SomMotA.
L. SalVentAumA
RH SomMorA
LH DefinitB
LH L LimbicB, OFC
RH THILLim LimbicA, TeeepPolee
SalVentAnnA
SomMort
Contt
LimbicH OFC
Defialt8
SomMorA
RH Limhicc,BCCC ,FCC
RH.SaaVemnAnm
HtLLimbiicR_OR
10
LH.Limbec BOFC LOCC
H.H. H.LLimbicBBOm
LH_DefaultB
LH.DefuutA
LH_DefaulAA
AttraBBAtta
19

---

Under review as a conference paper at ICLR 2022
6.33
0210
1400
6229
833
ml
Hime
025
676
0.31
(a) Resting state.
(b) Visual working memory task.
0
-00
-000
(c) Dynamic Attention Task.
(d) Dot Probe Task.
02200
00000
61.01
Math Task.
(1) Psychomotor vigilance task.
Figure 16: Brain subnetwork attribution distributions from Attrx-
20

---

Under review as a conference paper at ICLR 2022
A.2.6 ATTRIBUTION REPRODUCIBILITY
In order be used for downstream tasks, the extracted important regions should be reproducible across different initializations. Since the overall problem is non-convex, we test the reproducibility empirically with two models trained on two data splits, both initialized randomly and taking in length 256 inputs.
1.0
0.8
0.0
0.8
W W
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.8
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
150
150
200
50
200
(a) Column averages of Attra E IR
200
x:000
300
150
200
50
300
150
200
50
300
150
200
0.0
100
150
200
50
150
200
50
100
150
200
(b) Temporal averages of Attrx E 200
Reproducíbility validation: attributions obtained from two models trained on different data split and initialized randomly. Attribution values are normalized to [0, 1] and the larger the attribution value, stronger indicating power that ROI has for certain task. Tasks are Resting, VWM, DYN, DOT, MOD, and PVT.
Figure 17:
21

---

Under review as a conference paper at ICLR 2022
A.2.7 SIMULATION STUDY
To validate interpretation results, we perform simulation studies with know ground truth. For gener- ating graph signals, we first define graph structures. All graphs are generated with stochastic block model (SBM) using a same community structure (200 nodes, 10 communities). but each graph has its own adjacency matrix. This mimics brain structures in that samples share similar community structures but have different structural connectivities. Fig T shows a typical adjacency matrix of synthetic graph. All adjacency matrices are binary. Time-series on each node are then generated with codes adapted from this repo [ In particular, the value at each time step of each node is a small temporal Gaussian random noise plus signals from neighbors (a small Gaussian spatial noise is added to the adjacency matrix).
Simulation (I) We create two classes for this simulation: in class one, only the first three commu- nities (nodes - -00) generate small temporal noises and other nodes are only affected by neighbors; in class two, only the last three communities (nodes 141 200) generate small temporal noises and other nodes are only affected by neighbors. We visualize task aggregated Attrx and Audp and in relatively well reflected in Attrx. For the
(8b) [8c The signal importance differences generated series, signals are more important in node 1 - 60 for class and 141 - 200 for class 2: Aadp catches this up and help propagating signals in these regions better. We notice that Attr a are mostly random and no obvious patterns are shown. This also reflects the graph signal generation: when aggregating information from neighbors, all connected edges are weighted the same (binary), thus the connections not really affect generated signals. To see the opposite, we perform another study below.
Simulation (II) We again create two classes for the simulation: in class one, connections from nodes 61 100 are strengthened;; in class two, connections from nodes 101 140 are strengthened. The weights of strengthened edges are increased from to 5 during signal generation, but model still takes in binary adjacency matrices as inputs (processed as mentioned section 2. before feeding to the model). We visualize task aggregated Andp and Attr in figs. 18d and [8e We
this time the connection differences are reflected in Attr Signals in node 61 100 for class or 101 -140 for class are less important because they can be sent out by stronger connections: this results in smaller values for corresponding columns Aadp- Combined with previous simulation, regions that strong signal senders and connections from them weak not reflected in graph adja- cency matrices tend to have higher Andp values, In other words, Aadp complements both signals and connections to encode latent dynamics, while attributions obtained from IG are better interpreting the modalities separately.
22

---

Under review as a conference paper at ICLR 2022
0.8
0.8
303
0.4
10
0.2
350
100
150
200
100
150
200
(a) Adjacency matrix
(b) Simulation (I) Attrx
50
50
0.003
Jole
0.007
50
1.00
15.
50
100
150
25
50
100
196
150
175
300
26
75
160
135
(c) Simulation (I) Audp
0.8
0.8
100
100
m
0.6
6.4
puly
0.0
150
150
350
(d) Simulation (II) Audp
(c) Simulation (II) Attra
Figure 18: 18a A typical adjacency matrix for simulated graph signals.
18b
Task averaged Attrx of simulation (I) and
of simulation (I). Attribution values are normalized. [8c] Task averaged
Agip
its entry averages per column. [8d] Task averaged Auip of simulation (II) 18e
Task averaged Attr
of simulation (II). Attribution values are normalized.
23


Reviews:
[{"review_id": "xlPHJOIUquv", "reviewer": "Reviewer_zFSB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n- Clearly written and easy to follow.\n- Practically useful framework for fMRI data analysis. \n- Extensive experiment that validates the proposed framework. \n\nWeakness:\n- Novelty of the paper is quite unclear, especially in the introduction. Why is analyzing time varying signals on graphs challenging? What are the problems that recent methods are facing, and how are the authors are solving them?\n- Why is a normalized adjacency matrix used? Any technical / clinical motivation?\n- It is not clear why having individual adjacency matrix implies a latent graph cannot be learnt. A_adp is a functional graph derived from node signals; why can't each adjacency matrix can be deployed here? This is important as the authors use both AHW and A_adpHW later in the framework. \n- From anatomical perspective, relationships in local (structural) connections make more sense than those among far ROIs. What is the rational that it can be beneficial (rather than simply referencing Ying 2018)?\n- In the experiment, did the authors divide the given dataset in to a single set of partitions? Was cross-validation adopted?\n- Temporal data varies in their length per subject. How was this problem dealt? \n- During the method comparisons, I am not sure if it is valid to set the same epoch limit to derive evaluation measures as each model has different complexity. \n- Is the outcome properly validated? There are several labels according to memory, attention, dot prove and so on... are the label-specific variations detected from this framework really associated with those tasks? ", "summary_of_the_paper": "This paper proposes a deep learning method for temporal data on graph nodes, specifically designed for brain imaging data. It can be deployed for classification of data where the time-varying data and graphs are individual specific, and pinpoint subgraph structure where group-specific changes occur. ", "summary_of_the_review": "This paper tackles an important problem in neurosience. \nClinical motivation and the overall pipeline make sense, and extensive efforts were put on evaluation of the ideas and framework. \nThe paper is mostly clear, but I do have some concerns as mentioned in the Main Review above which can be very critical. \nI am willing to change my score once I go through the rebuttal and other reviews if needed.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pimSlMRCZNk", "reviewer": "Reviewer_Cuxq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The proposed method combines structural and functional brain information, which are often assessed separately.\n\n- The proposed network includes components for both temporal and spatial processing.\n\n- The proposed network utilizes an individualized graph structure (connections) for each sample, rather than defining one adjacency matrix for all like most standard approaches. The network can handle this in part due to the soft clustering and keeping the same number of nodes at each GNN level.\n\n- The paper is fairly clearly presented.\n\nWeaknesses:\n\n- The \"strided non-causal TCN\" I feel is a bit of a misleading name - when I think temporal convolutional network I imagine it needs to have the causal aspect, as it's trying to ensure appropriate analysis of temporal information. The \"non-causal\" TCN is just a strided convolution on 1D data that happens to be in time dimension. Also, it would be helpful if the authors briefly include the information/equation for how the gating mechanism works.\n\n- For the experiments, the partition of the dataset appears to be by scan, not by subject. Since there are 1940 scans, 6 different scan tasks and only 56 subjects, my understanding is each subject may have undergone multiple session of the same task. Thus, it does not seem fair to split by scan, but rather should perform a patient-wise split for the validation method to get a better estimate of generalization performance.\n\n- While the total number of scans is reasonably large for neuroscience study, given the smaller number of individual subjects, a single split validation study does not seem appropriate - a cross-validation framework would be more convincing. Furthermore, as there is only a single split, there is no variance information so it is difficult to assess if there are true differences between methods that perform similarly.\n\n- The extremely large gap in empirical performance between the standard GNN models and the proposed method seems rather jarring (up to ~45% difference in accuracy). This makes me wonder whether the other models are not being appropriately tuned, or are given appropriate depth? I did not find any details about the baseline models, e.g., how many layers are used? For the proposed method it seems the authors would have used 4 layers, did they match the same number for baseline models? Also, how are the graphs constructed for the baseline model - are the same connections used or did the authors vary connections per sample like in their approach? And are the connections based on structural or functional data? In general, how would the authors explain the large difference in performance? It could also be interesting to compare to other GNN methods that were developed specifically for functional brain data analysis, e.g. Li et al (as cited in the paper), or Li et al, Braingnn: Interpretable brain graph neural network for fmri analysis, bioarxiv 2020.\n\n- The authors compare to GNN-only baselines, but not TCN-only baseline. I think it would make sense to compare to TCN models given (over) half of their network is based on TCN, and also given the large gap in performance using GNN only models; this may point to the feature learning in the TCN layers as being very important compared to the GNN component for classification learning.\n\n- How is the cluster number c chosen? The authors note that c decreases with deeper layers, but there is no mention as to the values used or how they are determined. Considering the authors claim this cluster smoothing is essential, it is important to explain how related parameter c is set. Does changing c then greatly affect the results?\n\nMinor comments:\n- For the variable for structural connectivity SC, the use of 2 letters for 1 variable is confusing when reading the equations - considering changing to 1 letter.\n- More detail on the number of each type of task scan and how many scans per subject would be welcome (eg added to appendix).\n\n", "summary_of_the_paper": "This paper presents a graph neural network architecture for analyzing dynamic brain imaging data. The proposed approach uses both structural connectivity (via DWI) and functional connectivity (via fMRI) to setup the graph edge information, while the fMRI signals are used to define node features. The proposed network alternates between analyzing temporal information via gated temporal convolutional network layers and spatial information via GNN layers. A multi-resolution soft cluster smoothing is applied as the pooling operation in the GNN. Attribution of structural and functional imaging inputs is also explored using integrated gradients interpretation method. The proposed method is tested on a multi-task fMRI dataset and demonstrated better performance in both ablation studies and comparison to state-of-the-art GNN methods.", "summary_of_the_review": "My recommendation is based on the new proposed network for combining functional/structural brain data analysis in a meaningful way for processing temporal and spatial information, and allowing individualized graph structures to be learned/used within a GNN model. While the results appear good, they appear somehow incredulously higher than standard GNN methods, and important missing information about experimental setup and empirical comparisons further dampen my enthusiasm for the work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "llVgnmKzx5", "reviewer": "Reviewer_wRj8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the approach introduced in the paper interesting, and the motivation is clear. I enjoyed that the proposed approach aims to extract latent dynamics information, thus capturing temporal heterogeneity, which can be very useful in neuroscience as well as disease prediction research. While this contribution shows that the proposed method can generate better prediction results, I see several issues with it as it stands. The rationale for combining various existing approaches as proposed in the paper should be clearly justified emphasizing the novelty. The presentation of the manuscript can be improved, and a few details seem to be missing. I have some concerns, which are listed below.\n\n1) Existing approaches to estimate dynamic connectivity and approaches estimating spatiotemporal patterns in the fMRI data [1,2,3,4] are not mentioned. Authors could comment on what existing approaches (based on linear modeling) lack, relevance/irrelevance of the approaches in the given context, and compare the results.\n2) The model is not straightforward to understand in one go. Paragraphs in Section 2.2 could be rewritten such that they are more connected and give a better picture of the model. \n3) To validate the method and illustrate the motivation, please consider including some simulation studies on synthetic datasets. Since the method is intended to capture latent dynamics, the authors could borrow simulation settings from the related literature.\n4) It is mentioned that the F1 metric is used because of the imbalanced dataset, but the information about the imbalanced dataset is missing for understanding the data. Also, the number of frames for each task is missing. \n5) It seems that the experiments are performed only for one set of train, validation, and test set. Why are multiple runs of k-fold cross validation not used here? \n6) Not sure what to make of the qualitative analysis in Figures 6, 12, and 13. It would be helpful to discuss the importance of \"important\" nodes in the context of these tasks rather than just stating the region name. Also, the authors should describe the regions mentioned in the main text. The claim of the paper is that the proposed method can find good representations but the authors have not explained the importance of these subnetworks in the context of the task presented. Even a small discussion would be helpful.\n7) We notice that signal-important ROIs are not necessarily the same as connection-important ROIs:? What does it mean from the point of view of biological relevance? How to use these remarks as biomarkers?\n8) How reproducible are these important regions extracted? Since the complete problem is non-convex, depending on initialization, will the end result vary? Since these regions would be used in further downstream analysis, they would need to be highly reproducible.   \n9) Authors mention, \"We also find using adaptive adjacency matrices, and inner cluster smoothing can stabilize training, making the model less prone to overfitting and achieving close-to-best performance over a larger range of hyperparameters.\" Can you show results to support the claim made?\n10) Authors mention, \"Important brain regions obtained from $ATTR_A$ mostly comply with the previous literature.\" What are the important brain regions? Can you cite literature to back up this claim?\n11) Is there a reason that $A_{iadp}$ is sparse for each sample? Is this expected biologically or due to the way in which the model is defined? Would it be helpful to use sparsity constraints on $A$? This might be helpful in controlling the sparsity and in the interpretability.\n12) Can you comment on the biological relevance of the brain regions found in $ATTR_X$ \n13) How are $h_{adp}$ in eq 1 and $K$ in eq 2 set?\n14) X-axis and Y-axis ticks size can be increased to improve the visibility in Fig 3, Fig 5, Fig 10 and Fig 11.\n15) What is $\\tilde{D}$ and $v$ in section 2.1? Is $\\tilde{D}$ a diagonal matrix where $v,v$ entry is equal to the given sum?\n16) Published versions of the referenced arxiv papers should be cited.\n\nI am happy to raise my score if the above concerns are addressed.\n\n[1] Li, Lingge, et al. \"Modeling dynamic functional connectivity with latent factor Gaussian processes.\" Advances in neural information processing systems 32 (2019): 8263-8273.\n\n[2] Taghia, Jalil, et al. \"Bayesian switching factor analysis for estimating time-varying functional connectivity in fMRI.\" Neuroimage 155 (2017): 271-290.\n\n[3] Zhang, Gemeng, et al. \"Estimating dynamic functional brain connectivity with a sparse hidden Markov model.\" IEEE transactions on medical imaging 39.2 (2019): 488-498.\n\n[4] Bhinge, Suchita, et al. \"Extraction of time-varying spatiotemporal networks using parameter-tuned constrained IVA.\" IEEE transactions on medical imaging 38.7 (2019): 1715-1725.\n", "summary_of_the_paper": "This paper proposes a Graph Neural Network model to estimate latent dynamics in the human brain using functional Magnetic Resonance Imaging (fMRI) and Diffusion Weighted Imaging (DWI) while performing a classification task. The model consists of four parts 1) estimating each sample's adjacency matrix by learning a shared projection matrix 2) gated temporal convolutional neural network for learning information from time-series data 3) graph neural network to get embeddings of temporal and spatial information combined by 4) multi-resolution inner cluster smoothing. The authors use integrated gradients for estimating important features for the classification task helping with model interpretation. The authors show that the proposed method can perform better in the prediction task than GCN, GAT V2, etc. On the real dataset, it is shown that the method can capture temporal and spatial heterogeneity and provide regions important for the presented classification task.", "summary_of_the_review": "Based on the above assessments, I suggest rejection of this paper. The current experimental evaluations and discussion need to be stronger to be more convincing, and the presentation can be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "h0-mxSF77MS", "reviewer": "Reviewer_Whpw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The method in this article has made many improvements to graph neural networks. These improvements are to capture the spatial information and temporal information in brain dynamics. These improvements have also been verified in experiments to learn better temporal and spatial importance sanity.\n\n2. The author uses the method of graph neural network to learn spatial information and temporal information. This new proposal may be extended to different fields. Not only does it have the effect of improving the brain image, but it may also be useful for general computer vision in the future. . So the author's innovation in the method is quite meaningful.\n\n Cons:\n1. In the experiment, the author compared with other different graph neural network methods. In table 1, the accuracy of this method is about 20% higher than other methods. These results are quite confusing to me, and the GCN method only has 41% accuracy. I think the baseline method compared by the author may be too poor. It is more appropriate to compare it with other methods that are not graph neural networks. The ones in Table 1 The result is still unable to prove the effectiveness of the method.\n\n2. The improvement of several methods on graph neural network in this paper is mainly based on some previous work. But the main contribution is to make some methodological adjustments for brain imaging, which greatly reduces the innovation point of this article. Therefore, the overall innovation of the article still needs to be improved.", "summary_of_the_paper": "This paper proposes a method of graph neural network to jointly models dynamic functional signals and structural connectivities, to learn a good deep representation of brain dynamics. In this article, the author proposes some improved methods for graph neural networks, such as learning sample-level latent graph structures, temporal convolutional network and multi-resolution inner cluster smoothing methods to better learn insightful interpretations of brain dynamics. In the experiment, the author verified the effectiveness of the method in this paper on the fMRI signal data.", "summary_of_the_review": "This article improves the network method of graph neural network to learn a better representation for brain dynamics in the application of brain images. I prefer to reject this article because this article has made some small improvements on GNN, which is more like to combine them in different ways to improve the overall effect. In addition, the baseline effects compared by the author in the experiment are relatively weak, and the effectiveness of the method cannot be demonstrated.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a Graph Neural Network model to estimate latent dynamics in the human brain using functional Magnetic Resonance Imaging (fMRI) and Diffusion Weighted Imaging (DWI). The representation is tested on a classification task. While reviewers acknowledge the importance of this application, various concerns have been raised and partially addressed. The work focuses on graph deep learning and offers limited evidence of its superiority over more traditional ML or non graph based deep learning. Besides the methodological novelty is unclearly argued, which is not ideal for the audience of a conference like ICLR.

For all these reasons, this work cannot be endorsed for publication at ICLR 2022.
Decision: Reject