Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a GNN model (STAGIN) for learning from dynamic, spatio-temporal fMRI data. The model, based on GIN, takes a temporal sequence of brain graphs as input and includes a time encoding in the node features. The standard readout function is modified to incorporate spatial attention in two possible ways (scaled dot product attention and squeeze-excitation model). The series of graph features is then passed to a transformer-style self-attention encoder to incorporate temporal attention. Orthogonality of node feature vectors is also encouraged in the loss function. The method was tested on the public HPC datasets (~1100 subjects) to classify gender from resting state fMRI and decode task from task-based fMRI, and analysis of the temporal and spatial attention results was performed.
main_review: Originality:  
The authors present some new methods for learning dynamic graph representations in the context of brain connectomics. While prior work on methods for learning from temporally changing brain graphs are presented, I believe there is some discussion missing of prior work on the proposed methods of including attention in GNN networks, specifically for including attention in the readout layer (eg., [1,2]). Please discuss how the proposed methods differs from such prior work. 

[1] Fan et al., Structured self-attention architecture for graph-level representation learning, 2020.  
[2] Yan et al., Learning Multi-Attention Context Graph for Group-Based Re-Identification, 2020.

Quality:  
This is a nice paper that brings in some often used modules (scaled dot attention / squeeze excitation, self attention) to the GNN model to analyze dynamic graph data, specifically 4D fMRI.  The methods seem appropriate and is a well thought-out piece of work, with some nice posthoc analysis of the attention results with respect to connectome understanding (e.g., visualizations showing the changes in spatial attention following the changes in task, visualizations of network differences for males/females). My primary critiques lie in the experimental methods. Some detailed comments:

1. In Sec. 3.2.3, the authors discuss the use of orthogonal regularization for the node features, and say they "find it desirable to encourage the orthogonality of H". I am not sure I understand the motivation behind the orthogonal constraint, compared to some other form of regularization - it would be nice for authors to elaborate on that.
2. My biggest concern lies in the experimental portion of the paper. The authors do include results of ablation experiments in the supplementary materials, but in the proposed method is compared to only 1 other approach for the gender classification task and 2 variations of 1 approach for the decoding task.  Furthermore, while it seems the result for the gender task was produced by the authors, the decoding results were taken from the literature, and it is unlikely that the exact same subjects and preprocessing step/experimental methods were used. Thus, I would say the numbers are informative but not directly comparable. Also, given that the ST-GCN result was produced by the authors, I don't know why this method wasn't also tested on the decoding task.  
3. Authors appropriately list all the hyperparameter settings, and it seems that the same settings were used for all folds (no tuning). However, I am wondering about the setting for the time window length $\Gamma$. While authors give references to support their choice, in general window length in standard dynamic connectivity analysis is highly debated (actually as mentioned in one of their references) and the choice is generally arbitrary, with analysis results potentially greatly depending on the windowing. While the authors do provide a fairly complete study overall, I think it would be great to see the effect of different window sizes on the proposed method's results.
4. I am wondering for the k-means clustering of attended graphs, how/why the authors chose 7 clusters? 

Clarity:  
This is a well-written paper - it is well-organized with appropriate amount of background and the writing is very clear. Just a couple notes:
1. p.2, line 82: Authors wrote $f = g \circ q$ where $g$ is the GNN and $q$ is the transformer encoder - should be $q \circ g$
2. Fig. 3 caption: "attened" --> "attended"

Significance:  
Improved dynamic analysis of fMRI data could be very significant, allowing for better understand of underlying neurological processes and disease, and I think this work would definitely be of interest to readers in this field. I think the proposed method for including spatial and temporal attention to GNN analysis could lead to others building off of such ideas. 

limitations_and_societal_impact: Some limitations are addressed (threshold for determining attended regions), and they mention that the understanding brain connectome has both positive and negative societal impact, as linked to search for new biomarkers, e.g., for disease. However, it is not really clear what negative impact they mean by this, and I think some elaboration is needed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an attention based GNN method, which it names STAGIN. The new method handles dynamic FC matrices for gender and task classification using fMRI data and learns to focus on important time points and brain regions based on attention. The method achieves better accuracy and AUC for classification compared to the relevant competing methods that STAGIN is compared against. The temporal attention matches with the task-relevant time points in the input data.

main_review: The main contribution of the paper is in handling dynamic FC matrices and finding important time points and brain regions relative to the downstream task of classification. The paper uses GNN to model brain data into nodes representing brain regions and edges representing functional connectivity. The paper shows higher accuracy and AUC scores for the task of gender and task classification compared to the methods mentioned in the paper. The paper has the following major and minor drawbacks.

## Major Drawbacks:

1.  The idea of applying attention to graph networks is not novel as there have been many published studies for this e.g. GAT, GATV2 etc. The paper applies attention to the published method GIN but does not compare similarities or differences with existing such approaches e.g. “Understanding Attention and Generalization in Graph Neural Networks” ChebyGIN.
2.  The author(s) propose a novel attention based Readout method and add it in GIN to create a single embedding for the complete graph. The authors propose two variations of the method. Again, such methods are already well established e.g. topKpooling, SAG pooling, ASA pooling. The author(s) do not discuss or compare against such methods and rather mention the original method of GIN. The author(s) uses temporal data with a sliding window approach to create dynamic FC matrices but do not compare against methods which handle temporal data and dynamic graphs e.g. TGAT, TGN-attention, CTDNE etc. These methods may not have been applied for the same downstream classification task but still a comparison with them would reveal if the proposed method is improving over existing analogous approaches. The paper is well written and experiments are thorough and well structured, but it is unclear whether a technical novelty is introduced or largely existing methods are applyed to fMRI data for gender/task classification. (Also se item 2 in the minor drawbacks)
3.  The paper needs to report results from much more published studies, especially those which handle dynamic/temporal data as mentioned above. The author(s) also need to show how their attention mechanism is better than existing methods of attention in GNNs.
4.  The author(s) need to mention how they got the result of BAND and BaND++ and whether the presented comparison is made on the same data with the same processing. BAND and BaND++ are applied to 4D data of voxels and not on region based data and thus would be at a statistical disadvantage if compared with the method that uses region based aggregation.

## Minor Drawbacks:

1.  In Fig 3, the left three clusters namely "6,2,7" have more females than male but DMN does not have hyper connectivity compared to SMN. Whereas the last three clusters with more male do show hyperconnectivity in SMN. This should be explained in more detail.
2.  Since the FC is not learned rather is computed using covariance, the author(s) need to show how the connectivity of DMN and SMN looks in time points which are not attended by the model. It is quite possible that the DMN and SMN connectivity is different for males and females per the referenced literature in all time points and not just the attended ones. If this is true, then the value provided by the presented model is even less clear.
3.  The author(s) mention at line 237 that it's average of columns but looks like average of rows. Also the index j is not specified.
4.  The author(s) should explain why the 30th percentile was chosen when creating the adjacency matrix using FC.
5.  The author(s) should be more clear with what they mean by dynamics in this work. Notably, 1) The regions/nodes do not change through time 2) The number of edges also remain fixed. Please be specific and explicitly explain what does change?
6.  The ablation study in the appendix shows that attention doesn't improve AUC results by much. Also, the last row for the main atlas shows very high accuracy and AUC without any of the four components. This needs more explanation as it looks like the underlying classification task is not difficult.
limitations_and_societal_impact: Addressed
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The authors propose the Spatio-Temporal Attention Graph Isomorphism Network (STAGIN),  which can learn the dynamic graph representation of the brain connectome with spatio-temporal attention. The paper uses visualization for the interpretable display to a certain extent.
main_review: The paper proposes a new dynamic FC graph model and explores the interpretability of spatio-temporal attention. This is an interesting research work. But there are still some issues that need to be clarified.

1、The innovation of the paper is limited. Both the attention mechanism and transformer are existing technologies. Authors do not highlight their proposed technology details and technical novelty. In addition, the paper lacks theoretical innovation.

2、The authors claim that the proposed method achieves SOTA performance. The authors introduced a lot of related work in the Introduction, but only used 2 baseline methods. The authors should increase the baseline methods and report the method parameters and running time, etc.

3、The effect of hyperparameters on the results is not discussed. For example, λ and edge threshold (30%).

4、There are many methods to score attention. Why did the authors only choose SERO and GARO？

5、The author did not perform an interpretable analysis of the constructed dynamic graph (the graph before spatio-temporal attention). This is also important for verifying the authors' research motivation or interpretability.

6、More description should be added to the caption in Figure 1.

7、Figure 2 does not seem to be a complete display. For example, there is no node feature $\boldsymbol{x}_{v}(t)$.


limitations_and_societal_impact: Limitations
-Simple combination of existing technologies
-The performance of the proposed model is affected by hyperparameters.
-Interpretability
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method, STAGIN, to model dynamic connectivity. The architecture combines graph isomorphic network (GIN) with spatial and temporal attention. STAGIN is applied to HCP data to classify gender as well as tasks with higher accuracy than state-of-the-art methods shown.
main_review: Originality
While each component of STAGIN is based on existing works, the proposed combination shows improved performance over state-of-the-art.

Quality
The method seems sounded, but some claims and results raise concerns.
1. It is mentioned that “we address the issue that the node features of the input dynamic graph do not contain any temporal information and concatenate encoded timestamp with the node features”, but [11] uses windowed sub-sequences as node features of ST-GCN, which is analogous to the timestamp used in STAGIN, with the difference being not all sub-sequences are fed in simultaneously. Also, [3] uses temporal convolution network at the input, which would capture temporal dynamics. This latter method should be quantitatively compared and not brush off with one sentence.
2. Temporal explainability is claimed to be a limitation of existing methods. For resting state fMRI, why would someone care which time points are most relevant for classifying gender or other phenotypes. For task fMRI, while it is reassuring to see attention is paid at the time of stimulus, are we gaining any new information? Please provide examples of when temporal explainabiltiy is useful.
3. For interpretation, “k-means clustering analysis of the resting-state dynamic FC [1] and general linear model (GLM) statistical mapping of task fMRI [10] for interpreting the spatio-temporal attention learned from STAGIN” were used. The question is why can’t we apply these to latent representations or other models? This is not a particular property of STAGIN.
4. How sensitive are results to thresholding the top 30-percentile values of the correlation matrix? How were other hyper-parameters chosen?
5. How were subjects divided into folds? HCP is a twin study hence subjects are correlated. Were one of the twins removed for each pair?
6. How was temporal correlation accounted for when dividing the timeseries in the case of task classification? Or was classification performed at subject level with entire session timeseries taken as one sample, in which case, how was correlation between sessions from the same subject accounted for?
7. Would like to see STAGIN applied to harder tasks e.g. ASD classification.
8. The current experiments and the way results are visually presented (e.g. Figures 3, 6, and 7) do not provide much neuroscience insights. Would be nice to see less superficial neuroscience findings to show the gain in interpretability by STAGIN.

Clarity
Most parts of the paper are clearly written but technical details/contributions are deferred to references, partly since the proposed architecture is mainly based on existing works. A few minor points that might be worth considering below.
1. While it might be true that this is the “first study to demonstrate the capability to learn the dynamic graph representation of both resting-state and task fMRI data with a single framework”, why is this difficult? Any general DL architecture can be applied to both rest and task fMRI data.
2. The clear description of GIN is appreciated and the “encoder/decoder understanding of GIN” does provide some motivations to use attention instead of sum/average. However, that section seems to be an overkill to say attention is better than just sum/average.
3. Figure 3 is hard to follow. For example, cluster 7 is supposedly attended more by females but SMN has bright red blocks, i.e. not hypoconnectivity. The qualitative evaluation needs more work. Is the ratio of female/male attentiveness significantly different from 1 statistically?

Significance
It is a reasonable empirical study showing that graph convolution followed by spatial and temporal attention can improve fMRI classification. The current classification task might not be difficult enough to show the true gain of STAGIN. Could potentially be of broad interest if the result interpretation part is further developed.

Post Author Reponses
I have read through the reviews and responses, and will keep my original score since that score assumed correct statistical setup for the classification evaluation, which the authors have clarified and "promised" to acknowledge the issue with twin data. A point worth noting is that just finding which time points drive classification does not help interpretation for resting state data. What might help is the relative number of time points with "aberrant signals" between subject groups for example.
limitations_and_societal_impact: Description of limitations and negative societal impact is sparse. Might be helpful to mention some of the limitations stated above especially on result interpretation and the chosen experiments.
review_title: null

Meta-review: The paper introduces a method for learning dynamic graph representation of brain connectome that
achieves state-of-the art performance while providing spatio-temporal explainability.
The work was appreciated by reviewers, as it put together novel tools (scaled dot attention / squeeze
excitation, self attention) with a graph neural network model to analyze dynamic graph data, specifically
4D fMRI. The application to the brain imaging data analysis was thought to be convincing. The reviewers
also consider that the paper is well-written.
However, it was found that
• not enough baselines had been used in applications, questioning the practical usefulness of the
method.
• some technical details were still somewhat arbitrary, questioning the generalizability of the method.
• comparisons against alternatives, in particular, attention-based methods, were not sufficient
Overall, the authors did a good work addressing the comments used, in particular regarding
comparisons/benchmarks against state of the art. However, identifying more difficult benchmarks would
indeed make the paper more convincing overall.
The final consensus is thus a weak accept.
Decision: Accept (Poster)