Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed brain network transformer that learns representation from brain network with a transformer-based design. More specifically, the two main designs of the proposed brain network transformer are the multi-head self-attention module and the orthonormal clustering readout. In experiments, the proposed method showed performance improvements over multiple baseline methods.
strengths_and_weaknesses: S1. This paper is overall clearly written and easy to follow.

S2. Brain network analysis is an important field of study and the proposed method is interesting and reasonably designed.

S3. Both the experimental results in the tables and the case study features in Section 4.4 validates the effectiveness of the proposed method.

W1. In Introduction, the authors mentioned the advantage on efficiency of the proposed method over existing graph transformer models. Hence, runtime experiments would be helpful to validate this claim.

W2. Other than the transformer-base GNN methods, I wonder how does normal GNN perform on the brain networks. I would suggest the authors to also include vanilla GNN baselines for comparison. 
questions: Please address the above W1 and W2 if possible.
limitations: n/a
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: In this work, the authors have proposed a self-attention based graph model to classify a) ASD and b) gender using functional neuroimaging fMRI datasets. The authors also proposed a new readout method based on clustering for graph classification. The proposed method seems to perform a bit better in the classification of both tasks in comparison to other baseline methods.

strengths_and_weaknesses: ## Strengths
- The biggest strength is the classification performance of the model for ASD and the gender classification tasks. The classification performance is ~1% better in terms of accuracy when comparing with next best (FBNETGNN)

## Weaknesses
- There are some comments in the paper which are either incorrect or not properly justified. For example, lines 43-44 mention "First, a brain network is a correlation matrix defined on a complete graph."  This is not correct as correlation matric computed using methods like Pearson’s Correlation Cofficient is a way to measure brain connectivity. There can be many other ways to represent brain network e.g. Effective Connectivity, Directed connectivity, Entropy based, causality based (Granger causality), etc. Lines 49-51: it would be better to explain how a 1D time-series of a voxel has positional information.
- The authors do not mention how they compute the $X \in RV *V$ matrix.
- The authors cite multiple self-attention and graph based methods but do not compare with them, for example BrainGNN, STAGIN. It would be better to compare with these methods and others like these on the same task.
- There are others self-attention and graph based methods developed for neuroimaging datasets.  
   1. A Deep Learning Model for Data-Driven Discovery of Functional Connectivity https://doi.org/10.3390/a14030075
   2. Deep Dynamic Effective Connectivity Estimation from Multivariate Time Series https://arxiv.org/abs/2202.02393
- It is unclear how the results used for comparison of the methods are acquired. Are they taken from other papers? Was the code open source? How the hyper-parameters were tuned for these methods?
- There are many readout methods developed already which perform much better than the simple ones (add, max, avg, min) such as attention based, topkPooling, Edgepooling, SAG pooling and many more. These methods are also acquire a smaller graph as the readout method proposed by the authors. Other global pooling methods also exist such as GlobalAttention. Comparison with them is lacking.
- Similar concerns arise regarding the results in Table 2 as well. The manuscript compares against basic methods while many sophisticated and more powerful methods are already proposed. Table 2 also shows that the gains by the readout method proposed by the authors are minute when comparing against CONCAT method which is still not the most sophisticated among the available.
- The authors mention that they randomly selected $70$% data for training, $10$% for val and $20$% for testing. As I understand, there is no cross-validation or testing on multiple folds. ABIDE dataset has a high variation of performance across subjects and the single-point results cannot be fully trusted. This can be a reason that in Fig 5, it is seen that the proposed model performs better on test data than val data. It is very much possible that the authors considered test subjects that generally provide high classification performance.
- The paper provide very limited interpretation of the results.

questions: - Figure 2 shows multiple $X$ matrices and it is unclear how they were acquired. My guess is it's the same matrix, rather the self-attention module receiving rows of the matrix.
- How is the matrix $X$ computed? 
- If classification performance is the goal then why not use simple ML models like SVM, LR on FC matrices computed via PCC. With hand-crafted features (FC matrices), ML models give better classification performance.
- How were the results for comparing studies calculated? 
- Why not use HCP data for gender classification as it is publically available?
limitations: - Results were not compared against many other existing relevant studies. Mentioned in weaknesses.
- The readout method compared against basic methods such as sum, max, etc. but not against serious and widely available contenders.
- Marginal improvement in classification accuracy ($\approx1$%). 
- No cross-validation, no multiple test folds. 
- Missing details about input data ($X$).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a graph transformer for deriving effective representations of brain networks. In particular, this handles several caveats of brain networks: 1) use of connection profiles as positional embeddings and 2) deriving per-graph representation by projecting the node embeddings to orthonormal centroids for initialization. The downstream tasks using these embeddings on two public fMRI datasets show superior performance on autism spectrum disorder and gender prediction.
strengths_and_weaknesses: Strengths:
+ The paper is overall well-written and easy to understand.
+ The experiments are thorough with numerous baselines and ablation studies.
+ The theoretical justifications for the benefits of orthonormality in a statistical manner are interesting.

Weaknesses:
- Predicting the gender with the ABCD dataset is not particularly interesting or useful.
- If the soft clustering based on the proposed orthonormal bases shows empirical benefits, it would have been interesting to also see how the clusters imply additional neuroscientific findings based on their regions or functional modules.
- The stratified splitting based on the site information is a common procedure.
questions: The paper is overall solid. There are some questions related to the weaknesses section:
1. Structural connectivities are often not fully connected via heuristics (e.g., thresholding) and/or biology (e.g., no fiber bundles between certain regions). Would the proposed method also properly leverage the connectivity profile?
2. Was BrainNetTF tested with other readout functions?
3. Why was gender chosen to be predicted using the ABCD dataset? Were there no other meaningful response variables to study? I am not sure how meaningful gender prediction is in general.

[Update after the rebuttal]
I am keeping my score (6). Most of my concerns were addressed, so I appreciate the authors' effort. I still think the stratified sampling of the sites is still just a simple heuristic.
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a new transformer-based model for brain network analysis bringing better improvements on the ABIDE and ABCD datasets when compared to other transformer-based models. The model's architecture is motivated on insights from two previous papers [12, 25] to decide on the initial node features and to learn graph structures to achieve superior effectiveness. The paper makes a further contribution of providing a re-standardisation of an evaluation pipeline for the ABIDE dataset, given findings on unstable training.


--------------------------------------
[UPDATED AFTER REBUTTAL DISCUSSIONS]
In my opinion, the authors tackled my question in a satisfactory way, and I am making these changes to my review:

Soundness: 2 > 3
Contribution: 2 > 3
Rating: 5 > 7
strengths_and_weaknesses: Overall I think this is a good, well-written paper. Despite being yet another paper exploring transformer-based models, the results seem to be sound and the ablation analysis goes one step further and shows that the OCRead module can be useful in some other architectures. The proofs in supplementary material are a good addition and I thank the inclusion of a link for me to analyse the code during this review process. All these strengths are enough for me to recommend (borderline) acceptance, but I have some important concerns that make me not give a higher score at the moment, which I hope the authors can answer during the rebuttal period. I will first highlight what are my major concerns/doubts, and then leave a list of some other less important comments/suggestions.

Before going into it, **I must ask the authors to please double-check their terms to anonymise in the code shared** because while I was analysing the code, I found at least one `.yaml` file under the `setting` folder with an absolute path. It was quite obvious to me that it contained a username resembling quite closely some important papers referenced so I stopped looking more into it, but still would ask the authors to correct this in case we need to open the anonymised repo during the rebuttal process.

## Major weaknesses

1. The paper cites Kan et al. [25] to justify the idea of having the network to learn the graph structure, and uses the model presented in this reference (FBNETGEN) as another baseline for comparison. Although I know that choosing baselines can be a very subjective topic, I must say that selecting only one baseline for a learnable brain network is not enough. I have seen at least two possible papers learning a graph structure (arXiv:2002.04999, 10.1016/j.media.2020.101709) which were never evaluated on ABIDE/ABCD and could therefore be another interesting contribution of this paper to use this previous literature on unexplored datasets, even though the 2 papers I leave here had some analysis on medical images and were not used in the original FBNETGEN paper. From Table 1, I noticed that BrainGB and FBNETGNN seem to come from the same group as they come from the two references used to motivate the some choices on the model (refs 12 and 25), which is another non-subjective reason to include another baseline from another paper. Finally, the "learnable brain network" type in Table 1 is the one with the closest performance to BrainNetTF and therefore another baseline of the same type would improve the significance of this model. For all these reasons, I believe the authors need to add at least another meaningful baseline on the "learnable brain network" category in order for their results to be more sound.
2. I believe the authors are mistakenly using "gender" when they mean biological sex. As far as I know, these datasets do not provide information on the gender of the people in the datasets, but rather the biological binary sex. I therefore request the authors to correct this for example in figure 2 and in the other occurrences of "gender".
3. On lines 170-171 the authors claim that "none of the existing readout functions leverages the properties of brain networks that nodes in the same functional modules tend to have similar behaviors and clustered representations". After reading this sentence I immediately thought about the DiffPool paper (arXiv:1806.08804) and so many other papers adapted from this differentiable graph pooling module which, in practice, forms different clusters as a way to pool the nodes representations. I know of at least one recent paper (10.1016/j.media.2022.102471) using it in the medical context as readout function, so I believe the authors should probably downtown this sentence, and find a more complex (and who knows fairer) readout/pooling operator to compare OCRead with. From what I see in the code, the authors used DGL with pytorch, which I believe has implementations available for more complex pooling layers (https://docs.dgl.ai/en/0.8.x/api/python/nn-pytorch.html#global-pooling-layers) or DiffPool (https://github.com/dmlc/dgl/blob/master/examples/pytorch/diffpool/model/tensorized_layers/diffpool.py). 
4. I do not agree with the paper's justification in lines 270-272 that AUROC is the most proper performance metric, given for example that in the medical context a model needs to use a single threshold to decide on the positive/negative output, and AUROC tests all thresholds in practice. Besides, the model is trained such that the highest AUROC performance on the validation set is attained, which could artificially inflate these results and we do not know how they affect other very important metrics in the medical context like sensitivity, specificity, or even model calibration. I request the authors to at least add values on sensitivity and specificity (or equivalent) to show the strengths of this work. 
5. In Table 1 I find it difficult to understand how fair the comparisons are. Are these models similar in terms of number of layers? Or number of parameters? Given BrainNetTF seems to have had some sort of hyperparameter search, it is probably unfair to compare with out-of-the-box models?


## Other small weaknesses/suggestions

I order my comments in this section approximately by the order they appear in the paper. 

1. In lines 34 and 37, references [41] and [29] appear at the beginning of the line, which the authors can easily avoid in latex by using the tilde character ("\~"): `text~\cite{...}`.
2. Figure 1, pane B, indicates that orthonormal bases are able to distinguish nodes that were indistinguishable in non-orthonormal bases. However, it seems to me that those points were already distinguishable before? What am I missing here? Also, do these points correspond to points in the dataset (like pane A), or are these just for illustration purposes?
3. I like the highlight in lines 84-85 (and in other places in the paper) that ABIDE is the only dataset "fully" accessible online. However, I do not see this as a big issue in practice: most of the brain datasets like ADNI, PPMI, HCP, and NACC are quite easy to be accessed, and the logistics in place are more as a general control given we are talking about sensitive human data. Most of the times there's preprocessed data available as well.
4. In section 2.1 maybe the authors will like to know about this (possibly) interesting review on GNNs: arXiv:2105.13137
5. Line 111: Typo in "STOA".
6. From figure 2/section 3.1, I can see that the authors used a symmetric weighted matrix as input. Given the authors are using the Transformer model, have the authors thought of using only the upper-half of the matrix as input to reduce model complexity? Or maybe skip this step altogether with the models I mentioned in my point (1) from major weaknesses?
7. In line 145 the "Connection Profile" is capitalised. Authors probably want it non-capitalised (as in line 139)
8.  Typo in line 187: missing "initialization" after "Xavier uniform".
9.  Line 207 has a paragraph which starts as "The second method". I believe readibility would be improved if the method is actually specified here. 
10. Section 3.3.2 seems misplaced. It mentions possible generalisations of OCRead to other domains, but this is not explored in the paper. Wouldn't this be better fit in a discussion section or in conclusion?
11. RQ3, in line 244, uses the term "interpretability", but it seems to me the authors are actually looking to the term "explainability" as (for example) defined in 10.1016/j.inffus.2019.12.012. I am aware these terms might be used in different ways so I'd ask the authors to include a reference to which framework they are using.
12. In section 4.1, I think it is important to mention which tools were used, specifically DGL and pytorch from what I saw in the code (and reference accordingly).
13. Typo line 249: "Rs-fMRI" is not capitalised.
14. In line 263 and forward, is rs-fMRI also used from the ABCD dataset, or some task-based one?
15. In Table 1, why was a t-test performed (e.g., under what normality assumptions?)?
16. Given a statistical test was used in Table 1, why is there no statistical test in table 2?
17. The paper seems to provide an overly optimistic interpretation of figure 3. Sometimes variation in plot is very small (in the order of 1-2%), but the authors seem to ignore this fact? Maybe the authors could caution about this interpretation as, from what I see, a Random initialisation could be preferred for situations where model complexity can be a problem and a non-significant drop in performance is ok.
18. Typo in line 339: it is RQ3, not RQ4?
19. In Figure 4, I find it difficult to agree with the statement that orthonormal initialisations produce more discriminative values. The way I see it, I think the authors mean that visually (to the human eye) we can see some divisions on the *ordered* set of features, but that doesn't mean necessarily that the variability seen in the Random part of ABIDE is less discriminative (maybe the groups are there, not just ordered). I think this section needs to do more to be able to prove what is said.

questions: The five points in "Major weaknesses" are the ones that made me not give a higher score, therefore I believe those are the points which could change my opinion the most on the scores given.
limitations: The paper contains predictions using human brain data, with some labels being sex and diagnosis. Therefore, I find it incorrect that the paper does not follow the recommendations on writing about societal impact when the prediction of these labels in specific can have so many different impacts in society.  I believe the acknowledgement of these variables would be the bare minimum needed for a NeurIPS paper (maybe starting from the last sentence of conclusion).

With regards to limitations, I believe they were tackled properly in Conclusion and in other sections of the paper. However, as a consequence of the answers to my questions this will be be better after the rebuttal process.
review_title: null

Meta-review: metareview: This paper was reviewed by four reviewers. Three reviewers have participated in the discussions and they are all finally convinced and provided positive recommendations. The fourth reviewer was negative about this paper with some concerns. The authors provided very detailed rebuttals, but the reviewer did not respond to rebuttals though repeated reminders. I checked the comments and rebuttals and tend to believe most of the concerns of this reviewer have been addressed, at least to a large extent. Thus I recommend this paper to be accepted at this point.
confidence: Less certain
Decision: Accept