Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for graph-based learning from brain imaging datasets, where they 1) pretrain GNN using 2-level contrastive learning, which leverages some brain network knowledge, under a MAML optimization approach , and 2) perform a data-driven atlas transformation with various regularizations and variance sorting to provide some alignment of ROIs across different datasets. The method is tested using 3 datasets, where the large PPMI dataset is used for the pretraining and the transfer to task learning is tested on the other 2 datasets. Experiments are performed comparing multiple baselines with and without pretraining and studying effect of model components through ablation studies.
strength_and_weaknesses: Strengths:
1. The method considers leveraging additional brain network information not previously considered in GNN pretraining - using samples from corresponding node positions in other graphs as positive examples for the anchor node. This seems like a reasonable way to define more positive samples utilizing other graphs for contrastive learning.

2. The approach considers the problem of combining brain imaging datasets that used different parcellations and thus have different graph node configurations.

3. The experiments have meaningful comparisons to baselines separated into different groups (e.g., no pretraining, baselines using different levels of node/graph representation in contrastive learning) to help understand the improvements in model decision.

4. There are several ablation experiments included to see the effect of different parts of the design approach.


Weaknesses: 
1. The motivation for the data driven atlas transformation is to get a common graph configuration for all the datasets, since different datasets may use different parcellations. However, what if one fixed atlas were used and applied to the existing imaging data - this is not compared? Is the idea that it is not always available? Furthermore, my understanding is each dataset transformation is learned separately - then how can the transformed space be aligned? I understand that the "virtual ROIs" are set according to some variance ranking, but this does not mean the ROIs would be aligned across datasets - as seen in Fig. 6 which is supposed to demonstrate overlap, I would argue that the ROIs do NOT look aligned. Then, if these ROIs are not aligned, does the pretraining of a model to use on a different dataset make sense?

2. It is unclear to me if the atlas transformation is learned separately from the pretraining step? And whether each transformation mapping is learned separately for each dataset? Also, is it based on all the data, or trained using just training data? It would be unfair using all data as the representation is then fit to the test data as well.

3. In comparing the proposed method to other methods with pretraining, I presume this is with the proposed atlas transformation, while the comparison method does not have any such mapping estimated - is this correct? Or are other methods also learned using the atlas transformation? If not, then it is unclear the contribution of the proposed contrastive learning formulation vs. the atlas transformation. If could perform the atlas transformation for comparison methods, then can more fairly compare just the contrastive learning for pretraining. Or, do not include any atlas transformation, and then compare the contrastive learning approaches.

4. Some of the experimental details are confusing to me. For example, the plots in Fig. 5 show pretraining and learning on the task dataset for 150 epochs, while the appendix says pretraining goes for 400 epochs and the task learning for 200 epochs. In Fig. 5b, it seems that the curves for all the approaches may not have converged yet, so the comparison is not quite fair, even though at that epoch (which is not 200) the proposed method is most accurate. 

5. At a higher level, I wonder whether expecting the same parcellation of brain regions for fMRI vs DTI data makes sense - it may be that the natural groupings based on fMRI data may not align with natural groupings based on structural DTI data. Yet both these types of datasets are simply combined for analysis. Could the authors comment on this?


clarity,_quality,_novelty_and_reproducibility: Quality:
It is clear the authors have put a lot into the evaluation of the proposed method. Some implementation and dataset details are in the appendix. However, there are no citations for the datasets used in the main text (and limited info is found in appendix)- are these public or private, where are they from, were they properly collected with IRB approval, etc. At a minimum citation for the datasets need to be included in the main paper if they have already been previously introduced elsewhere.

Clarity:
The organization of sections is good, and the writing is generally ok. However there are some details that need to be clarified (see above weaknesses).
Some additional notes on writing: the references are all within the text which is difficult to read - they should be in parenthetical form (eg \citep)

Originality:
The contribution for the GNN pretraining is that the contrastive learning framework considers both node and graph level information, using nodes in a k-hop neighborhood of the target node i in other graphs in the dataset as positive samples. Other parts of the pretraining are from prior work (MAML optimization, definitions of pos/neg sample within graph, cost function).
I think the originality of the transformation mapping portion lies in the specific combination of regularizers applied, as I believe these have been used before. 

summary_of_the_review: While the work largely builds off of existing prior work, the new approach to defining positive samples for contrastive learning of GNNs for brain datasets is interesting and makes sense, and a number of experiments have been performed to demonstrate the potential advantage of the proposed methods. However, some clarifications are needed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an unsupervised GNN pre-training technique for brain networks. The authors first designed a two-level contrastive learning method and then propose a data-driven atlas mapping technique for mapping different ROI systems. The method was evaluated in three brain network datasets. The results showed the proposed method outperformed other pretraining frameworks in disease classification problems.
strength_and_weaknesses: The second work of this paper about brain atlas mapping is problematic. The author states that they need to convert networks between different ROI systems because the brain networks in the two datasets will have different numbers and semantics of nodes. However, to train the model, you do not need to convert the network between different brain templates, you only need to generate data with the same brain template. Because raw MRI image data is voxel-based, an easy way is to generate a brain network based on the same brain template. So, I do not see a reasonable motivation for using deep learning methods to solve these simple problems. The authors may argue that the available data are based on different brain templates and need to convert. However, keep in mind that brain networks are just an analytical representation of neuroimaging data, and different networks can be generated based on different templates and parameters. Therefore, it is simpler and more accurate to generate a unified brain network directly from raw image data using the same brain template. 
clarity,_quality,_novelty_and_reproducibility: The authors seem to lack some knowledge about brain disease and neuroimaging data analysis, so many disease-related analyses are inaccurate. For example, in the second sentence of the abstract, the authors state that ” Recent studies in neuroscience and neuroimaging analysis have reached a consensus that interactions among brain regions of interest (ROIs) are driving factors for neural development and disorders.” It is not true. There is still much debate about the role of neuroimaging, especially functional brain networks based on fmri in brain disease. Conclusions from existing MRI imaging works are also difficult to replicate.(Botvinik-Nezer, R., Holzmeister, F., Camerer, C.F. et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582, 84–88 (2020). https://doi.org/10.1038/s41586-020-2314-9)
In 3.3.2, the authors state that “Therefore, it is reasonable to assume a shared virtual ROI system underlying different parcellation systems”. There is no basis for this statement. The authors believe that different diseases and different parcellation systems may share the same virtual ROI system. This hypothesis is difficult to be convinced unless the author has a large amount of clinical data to support it.

summary_of_the_review: The second motivation of this paper is problematic, so it is not suitable for published in ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposed data-driven brain atlas mapping and contrastive pretraining for brain network analysis. The authors show improved results over previous baselines, ablate the proposed method based on different strategies, and perform a visual examination of ROI alignment. Contrastive learning is based on Jensen-Shannon divergence with positive and negative pairs defined based on the graph. The data-driven brain atlas mapping incorporates an autoencoder with brain-oriented regularizers based on locality, brain modules, and sparsity. In addition, variance-based dimension sorting is proposed to align ROIs of the different dataset and their atlases.
strength_and_weaknesses: #### Strength
- The experiments were performed on multiple datasets.
- There are multiple graph baselines.
- There are ablation studies for choosing contrastive sampling strategies and atlas mapping regularizes.

#### Weaknesses
- It is not clear what statistical significance test has been used. The t-test should not be used as you can use it only if you pass the normality assumption. Further, you need to report statistics in a valid format. In most scenarios in Table 2 and Table 4, the variance is so high that most performance metrics should overlap, so I don't think the metrics should pass the significance testing. In addition, have you explored the reason behind such a huge variance?
- There is no final holdout-test set, only 5-fold cross-validation. The validation set has to be used only to select checkpoints or hyperparameters.
- Alignment of the virtual ROIs is only done visually. Consider segmentation measures (e.g., DICE (F1)) or other measurable methodology to evaluate the performance of data-driven brain atlas mapping concerning the original atlas. 
- There are no non-graph-based baselines or discussions. 
- There is no discussion on related work for Neuroimaging domain:
  - That describes the use of GNNs or pretraining:
    - Mahmood, Usman, et al. "Multi network InfoMax: A pre-training method involving graph convolutional networks." arXiv preprint arXiv:2111.01276 (2021)
    - Mahmood, Usman, et al. "Deep Dynamic Effective Connectivity Estimation from Multivariate Time Series." arXiv preprint arXiv:2202.02393 (2022).
  - That describes data-driven brain atlas mapping (e.g., via spectral clustering):
    - Geenjaar, Eloy, et al. "Spatio-temporally separable non-linear latent factor learning: an application to somatomotor cortex fMRI data." arXiv preprint arXiv:2205.13640 (2022)
- Could you clarify how the matching of the nodes for a positive pair between the source datasets is enforced? It seems that nodes should be paired based on spatial alignment. How do you guarantee that nodes are spatially aligned to the same brain region after the autoencoder?
- I am unsure if you used the same MAML framework for all the baselines. Could it be the main reason for the improvement? Could you compare your model by pretraining on two datasets and perform transfer learning by fine-tuning on the third?






clarity,_quality,_novelty_and_reproducibility: #### Clarity
- Does the number of subjects in the dataset reflects the number of images? And it is not clear how cross-validation has been performed. It must be performed subject-wise to ensure that the subject is only in one subset. 

#### Quality
- There are some issues with technical sophistication as there is no hold-out set and no methodology to evaluate the data-driven atlas mapping.
- The authors should address huge variance issues in the performance metrics.

#### Novelty
- The proposed solution seems novel.

#### Reproducibility
- The statistical significance testing details are missing.

summary_of_the_review: Overall, I like the proposed approach and comparison with multiple baselines. The proposed framework seems to improve over previous baselines. However, I have some minor issues that I want to clarify. 

Further, I think the authors should propose some methodology for the data-driven atlas mapping, as the visual evaluation does not seem enough.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose to leverage an unsupervised pre-training strategy to capture the intrinsic brain network structures regardless of specific clinical outcomes for obtaining GNN models that are easily adaptable to downstream tasks. The method utilizes large-scale brain imaging dataset that may not be related to a specific target task to improve the target task, and a data-driven parcellation atlas mapping pipeline is proposed to interpret knowledge transfer across studies with different ROI systems. 
strength_and_weaknesses: Strength:
- The proposed method can propose a globally sound initialization for a graph neural network (GNN) to be trained. 
- The proposed method let one combine multiple datasets to come up with the initial representation.
- Experiments on various datasets demonstrate that the method improves training of other GNN module. 

Weakness:
- The authors mentioned that obtaining a large-scale brain imaging dataset is difficult but the model is proposed assuming that a large-scale dataset exists for training the proposed method. 
- Problem definition is not formally written. According to the introduction, as $D_i$ are all different, I assume that dataset sizes are different across $D_i$ but all of them are presumably given as $N$. It is not clear if there exists one $A_j$ per dataset, and the objective of GNN $f()$ (i.e., input and output) is also not clear. Moreover the description of $\theta^{t+1}$ seems not right with $L_{query_i} f(\theta\prime_i^t)$ as the $f()$ should have a graph $G$ as an input and multiplying L and f doesn't make sense; please correct me if I am wrong. Is there any description of measurement on the nodes? 
- Lack of problem description lead to confusion in the method section. Contrastive learning is performed with $X$ but it is not clear what the $X$ is. Is this measurements on the graph nodes? 
- At the beginning of 3.2, X was described as a sample. Then, $\bf S$ is named as an arbitrary sample but eq (2) sums over $\bf S$ where $X_S$ are its elements and not samples which is causing confusion. 
- S1 and S4 are considered as positive samples but S4 is a subset of S3 which is a set of negative samples. Is there a reason why the data were divided into subsets such that they contradict to each other? Also, there is no equation or loss function explaining how these subsets are contrastively trained. 
- It is not clearly explained how the authors deal with dataset heterogeneity. 
- A summary table of individual dataset can be helpful.  
- Ablation study is performed on the variants of subsets for contrastive learning. A reader perhaps would be more interested how much each regularization term designed by the author is contributing to the performance. 

clarity,_quality,_novelty_and_reproducibility: - Quality of writing is quite poor and it is difficult to judge if this work is really novel or not. In a high-level sense, I think simply getting a good initialization doesn't sound very interesting but it can be better presented if mentioned as learning better initial representation.
- Reproducing the overall framework will not be easy based on the paper itself. 
 
summary_of_the_review: The proposed model has some merits as shown in the performance improvement but there are many places where the description of the framework are not very clear. The authors should better demonstrate their ideas such that this paper can be read and understood by a broader audience.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper suggests a method for pre-training GNNs for brain connectivity using a) Using 2-level contrastive learning to pretrain GNNs using prior knowledge about brain networks, and b) performing a data-driven atlas transformation to provide alignment across datasets.


Strengths:
The use of the contrastive learning to encode prior knowledge seems novel and useful


Weaknesses:
The second alignment part of the algorithm seems unnecessarily complicated and weakly motivated -- why did the authors not just use a single atlas for alignment? There were seeveral concerns regarding clarity. Finally, the neuroscience hypotheses and conclusions drawn were found to be problematic in comparison with the existing literature.
justification_for_why_not_higher_score: All reviewers recommend rejection.
Decision: Reject