Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors present VesselGraph, a whole brain dataset of murine blood vessels. Vessels are rendered volumetrically from whole brain image volumes and graph representations of the vasculature are extracted. The authors train baseline geometric deep learning models on the dataset for link prediction and node classification tasks and make the dataset available in formats commonly used by the geometric deep learning community.
strengths: * This is a meaningful problem and this work sets a foundation for the adoption of geometric deep learning by the connectomics community. We can expect a huge increase in the volume of whole brain datasets available in the coming years and robust graph extraction techniques are a solid contribtion.

* The datasets are well documented

* The authors are thorough in their baseline experiments.
weaknesses: * More details on the registration proceduce would be useful. Whole brain image registration is a non trivial step in generating this dataset and it is not clear how the registration was performed (eg rigid/affine?). 

* Allen brain regions have hierarchical organisation (as the Authors acknowledge). It is likely that this can provide meaningful signal if leveraged appropriately. It would be useful for the authors to make this available . Furthermore, subregion labelling may also prove useful to users in addition to the 71 regions discussed.

* Dataset curation - SEAL. The authors present a negative sampling strategy based on the average length in the graph. This is a reasonable start (and is in response to the obvious random sampling problem which they acknowledge and try to address). However, this seems very coarse - really this should be tuned to a more nuanced strategy, for instance using the average length in a given brain region as a first step in this direction. This should be a straightforward modification, given the this information is already available.

L144: Missing space: inSection
correctness: The paper seems correct and claims well-supported. The authors are thorough in discussing the limitations of their work. The baseline evaluations would benefit from repeated runs.
clarity: Overall the paper iswell-written. Figure two, whilst pretty, is not especially information dense and takes up a lot of real estate that could be used to expand on the methodolgy. The graph extraction protocol (L128) can be more more detailed. Table 2 would benefit from the use of boldface to indicate the best results & improve readability
relation to prior work: Previous work is discussed adequtely.
additional feedback: I would like to commend the authors for their interesting work. This was an enjoyable paper to read.
review_title: A good contribution to the geometric deep learning and computational neuroanatomy communities.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This work introduces a new dataset describing blood vessels in the mouse brain based on whole-brain imaging. 
The authors use several existing graph learning algorithms to describe the dataset.
This dataset is useful for those interested in blood vessels in the brain. 

strengths: This is a novel dataset, it is well documented, and is based on sound methods of data collection.
The authors also provide benchmarks for link predictions and for classification of blood vessels into capillaries, arterioles/venules and arteries/veins. 
weaknesses: No weaknesses noted. 
I am not aware of other similar datasets and benchmarks. This might reflect my own ignorance, perhaps other reviewers are more familiar with other comparable datasets and benchmarks. 
correctness: The evaluation methods, experiment design and data collection are all sound and rigorous. 
clarity: The paper is very well written. 
relation to prior work: I am not aware of other similar datasets and benchmarks. This might reflect my own ignorance, perhaps other reviewers are more familiar with other comparable datasets and benchmarks. 
additional feedback: None
review_title: Cool dataset of mouse brain blood vessels

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper extracts a set of standardized whole-brain vessel graphs based on murine brain images. The dataset is easily accessible in widely-used OGB and PyTorch Geometric data loaders. Benchmarking experiments are conducted on link prediction and node classification.
strengths: (1)	This paper presents a novel whole-brain vessel graph dataset, which can potentially promote the research in graph learning and brain analysis.
(2)	The dataset is easily accessible in widely-used OGB and PyTorch Geometric data loaders.
(3)	Benchmarking experiments are conducted on link prediction and node classification.

weaknesses: This paper did not discuss the data bias in the presented dataset. Considering data bias always occurs in human-constructed datasets, it is worthy to discuss these bias, such unbalance among different genders, regions and races.
correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback: (1)	Is there data bias in the proposed dataset?
(2)	Only link prediction and node classification are benchmarked, is the graph classification task also useful for the benchmarking of vessel graph dataset? Because I think graph classification is more related to disease diagnoses.
review_title: A novel dataset for graph learning with applications in brain analysis.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The dataset presents unique challenges for both brain connectomes and graph neural networks (or general graph theory). The extraction method could also be useful for future brain image datasets. The manuscript could benefit more by addressing some of the minor concerns related to the graph generation methods detailed in later sections.
strengths: **1. Uniqueness:** The dataset is pretty unique and would be useful for graph study of the brain connectome. \
**2. Graph extraction method:** The manuscript considered and compared multiple venues of extracting the graph from the data source. \
**3. Demonstrate the use of GNN:** The author benchmarked the dataset with fairly up-to-date GNN methods.
weaknesses: **1. Data Exploration:** For such a new dataset, I would really love to see preliminary data exploration and stats of it, e.x. distribution vessel features, number of nodes/edges, distribution of other node/edge features, stats when stratified by brain regions, and especially those related to graph measurements, such as centrality measures, to name a few. This would also help to visualize imbalance inherent in the data. \
**2. Clarifying Imbalance in the Dataset:** The manuscript mentions the dataset results in a large imbalance dataset but did not go into much detail about it. This is critical and should be discussed/visualized more. \
**3. Lack of model explainability/insight:** Given that the introduction eluded drawing insights from vessel graphs, the authors could consider using at least some GNN model explanation to explain the model results. \
**4. The simulated graphs are not used or mentioned:** It is ambiguous what the use of the included simulated graphs is. Its characteristics compared to the real ones are also not clear. \
**5. Metrics should address the imbalanced nature of the dataset:** As the dataset is imbalanced, at least AUPRC should also be reported in Table 1.
correctness: Most of the claims in the manuscript are acceptable and the author also noted the limitations of the generation method on vessel features. However, some of the claims/experimental design should be clarified, such as
1. The claims in the paragraph starting in line 41 could be toned down as, while the dataset is novel, it is still far from those diseases and it is unknown how pathologies in different diseases, such as tangles/plaques, will affect the graph generation method. 
2. Why is a degree 2 nodes are generated when the vessels exhibit large curvature?
3. In line 171, the node is assigned to a certain region if x, y, z are all in the cth region of the brain atlas. But these coordinates could vary by sample, what is the region assigned if, say, x and y in one region, and z in another?
4. Experimental setup for predictions need more clarification, ex. how is the train/test set split performed in that one graph?
5. Is there any reason why cartesian coordinates are preferred over polar? Intuitively, I would think polar coordinates would be better at distinguishing, for example, left/right hemisphere.
clarity: The manuscript is very well-written and is easy to follow despite being a rather specialized topic.
relation to prior work: It is pretty clear that this work contributed by extracting graphs from previous works on vascular iamges.
additional feedback: All suggestions (if needed) have been written right after the comment in each section.
review_title: Unique dataset for benchmarking of GNN for brain connectomes

Meta-review: This paper introduces a novel whole-brain vessel graph dataset, which is likely to be very useful for the graph learning community. All reviewers gave acceptance scores, several of them very enthusiasticly so. Clear accept.
Decision: Accept