Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this paper, the authors propose a benchmark for graph neural architecture search. Based on a tailored search space, they evaluate different search algorithms of existing methods on graph neural architecture search. They further give analyzes of the benchmark in terms of different aspects.

This is the first work to design a benchmark for graph neural architecture search and extensive experiments are conducted and comprehensive analysis is given, which can help us better understand the progress of this topic. Further, it can provide a database of GNN architectures for quick look-up, which can accelerate the following-up research of graph neural architecture.
strengths: Please check the above contribution part.
weaknesses: 1. The search space is kind of smaller, and the datasets are only for node-level tasks, thus may limit the usage of the benchmark.

2. In Section 4, it will be more interesting if the authors can analyze the influences of differentiable search algorithms beyond RL and EA.

3. Some related works are missing, e.g., 

### Refs
1. Qin et al., Graph Neural Architecture Search Under Distribution Shifts. ICML 2022
2. Wei et al., Designing the topology of graph neural networks: a novel feature fusion perspective. WebConf 2022
3. Cai et al., Rethinking graph neural architecture search from message-passing. CVPR 2021
4. Zhao et al., Search to aggregate neighborhood for graph neural network. ICDE 2021
5. Want et al., AutoGEL: An Automated Graph Neural Network with Explicit Link Information. NeurIPS 2021
correctness: Good.
clarity: The presentation of the paper is very good.
relation to prior work: Yes.
additional feedback: None
review_title: An interesting work

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This article presents a benchmark for Neural Architecture Search aiming at giving a baseline protocol to efficiently compare different architecture, in a reproducible manner, and a easily usable database for any practician, mimicking the progresses made in other NAS domains.
It presents its way of searching an efficient architecture using 7 base and typical Graph Neural Network (GNN) layer, and tests them on 9 open datasets, also well-known and relevant in GNN research, using different approaches (namely evolutionary algorithms and reinforcement learning).
Another part presents insights on the distributions for the models, with their performance, how much each was used. Some additional information is provided on the correlation between datasets.

strengths:  - Following a procedure that has worked in other fields is very sound and good practice
 - Can be very useful for many practical purposes
weaknesses:  - Not completely clear/insisting on the main point of the article which seems to be the benchmarking of different methods, as it seems that there is not much novelty on the different methods of NAS
correctness: The article seems constructed in a sound way, and is extensive on the methods used to create the data for the benchmark
clarity: It is well-written, but it feels like it gets a bit lost in technicality and loses a bit of clearness on its main point.
relation to prior work: Related works are presented in part 2 quite clearly.
additional feedback: Are the best structures for the similar datasets (from the Cross-Dataset Correlation part) correlated too ?
review_title: Review for NAS-Bench-Graph

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The authors propose NAS-Bench-Graph, a tailored benchmark for GraphNAS. The authors trained and evaluated different GNN architectures on nine graph datasets. The authors showed that the proposed benchmark is compatible with existing GraphNAS libraries such as AutoGL and NNI.
strengths: The studied problem is novel. The authors tried to benchmark GraphNAS algorithms, which haven’t been studied by anyone else.

The proposed benchmark is compatible with existing GraphNAS libraries such as AutoGL and NNI.

weaknesses: The studied problem is novel, but not important/interesting. Different NAS algorithms might have different performances on different problems/tasks/datasets. Considering the NAS is usually used to find the best model structure for a specific problem/task/dataset, what are the advantages of benchmarking different NAS?

The author should compare with GraphGym [1], considering this is one of the most important and most common-used GraphNAS libraries, and a lot of designs in this paper are inspired by GraphGym, e.g., the search space.

The authors only conducted experiments on node classification tasks. Link prediction and graph classification are also important tasks in graph learning.

The authors just generally compared the “evolutionary algorithm” and “RL algorithm” in the paper. However, there are a lot of specific GraphNAS baselines that the authors ignored and didn’t compare [2, 3, 4].

Lack of analysis and insights from the experiments. What are the take-aways from benchmarking different NAS algorithms?


[1] You, Jiaxuan, Zhitao Ying, and Jure Leskovec. "Design space for graph neural networks." Advances in Neural Information Processing Systems 33 (2020): 17009-17021.

[2] Gao, Yang, et al. "Graphnas: Graph neural architecture search with reinforcement learning." arXiv preprint arXiv:1904.09981 (2019).

[3] Zhao, Huan, Quanming Yao, and Weiwei Tu. "Search to aggregate neighborhood for graph neural network." arXiv preprint arXiv:2104.06608 (2021).

[4] Shi, Min, et al. "Evolutionary architecture search for graph neural networks." arXiv preprint arXiv:2009.10199 (2020).

[5] Zhou, Kaixiong, et al. "Auto-gnn: Neural architecture search of graph neural networks." arXiv preprint arXiv:1909.03184 (2019).

correctness: Good.
clarity: Fair. The authors should provide more illustrations and clarifications regarding the motivation and analysis.
relation to prior work: The authors claimed that they were the first to benchmark GraphNAS algorithms, but didn’t compare with a lot of specific algorithms. Please see the comments above.

additional feedback: Please see the comments above.
review_title: ok but not good

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: **Edit 08-25**: I increase my score from 7 to 8 as the authors addressed all my concerns and acted upon my suggestions for improvements. I  am also happy about several improvements in response to other reviewers / the AC, especially including a discussion on limitations in the main text. Concerns raised from other reviewers are either not convincing to me or adequately addressed by the authors.

---

The paper presents a tabular NAS benchmark for GNN architectures spanning nine graph datasets. The tabular data includes several metrics and will allow future research to study NAS for GNNs in an efficient and principled fashion. The authors also use their benchmark data to derive insights such as different graph datasets requiring different architectural choices for best performance.

strengths: 1. **Impact and relevance to NeurIPS community**: NAS researchers in general will find the benchmark useful and build on it. NAS for GNNs in particular has seen several works in NeurIPS and other top conferences, so NAS-Bench-Graph is highly relevant to the NeurIPS community.
2. **Originality**: NAS-Bench-Graph follows many previous tabular benchmarks for NAS, it is, however, the first such benchmark for NAS for GNNs and the first NAS benchmark for the graph modality. The benchmark features a macro and micro component for the search space, which not many benchmarks do.
3. **Accessibility and accountability**: The main benchmark data is hosted on github, with the complete data available on Figshare with its maintenance guarantees etc. The authors provide code to read the data in the common data format of a python dictionary. Further, some documentation is provided in the form of a README that shows the basic usage of the benchmark. License for code and data is given. I did not try running the code.
4. **Miscellaneous**: NAS-Bench-Graph features a large number of (varied) datasets (9) compared to many other NAS benchmarks, which allows for more comprehensive benchmarking. Performance metrics are recorded for all epochs, allowing, e.g., also multi-fidelity algorithms to be benchmarked with NAS-Bench-Graph. The paper includes several insightful analysis that provide a good intuition of the benchmark characteristics and, e.g., add evidence for the surprisingly strong performance of random search. The authors include a detailed explanation for the systematic setting of hyperparameters.
weaknesses: 1. **Accessibility and accountability**: The code that was used to create the benchmark data is not open sourced. The same goes for the experiments and empirical analysis. The python dependencies for the benchmark are not specified. Ideally, NAS-Bench-Graph could be installed via pip from github or pypi. Installation instructions in general are not provided.
2. **Empirical protocol**: The empirical evaluation does not quantify meassurement error, e.g., by providing error bounds, making it hard to assess how statistically sound their analysis is. This is acknowledged in the checklist which I appreciate. Further, the empirical evaluation only concerns final performance after ~200 evaluations and not full optimization curves.
correctness: * The claims made in the submission are correct and sound.
* The experiment design is lacking in some aspects as described under weaknesses.
clarity: 1. The paper is clear in its writing, well organized, and supported with helpful figures and tables.
2. Some background on GNNs is missing, as the benchmark is relevant to NAS researchers in general and not just researchers familiar with GNNs.
relation to prior work: 1. Related work on NAS for GNNs is covered well.
2. A more comprehensive comparison to existing NAS benchmarks, e.g., in the form of a table in the Appendix would help, but the main discussion on how this work differs from previous contributions in its data modality is sufficient.
3. The paper does not include a discussion on the drawbacks and positives of tabular NAS benchmarks compared to surrogate benchmarks.
additional feedback: 1. A discussion on potential negative societal impacts was crossed as yes in the checklist, but I did not find it.
2. Can make github links clickable in paper
3. Do not say "we do x as in [11] but "we do x as xy et al. [11]"
4. The choice of HPs involves optimizing for random samples and common architectures, which introduces a bias. This is a minor point and HPs have to be chosen somehow.
5. How much compute resources did go into creating this benchmark and how costly is it to run a training from scratch for a typical architecture?
6. Figure 5 and 6 are not legible without zooming in, please fix this. Ideally the font should be the same size as the main text.
review_title: Accept with some more room for improvement in the rebuttal

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: To address two key challenges in GraphNAS 1) unified and reproducible experiments 2) excessive computation requirements. The authors propose NAS-Bench-Graph, a benchmark for graph neural architecture search (GraphNAS) including data points from 26K GNN architectures on nine prevalent graph datasets. In addition, the authors conducted detailed analysis such as the visualized distribution of accuracy, latency, the number of parameters, and the compatibility of NAS-Bench-Graph to existing NAS algorithms.  The authors also demonstrate the effectiveness of NAS-Bench-Graph by searching GNN architecture with open-sourced libraries on NAS-Bench-Graph and report the performance of the searched network.
strengths: +Clear GraphNAS search space abstraction and benchmark construction procedure.
+Detailed and intuitive analysis of the architecture space.

weaknesses: The authors said NAS-Bench-Graph contains 26,206 different architectures, and a detailed explanation of how the number is derived would be helpful.

Typo in line 30: “...key challenges that seriously hind the”, hind->hinder.

correctness: The claims made in this paper are reasonable to me.
clarity: This paper is well written and easy to follow.
relation to prior work: I am satisfied with the discussion on the relation to prior work.
additional feedback: N/A
review_title: An interesting and useful benchmark for graph neural architecture search 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper presents a NAS for GNN benchmark, that is fair, comprehensive, and reproducible. The authors provide a framework for GNN NAS unifying the codebase hyperparameters, datasets, and evaluation protocols. They propose a new search space and conduct analysis and visualization on the searched architectures and their transferability.

strengths: + The motivation of this benchmark is good, which calls for a unified and fair benchmark for GNN NAS algorithms. 

+ The proposed benchmark is first presented in GNN NAS domain, to the reviewer’s best knowledge.

+ The visualization Fig. 2 and 4 provide good insights to analyze the latency-performance trade-off and cross-dataset architecture transferability.

weaknesses: - Most of the experiments are done on their proposed search space, which may vary across different methods. This seems to be a mismatch with the goal of a benchmark, which is often designed to compare different methods under a fair experimental setting.
- The chosen datasets are not diverse enought. Only homophily data are considered. It would be better to consider heterophily data.
- The benchmark paper does not compare with enough baseline methods, such as SANE, GASSO, AutoCoG.

correctness: As far as the reviewer understood, the claims are correct though the goal of a benchmark somewhat mismatches with what this paper presents.
clarity: The paper is well written and easy to follow.
relation to prior work: This paper is the first GNN NAS benchmark as far as the reviewer knows. But some related baselines is missing.
additional feedback: 1. The macro search space presents some multi-branch architectures. How do authors combine their outputs?
2. This paper presents random, EA and RL based methods. Why differential architecture search based methods are not presented.

The major limitation of this paper is the insufficient compared baselines. Most of the experiments and discussion are done on their proposed search space and methods. As a benchmark paper, the reviewer may suggest including the comparison with more existing methods. Also, it is recommended to involve more heterophilic datasets.
review_title: Good analysis and codebase, but needs benchmark more existing methods.

Meta-review: metareview: This paper introduces the first NAS benchmarks on graphs. Ratings were quite diverse, with scores of 4,5,6,7,7,8.
The many positive reviewers highlighted that GraphNAS is very relevant and that a benchmark for them would be very useful for the community. The benchmark includes comprehensive evaluations on as many as 9 different datasets, which may also help facilitate research on meta-learning for (Graph)NAS. Initially, the code for creating the benchmark and analysis was not available, which is a no-go for tabular NAS benchmarks, but the authors fixed this in the rebuttal period.
The most negative reviewer, 8asK, appears to not be familiar with tabular NAS benchmarks and their usefulness, and did not react to the discussion about it, as well as to the modification of the paper to highlight it. The other borderline negative review, by reviewer LkTe, had as their main ciriticism that NAS methods should be benchmarked on many spaces, but the point of tabular NAS benchmarks is *not* to provide a conclusive assessment of the performance of NAS methods that holds in general across search spaces (as a "horserace" paper might try to achieve), but rather to facilitate the cheap evaluation of current and future NAS methods on a single search space. For these reasons, I do agree more with the positive reviewers and recommend acceptance of this work as a poster.


confidence: 5: The area chair is absolutely certain
Decision: Accept