Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a new benchmark to understand the performance of GNNs on graphs with different characteristics, which are class sizes distribution, the relationship between classes and topology, the relationship between classes and attributes, and graph sizes. The authors leverage a synthetic graph generator that can generate graphs having controlled characteristics above as references to real world graphs. Rigorous and comprehensive benchmarking experiments using a variety of representative GNN architectures are conducted on those graphs, demonstrating some interesting insights with respect to graph characteristics.
strengths: - The paper is well-organized and clearly written, with clear motivations of existing evaluations not considering varying characteristics among graphs.
- The selected graph characteristics are useful. Further study on those characteristics will be beneficial for understanding GNNs better.
- The experiments are rigorously conducted with clear details of experimental setups.
weaknesses: The evaluations designed and conducted for the proposed 4 graphs characteristics are meaningful overall, but there are several limitations that can be further improved.
- For $\rho^{\mathrm{conf}}$ and $M^{\mathrm{conf}}$, it is not easy to see how varying the hyperparameters will change those values or change how the graphs look like. Providing some simple examples (e.g. $\rho^{\mathrm{conf}}$ for each class with certain $\alpha$s, visualization of a demo graph with only <10 nodes and edges) will be highly appreciated.
- For graph size experiments, the gaps between different $(n,m)$ pairs are too small. The efficiency and performance changes of a certain method will be much more significant when it is run on two graphs largely different in sizes e.g. cora and ogbn-arxiv, revealing some meaningful limitations. I would suggest authors at least add a graph with 100000 nodes, a size of the same magnitude as ogbn-arxiv (~170000 nodes), and it is acceptable to just put on a note (e.g. OOM or unfinished after 48 hrs) for methods that are not scalable.
- It would be interesting to see how Figure 3 (different attribute-class correlations) looks with 1 or 2 more examples from other domains than citation networks to study how attribute values affect the performance i.e. changing $H^{\mathrm{Cora}}$ to $H^{\mathrm{(a \ non \ citation \ network \ dataset)}}$. 

correctness: The evaluation methods and experiment design appropriate and performed correctly mostly, with a few limitations mentioned in the above weakness session.
clarity: Yes, the paper is well written.
relation to prior work: The literature review on GNNs for node classification, datasets and benchmarks, graph generators are comprehensive. The unique contributions of this work are highlighted clearly.

additional feedback: As evaluated nonhomophilous GNNs do not utilize the topology information in the most heterophilic settings well, only achieving marginal improvements to MLP, it will be interesting to add **LINKX** and investigate how it performs on those synthetic graphs (especially the heterophilic settings), as it represents a simple fusion between the MLP learned node features and LINK regression learned graph topology.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper presents an empirical study of GNNs on node classification tasks by using a synthetic graph generator with controlled characteristics. Four major dimensions of such characteristics are discussed and are used for the controlled experiments. The paper conducts extensive experiments using such a setup and presents insights on how different settings affect the performance or capabilities of a GNN model.
strengths: -The paper writing is easy to follow.    
-Several limitations on the existing use of datasets for node classification and the challenges in developing fine grained analysis with those are discussed, which forms the motivation of the paper.  
-The method adopted or proposed in the paper can be used to test future GNNs' abilities in the corresponding 4 characteristics as stated.  
-Extensive experiments are performed on the characteristics stated by using numerous GNN models that represent the progress of the field.   
weaknesses: -Some existing work in this regard, such as GraphWorld [1] or the synthetic datasets used in [2], [3], among others, already use synthetic datasets for fine grained analysis or testing some particular graph properties. In such case, the use of synthetic datasets for deeper investigation is not novel to some extent.   
-The objective of using a synthetic graph generator to test specific graph characteristics which may not be readily available in real-world dataset has already been explored in [1]. For example, the case of higher or lower homophily is explored in [1]. In this regard, the contribution of this paper seems limited.     
-If graph generator is somehow a contribution of the paper, it is already presented in a prior work. 

Hence, even though the insights presented after the extensive experiments seem interesting, the contribution of the paper look limited and not novel, and researchers may rely on the prior generator tool to assist in their experiments. Overall, this paper seems to be an experimental demonstration, instead of a framework presented, as far as this track is concerned.

[1] Palowitch J, Tsitsulin A, Mayer B, Perozzi B. GraphWorld: Fake Graphs Bring Real Insights for GNNs.    
[2] Corso G, Cavalleri L, Beaini D, Liò P, Veličković P. Principal neighbourhood aggregation for graph nets.    
[3] Vignac C, Loukas A, Frossard P. Building powerful and equivariant graph neural networks with structural message-passing.   

Question:
- In line 206-207: "In fact, existing studies use only accuracy to evaluate the classification quality since they do not consider class size distributions." => I understand that the metric is altered other than accuracy to account for class imbalance generally in existing works. Are there particular set of references which support this statement in line 206-207?
correctness: The evaluation methods seem correct and appropriate, such as using appropriate metrics for evaluation, among others.
clarity: Yes the paper is fairly well written, except that Section 3 is longer than required and may not be needed in detail in such a paper.
relation to prior work: Mostly covered, except for [1] which is referenced in above in "Weaknesses" field.
additional feedback: see "Weaknesses" field.
review_title: An empirical demonstration with experimental insights instead of a framework

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors provide a dataset based on the GenCAT graph generator, for the purpose of evaluating node classification by Graph Neural Networks. They vary four characteristics of the graph: class size distribution, edge connection proportions, attribute values, and graph size, and provide benchmark results using 16 GNN techniques.

I feel that this paper accomplished its goals, and makes a good contribution. The authors make good use of the space provided. For this reason, I'm giving it a rating of "accept".
However, as described in "Weaknesses", I am still unsure if the benchmark is extensive enough for inclusion in the track.
strengths: - A synthetic dataset (with suitable tuning parameters) and benchmark for GNNs is a valuable contribution.
- A good collection of results comparing an impressively large set of 16 GNN algorithms.
- The authors provide insight (based on experiments) regarding which techniques work well in different settings.
- The code appears solid, and reproduces raw data and plots found in the paper
- Observations regarding the four sets of results provide at least some insight into the strengths/weaknesses of the GNNs.
- There are some results in the supplemental giving insight into the dataset's characterisitcs.
weaknesses: Overall, I feel that this paper accomplished the goals it set. However, there are some additional aspects that could have been explored:
* More thorough analysis of the GNNs with respect to the benchmarks to better pinpoint why the GNNs do or not do well. The observations in Section 6.1 are nice, but don't provide deep insight into strengths/weaknesses. This would be difficult due to time and space limitations, since many GNNs were used.
* The benchmark is only based on characteristics from the Cora dataset. I think providing multiple benchmarks based on other datasets (like the Texas dataset) would provide a more comprehensive set of tests. To save space, the paper could just focus on analysis of the Cora dataset.



---
My original questions follow, which the authors since answered in their comments.

The choice of characteristics to study and tuning parameters seems somewhat ad-hoc. It would be useful if there was more insight into the choices. Specifically:
- Lines 144-147 say “Node classification results depend on four major characteristics of graphs having node labels…”. The paper does justify the importance of these four characteristics. But, what was the rationale for choosing these and not others? For example, why not vary class preference deviation to demonstrate core/border phenomena (see [20]). Why not vary number of classes, or degree distribution?
- Why was the Cora dataset used as a basis for the parameterizations? 
- Why was (3) defined in this way, as opposed to using some other distribution?
- For the homophily/heterophily results, how was (4) derived? Why not interpolate, let’s say, between M^Cora (which displays homophily) and M^Texas (the Texas dataset which displays heterophily).

I also find it difficult to develop an intuition for the benchmark graphs. Given that these are the core of the paper, I think this is important. The reader could study the dataset themselves, but I feel that a large value of the publication would be to provide the reader more insight into the characteristics of these graphs. I realize that there is limited space, but perhaps some detail could be put in the supplemental? For example:
- What exactly does the class size distribution (3) look like for Fig. 1. Perhaps you could show a 3D graph that shows how the size of each of the seven classes varies as \alpha is varied. I am also not clear on why seven classes implies that the graph is imbalanced for \alpha = 0.5, 0.7. Isn’t it always imbalanced if it is generated according to (3)? Why were 0.5 and 0.7 chosen as endpoints?
- What exactly are the values of M^Cora? Can you somehow quantify or show the edge density for the different values of \beta?
- What do the attribute distributions actually look like E.g., Figure 6 or 7 in [20]

There are only 3 or 4 datapoints in each of the plots. Was this due to time limitations? It would be nice if there were at least a few more datapoints each. 

-  It seems like understanding the generative model requires looking at [20] and/or the code. Is there a way to describe, let’s say, the attribute distributions exhibited by the benchmark (without having to add pages of detail)?

- What were the sizes of the train/validation/test sets? The supplemental lists the epochs for each algorithm, but not the size of the training set (unless I missed it).

- One concern: In [3], the cSBM model is used to generate graphs with both heterophily and homophily. Perhaps you should discuss how this compares with GenCAT? 
correctness: The work appears correct.
clarity: The paper is overall well written.

relation to prior work: The summary of prior work is good.



additional feedback: - For Section 6.1.2, would there be value in including a similar result but using a purely synthetic M matrix (not based on Cora), where \beta=0 would represent a graph with neither homophily nor heterophily (a so-called null model)? Negative values of \beta could generate heterophilic graphs, whereas positive values could generate homophilic graphs. I feel that this type of tuning parameter would be easier for researchers to use.

- I see that the datasets are available for download from Google Drive, which is great. Do you have a plan for ensuring these remain available in the long-term?
review_title: Fills an important need for evaluation of GNNs

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper argues using generated graphs to evaluate GNN performance fully. Making use of their previous work GenCAT as the graph generator, the authors develop a benchmark to test GNN’s performance with respect to 4 graph attributes: Class size distribution, edge connection proportions, attribute values, and graph size. From my perspective, the motivation that generating graphs to test GNN’s performance is incorrect, and the contribution of this paper is also very limited. 
strengths: I would like to leave it to further discussion.

weaknesses: 1. Incorrect motivation. GNNs are designed to cope with real-world graph-related problems, which should be tested on real data. From my perspective, testing GNNs performance on the generated graph is not a good motivation.
2. The contribution is not big enough. It seems like the authors simply adopt previous research GenCat to generate graphs as the benchmark. The contribution of this paper is not big enough.
3. How do authors identify the four characteristics? The topology characteristics (e.g. density)  are not studied. Topology structure is the core difference between graph data and other kinds of data.

correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: N/A
review_title: Incorrect motivation with not enough contribution

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper points out that the existing evaluation of GNNs lacks fine-grained analysis from various characteristics of graphs. Based on this, the authors conduct extensive experiments with a synthetic graph generator on different datasets and clarify the strengths and weaknesses of GNNs from four chosen characteristics.
strengths: This paper is well motivated. As the authors point out, the existing evaluation of GNNs lacks fine-grained analysis from various characteristics of graphs.  I applaud the authors' efforts in experiments. To clearly and comprehensively clarify the strengths and weaknesses of GNNs, they conduct quite enormous experiments including 16 GNNs with different features. The authors also use nearly every dataset commonly used in the node classification task. This paper has many contributions from both technical and conceptual perspectives. I applaud the authors' efforts to clearly divide the characteristics of real-world graphs into four parts and point out how to manually control the characteristics in details.


 
weaknesses: As the authors imply, they only conduct relative experiments on node classification tasks. The GNNs can be applied to various tasks including node classification, link prediction, etc. Large datasets like Reddit are not included according to the paper because of the space limit. Nowadays there are many sampling functions in PyTorch to do with large-scale datasets so experiments can be conducted with limited space. Also, as the main contribution of this paper, the figures are set quite small and indistinguishable. In Figure 1, the setting of the size is not sufficent enough compared to other figures.
correctness: Yes, I think the evaluation methods and experiment design appropriate and performed correctly.
clarity: Yes, I think the paper is well written.
relation to prior work: This work mainly focuses on four characteristics of graphs 
additional feedback: Anything to replenish.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: To analyze the impact of different chracterisics of graphs on  the strengths and weaknesses of real word GNNs,this paper proposes to use a synthetic graph generator to generate graphs with specified features. They mainly explored from four aspects:1. balanced or imbalanced class size distributions 2.homophilic or. heterophilicedge connection proportions between classes 3. biased or. random attribute values and small or large graph sizes  
strengths: This work briefly summarized four different characteristics  and their influencies on diffenent kinds of GNNs. They discover that  by generalizing to  heterophilic graphs , the networks  may perform better  classification in a heterophily setting from a graph-agnostic
 classifier MLP.
weaknesses: A variety methods has been adopted to solve the imbalance problems in positive and negative classes, such as negative sampling, and has achieved great performance in node classifiaction[1] . It maybe more appropriate and fair to adopt negativate sampling while making comparisons between different GNNs.



[1] Zhen Yang, Ming Ding, Chang Zhou, Hongxia Yang, Jingren Zhou, and Jie Tang. 2020. Understanding Negative Sampling in Graph Representation Learning. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20). Association for Computing Machinery, New York, NY, USA, 1666–1676. https://doi.org/10.1145/3394486.3403218
correctness: According to the author, there are two main claims made in the paper:
"GNNs including the state-of-the-art methods suffer from a class imbalance problem that typically deteriorates the performance of multi-class classification. Interestingly, the simplest algorithm SGC outperforms recent complicated GNN algorithms in a class imbalanced setting because the complicated algorithms tend to over-fit to major classes."
"GNNs generalizing to graphs with few edges within each class (we call heterophilic graphs),provide marginal classification performance gains in a heterophily setting from a graph-agnostic classifier MLP. This indicates that such GNNs almost ignore the topology information in the setting."

The above are plausible proposals and have some experiments with finetuned parameters as backups.

clarity: This paper clearly describes the experienment
relation to prior work: This work clearly discusses the differences between previous work
additional feedback: N/A
review_title: A Review of Beyond Real-world Benchmark Datasets: An Empirical Study of Node Classification with GNNs 

Meta-review: metareview: This paper proposes a benchmark for evaluating node classification of GNNs, considering several characteristics of the graph, including the class size distribution, edge connection proportion, attribute value, and graph size. The concerns lie in that this paper mainly builds the synthetic graph datasets without very significant differences or new insights from existing works. The motivations for "beyond real-world benchmark" stated in the title are also not very clear. I expect the author(s) could well address these concerns to improve this paper.

All in all, I recommend a borderline accept if there is room for the proceedings.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept