Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper collects graph datasets that are not as homophile as the regular benchmarks for GNNs. Furthermore, it shows that a rather simple new baseline achieves quite good results compared to much more involved models / the SOTA. The paper presents important contributions to the community, is well and thoroughly written, related work is covered adequately, the experiments are overall thorough, and I have few to criticise. Altogether, I suggest to accept the paper.

main_review: As outlined above, I have few to critisize, I summarise the most important contributions below.

- The collected (and sometimes adapted) datasets represent a useful set to move research in the field forward.

- The paper actually introduces a new metric to measure homophily which is quite interesting since it overcomes issues with existing ones. I would strongly suggest to move parts of all the explanations in the appendix into the paper, especially the descriptions illustrating the latter issues. I personally consider this metric contribution at least as important as the new model.

- The proposed approach is simple but effective which is quite nice, and the experiments over wiki show that such a solution was needed.

- The appendix contains much additional information and insights.

-------------------------------------------------------------
Questions & Smaller Comments
-------------------------------------------------------------

I wonder if the prediction of the year for the arxiv and patent data is a realistic task. Why would someone care about this in particular, and is it even possible to do this (ie do we expect enough variability in the data to derive such conclusions)?
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes 7 new large heterophily graph datasets (up to 2 million nodes) and also provides a simple, scalable yet effective baseline LINKX which combines the information from topology and node features.
The paper extensively evaluates the datasets on various baselines, including typical GNNs and GNNs specifically designed for heterophily graphs. They find that many baselines cannot compete with the simple baseline LINKX and some of them run out of memory.
It also points out the minibatching problem and provides thorough results for fullbatch and minibatch training.

main_review: I appreciate the authors' effort in proposing large-scale heterophily graph datasets and along with a strong baseline LINKX.
Recently, many researchers have looked at the performances of GNNs on heterophily graphs but most of them are testing their models on very small heterophily datasets. So this work will potentially have a great impact on the community.
The authors not only provide various baselines (including typical GNNs and GNNs specifically designed for heterophily graphs) but also provide thorough results for fullbatch and minibatch training.

This paper is well written and easy to follow.
limitations_and_societal_impact: I have a few questions regarding the new datasets.
1. From Table 3, it seems that GCN, in general, performs well than MLP in your heterophily datasets. It means, unlike the experiments did in small heterophily datasets, GCN can utilize graph structures in the new datasets. In the H2GCN paper, it mentions a similar situation for chameleon and squirrel datasets and points out that it is because those two datasets contain many nodes that share common neighbors. I am wondering for the heterophily datasets you provided, does this phenomenon occur in your datasets: nodes of same label share less connections but they share much more common neighbors than nodes from different classes. Can you also provide information on the 2-hop heterophily level, that is the portion of nodes from the same class in the 2-hop neighborhood but not the 1-hop neighborhood?
2. Recent works also find degree distribution affect the performance of node classification task. I am wondering if you can also provide the information on degree distribution on those datasets?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies the performance of graph methods on heterophilous datasets. In particular, they propose several new graph datasets, a method for such datasets, and an evaluation of many graph methods on these new datasets. The paper is written in a clear and thorough manner, there is an exhaustive list of references and introduction makes a good job of motivating the problems studied in the paper. 
main_review: The paper's contributions can be split into three parts: new datasets, new method, and empirical comparison. 

1. I have strong concerns about the datasets and their possible misuse in the future. In particular, 4 of the proposed datasets are social networks or alike, which could be used to train machine learning models for assessing properties of individuals or the communities involved in the network. Besides, the tasks of predicting properties such as gender could lead to the situation when the trained models discriminate certain groups of people. As one of the main paper's contributions are the new datasets, I am worried that such datasets could be used in many other future related works that would eventually lead to unfair treatment of people by the machine learning models. Creating a precedent in this work, could also lead other works to include datasets that mistreat people's interests. I think there is [a related track](https://neurips.cc/Conferences/2021/CallForDatasetsBenchmarks) on benchmark and datasets, where this work can get more support and feedback on the validity of the proposed datasets. 

2. The authors propose a new method called LINKX, which builds on a well-known method LINK. However, the previous LINK method was not used with node features and the authors propose to combine LINK embeddings with MLP embeddings on the node features. This allows the algorithm to scale easily to large datasets such as Wiki, whereas GNNs need more complex sampling solutions such as graphSAINT. 

3. The empirical comparison is done nicely across the proposed datasets and many graph models including vanilla GNNs, heterophilous GNNs, MLPs, LP, and others. However, I would like to see the performance of LINKX on other datasets, homophilous and heterphilous (see [here](https://arxiv.org/abs/2102.06462) for example) for the fair comparison. 

After rebuttal: Given a significant discussion between authors, reviewers, and ethics committee I tend to increase the score from 4 to 6. I still believe this paper would make a better fit to the dataset track of NeurIPS 2021. 
limitations_and_societal_impact: As discussed above, one of the consequences of proposing real-world social networks and corresponding machine learning tasks of predicting properties of individuals (such as their gender) might be the misuse of the trained models on such datasets as well as promoting other works to propose datasets with sensitive information. As the field of GNNs mature, I can easily see how end users in the future use these models and datasets for the applications that treat unfairly certain groups of people (e.g. in credit scoring, institution admission, etc.).
review_title: null

Meta-review: The paper proposes a new bechmark for graph neural networks for non-homophilic graphs. This is an important aspect currently missing in existing datasets. The reviewers were generally positive about the paper, and most of the review comments raised ethics concerns which in the opinion of the AC have been properly addressed by the authors. The authors provided detailed responses in the rebuttal. We recommend acceptance. 
Decision: Accept (Poster)