Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new benchmark for evaluating GNNs under heterophily. Adjusted homophily and label informativeness are used to characterize new datasets from diverse domains. Extensive experiments are conducted to evaluate various GNN methods on these new datasets, and results show that standard GNNs outperform heterophily-specific GNNs.
strength_and_weaknesses: Pros:

1. This paper comprehensively summarises the related work on GNNs under heterophily.
2. This paper analyses the drawbacks of the widely used benchmark (Pei et al. 2020) and performs experiments to validate the effect of removing duplicate nodes.
3. The novelty of the proposed new datasets is reasonable to me (using different homophily measure; from diverse domains).

Cons:

Regarding another recent non-homophilous graph benchmark (Lim et al. 2021), this paper only mentions its scale. I expect a more detailed comparison with it (e.g., what are the characteristics of those datasets? Are they suitable for evaluation under heterophily?). This may help people differentiate them and make them complement each other.
clarity,_quality,_novelty_and_reproducibility: Please see Strength And Weaknesses
summary_of_the_review: This paper will be of interest to the non-homophilous GNN community
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Studying heterophily of graphs and their potential impact of learning GNN models is an important area of research. However, the datasets currently used for evaluating the various models could have potential biases and issues. This paper identifies one such issues with these datasets and finds that mitigating the issue causes several SoTA models for heterophily to do poorly. The paper then goes ahead and creates new heterophilic datasets which have wider variety in terms of domains, sizes and heterophily. The paper shows that simple modifications of GAT architecture on these datasets work better than most of the SoTA models for heterophily.
strength_and_weaknesses: # Strengths:
1. The paper is clearly written and easy to follow.
2. The raised issue of duplicates is very interesting.
3. The reported results significant gap in performance.
4. The new datasets presented show wide variety in domain, size, heterophily and class balance.
***
# Weaknesses:
1. There is no reported statistics on the filtered graphs. 
2. GPRGNN can be reduced to SGCN model [A]. Typically SGCN model is quite close in performance to GCN. Also, note that for homophilic datasets, GPRGNN results are on par with GCN. Additionally, GPRGNN was also evaluated on synthetic benchmarks built from stochastic block models which are not likely to have the duplicate issue. It showed good performance even on those. All of this suggests that GPRGNN model should be at least as competitive as GCN on any dataset. So, it is quite odd that GCN gets 40.70% on filtered squirrel whereas GPRGNN is at 33.57%. Similar issues are present in results reported for the new datasets. This makes the reported numbers less believable.
3. The paper does not report whether there are duplicate issues in the new datasets.
4. The wiki-cooc dataset already report pretty high numbers. Despite having low homophily, GCN numbers are already pretty high suggesting quite possibly strong node features. So, the value of the dataset is somewhat unclear. Same with minesweeper dataset.
***
# References:
[A] Simplifying Graph Convolutional Networks, Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr., Christopher Fifty, Tao Yu, Kilian Q. Weinberger, ICML 2019.
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow, but there are several things that need clarification:
1. How connected is the filtered graph? 
2. Do all the duplicate node also have the same label?
3. What fraction of the duplicates are present in train, val and test? If you take the original graph and run any of the models, then split the test set into duplicate nodes and non-duplicate nodes, and measure performance independently, what is the performance split that you obtain? The paper claims memorization as the possible cause of degradation, it will be good to verify this. 
4. Is the filtered graph directed or undirected? If directed, is it converted into undirected graph before using in these models? These models cannot handle directed graphs and making the graph symmetric using $A + A^T$ helps.
5. How is the train/val/test created for the filtered graph? Do you create a fresh train, val, test split or simply remove the duplicate nodes from the previous train, val, test split?
6. Are all the baseline models tuned for filtered graph or the previous parameters are utilized for measure the filtered graph performance? These models have sensitivity to hyper-parameter tuning, thus it is important to understand.
7. Are there duplicates in the new dataset? If yes, what fraction?
8. The wiki-cooc and minesweeper dataset already reports pretty high numbers for baseline model like GCN. Is there any way to quantify value of these datasets. For example, an upper bound on performance that might be achieved on these datasets? That will help understand the value of these datasets. Same can be done even for other datasets for uniformity.

If the above questions are clarified and If the reported numbers are properly verified and double checked, then the paper shows a very interesting finding and proposes several new interesting datasets on which newer model that can address heterophily can be built. This will be of great help to community. 
summary_of_the_review: Overall, the paper two important points:

1. The current heterophilic datasets have issue of duplication. And that current research using these dataset is flawed because of it. However, the numbers reported for various models are not quite believable. And it would be worthwhile to double check and verify the correctness of these numbers. Also, some more analysis on whether duplicates are present in large proportions in train and test and not only in large proportions but possibly skewed proportions. Any other analysis, in addition, would be helpful. If all of this leads to the claim that memorization is truly the cause of drop in performance, then the numbers might be more believable.

2. The paper presents 5 new datasets. However, for two of these datasets, the numbers for base models like GCN are already pretty high. So, the value of these datasets for heterophilic graph neural network research is unclear.

Based on these, I am inclined to mark this paper as marginally below acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of GNNs under heterophily. Different from previous methods, this paper investigates the issues from the dataset perspective. The authors analyzed the issues in five widely used heterophilous datasets and provided some empirical results to illustrate these issues. To solve this problem, a list of new benchmark graphs from different domains as been constructed for evaluating GNNs under heterophily.
strength_and_weaknesses: Strengths:
- Research on Graph neural networks that are able to capture heterophily is an important research problem. This paper studies this problem from a different and novel perspective, i.e., analyzing the issues in these widely used datasets.
- Based on issues of some widely used datasets for heterophilous GNNs, several new benchmarks have been proposed.

Weakness:
- What is the evaluation metric used in Table 2, Accuracy or ROC AUC?
- When introducing Minesweeper and Questions datasets, what's the meaning of additional binary features? Will this feature influence the quality of the dataset, i.e., the correlation between such features and labels can be guaranteed?
- The author mentioned the possible issue in Cornell, Texas, and Wisconsin datasets, i.e., imbalanced distribution. I suggest adding some empirical analysis to better illustrate the impact of such imbalance on previous methods similar to Section 3.1.
- Although the authors claimed an existing benchmark [1] is different from this one ([1] is designed for scalability), more discussions on the differences can be provided from the perspective of data construction.

[1] Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods
clarity,_quality,_novelty_and_reproducibility: Clarity: This paper is well-organized and easy to follow.

Quality: This paper is of good quality with detailed discussion and comprehensive experiments.

Novelty: Studying GNNs under heterophily from the perspective of datasets is novel.

Reproducibility: The constructed new benchmark graphs are not provided so the reproducibility could be a question.
summary_of_the_review: This paper studies an interesting and important problem, i.e., GNNs under heterophily. Different from previous methods, this paper investigates the issues from the dataset perspective which is novel and interesting. Although benchmark datasets for similar tasks have been proposed, this paper shows some differences from the existing datasets. Thus, I will recommend a weak acceptance of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper examines the existing heterophilous data and finds out that they exist some issues. Afterwards, the authors propose several new data. Evaluations show that standard GNNs achieve competitive results outperforming most of the specialized models on heterophilous graphs.
strength_and_weaknesses: The strong points of this paper include:
1) This paper is well-motivated and clearly written. The authors examine the datasets and existing methods for heterophily.
2) Five new datasets are developed for heterophily.
3) The evaluation is comprehensive.
The weaknesses include:
1) This paper doesn't provide any new algorithm. The authors just evaluate existing models on their data.
2) The baselines should include some recent works that work on both types of data. For example, JacobiConv, GBK-GNN.
clarity,_quality,_novelty_and_reproducibility: The authors investigate the recent progress on the heterophily issue. The authors find that the widely used benchmarks have serious drawbacks. Based on the newly developed datasets, the authors point out that standard GNNs achieve competitive
results on heterophilous graphs outperforming most of the specialized models. Overall, this paper is well-written and easy to follow. However, there is no new algorithm. Moreover, it seems that the new data are not publicly available.
summary_of_the_review: My concern about this paper is that there is no new algorithm. The main contribution of this paper lies in the new datasets. The authors point out the drawbacks of current data, which could be meaningful for the community.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors of this paper reveal an unexpected point about existing heterophilous datasets---that one of the key points making them challenging for classical GNNs may be issues such as duplicated nodes in the data. They follow up by testing their method on "cleaned up" versions of these benchmarks, and including additional tasks with comprehensive evaluation. All reviewers agreed this method is valuable, and after thoroughly discussing the pros and cons with them, I am confident that this paper should be accepted. More details may be found in the AC-Reviewer meeting summary.
justification_for_why_not_higher_score: While this method would have potential for a spotlight, the authors would have probably needed to conduct a significantly more thorough study of the effects they discovered in order to warrant it, in my opinion.
justification_for_why_not_lower_score: My opinion, concurred by other reviewers, is that this work passes the bar for ICLR, and will be valuable to several groups within graph representation learning. I am absolutely certain about this assessment.
Decision: Accept: poster