Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper addresses the need for graph out-of-distribution datasets and a systematic graph OOD benchmark. They curate 8 datasets: GOOD-HIV (predicting HIV inhibition from a real-world molecular dataset), GOOD-PCBA, GOOD-ZINC (predicting constrained molecular solubility), GOOD-CMNIST, GOOD-Motif, GOOD-Cora (predicting publication type from a citation network), GOOD-Arxiv (predicting subject area from a citation network of CS arXiv papers), and GOOD-CBAS (classifying node types in motif-like graphs). 

They provide a total of 14 different domain selections. Each domain selection can be combined with a covariate shift (which they refer to as “diversity shift”), concept shift (“correlation shift”), and no shifts to obtain 42 different splits. They create concept shift splits. On synthetic datasets, they set a domain-output correlation and generate graphs where a specific output is highly correlated with the domain feature according to this correlation.

They evaluate ERM and 6 other major domain generalization baselines (GroupDRO, VREX DeepCORAL, IRM, Mixup, on these datasets), and show that current OOD algorithms do not significantly improve generalization abilities, emphasizing a need for graph-specific OOD algorithms.
strengths: GOOD provides a graph OOD benchmark to facilitate further research in robustness for graph neural networks. GOOD contains real-world, semi-artificial, and synthetic datasets. Each dataset offers both covariate and concept shift settings, which the authors state is unique among existing OOD benchmarks. 

The authors evaluate several OOD methods (including 1 graph OOD specific method) on all datasets.

The authors provide a script to load GOOD datasets and training and evaluation pipelines.

weaknesses: Though GOOD is proposed as a graph OOD benchmark, the authors only evaluate one graph-OOD specific algorithm in baseline evaluations, Mixup.

The concept shift splits may contain covariate shift.

The real-world datasets do not come from diverse domains (most are molecular prediction tasks).
correctness: The evaluation methods and experiment design are designed appropriately and performed correctly.

A minor comment: line 47 should be “WILDS [21] collects real-world data from wild” not “… from wilds”


clarity: Overall, the paper is written fairly clearly.
relation to prior work: Yes. The authors propose several OOD datasets specific to graph neural nets, and they include a few non-synthetic, real-world datasets in their benchmark. They modify existing graph datasets.
additional feedback: See above.
review_title: Graph OOD benchmark that requires more graph-specific baselines in evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper targets the lack of GOOD benchmarks and curated a GOOD benchmark from different datasets. The motivation is good and the proposed datasets should contribute to the community.
strengths: 1. Targets the lack of GOOD benchmark, which is a meaningful direction.
2. The explanation of the problem background is pretty clear and is very friendly for people not familiar with this field.
weaknesses: The analysis on the concept in Section 3.2 seems problematic, which will be detailed in the following.
correctness: 1. When P(Y|X) does not change as in the covariate shift circumstance, it is reasonable to say there may exists X_inv so that P(Y|X_inv) doesn't change. But in concept shift scenario, in which P(Y|X) does change, it is not convincing to still assume the existence of such an X_inv that guarantees P(Y|X_inv) doesn't change. For example, suppose we do find X_inv in the concept shift scenrio so that P(Y|X_inv) doesn't change, then what if we deliberately apply the concept change to P(Y|X_inv)? It seems this operation still satisfy the definition of concept shift that P(Y|X) changes while P(X) does not. Or more seriously, why cannot P(Y|X) just change for every possible X?
Besides, as mentioned in line 95, P(Y|X)=P(Y|X_inv), which means X_var does not change the conditional probability of Y, then why would the shift of P(Y|X) be caused by P(Y|X_var) as mentioned in line 118? Is there any mathematical difference between the causal correlation and the spurious correlation in terms of the P(Y|X_inv) and P(Y|X_var)?

2. Concept shift seems to be same as the commonly known concept drift (I cannot find concept shift anywhere on the Internet), what is the point to change the name?

clarity: Above average.
relation to prior work: Relation to previous works is discussed.
additional feedback: Minor problems:

In line 44, Domainbed -> DomainBed
review_title: Good paper but with some potential major mistakes

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This work presents a comprehensive out-of-domain generalization benchmark for graphs. The proposed GOOD benchmark contains 8 datasets on node/graph tasks with covariate and concept shifts. Moreover, a series of experiments have been conducted to demonstrate the in-distribution and the out-of-distribution performance gap.
strengths: + The OOD problem is important to the graph learning community.
+ Both covariate and concept shifts are considered.
+ Comprehensive experiments have been done to demonstrate the ID/OOD performance gap and state-of-the-art OOD models performance.
weaknesses: - This work extends existing datasets (e.g., OGBG-PCBA in WILDS) by additionally considering the concept shift. While I agree that this work makes a comprehensive benchmark on OOD problems, the real-world datasets are all based on very limited domains, namely molecular graphs and citation networks. I still feel the significance of this contribution is a bit incremental.
- Some domain splitting could be further improved to add practical values to the evaluation.
- It seems that the adoption of existing datasets do not strictly follow license of prior works. See my detailed comments in the following sections.
correctness: Most datasets are adopted from existing datasets. For semi-synthetic and synthetic datasets, as far as I can see they are constructed in a technically sound way.
clarity: The organization of this paper can be improved. Firstly, the definition of covariate and concept shifts should be put in prior to related work, so that readers could avoid possible confusion in the terminology. Secondly, the idea of environments look a bit vague to me. In Section 3.3, the connection of the OOD problem with causal inference models can be elaborated further.
relation to prior work: Most datasets are adopted from prior work. The novelty of this work feels limited to me.
additional feedback: 1. It could better to consider evolvement over time as another factor of variance. For example, the Pubmed database provides a unique ID of each molecule that is closely related to the year when it is found. OGB also follows this way of splitting molecular graphs.
2. The authors may further justify the selection of degree as domain factors in the Cora and arXiv datasets. Is this of particular semantic meaning in practice?
3. Also, the authors may elaborate on the practical values of using the CMNIST dataset. It appears that very few graph models will cope with such graph datasets converted from images.
4. I do not see any relationship between the baseline models in Section 5.2 and graph learning models. It feels to me Section 5.2 is general to any existing machine learning models and is deviated from the graph domain.
review_title: Official review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper developed GOOD, an OOD benchmark focusing on covariate and concept shifts for graph data. The authors provide carefully designed data splits to create different shifts and non-trivial performance gap, making it suitable for node and graph classification. Extensive experiments have been conducted on 8 datasets with up to 42 different splits using mutiple baselines such as VREx, GroupDro and Mixup, which demonstrates significant gaps caused by OOD and further sheds light on the future research trend of graph-based OOD topic.

strengths: 1. The motivation of systemmatically investigating the impacts of different graph OOD scenarios is clearly described. This paper seems to be the first to propose well crafted split design for different shifts easily applied to general datasets.
2. The benchmark covers a wide range of datasets from various domains. The thorough evaluations also provide insightful suggestions such as shift-sensitive model architecture and task-related data augmentations for future graph OOD research.

weaknesses: 1. The math notations in concept shift derivation are slightly abused. To align with the design in Figure 2, the notation in line 122 should be P(Y = y_{j} | X_{inv}, X_{var} = d_i). Subsequently, it is more rigorous to rewrite equation (2) as P_{c_{k}}(Y | X_{inv}, X_{var} = d_{i}) = \sum_{j=1}^{|\mathcal{Y}|} following the definitions stated in line 132. 

correctness: Yes
clarity: Yes, easy to follow.
relation to prior work: Yes, the authors provide thorought discussion with related works in the main body.

additional feedback: See above.
review_title: GOOD: A Graph Out-of-Distribution Benchmark

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a new benchmark GOOD, which focuses on the graph out-of-distribution learning task, where the distribution of the training set is different from that of the test set. Specifically, the benchmark consists of 8 datasets and 14 domain selections on the node classification task and the graph classification task. The authors consider both covariate and concept shifts and obtain 42 different splits. The experiments demonstrate significant performance gaps between in-distribution and out-of-distribution settings.
strengths: This benchmark, to the best of my knowledge, is the first one to focus on graph out-of-distribution learning tasks. The datasets cover covariate and concept shifts. This could be a new direction in the graph learning domain. The experiments show the effects of out-of-distribution on performance and motivate future works to propose approaches to narrow the performance gap. The datasets are accessible on GitHub and well documented.
weaknesses: 1. One of the major concerns is the motivation for this benchmark. Specifically, there exist several benchmarks for OOD. It is unclear what the difference between the traditional OOD and the OOD on the graph is. This should be explained better because it leads to the motivation of this work. More specifically, what is the role of the graph on the covariate and concept shifts?
2.  It is much better if the authors can compare the difference in data generation for concept shifts and covariate shifts for real-world, semi-artificial, and synthetic datasets.
3. Existing datasets only cover two types of graph learning tasks. I am happy to see more tasks involved such as link prediction.
correctness: The datasets seem not particularly novel. They are derived from existing datasets via different splits of domain selections. But the performance evaluation on the performance gap between ID and OOD is convincing.
clarity: The paper is overall well-written and clearly presented. One quick question is how to choose the training samples and test samples. Since nodes in the graph are not IID. It is unclear whether this may make some difference.
relation to prior work: In the related work section, the authors clearly show the differences between their benchmark and existing benchmarks. One quick question is whether they may do the same on other domains rather than graphs. This question is related to the unclear motivation of this work.
additional feedback: Please reply to the concerns in the above section, and I will raise the score if the authors address my concerns properly.
review_title: A Well-written Paper but Motivation Can be Improved

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper aims to develop an OOD benchmark for graphs. By dividing distribution shifts on graph into two types, the authors use existing datasets to artificially inject corresponding shifts. They evaluate existing OOD methods mostly from Euclidean data and demonstrate the IID-OOD generalization gap in the curated benchmark.
strengths: The authors systematically divide and discuss the distribution shifts into two categories, i.e., covariate shift and concept shift, to differentiate this work from other benchmarks for OOD generalization on graphs.
weaknesses: The main weakness of this work is the lack of graph specific discussions. In fact, throughout the paper, there are few specific designs and discussions about the unique challenges and distinctions of OOD generalization on graphs, which weakens the significance and novelty of the benchmark as it may not reflect the difficulties of OOD generalization on graphs. See more details below:
- The discussion about relevant works are limited. This paper has many overlaps with [1] and takes similar split strategies existing in the literature [1,2,3,4,5]. Why would the community prefer the proposed benchmark in this work instead of [1]? More discussions and comparisons are expected.
- The usage of causal languages in this work seems to be confusing. Given no causal graphs presented, how to determine the invariant features and variant features? Why the variant feature is causally irrelevant to the labels? If so, how would they correlate with the labels? The authors are suggested to refer [3,6] as well as the invariant learning literature for precise use of the causal languages. 
- The definition of covariate shifts and concept shifts seem to be problematic too. For example, covariate shifts require P(Y|X) invariant across different environments/domains, but the introduction of X_inv and P(Y|X_inv) invariant, makes the P^tr(Y|X)=P^te(Y|X) confusing.
- In graphs, how to select invariant and variant features by index?
- This work seems to have weak relevance to graphs. The design of distribution shifts (Sec. 3) actually applies to both regular data and graph data, and has a poor reflection with the curation process of the datasets. For example, what kind of the scaffold, degree, size shifts belong to and why? 
- This work does not benchmark any existing graph OOD methods, making the OOD gaps suspected. If the existing OOD methods on graphs [3,4,5,6] can solve the OOD shifts introduced in this benchmark already, developing follow-up works based on this benchmark seems to be uninteresting.

==== Post Rebuttal ====

I thank the authors for the detailed reply. Although they didn't fully resolve my concerns, I am convinced that this benchmark could be a good initial testbed for OOD algorithms on graphs. Thus I'd like to raise my score to Weak Accept.

My remaining concerns are still about the weak relatedness to graphs and the design of covariate and concept shifts as partially admitted by the authors, but I also admit that the novelty and contribution of the current design outweigh its weakness.

===================

References:

[1] Mucong Ding, Kezhi Kong, Jiuhai Chen, John Kirchenbauer, Micah Goldblum, David Wipf, Furong Huang, and Tom Goldstein. A closer look at distribution shifts and out-of-distribution generalization on graphs. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.

[2] Yuanfeng Ji, Lu Zhang, Jiaxiang Wu, Bingzhe Wu, Long-Kai Huang, Tingyang Xu, Yu Rong, Lanqing Li, Jie Ren, Ding Xue, et al. DrugOOD: Out-of-distribution (OOD) dataset curator and benchmark for AI-aided drug discovery–a focus on affinity prediction problems with noise annotations. arXiv 2022.

[3] Beatrice Bevilacqua, Yangze Zhou, and Bruno Ribeiro. Size-invariant graph representations for graph classiﬁcation extrapolations. ICML 2021.

[4] Yingxin Wu, Xiang Wang, An Zhang, Xiangnan He, Tat-Seng Chua. Discovering Invariant Rationales for Graph Neural Networks. ICLR 2022.

[5] Qitian Wu, Hengrui Zhang, Junchi Yan, David Wipf. Handling Distribution Shifts on Graphs: An Invariance Perspective. ICLR 2022.

[6] Yongqiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, Kaili Ma, Binghui Xie, Tongliang Liu, Bo Han, James Cheng. Invariance Principle Meets Out-of-Distribution Generalization on Graphs. arXiv 2022.


correctness: The languages used to describe distribution shifts on graphs are confusing. The experiments are not convincing given the lack of benchmarking existing OOD methods on graphs. See the Weakness for more details.
clarity: The paper is well-organized but hard to follow, given the confusing descriptions of the distribution shifts.
relation to prior work: The discussion are limited. See the Weakness for more details.
additional feedback: Overall, I believe this work has the potential to be impactful, given the lack of a systematic benchmark of OOD generalization on graphs. 
- However, the current version still requires more revisions to introduce more distinctions of OOD generalization on graphs compared to that on regular data. 
- The languages to describe the distribution shifts seem to have conflicts with the literature on covariate shifts/concept shifts, and invariant learning.

The authors are suggested to refer [6] for an in-depth discussion of the two points above.
review_title: A potentially impactful work but reuqires more improvements 

Meta-review: metareview: This paper proposed a benchmark for out-of-distribution generalization in graph data. The problem is interesting and the authors made a lot of efforts in addressing the reviewers' comments. I do think the authors addressed most of the reviewers' concerns. In addition, graph OOD benchmark can indeed benefit the community, which is still underexplored in existing benchmarks, such as WILDS, MetaShifts, NICO, and Shifts.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept