Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The Open Graph Benchmark - Large Scale Challenge (OGB-LSC) is a set of three large real-world datasets (between 55M and 1.7B edges) focusing on three different graph ML task types (node-, link-, and graph-level), and including the task metrics, competitive baselines, and state-of-the-art reference results (from Kaggle challenges winning entries). The datasets, baseline code, and documentation is also made available. The OGB-LSC fills a gap in the graph ML community by providing datasets and competitive results at a very large scale.
strengths: - The main strengths of the work are:
    - Providing three large, realistic datasets for ML graph tasks
    - Providing results of multiple competitive baselines on those tasks
    - Providing documentation and code to run those baselines
    - Providing results of state-of-the-art winning entries of Kaggle challenges on those same results
    - A clear, well written paper
weaknesses: - I couldn't find any relevant weakness.
- There is a typo on line 98 (annotating → annotate)
correctness: The datasets, benchmarks, baselines all appear correct.
clarity: The work is very clearly presented.
relation to prior work: The relation with prior work is clear.
additional feedback: Great work, congratulations!
review_title: A good set of realistic, large ML graph benchmarks, with datasets, baselines, and high performance results

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper introduces three large-scale graph datasets that are significantly larger than other existing datasets. Each dataset corresponds to different graph ML tasks: node level, link level, and graph level. Exploring how graph-based machine learning methods operate on large-scale data is essential for the field. However, there are few open benchmark datasets of the size comparable to the industrial data. This paper fills this empty room and, in addition to datasets themselves, brings a benchmark of the state-of-the-art methods of learning on large graphs.
strengths: 1. MAG240M, WikiKG90M, PCQM4M are actually large — this is, in my opinion, the main and solid contribution of this paper.
2. Each dataset has reasonable and realistic splits.
3. Carefully conducted baseline evaluation. The authors describe the hyper-parameter selection process and evaluate each method with different depth, hidden dimensionality, and other parameters. In addition, the authors include simple baselines such as graph-agnostic MLP and label propagation. Including these baselines allows evaluating the impact of using graph deep learning methods. Also, this evaluation might reveal peculiarities of methods for learning on large graphs and compares all the state-of-the-art approaches under similar realistic conditions. This contribution is crucially important for the graph ML community.
weaknesses: Although I think that the following do not downplay the paper's main contributions, I find it necessary to point them out to start a reasonable discussion.
1. The paper suffers from the lack of the related work review and comparison with other datasets. The authors state that the introduced datasets are a magnitude larger than the existing ones but do not provide a corresponding comparison.
2. The paper nor the supplementary materials do not provide the graphs' statistics such as graph diameter, connectivity, etc.
3. The provided datasets look like the (significantly) larger of the existing ones — citation networks, Wikipedia articles, that are significantly different from the industrial data, e.g., transaction networks. This issue does not downplay the impact of this benchmark but potentially could raise a concern about its applicability to the industrial conditions.
4. The nodes' features of MAG240M and WikiKG90M are obtained through a sentence encoder. This reveals several concerns: first, as was mentioned above, the real-world datasets often contain heterogeneous features such as categorical data; second, the benchmark is encoder dependent, but, as far as I know, the study on encoder choice was not conducted as well as fine-tuning the encoder along with the training baseline models. Also, it would be interesting to see pure text classification baselines here, like fine-tuning SOTA transformers.
5. Although the authors show the plans on publishing the 3D structure information for PCQM4M, this data is essential for learning on molecular graphs and is not presented in the current version. Many methods that incorporate the graph's 3D structure or angular information were developed (some examples are [1, 2, 3]) and showed their superiority over straightforward graph DL approaches and found applications in real-world tasks such as CASP MQA. So, it would be interesting to see a comparison of these methods on large-scale data, but it seems to be hard without having 3D structures for the test split.

[1] SchNet: a continuous-filter convolutional neural network for modeling quantum interactions. K. T. Schütt et al. NIPS’17

[2] Directional Message Passing for Molecular Graphs. J. Klicpera et al. ICLR 2020

[3] Spherical convolutions on molecular graphs for protein model quality assessment. I. Igashov et al. Machine Learning: Science and Technology, Vol. 2, Num. 4

correctness: I have not found any issues with correctness in this paper.
clarity: The paper is clear and well-written.
relation to prior work: The relation to prior work is clearly discussed.
additional feedback: No additional feedback.
review_title: Datasets and Benchmarks for Learning on Large Graphs

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper presents the OGB-LSC datasets, a large-scale dataset collection that focuses on node-, link-, and graph-level predictions. Baseline performance and interesting findings regarding experimental results have been provided. Future update plans are also discussed with preliminary observations.
strengths: + Large-scale evaluation of graph neural networks is a very important direction. The authors collect three datasets from various domains at different levels, which will facilitate development of algorithms for both academia and industry.

+ The collected datasets have been verified through a KDDCup conference that have attracted a lot of competitors around world, which demonstrates its real-world impact.

+ Detailed discussions of baseline performance, common practices, and winning solutions are provided.
weaknesses: I am overall satisfactory with the presentation by the authors. Please see my minor comments in the "additional question" section.

correctness: No technical flaws are spotted.
clarity: Yes.
relation to prior work: Large-scale graph datasets are lacking in current open literature.
additional feedback: - The authors may discuss how the data is cleaned and how to ensure the label quality, especially for the MAG240M dataset, where arXiv subject areas may not be always accurate and moderators often change the initial category chosen by the authors.

- Since there are different challenges for different datasets, I wonder whether the OGB team will adopt more evaluation metrics, not only limiting to performance. For example, I am in particular interested in resource-constrained scenarios for giant graphs, considering common practices like the neighborhood sampling are computation-intensive.

- Some key takeaways summarizing common practices could be provided at the end of Section 2.
review_title: Solid and impactful contribution to graph learning community

Meta-review: In this paper the authors present three real-world datasets in different domains to test graph ML capabilities in three distinct tasks. The authors present baselines and learnings from these datasets being used in the 2021 KDD Cup. All reviewers found the paper to be strong and valuable, and as such I believe it should be accepted.
Decision: Accept