Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper provides a relatively comprehensive survey on training scalable GNN for large-scale graphs, and proposes an ensemble-based training strategy dubbed as EnGCN, which achieves promising performance on several large-scale graphs. 
strengths: This paper not only involves major representative methods on training scalable GNN on large-scale graph, but also categorize these methods into two branches: Sampling-based and Decoupling-based methods. This paper further categorizes the decoupling-based methods into two varieties (the Pre-processing methods and Post-processing methods). To the best of my knowledge, there are surveys and books focus on sampling-based methods, but few surveys concentrate on the decoupling-based methods. This paper provides a fair and consistent evaluation on various representative scalable GNN training methods for large-scale graph. This paper also provides empirical analysis (including empirical Pros. & Cons.) based on the evaluated performance and efficiency. This paper further proposes an ensemble-based training strategy, dubbed as EnGCN, which achieves promising performance compared to various existing scalable GNN training methods. 
weaknesses: Despite the paper provides a relatively "fair and consistent" evaluation (benchmark) for various representative methods, nevertheless, the benchmark has at least two major drawbacks: The HPO strategy is not inevitably insufficient, and the demonstration of evaluation results are not intuitive. Particularly, the linear greedy search strategy (i.e., iteratively searching the optimal value for a HP while fixing the others) is adopted to obtain the best combination of HP, however, the iterative nature possibly not capable to derive near-optimal HP configuration in the search space, besides, the number of trials of HPO for one method is relatively insufficient (less than 30 trials per method on specific dataset). It's understandable that the sufficient HPO process incurs large computational cost. But I am afraid that the empirical results derived by insufficient HPO process are potentially not convincing to support the following empirical analysis, including the empirical Pros. 

Cons. 
The demonstration of the evaluation results are no sufficiently intuitive and straight-forward, thus it's not easy to show the following empirical analysis. Maybe some statistical properties can be shown to demonstrate the empirical analysis.  Despite the paper introduces an ensemble-based scalable training scheme, however, similar ideas have been proposed (e.g., AdaGCN). Thus the novelty in relatively limited. Nevertheless, the introduced manner has considerable practical capability, that I have already claimed in the Strengths section. 
correctness: A benchmark is presented in the submission, the evaluation methods and experiment design seems to be correct, but the results are potentially inaccurate, due to the insufficient HPO process. 
clarity: Yes, the paper is sufficiently well-written.
relation to prior work: No, this paper has not discussed the difference with previous related works. Only researchers familiar with related directions are able to recognize the novelty and differences compared with previous works. 
additional feedback: Suggestions1: Why not use BO and maybe more computational resources to obtain more convincing evaluation results, by adopting sufficient HPO process. 

Suggestion2: Revise the demonstration of the evaluation results. The results should intuitively support the following empirical analysis. Please note: intuitive, straight-forward, and direct.
review_title: A survey with empirical analysis, and an ensemble-based training scheme.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper studies the important question of the scalability of GNNs on the large-scale node classification task. In the first half, the paper clearly categorizes literature into sampling- and decoupling-based methods and provides an extensive benchmark in terms of performance and efficiency. In the second half, the paper leverages the intuition learned from the first half and proposes their ensemble method (EnGCN) that registers strong performances. Both parts are of valuable contributions.
strengths: This paper has a good summarization and review of previous works.

This paper provides extensive experimental benchmark results of performance scores and time & memory overhead.

This paper provides interesting observations and intuitions from the benchmark.

This paper proposes EnGCN which achieves SOTA performances.
weaknesses: It seems that EnGCN is a variant of SIGN with three additional components: 1. label Y as feature, 2. self-training, 3. voting (boosting). The performance of EnGCN is quite strong but the authors fail to ablate on those components to deliver intuitions behind the designs. Also, EnGCN will share the “Cons” of the preprocessing method, which is consuming a large volume of CPU memory when training on large-scale graphs, and the authors do not cover the efficiency of the proposed method in the paper. Lastly, the authors need to provide a more detailed description of the methodology. Currently, the writing is vague and hard to understand.
correctness: Claims are correct and evaluations are appropriate.
clarity: At large, this paper is clearly written and easy to follow, but the writing of the methodology of EnGCN needs further polishing.

Typos: page 7, GINAT→GIANT, Lable→Label

Section 4.2: Does the speed meassure of decoupling-based methods count the operations on CPU? How are the hyperparameters chosen to ensure a fair efficiency comparison?
relation to prior work: The authors may also consider memory-based scalable algorithms for reference.

GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddings, ICML2021

VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantization, NeurIPS2021
additional feedback: No further comments.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this work, authors perform a benchmark analysis by applying a selection of models to three common datasets. They provide a basic theoretical description of the two major groups of algorithms, sampling and decoupling, and compare the pros and cons of the various approaches. Their empirical evaluation provides evidence for a series of practically useful observations that align with theoretical claims. Finally they propose a semi-novel ensemble method that is competitive on the datasets considered, from a performance perspective.


strengths: The main strengths of this work are 1) the clarity of the taxonomy of the benchmarked methods 2) the detailed observational results, of use as a reference by the community, and 3) the strong performance of the proposed method, EnGCN.

On 3) The performance of the EnGCN method on ogbn-products is impressive, conditional on replicability and comparable computational cost (see related comment).
weaknesses: Weaknesses are limited, and mostly do not compromise the overall value of the work - 1) hyperparameter search setup is simple, potentially biased, and 2) presentation of empirical efficiency results could be improved, and such results for EnGCN are missing.
correctness: Overall the work is sound and likely presents correct results.

On Strength 2) - Obs 2 matches the standard takeaway in the literature, and Obs 3 is plausible, their hparam search seems fruitful. Obs 7 is a very practical summary takeaway, backed up by their data, the sort of thing a practitioner will appreciate.

On Weakness 1) - It is possible that a more robust sampling strategy over hyperparameters should have been employed. Minimally it might be useful to ablate over the order (of the different parameter columns in Fig 1) in which the greedy optimization is performed. Certain parameters like batch size and depth are interrelated with respect to optimization behavior. 

Related to the latter comment, while benchmarking on a commodity GPU like a 3090 is valuable since it is accessible to most researchers, it does limits the hparam values that can be explored, ie. the OOM errors. Some extra runs on a larger card would be nice to exhaust the batch and depth degrees of freedom.

On Weakness 2) - It would be important to understand the relative efficiency or cost of the proposed method EnGCN to the others given that the work presents it as SOTA, however efficiency results are not included (unless I missed something?).
clarity: Generally the paper is well organized and clearly written. A close pass for typos will be helpful.

On Strength 1) - Notational conventions are simple but adequate to describe the differences between the methods as relevant to their benchmark analysis. The hyperparameter search space and results grids are clearly presented in Table/Fig 1, and the best settings are collated well in appendix A5. 

On Weakness 2) - The empirical efficiency analysis is presented using "Throughput" in (iterations/s) which is not clearly defined. Stated differently, it is hard to understand the comparison, or relative point between those numbers in Table 3. Potentially a "pareto-frontier" curve based on different methods training time and performance could be generated, though technically one can extract this information from Table 2.

Related to the above, the Appendix section A2.3, Figure A4 and Table 7, may have some typos in how the accompanying prose references them. I was a bit confused.

Selected typos:
- I suggest replacing the "sweet point" motif with "sweet spot" which is the more common english phrase.
- Pg 4, Post-processing section, Zeros method definition, script T missing the "train" subscript
- I believe Table 1 should not say "sampling" methods only
- Obs 5, remove "Counter-intuitively" or explain why it is framed this way?


relation to prior work: This work does not present any single component or analysis that is particularly novel standing alone (except the new method) however, the consolidation of the different theoretical and empirical analyses is of significant value as a single point of reference. 

As the authors state, EnGCN is relatively similar to AdaGCN, however, they appear to perform at quite different levels.
additional feedback: N/A
review_title: Benchmarking of standard models on a few common datasets, adding new strong results based on an ensemble method.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors benchmark large-scale graph training for both sampling-based methods and decoupling-based methods. The authors also compare the efficiency and analyze the pros and cons of different GNNs. Besides, the authors present EnGCN, a new ensembling training manner.
strengths: The problem is interesting and important. The scalable GNNs have been widely studied in the machine learning community. It could be beneficial to benchmark all the scalable GNNs and summarize the pros and cons.

Insightful observations are provided. The authors claimed various observations with respect to effectiveness and efficiency. Some of them are interesting findings.

The authors propose a new GNN to solve the constraints of sampling-based methods and precomputing-based methods. 

weaknesses: More experiments can be done to validate the effectiveness of EnGCN. For example, what is the parameter sensitivity of EnGCN, considering the reported performance strongly relies on hyper-parameter search? What is the performance of EnGCN on other public benchmark datasets (e.g., ogbn-proteins, ogbn-arxiv), besides Flickr, Reddit, and ogbn-products?

Ablation studies are needed to verify the effectiveness of each component in EnGCN, e.g., pre-processing, post-processing, and majority voting.

correctness: Good.
clarity: Good.
relation to prior work: Good.
additional feedback: Please see comments above.
review_title: An interesting benchmark paper but more experiments are needed

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The scalability issue of graph convolutional networks has been being a notoriously challenging
research problem. This paper establish a benchmark for large-scale graph training w.r.t effectiveness and efficiency. And a unified formulation is given for dozens of works and further assess on the basis of accuracy, memory usage, throughput, and convergence. And rethinking of the scalability of GCNs is provided from the perspective of ensembling and ensembling-based trainer scheme. 
strengths: 1. The provided benchmark is comprehensive and looks fair. 
2. The evaluation matrics are helpful and easy to obtain.
3. The experimental results are comprehensive. 
4. The study is solid and helpful. 
weaknesses: I think the consistency and robustness should also be considered in the evaluation. 
correctness: I think the claims in the paper are correct. 
clarity: The writing of the paper is good. The presentation is clear and easy to read.
relation to prior work: The paper clearly discusses its relation to prior work. It's obviously a new work. 
additional feedback: No.
review_title: This paper is comprehensive and meaningful. This paper introduces the large-scale graph training vey well.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes to study GNNs for large-scale graphs. Specifically, the paper systemically compares sampling-based approaches (including node-wide, layer-wise, and subgraph-wise) and decoupling-based approaches (including preprocessing and post-processing) on three datasets. Besides reporting several observations for the existing methods, this paper also proposes EnGCN, an ensembling-based approach that achieves SOTA performance.
strengths: (+) Large-scale GNNs are a trending and critical problem.  
(+) The authors have compared various GNN methods with a unified evaluation protocol, including how to tune hyperparameters.  
(+) The proposed EnGCN shows impressive performance.  
(+) The proposed benchmark is publicly available at Github.  

weaknesses: Limitations and suggestions/questions:  
(-) The adopted datasets are somewhat limited. I acknowledge that two datasets (Flickr and Reddit) are social networks and ogbn-products is a co-purchasing graph, but more diverse datasets (e.g., citation and biochemical) and tasks (e.g., link prediction or graph classification, besides node classification) could provide a more comprehensive picture for large-scale GNNs.  
(-) Since this paper aims to provide "a fair and consistent benchmark", another essential part besides showing results for the existing methods is to allow new methods to be compared using this benchmark. Therefore, I think some discussions regarding this aspect should be provided, e.g., how others should use this benchmark and whether there are limitations.  
(-) Besides separately showing accuracy and efficiency (in Table 1 and 3, respectively), I think it would make the tradeoffs more straightforward if these metrics could be shown jointly, e.g., in a Pareto surface.  
(-) I wonder how the exact implementation would affect the efficiency of different methods. For example, it seems that the authors have adopted PyG, and would another library, e.g., DGL, change the results?	 

correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: See comments above
review_title: Official Review for "A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking"

Meta-review: metareview: All the reviewers appreciate the extensive survey on training GNNs for large-scale graphs, the useful experimental observations, and the new ensemble-based training strategy (EnGCN) of this work. The main concerns were insufficient hyperparameter optimization, marginal novelty (compared to SIGN), lack of efficiency comparison, missing analysis on parameter sensitivity, and limited graph datasets in the experiments. In the rebuttal and discussion, the authors addressed most of the concerns raised, and the manuscript has been updated accordingly. AC thus recommends accepting this paper. 
confidence: 5: The area chair is absolutely certain
Decision: Accept