Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work proposes IGLU, an algorithm to scale-up GNNs using stale computations instead of traditional neighborhood sampling.
IGLU has bounded bias if the loss and activation functions are smooth. Results on large-scale benchmarks show IGLU achieves SOTA and scales better than previous methods.
main_review: # Strengths
* I like the idea of holding some variables constant in gradients and using stale computations to speed up computations. Somehow similar ideas have previously been used to, e.g., speed up the convergence of SGD (1) and SGLD (2).
* Strong empirical results show that IGLU scales well and achieves SOTA.
* The authors provide a convergence analysis of IGLU.

# Weaknesses
* In practice, many tricks are used in conjunction with GNNs. Some examples are long-range residual connections, BatchNorm, and virtual nodes. These are especially important to train deeper networks. It is not clear how IGLU could incorporate such tricks. Can the authors elaborate?


1: https://proceedings.neurips.cc/paper/2013/file/ac1dd209cbcc5e5d1c6e28598e8cbbe8-Paper.pdf

2: https://proceedings.neurips.cc/paper/2016/file/9b698eb3105bd82528f23d0c92dedfc0-Paper.pdf




summary_of_the_review: I appreciate the use of stale gradients to speed-up backward computations in GNNs. In principle, the IGLU is architecture-agnostic and can be applied to many GNNs. Therefore, I am more inclined to acceptance than rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies tries to tackle the scalability challenge of training GNNs on large graphs. The authors propose IGLU, an architecture-agnostic method. IGLU caches intermediate computations and uses a lazy update strategy. Convergence analysis on IGLU is provided and empirical results show that IGLU has better performance than baselines such as GraphSAINT, Cluster-GCN, VR-GCN.

main_review: Strengths:
1. The scalability issue of GNNs is challenging and an important research topic. IGLU is simple and the motivation is clear and intuitive.
2. Compared to strong baselines, IGLU has better empirical performance in both accuracy and training time.
3. Under certain assumptions, IGLU has non-trivial convergence guarantees, which is an advantage against baseline approaches.


Weaknesses:
1. Caching intermediate results to speed up GNN training has appeared in multiple prior works. The main difference is that this paper considers two variants: cache embeddings or gradients. However, I still think the technical novelty of IGLU is not significant. 
2. The GNNAutoScale method of Fey et al. uses similar ideas and make uses of stale embeddings to scale up the training of GNNs. From their experimental results, GNNAutoScale achieves much better performance compared to GraphSAITNT, Cluster-GCN, etc.  So, I think the authors should provides a more detailed discussion on the difference and connection between IGLU and GNNAutoScale. Moreover, empirical comparison between them would make the results more complete and convincing.

summary_of_the_review: Overall, this paper has some insights on tackling the scalability challenge training GNNs on large graphs. And IGLU is an effective method for scaling up GNN training and shows better empirical performance compared to well-known techniques. On the downside, the technical novelty is not significant. Moreover, I think authors should provide a more detailed comparison between IGLU and GNNAutoScale.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new method, IGLU, that caches intermediate computations at various GCN layers.
This enables IGLU to perform lazy updates that do not require updating a large number of node embeddings during descent and offers much faster convergence without significantly biasing the gradients.

Overall, this paper represents a novel solution towards efficient GNN training.
main_review: Overall, IGLU represents a novel method for achieving efficient GNN training. IGLU's idea is quite different from the common scalable GNN training methods, such as GraphSAGE and ClusterGCN.
The authors provide theories to show that the bias of IGLU is bounded.
The experimental results are promising as well.

Main questions:
1. The overall algorithms are still vague for me. I can understand the main results that we can use the previous layer's embeddings to approximate the exact backpropagated gradients. 
I appreciate the authors presenting Algorithm 1 and 2 as well. However, the exact computation is vague. For example, in the algorithms it said things like "Compute G", "using Lemma 1 (1)", but it's unclear how they are used.

2. The evaluation setup is not rigorous. In Table 1, different GNN architectures are used to produce the results, which makes things unclear how much gain IGLU training is offering. 
I believe to show that IGLU is really model agnostic, at least one set of experiments should be made, where everything about GNN architecture is fixed, but only the training algorithms vary. 
For example, you may refer to the paper "Design Space for Graph Neural Networks" on rigorously comparing different GNN design options.
I believe only with rigorous controlled experiments, the results in Table 1 Figure 2 can assure the readers of the effectiveness of IGLU.

3. In Table 1, the percentage of speedup is quite significant. Could the authors explain how they are computed? What is the baseline speed?

4. Could the authors explain the rationale for IGLU outperforming full-batch GNN training? I believe full-batch training can offer exact gradients.

5. In Figure 2, the results on "OGBN-Proteins" seem suspicious. It seems that without training, IGLU can already achieve much higher ROC AUC over other methods. I hope the authors explain the reasons behind it.



summary_of_the_review: Overall, I like the technical contributions made in this paper.
My concerns are:
1. The algorithms are not clearly described, making it hard to understand the implementation of IGLU
2. The evaluation setup is not rigorous, making the claims on the effectiveness of IGLU questionable
review_title: null

Meta-review: Overall the paper present the idea of caching and using stale information to update instead of sub sampling for speeding up graph convolution neural network. Reviewers liked the idea but also there were concerns about experimental comparisons.  In the rebuttal the authors did provide more evidence of comparison with other caching based and other relevant baselines. Overall the importance of scaling up GCNN and empirical results helped the paper cross the high bar.
Decision: Accept (Poster)