Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a distributed full-graph GCN training method to speed up GCN training for large-scale graphs.  Experiments demonstrate its performance and efficiency. Convergence proof is also provided.
main_review: This paper proposes a distributed GCN training on large graphs named PipeGCN. Specifically, the authors hide the communication time by parallelizing the communication and computation process in each layer and using the stale information for parameter updates.  The idea seemed simple and straightforward, and experiments show that the algorithm can achieve up to 2.2x speedup. The authors provide the convergence proof for PipeGCN and propose two smoothing methods for faster convergence. 

 Several limitations of this paper are listed as follows:
1. The proof of convergence should be justified better. The author didn’t claim how can the proposed model satisfy Assumptions 3.1-3.3, such as whether the loss function they chose satisfies the Lipschitz continuous condition mentioned in Assumption 3.1. In addition, the convergence proof is not applicable if PipeGCN uses the most commonly used ReLU activation function, as ReLU doesn’t satisfy Assumption 3.2.
2. Some statements in the paper are not clear enough.  For example, in Table 1 the authors didn’t mention AllReduce of Weight Gradient PipeGCN, whereas it appears in line 32 in Algorithm 1. 

summary_of_the_review: The authors designed a pipelined distributed GCN training algorithm to speed up large-scale GCN training and demonstrated the performance and efficiency with abundant experimental results. The paper can be accepted if the authors improve the convergence proof and modify the uncleared statements.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes PipeGCN, which pipelines communication and computation in distributed GCN training to improve training throughput. Analysis is conducted to show the convergence speed when using both stale gradient and feature vectors. Extensive experiments are conducted to show that PipeGCN significantly improves the efficiency of vanilla distributed GCN training without hurting model accuracy.  
main_review: I think the paper is a well-executed work: the discussions about related works are extensive, the idea behind PipeGCN is clearly explained, the convergence speed of PipeGCN is analyzed theoretically and the experiments are comprehensive. 
I have some concerns on the novelty of the paper. As mentioned by the authors, Dorylus considered using stale features. To my knowledge, using stale gradients is well studied in Pipe-SGD related works. PipeGCN seems to combine the two ideas by using both stale features and stale gradients. Thus, the core problem is the technical contribution of the analysis of PipeGCN, which I do not have the technical expertise to evaluate. 
I think the paper can be improved by fixing the following problems.
1. Experimental comparisons with Dorylus may be included to show the benefits of using stale gradients.
2. Figure 1 and Figure 2 can be improved. In Figure 1(b), the “…” for Part 2 should be removed and all communication tasks should be explicitly listed (e.g., communicate features, communicate gradients, and note that communicate gradients happen after some computation). In Figure 1(b) and (c), the computation and communication tasks should have iteration counts such that the pipelining idea can be understood. In Figure 2, “communicate for Next L1 backward” happens before the L1 backward of the current iteration, how is that possible? Is it a typo?
3. The influence of the smoothing technique can be better analyzed. In Table 4, there is not a consistent winner among PipeGCN, PipeGCN-G and PipeGCG-F in model accuracy. 

summary_of_the_review: The paper is complete but lacks comparison with an important related work and a justification of the technical challenges.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose PipeGCN - a method for efficiently distributed full-graph GCN training. The method pipelines communication with computation in distributed GCN training to hide substantial communication overhead. The paper leads an effort to study pipelined and asynchronous distributed training of GCNs with a new smoothing method aimed at reducing the error incurred by stale features/feature gradients at minimal overhead.
main_review: **Strengths**
1. It analyzes two efficiency bottlenecks in distributed training, including communication overhead and synchronization, and proposes PipeGCN to pipelining the inter-partition communication with intra-partition computation.
2. The paper provides novel theoretical proof of the convergence of GCN training with stale feature and feature gradients, which is useful for future work.
3. The experiment results show a significant speedup of the training for the GCN model on multiple reported datasets and compares it with the other latest methods.

**Weakness**
1. The PipeGCN aims at scaling graph neural network training, but the setup of the largest dataset, ogbn-papers100M, is not practical. With only 2 layers and 8 hidden units, the GNN may not learn anything from the graph. Also, the accuracy on ogbn-papers100M is missed in the result table. With that in mind, I am concerned about the scalability of the system.
2. For each dataset, the paper only shows the results for a fixed number of partitions. It would be great to see how the speed and convergence differ when the number of partitions and computation nodes increases. 
3. The results in Table 6 are vague, without showing the dataset used and what is the dist GCN method.
summary_of_the_review: The paper proposes an interesting framework to speed up the GCN training with the theoretical proof for the complexity. However, as mentioned in the weaknesses above, the weird setup on the largest dataset, ogbn-papers100M, and lacking the evaluation on a different number of computation nodes, it is hard to understand the scalability of the system.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A method of partitioning GNNs, and using stale versions of activations across partitions. This allows activations to be communicated in parallel (rather than sequentially) across GPUs, which improves throughput. Theoretical results show that the stale activations still result in convergence, and empirically the method appears to perform better than the rate given by the theory.
main_review: Strengths
- Simple method that is relatively easy to implement compared to other parallel GNN systems. Does not require complex memory management (e.g. involving CPU memory). That makes it potentially easy to extend to a distributed network setting (multiple machines).
- The staleness is in activations/gradients, not weights. Intuitively, this leads to more stable convergence (vs stale weights) because (1) the stale activations/gradients only occur on the partition boundaries; (2) the activation/gradient for a neuron on the boundary is a combination of stale (inter-partition) and non-stale (intra-partition) activations/gradients
- I did not evaluate the theoretical proofs or convergence rate for mathematical correctness, but the fact that the method converges at all is intuitively reasonable, given that stale gradient systems are already known to converge.

Weaknesses
- It is rather surprising that vanilla partition parallel, a simple baseline that many other GNN systems are adapted from, beats recently published (2020) methods by a very wide margin, such as ROC and CAGNET. Although the authors did give some analysis as to why this is the case, I am not convinced those could be the major factors. It is more likely that the baselines have been misconfigured. For the avoidance of doubt, the authors should provide numerical evidence that their obtained results are consistent with the results reported in the ROC and CAGNET papers.
- The method relies on a somewhat brittle assumption that inter-partition communication time is roughly equal to activation/gradient computation time. If the communication time for a given model and partitioning were significantly smaller or larger than the computation time, the throughput gain from parallelizing computation with communication would be much less impressive. Ultimately, the method does not truly solve the issue of overwhelming communication volume - which sampling-based methods do address. The paper should acknowledge this limitation, rather than claiming superiority in all scenarios to sampling based methods.

Minor issues and suggestions
- The authors should not use the word "distributed" unless the implementation and experiments support networked machines. The hardware configuration in the experiments is a single machine with multiple GPUs.
- It is interesting that the experiments were performed on commodity hardware with what I understand to be no NVLink, and only PCIe v3 x16 connections. The bandwidth of PCIe v3 connections is roughly comparable to 100-200Gbps network connections. I take that as a promising sign that the method will continue to perform well with distributed network machines.

summary_of_the_review: While the idea of stale activations/gradients is not new (as the authors have acknowledged in the related work), the convergence result seems new to the best of my limited knowledge. I am positive that (1) validation accuracy is essentially equal to the non-stale version; (2) based on the PCIe bandwidth in the experiments, the method should also work in distributed inifiniband or 100+Gbps ethernet settings. However, I do have concerns about (A) how the ROC and CAGNET results were obtained, and (B) the method only performs well in a narrow "Goldilocks zone" where computation time is roughly equal to communication time.

On the last point - the authors might want to consider what happens at the billion+ node scale. In my experience, the distribution of very high degree nodes increases, and it is likely that nodes with an extreme number of neighbors will end up on the boundary. This could greatly increase communication time relative to computation, and the method would no longer perform well.

Overall, it is an interesting if slightly flawed paper, and I lean towards acceptance.
review_title: null

Meta-review: The paper proposes PipeGCN, a system that uses pipeline parallelism to accelerate distributed training of large-scale graph convolutional neural networks. Like some pipeline-parallel methods (but unlike others), PipeGCN involves asynchrony in the sense that its features and feature-gradients can be stale. The paper provides theoretical guarantees on the convergence of PipeGCN in the presence of this staleness, which is a nice contribution in itself. In discussion, the reviewers found the work to be well-executed and sound. All reviewers recommended acceptance, and I concur with this consensus.
Decision: Accept (Poster)