Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Summary**
This paper introduces a method for decoupled layer wise training of graph neural networks. This method has the potential to be parallelized and offers computational and memory savings. The authors provide experimental results on 4 datasets and compare against 4 baselines. Results suggest that the approach achieves comparable results to other methods while being faster by parallelization.

**Strengths**
a) Well written paper with clear notation and illustrations

b) Good analysis of the different algorithms. Lays out complexity and memory costs of related work.

c) Analogy to block coordinate descent intuitively makes sense.

d) Nice contribution towards achieving asynchronous/lazy updates for GNNs. 

**Weaknesses**

*Novelty and impact need to be strengthened*
a) As mentioned by the authors, greedy layerwise pre-training is an old idea. When it first came out for Deep NNs, everyone was excited about it but nowadays hardly anyone does it. So if this method has to make a comeback for GNNs then the benefits have to be very compelling.

b) Wrt benefits, the presentation around why this idea is compelling needs to be concretely laid out. It seems like there are some computational and memory benefits via parallelization. But is the additional complexity of parallel training worth it? As the authors mention, there can be non-trivial communication costs between GPUs in addition to the additional code complexity.

c) Absent theoretical convergence guarantees, the proposed approach is a heuristic as it relies on an auxiliary function that’s somewhat arbitrary. Further discussion is needed on why the chosen auxiliary function is a good idea. Some ablation studies with different auxiliary functions are needed to shed light on this particular choice of auxiliary function.

*Experiments can be strengthened*
a) The results seem to trade off accuracy with speed and saved memory and results are reported on 3 small and one large dataset. To truly claim that this method generalizes, the authors would need to strengthen their results across some different tasks (not just semi-supervised classification)  and models (not just GCN).

b) More details are needed regarding the experimental setup. Was there a multi-gpu setup? E.g. OGBN-arxiv has 7 layers. The authors report a total running time of 23.6s. Was this on a 7 GPU setup? 

c) More ablation and convergence studies. The authors stop at 200 epochs for the smaller datasets. Does the proposed method reach full-batch GCN accuracy at higher epochs?

**Typos and other fixes**
a) Typo abstract: Should be “ Graph Neural Networks (GNNs) *have* become”

b) Table 6 is missing.

c) Results: “... but higher than all the other efficient GCN training strategies…” ; LGCN is better for pubmed and citeseer

d) “....when comparing the accuracy curves of different Tlayer shown in Figure 3, we find that large Tlazy makes the training more stable…”
i) Hard to see this from Figure 3. Consider plotting running variance as a shaded overlay. 
ii) Plots are too busy. Hard to draw conclusions.


**What can the authors do to make the paper better**
a) More thorough experimentation as described in the *Experiments can be strengthened* section.

b) I did not find the code release with the paper. For a paper whose primary claim is computational and memory savings, I think it would be a good idea to include it.

c) Emphasize the novelty of the method otherwise, greedy pre training seems like a lukewarm idea.
review_title: Well written paper with some interesting ideas. Impact and experiments can be strengthened. Otherwise paper seems of lukewarm interest.

final_score: rating: 6: Marginally above acceptance threshold
review_content: GNN is known to be computationally complicated and it requires significant amount of time and memory. Unlike CNN, inference of the node embedding in GNN requires information be propagated throughout entire graph for each epoch. In this paper, the authors proposed to decouple the inference and aggregation in each layer and learn the layer-wise latent representation separately. The result is to decompose the deep GNN into a series of shadow networks that are connected sequentially. Since the optimization is done independently, it requires significant less time and memory.

This paper is clearly written and the contribution, literature review is sufficient for reader to understand. The analogy to block coordinate descent is a good part for reader to connect this work with the well-established work in optimization theory. The idea of layerwise decomposition is published in [You et al., 2020] and this paper improve over that by introducing more parallelization. 

Pros:
1. The layer-wise decomposition is not a new idea but to optimize GNN, it is not as popular as in normal DL framework. The study on this topic will definitely brings some attention to the community and is enouraged.
2. The parallelization strategy and lazy update scheme is useful and easy to implement
3. In the experiment, it shows that the time reduction of LU-DGL-GCN is significant and it does not depends the depth of model. Therefore it is applicable to larger graph and deep structure.

Cons:
1.  Unlike the traditional convex optimization where the block coordinate decent is know to converge to optimal point. In the settings of GCN, there is no proof on how close the block coordinate descent would be towards to traditional GCN. 
2.  Following the above question, it is known that the message-passing and aggregation in one layer only accumulate the information in 1-hop neighborhood and as a result, the embedding it learned only encodes the local structure within the 1-hop neighborhood. In many applications, the structural information of the graph can only be obtained through a wider p-hop neighborhood where p > 1. Traditional GNN take inference on p-hop neighborhood through  optimization of p-layers all together in backpropagation.  While the layer decomposition is attractive in time, LU-DGL-GCN loss the accuracy to infer the structural information in larger neighborhood. And because each layer is optimized using independent classifier on the same target, it is not hard to expect the node representation learned in the bottom layer to be less representative in LU-DGL-GCN compared to the traditional GCN as it does not encode the information beyond the 1-hop neighborhood.
3.  In early DL, the layer-wise optimization only used as pre-training to obtain a good weight initialization, and after that, a further full training is needed to obtain a representative embedding and model. It is suggested to conduct a full training with weight learned by LU-DGL-GCN and to compare with traditional GCN and to see if there are better solutions. 
4.  It is better to discuss the limitation of the decoupled greedy learning, i.e. the model is myopic in that the resulting layer-wise node embedding only collect information in nearest neighbors. This is esp. an issue for lower-level representation since it lacks of prior learned from upper layer to guide its inference. 
5.  The LU-DGL-GCN is a cascade  of  L independent shadow network models as opposed to one L-layer deep model. It is known that the representative and expressive power of the deep models is higher than that of a sequential connected independent shadow models. Thus compared to LU-DGL-GCN, it sacrifices the representation accuracy and model complexity to obtain a faster training and less memory. This is the tradeoff. 
review_title: The greedy layer-wise decomposition of GNN in this paper is efficient in memory and time but the optimality of solution is of concern

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper gives up end-to-end training and instead trains graph convolution nets layer by layer. The authors call such layer-by-layer training process decoupled greedy learning (DGL-GNN).  In the decoupled greedy learning process, a lazy-update scheme is adopted, that is to exchange information between layers after a certain number of epochs instead of every epoch. The aggregation and transformation process are also separated and performed respectively.  Authors show their proposed engineering methods can save a constant fraction of computation time for graph convolution nets, but for the cost of sacrificing some predictive performance. Authors conduct some experiments on semi-supervised node classification data, reporting less memory and runtime but worse performance.

While the paper attempts to study an important problem of scaling graph networks to large graphs, and authors show the proposed methods is capable of saving in terms of memory and runtime by sacrificing predictive performance (which is not surprising at all), it is not suitable for publication at ICLR. 

Comments: 

Novelty is clearly below the bar of ICLR. The proposed method "layer-by-layer training" is rather trivial. 

No theoretical or empirical justification is given for giving up end-to-end training. In fact, there are many easy counter-examples to show this approach would fail. Moreover, another problem with this method is the lack of auxiliary labels for each layer. It is easy to find problems where we do not have such layer wise auxiliary labels, and the proposed method would not work. 

Moreover, even by doing layer by layer training, the proposed method cannot solve the scaling issue for billion scale networks, so some sampling is still needed.

This paper is purely experimental; however, the experiments are unsatisfactory:
- The reported performance is not competitive. In Table 2 and Table 3, test performance of proposed method is worse than prior works, e.g., LGCN, despite consuming even more memory cost. In this sense, practitioners would simply prefer LGCN to the layer-by-layer training. 
- The datasets in experiments are very small, e.g., cora only have a few thousand nodes. What is the value of acceleration (while sacrificing performance) on such small graphs? 
-  More graph neural network architectures need to be tested to show the effectiveness of the method: 1. graph attention network, 2. graph isomorphism network, 3. graphsage, 4. chebyshev GCN. 
-  Missing link prediction benchmarks. 
- Missing baselines: SIGN, GraphSAINT.


Grammar mistakes: 
then after it get fully converged -> gets

optimize next layer -> the next

we do the same things -> thing

which scales exponentially with the number of layers -> factual error
review_title: Review

Meta-review: In this paper, the authors propose a new layer-by-layer training approach for GNN in particular for a large graph. The proposed approach can be easily parallelizable and scale well to a large graph. Reviewers are concerned about the novelty of the approach and the lack of theoretical analysis, and it is not well addressed by the rebuttal. Therefore, this paper is below the acceptance threshold of ICLR. I encourage the authors to revise the paper based on the reviewer's comments and resubmit it to a future venue.
Decision: Reject