Decoupled Greedy Learning of Graph Neural Networks
Authors: YEWEN WANG
Jian Tang
Yizhou Sun
Guy Wolf
Venue:  (2021)

Abstract: Graph Neural Networks (GNNs) become very popular for graph-related applications due to their superior performance. However, they have been shown to be computationally expensive in large scale settings, because their produced node embeddings have to be computed recursively, which scales exponentially with the number of layers. To address this issue, several sampling-based methods have recently been proposed to perform training on a subset of nodes while maintaining the fidelity of the trained model. In this work, we introduce a decoupled greedy learning method for GNNs (DGL-GNN) that, instead of sampling the input graph, decouples the GNN into smaller modules and associates each module with greedy auxiliary objectives. Our approach allows GNN layers to be updated during the training process without waiting for feedback from successor layers, thus making parallel GNN training possible. Our method achieves improved efficiency without significantly compromising model performances, which would be important for time or memory limited applications. Further, we propose a lazy-update scheme during training to further improve its efficiency. We empirically analyse our proposed DGL-GNN model, and demonstrate its effectiveness and superior efficiency through a range of experiments. Compared to the sampling-based acceleration, our model is more stable, and we do not have to trade-off between efficiency and accuracy. Finally, we note that while here we focus on comparing the decoupled approach as an alternative to other methods, it can also be regarded as complementary, for example, to sampling and other scalability-enhancing improvements of GNN training.

Under review as a conference paper at ICLR 2021
DECOUPLED GREEDY LEARNING OF GRAPH NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Graph Neural Networks (GNNs) become very popular for graph-related applica- tions due their superior performance. However, they have been shown be computationally expensive in large scale settings, because their produced node embeddings have be computed recursively, which scales exponentially with the number of layers. To address this issue, several sampling-basee methods have re- cently been proposed to perform training on a subset of nodes while maintaining the fidelity of the trained model. In this work, introduce decoupled greedy learning method for GNNs (DGL-GNN) that, instead of sampling the imput graph, decouples the GNN into smaller modules and associates each module with greedy auxiliary objectives. Our approach allows GNN layers to be updated during the training process without waiting for feedback from successor layers, thus mak- ing parallel GNN training possible. Our method achieves improved efficiency without significantly compromising model performances, which would be impor- tant for time or memory limited applications. Further, we propose a lazy-update scheme during training to further improve its efficiency. We empirically analyse our proposed DGL-GNN model, and demonstrate its effectiveness and superior efficiency through a range experiments. Compared to the sampling-based ac- celeration, our model is more stable, and we do not have to trade-off between efficiency and accuracy. Finally, we note that while here we focus on comparing the decoupled approach as complementary, for example, to sampling and other scalabilit--enhaccin im- provements of GNN training.
alternative to other methods, it can also be regarded
INTRODUCTION
Graph Neural Networks (GNN) have been shown to be highly effective in graph-related tasks, such as node classification Kipf & Welling 2016), graph classification Ying et 2018b), graph match- ing (Bai et al. 2019), and recommender system (Ying et al.) (018a)) Given graph of arbitrary size and attributes, GNNs obtain informative node embeddings by first conducting a graph convolution operation aggregate information from the neighbors of each node, and then transforming the aggregated information. As a result, GNNs can fuse together the topological structure and node features of graph, and have thus became dominant models for graph-based applications.
Despite its superior representation power, the graph convolution operation has been shown to be expensive when GNNs become deep and wide (Chen et al., 2017). Therefore, training a deep GNN model is challenging for large and dense graphs. Since deep and wide GNNs are becoming in- creasingly important with the emergence of classification tasks on large graphs, such as the newly proposed OGB datasets (Hu et al. (022)). and semantic segmentation tasks as introduced in (Li al. 2019), we focus here on studying methods for alleviating computational burdens associated with large-scale GNN training.
Several strategies have been proposed during the past years to alleviate this computation issue of large-scale GNNs. GraphSAGE (Hamilton et al. 2017) took the first step leverage neighbor- hood sampling strategy for GNNs training, which only aggregates a sampled subset of neighbors of each node in the graph convolution operation. However, though this sampling method helps reduce memory and time cost for shallow GNNs, it computes the representation node recursively, and the node's is receptive field grows exponentially with the number of GNN layers, which may make

---

Under review as a conference paper at ICLR 2021
the memory and time cost even goes larger for deeper GNNs when the sample number is big. The work of Chen et al. (2017) 2 Zou et al. (2019) developed sampling-base stochastic training methods to train GNNs more efficiently and avoid this exponential growth problem. CChiang et al. (2019) proposed a batch learning algorithm by exploiting the graph clustering structure. Beyond the aforementioned methods, recently, You et al. (2020) proposed layer-wise sequential training al- gorithm for GNNs, which decouples the aggregation and transformation operations in the per-layer feed-forward process and reduces the time and memory cost during training while not sacrificing too much model capabillit, this indicates that the GNN layers do not have to be learned jointly. However, the sequential training would bring some inefficiency.
In addition to the inefficiency brought by the graph convolution operation, as discussed in (Belilovsky et al., 2019a). the sequential nature of standard backpropagation also leads to ineffi ciency. As pointed out in (Jaderberg et al., (017), backpropagation for deep neural networks suffers an update-locking problem, which means each layer heavily relies on upper layers' feedback to up- date itself, and thus, it must wait for the information to propagate through the whole network before updating. This would be a great obstacle for GNN layers to be trained in parallel to alleviate com- putation pressure under time and memory constraint, and would prohibit the GNN training to be trained in an asynchronous setting.
In this work, using semi- superviieee node classification as example, we show that the greedy learn- ing would help to decouple the optimization of each layer in GNNs and enable GNNs to achieve update-unlocking, i.e., allow the GNN layers to update without getting any feedback from the later layers. By using this decoupled greedy learning for GNNs, we can achieve parallelization of the network layers, which would make the model training much more efficient and would be very im- portant for time or memory limited applications. Moreover, we propose to use lazy-update scheme during training, which is exchange information between layers after a certain number of epochs instead of every epoch, this will further improve the efficiency while not sacrificing much perfor- mance. We theoretically analyze the computation complexity of our proposed method, and analogue our method to the classic block coordinate descent optimization to enable further analysis. We run set of experiments to justify our model, and show its great efficiency on all benchmark datasets. On the newly proposed large OGBN-arxiv dataset, when training 7-layer model, our proposed method even saves 85% time and 66% per-GPU memory cost of the conventionally trained GCN.
Our main contributions can be summarized follows. First, introduce decoupled greedy learning algorithm for GNNs that achieves update-unlockin and enables GNN layer to be trained in parallel. Next, we propose leverage a lazy-update scheme improve the training efficiency. We evaluate our proposed training strategy thoroughly on benchmark datasets, and demonstrate it has superior efficiency while not sacrificing much performance. Finally, our method is not limited to the GCN and the node classification task, but can be combined with other ccalabiity--enhaniine GNNs and can be applied other graph-related tasks.
2 RELATED WORK
Before discussing our proposed approach, we review here related work on efficient training strategies for GNNs. The computational complexities the discussed methods are summarized in Table [ and we refer the reader to Appendix A for detailed computation.
2.1 DEEP GRAPH CONVOLUTIONAL NETWORK (DEEPGCN)
Graph convolutional network (GCN, Kipf & Welling 2016) is one of the most popular models for graph-related tasks. Given an undirected graph & with node feature matrix X E R x and adjacency matrix A E NNN where N is node number and D is feature dimension, let À = A
D be a diagonal matrix satisfying D., = ビ Åig. and F = 1-1 AI-AAD-11 1/2 be the normalized A, then, the 1-th GCN layer will have the output ( as O = non-linear transformation, and (1()) is the trainable weight matrix at layer 1.
where a is the
As pointed out in Li et al. (001)), when GCN becomes deep. it will suffer severe over-smoothing problem, which mean the nodes will become not distinguishable after stacking too many network layers. However, for applications such semantic segmentation (Li al. 2019) classification
2

---

Under review as a conference paper at ICLR 2021
Table I: Summary of Complexity. Here denotes the average degree, b denotes the batch size, Snode and Stayer are the number of sampled neighbors in NS and IS respectively, K the dimension of embedding vectors (for simplicity, assume it is the same across all layers), L is the number of layers, N the number of nodes in the graph, A is the adjacency matrix, is the number of
iterations. Twoit is the waiting time for LU-DGL-GCN,
Methods
Memory (per GPU)
Time
Full-Batch GCN (Kipf Welling 2016
O(LNK + LK*)
A Jo + TLNK*)
GraphSage (Hamilton al.
2017
Obb mode.
L-1 LK2)
O made de + BT BTK22 L-11
uode!
VR-GCN Chen ct al.
2017
OLLN + LK2)
O(bDT 1--1
+ bTK-* &node!
2 node
FastGCN Chen el al
2018
OLL Stager + LK?)
O Jager
+TLK* "TLK2 Stayer
LADIES (Zu al.
2019
OLLK Stager + +LK2)
O(TLK S Saager
+ .TLK* Stayer)
ClusterGCN (Chiang et
2019)
O + +LK²
O(TL + +TLNK*)
1,2GCN You et al.
2020
ONNK +2K²)
[ Alla K 2TLKK²
LJ-DGGI-GCN (ours)
O(NK + +2K)
A|o]) + +2TNK3)
tasks large datasets (Hu et al., 2020), we do need deeper GCN models. Therefore, we follow the work of Li et al. (2019), alleviating over-smoothing problem by adding residual links between GCN layers and obtain the deepGCN model. The (-th layer our network model will be H (F ((() H
2.2 EFFICIENT GNN TRAINING
To alleviate the expensive computation issue of GNN introduced in previous section, lot of litera- ture has proposed sampling based batch-learning algorithms to train GNNs more efficiently.
GraphSAGE (Hamilton al. 2017) introduced node sampling strategy (NS), which is to domly sample neighbors for each node at each layer, then, for each node, instead of aggregating embeddings of all its neighbors, we only aggregate the sampled ones. VRGCN (Chen et al.) 2017) also followed this NS strategy, but it further proposed to leverage history activation to reduce the variance of the estimator. Though NS scheme has smaller complexity compared full-batch GNN, there exists redundant computation and the complexity grows exponentially with the layer number.
Layer-wise importance sampling strategy (IS) would more advanced method for efficient GNN training. FastGCN (Chen et al., 2018) proposed to sample nodes for each layer with a degree- based sampling probability in order to solve the scalability issue NS. The work of LADIES Zou al.) 2019) leveraged IS idea as well, but it proposed a layer-dependent importance sampling scheme, which enjoys a smaller variance while maintaining same level complexity as FastGCN. Though this IS is better than NS in general, but we may still have to trade-off the complexity with the performance (i.e. accuracy for classification tasks), since using a large sample number would be helpful for the performance and increase the computation cost and vice versa.
Except for ther aforeementioned methods, we also has ClusterGCN (Chiang et al., 2019), which proposes to partition the graph into several clusters then randomly select multiple clusters to form batch to train the GNN. Though this would allows us to train much deeper GCN without much time and memory overhead, the stability of the performance of this approach would be hard to guarantee, since the performance would heavily depends on the graph clustering settings.
2.3 LAYER-WEL GNN
Layerwise learning for neural networks first introduced by the work of (Hinton et al. 2006), and was further discussed in (Bengio et al. (007]. The work of (Belilovsky et al. 2019a b) explored the layerwise CNNs and achieved impressive results.
Recently, (You et al. 2020) proposed a layerwise algorithm for GNN training. The key idea is to train GNNs layer by layer sequentially. Figure | illustrates the sequential training framework for layerwise GNN. For L-layer GNN, first train its first layer with auxiliary classifier, then after it get fully converged, we fix this layer, and start to optimize next layer, we do the same things for all T. layers. This layerwise training saves lot of memory, since this method only requires us
3

---

Under review as a conference paper at ICLR 2021
to focus on one layer and only need to store one layer's activation results. Besides the clear memory saving, this layerwise training scheme also saves us a lot of time. During the learning process, it can decouple the two key components in the per-layer beed-forward graph convolution: aggregation and transformation. Then for each layer, it only needs to conduct the aggregation once at the beginning of the training, and then only need to do the transformation step each iteration, which greatly reduce the time cost. According to the reported results, this sequentially-trainee layerwise method efficient than the joint learning strategy and can get us good performance. However, its sequential scheme would bring some inefficiency because one layer has to wait until its previous layers to get fully converged start training. In our work, we solve this problem and enable the model parallelization to further improve the effciency.
PROPOSED APPROACH
3.1 MODEL ARCHITECTURE
lost
conufarmatio
dantier ( sireration O uee layer
iComentional
As mentioned in section 2.1 we introduce our proposed algorithm with deepGCN model (i.e., the GCN model with residual link) since the residual link would help to alleviate GCN's over- smoothing problem when goes deep, which
T
Laperwiiee
ONN
Stage
Pagg
T
A
A
would be important for large scale scenarios.
2
There exist two ways to train such GCN model: conventional training and sequential layerwise training. We illustrate these two strategies with the high-level framework shown in Figure For conventional training, we jointly optimize the learnable parameters in all layers and in the clas- sifier. For layerwise training. we break the train- ing for a L-layer GNN into L sequential stages, each stage has to wait all its previous layers to get fully converged start training.
Figure 1: High level framework of conven- tional deepGCN (upper) and layerwise deep- GCN (lower). The aggregation step (A) cor- responds to F (H-11)) operation and the trans- formation step corresponds to (--W()) oper- ation. In conventional training, both steps are done for every iteration, but for sequential layer- wise training, we can decouple these two opera- tions to conduct the transformation step in every iteration, but the aggregation step only once at the beginning of each layer, which results in the demonstrated time saving.
Note that, the sequential layerwise training has the advantage that can save time and memory while not compromising too much performance, this suggests its promising applications in large scale models under hardware and time constraints. We now consider, whether we can extend it to parallel version, so that the efficiency can be further improved? Interestingly, as shown the following sections, we find the answer is affirmative.
3.2 DECOUPLED GREEDY LEARNING ALGORITHM
To enable parallel GNN training, the most challenging problem is update-locking. Before updating one layer, we have to wait after the signal has been passed through all its successors, which would bring ineffi- ciency. To alleviate this problem, fol- low the design of layerwise GNN: decou- pling the GNN model into different layers. associating each layer with an auxiliary classifier, which is a MLP layer with soft- max activation, and assigning per-layer greedy objective. Then, with the output activation of given layer, we can lever- age the auxiliary classifier to optimize the per-layer objective and therefore can up- date the current layer without any feed- back from its successors while the rest lay-
doostiee
cammetion
use GaNel
- forword Sodsward
Laperd.
Layert
Layert.
Iaye22
lxper2
Iayer
Jayer1
Iipert
layer1
Flans
6m/
(a) Conoecatonal Training
(b) Sequental Traising
(4) Puralia Training
Figure 2: Signal propagation process for convention- ally trained GNN, sequentially trained layerwise GNN, and the parallel trained GNN, Arrows of different col- ors represents different batches of data (can be either mini batch or full-batch). We assume the forward pro- cess and backward process have the same time cost, we also assume the auxiliary classifier computation is neg- ligible, these are only for simpler illustration purposes.
4

---

Under review as a conference paper at ICLR 2021
Algorithm Decoupled Greedy Learning (DGL) of GNNs
Require: Normalized Adjacency Matrix F; Feature Matrix X; Labels Y; Total Number of Itera- tions 7'; Total Number of Layers L.
1: Initialize: H = x:;
2: for = 1 to T do
3: for = 1 to L do
4: H = (F w
1/ Get node embeddings and store them as H ( Update with (wo, HH (0000")) // Update pa-
5:
rameters.
6: end for
7:
end for
Algorithm 2 Decoupled Greedy Learning (DGL) of GNNs with Lazy Update Scheme
Require: Normalized Adjacency Matrix F; Feature Matrix X; Labels Y; Total Number of Itera- tions T; Total Number of Layers L; Waiting time Tiazg-
1: Initialize: H (0) FX;;
2: for = 1 to T do
3: for = 1 to L do
4:
( =o ((())
// Get node embeddings.
5: (W (00) + Update with (loss (Y, H1 w()) e(1) // Update param- eters.
6: if (t mod Tuzz == 0) then
7: H FH // Get propagated node embeddings and store them H
8: end if
9:
end for
10:
end for
ers are still in the forward process. We name our training strategy as Decoupled Greedy Learning of GNNs (DGL-GNN).
With our DGL-GNN, we achieve update-unlocking, and therefore can enable parallel training for layerwise GNNs. For clarity, we provide Figure to compare the signal propagation process of the conventionally trained GNN, sequentially trained layerwise GNN, and the parallel trained GNN. With this illustration, case in which one layer is forwarding or back-propagating the signal while other layers are idle. Therefore, that given same number of batches of data, the parallel version would finish training much earlier than the conventional and the sequential version.
can observe that the parallel training of layerwise GNN can avoid the
Following our notations in section 2 we denote by F the normalized adjacency matrix, ( the output activation of [-th layer, and (!() the learnable parameters for 1-th layer. Plus, let Y be the labels, (1) be the parameters for 1-th layer's classifier, and loss be the cross-entropy loss which is frequently used for classification tasks. Then, we have the per-layer objective func- tion: loss(w:t) e (Y, (I-1), F; ( (()). We now formally define our DGL-GNN training method in algorithm [1] Note that, the inner for-loop can be done in parallel manner, i.e., when the !-th layer is working on the backward process as given in line 5, the (1 + 1)-th layer can start forward propagation as given in line Therefore, we claim our DGL-GNN algorithm can achieve
We then empirically observed that, without passing the signal to next layer immediately after the forward process, we still get same-level performance. Thus, we find that the efficiency of DGL- GNN can be further improved by leveraging an Lazy Update scheme (LU-DGL-GNNN) Instead of using the up-to-date activation output from its predecessor, one layer can use the history activation to Iearn its parameters and only update the history activation a few times during the overall training process. Then, same as sequential trained layerwise GNN, we only need conduct the aggregation step for one time after each update, this saves a lot time. We denote by (!) the aggregated stored history activation for layer I. We now formally define the LU-DGL-GNY method in algorithm 2 marked its difference with DGL-GNN in blue.

---

Under review as a conference paper at ICLR 2021
To sum-up, our proposed DGL-GNN and LU-DGL-GNN methods enjoys very high efficiency be- cause we introduce an auxiliary greedy objective for each layer and thus achieve pppdate-unlocking then decouple the model into layers and therefore enable the model to be trained in parallel, and we finally propose to leverage the lazy-update scheme, with which we can avoid redundant computation in the aggregation step and further reduce the training time.
COMPLEXITY ANALYSIS
As shown in Table [ our proposed methods achieve a lower complexity compared to the conven- tional training and other baselines. Note that DGL-GCN can be regard as a special case in which Twast = 1, i.e. we update the stored activation every epoch. Therefore, we focus on the complexity justification for LU-DGL-GCN.
For time complexity, first, we know that the training process consists two fundamental operations: aggregation and transformation. The time complexity of aggregation is O(I) and the time complexity of the transformation step is O(N Then, we note that, for LU-DGL-GCN, during the full learning process, for each layer, we have to do aggregation T T/Twait times, and we have to do the transformation 27 times because this step should be conducted for both the GNN layer and the auxiliary classifier. Since the computation for each layer is done in parallel, we know that the overall time complexity for LU-DGL-GCN should be (T]) Allo) //mmitt + N K?). In practice, we put different layers on different GPUs and do the training in parallel, there would be some extra non non-negligi time cost for GPU communication.
For memory complexity, it also consists components. We have to store two things for each layer during the training: the history activation and the intermediate learnable weight matrices for GNN and for the auxiliary classifier. The activation takes ONNK), and the two types of weight matrix take (²2 space. Again, when we do the training in a parallel fashion and assign the layers to different machines, the Per-GPU memory would only be O(NK + 2K2), which is significantly reduced compare to most of the existing baselines.
ANALOGY TO BLOCK COORDINATE DESCENT
To justify the rationality of the proposed model, present here analogy of decoupling approach to the classic Block Coordinate Descent (BCD) optimization strategy and its variants (Wright 2015 Wu et al. 2008 Shi et al., 2016), which for completeness are discussed in Ap- pendix A.2, We observe that, the sequential layerwise GNN share similar high-level idea with the BCD method. We regard each layer and its associated auxiliary classifier as module. Then, for each module, all its parameters can be treated as a coordinate block, we order the coordinate block according to which layer it corresponds to. Note that, in BCD, for each iteration, we choose one coordinate block and optimize the overall training objective with respect to the chosen block. So if we keep choosing the first coordinate block until it fully converged, then keep choosing the second coordinate block, etc., until the last coordinate block fully converge, then this optimization process is the same as the learning process of a sequentially trained layerwise GNN.
We also oberve that, the DGL-GCN can be analog the synchronous parallel BCD and the LU- DGL-GCN can be regard as an analogy of asynchronous parallel BCD. Note that our DGL-GCN and LU-DGL-GCN can be implemented in a parallel fashion, and their key difference is whether all the layers share a consistency and up-to-date information. Therefore, if we make the same analogy of learnable parameters and the coordinate blocks as in the above sequential version, then would be easy to find the similarity between parallel BCD and our decoupled greedy learning methods. With such analogy, it would allow us to leverage existing theorems for BCD optimization to better understand and analyze the DGL-GCN and LU-DGL-GCN,
EMPIRICAL RESULTS
We evaluate our proposed algorithms with the multi-class node classification task. However, it should be noted that the decoupled greedy learning method can also be applied in other graph-related tasks and is not limited to node classification.

---

Under review as a conference paper at ICLR 2021
Table 2: Comparison of LU-DGL--GCN with baseline methods benchmark datasets. For small datasets (Cora, Citeseer, Pubmed), we set the number of layers as 2, for large OGBN-arxiv dataset, set it as 7. We set Tlazy 50 here for LU-DGL-GCN, Results show that LU-DGL-GCN can achieve superior efficiency without much performance sacrifice.
Dataset
Sample Method
Accuracy ($6)
Total Time(s)
Mem(MiB)
Full-Batch GCN
81.1 0.5
4.8 +
1275
LADIES
79.2 + 1.2
3.1 + 0.6
1249
FastGCN
78.9 + 3.2
4.7 + 1.3
1257
Cora
LGCN
78.9 0.8
2.9 + 0.3
1239
LU-DGL-GCN
79.3 1.0
1.4
1269
Full-Batch GCN
71.6 0.3
5.6 0.2
1401
LADIES
66.5 1.1
2.4
1327
FasIGCN
62.9 2.3
5.8 0.9 0.9
1335
Citeseer
LGCN
69.7 0.7
3.7 0.3
1335
LU-DGL-GCN
68.8 - 0.8
1.5 0.0
1331
Full-Baich GCN
79.3 + 0.3
10.8 0.2
1445
LADIES
78.1 0.6
2.7
1333
FasIGCN
42.6 3.4
2.7
1341
Pubmed
LGCN
77.8 0.7
3.5 0.2
1327
LU-DGL-GCN
77.2 = 0.5
2.2 0.1
1403
Bull-Batch GCN
71.9 - 0.2
162.9 123.44
4327
LADIES
49.1 + 2.8 2.8
50.4 + 16.8
1301
FasIGCN
21.6 + 0.0
24.0 + 6.3
1293
OGBN-arxiv
LGCN
68.8 0.1
122.5 + 17.2
1424
LU-DGL-GCN
69.3 0.3
23.6 0.1
1473
We use the following four public datasets for evaluation: cora, citeseer, pubmed (Sen et al.. 2008
and OGBN-arxiv (Hu et al., 2020). We briefly introduce these datasets and summarize their statistic
in Appendix A.3 We compare our method against several baseline models introduced in Section
Bull-Batch GCN (Kipf & Welling 2016), FastGCN (Chen et al.) 2018). LADIES (Zou et al. 2019)) and LGCN YYou al. 2020)) For all the methods, set all the hidden-dimension as 1 128. We follow the public implementations of all the baselines, and use their parameter settings. We conduct the training for 10 times and take the mean and variance of the evaluation results. Further implementation details are provided in Appendix A.3.
use the same deepGCN model architecture and
We evaluate the performance of different methods with the following evaluation metries: Accuracy (%): The micro F1-score of the test data at the convergence point. Memory (MiB): The maximum per-GPU memory cost during training. Total Running Time (s): The total training time (exclude validation) before convergence.
We summarize the classification performance results in Table 6 which demonstrates the efficacy of approach. We can see that, for small-scale datasets such as Cora, Citeseer, Pubmed, LU-DGL-GCN is the fastest among all the baselines, and its accuracy is only lower than Full- Batch GCN but higher than all the other efficient GCN training strategies. Since only 2-layer GCN architecture for these small-scale datasets, we can not see much memory advantage. deeper 7-layer GCN architecture, then we can
For the large-scale OGBN-arxiv dataset, find that, compared to Full-Batch GCN, our LU-DGL-GCN is much faster and can save per-GPU memory while only sacrifice minor performance. Compared to sampling-base methods (LADIES and FastGCN), our LU-DGL-GCN has a better accuracy and is faster, while only need slightly more per-GPU memory. Note that for here, we set the sample number 64 for FastGCN and LADIES, we increase the sample number, we will obtain a higher accuracy but the time cost and memory cost would be larger. Compared LGCN which a sequantially-traine
layerwise GCN, our LU-DGL- GCN has clear advantage the time cost, and can even obtain little accuracy improvement. Our experimental results together with our analysis previous sections indicate that our LU-DGL- GNN would be very helpful for time and memory limited applications, especially for the scenarios in which we need deep models. Furthermore, to establish the importance of different components of method, isolate certain aspects of it via the following ablation study.

---

Under review as a conference paper at ICLR 2021
Table 3: Comparison of LU-DGL--GCN with different Taasy Note that, Tlasy corresponds to the DGL-GNN model. Results show that with proper Tlasy. we can reduce time cost and improve accuracy while maintaining same memory cost.
Dataset
Performance
Tlozy
Thaay
Tlozy 10
Tiuzy 20
Tiosy 50
Accuracy (%)
78.9 + 0.8
79.1 + 0.5
79.0 + 0.1
79.2 + 0.6
79.3 + 1.0
Cora
Total Time (s)
1.7 0.1
0.07 1269
1.6 0.2 1269
1.4 0.1 1269
1.4 = 0.1 1269
Memory (MiB)
1269
Accuracy (%)
69.1 0.3
69.5 0.2
69.4 + 0.2
69.3 0.3
69.3 + 0.3
OGBN-arxiv
Total Time (s)
332.5 + 14.1
85.8 + 1.4
48.1 = 0.5
33.1 0.1
23.6 = 0.1
Memory (MiB)
1473
1473
1473
1473
1473
Importance of Lazy Update Scheme. We illustrate the advantage of the Lazy Update Scheme and show how the waiting time Tizzy will influence the performance. We use the cora and OGBN-arxiv datasets examples for small-size and large-size scenarios, and follow previous model architectures and parameter. We run the model for 200 epochs and compare the obtained accuracy, total running time, and memory. We summarize the results Table We find that, with the lazy update scheme, we can greatly reduce the time cost, and a proper waiting time Tiazy would help to improve the addition, when comparing the accuracy
accuracy little bit since it can alleviate overfitting. curves of different Tlayer shown in Figure we find that large Tlazy makes the training more stable.
Sequential Training v.s. Parallel Training. Finally, we briefly compare the sequential training and our parallel training of the greedy objective. Again, we use cora and OGBN-arxiv as examples and follow the above experiment settings. shown Figure we find that in terms of aceuracy, parallel training can quickly catch up with the sequential training, and is less likely to overfit.
layer,
layers,
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
Sepeenbal Just
SecuerraLii Lasy 144:
0.3
Layy
0.2
LazyS
Larp5 cem:
0.1
0.1
0.0
0
25
50
75
125 150 175 200 Epoch
100
0
100
200
300
400
500
Epoch
Figure 3: Comparison of sequential and parallel training. Results show that parallel can quickly catches up to sequential training within few epochs.
CONCLUSIONS
In this paper, focus the efficiency issue of GNN training in large-scale applications and present a decoupled greedy GNN learning strategy. Our proposed DGL-GNN model achieves update- unlocking by introducing greedy auxiliary objectives during training, and enables parallelization by decoupling the GNN into smaller modules. We also propose the LU-DGL-GNN method, which leverages a ladate scheme during training further improve the model efficiency. We empir- ically analyze our proposed DGL-GNN model and demonstrate its effectiveness and superior effi- ciency through a range of experiments. We note that while we introduce our proposed method with deepGCNN model and use the semi-supervised node classification task as example, the DGL-GNN and LU-DGL-GNN methods are not limited to this setting, and can be applied to other GNNs and graph-related downstream tasks. Further, while here we focus comparing the decoupled approach as an alternative to other sampling based methods respect to their accuracy and efficiency, these ap- proaches be regarded complementary each other. By combining the decoupled greedy learning method with other eenhaiiitt-eehancinng improvements GNN training, the computation cost would be further reduced, which poses promising direction for future work.
baseed
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yunsheng Bai, Hao Ding, Song Bian, Ting Chen, Yizhou Sun, and Wei Wang. Simgnn: A neu- ral network approach fast graph similarity computation. Proceedings of the Twelfih ACM International Conference Web Search and Data Mining. pp. 384 392, 2019.
Amir Beck and Luba Tetruashvili. On the convergence of block coordinate descent type methods. SIAM journal on Optimization, 23(4):2037-206) 2013.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Decoupled greedy learning of cnns. arXiv preprint wwXiv:1901.011600 2019a.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can scale to imagenet. In International conference on machine learning, pp. 583-593, PMLR, 2019b.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances in neural information processing systems, pp. 153-160, 2007.
Jianfei Chen, Jun Zhu, and Le Song. Stochastic training of graph convolutional networks with variance reduction. arXiv preprint arXiv: 10568, 2017.
Jie Chen, Tengfei Ma, and Cao Xiao. Fastgen: fast learning with graph convolutional networks via importance sampling. arXiv preprint arXiv:180/ /0247, 2018.
Wei-Lin Chiang, Xuanqing Liu, Si Si. Yang Li. Samy Bengio, and Cho-Jui Hsich. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp- 257-266, 2019.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances neural information processing systems, pp. 1024-1034, 2017.
Geoffrey E. Hinton, Simon Osindero, and Yee whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:152 18:1527-1554, 2006.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00685 2020.
Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Inter- national Conference on Machine Learning, pp. 1627-1635, PMLR, 2017.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv:1609.02990 2016.
Guohao Li, Mathias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go deep as cnns? Proceedings of the IEEE International Conference on Computer Vision, pp. 9267-9276, 2019.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. arXiv preprint / 076606, 2018.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Ran Rad. Collective classification in network data. Al magazine, 29(3):93-99 2008.
Hao-Jun Michael Shi, Shenyinying Tu. Yangyang Xu, and Wotao Yin. A primer on coordinate descent algorithms. arXiv preprint
2016.
Stephen J Wright. Coordinate descent algorithms. Mathematical Programming, 151(1):3-34, 2015.
Tong Tong Wu, Kenneth Lange, et al. Coordinate descent algorithms for lasso penalized regression. The Annals Applied Statistics, 2(1):224-2444 2008

---

Under review as a conference paper at ICLR 2021
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 974- 983, 201
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hi- erarchical graph representation learning with differentiable pooling. In Advances in neural infor- mation processing systems, pp. 4800-4810, 2018b.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. L2-gen: Layer-wise and learned efficient training of graph convolutional networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2127-2135, 2020.
Difan Zou, Ziniu Hu, Yewen Wang. Song Jiang, Yizhou Sun, and Quanquan Gu. Layer-dependent importance sampling for training deep and large graph convolutional networks. In Advances in Neural Information Processing Systems, pp. 11249- 11249-11259, 2019.
A APPENDIX
A.1 COMPLEXITY ANALYSIS FOR BASELINES
In this section, we explain how we compute the memory and time complexity in Table !
All the aforementioned methods's 's memory cost consists two parts: intermediate embedding matrices storage and weight matrices storage. The weight matrices always need (LK)) since it has to store L weight matrices with dimension K x K. The intermediate embedding has different memory cost for different methods. In terms of time complexity, it also consists two parts: the aggregation time cost and transformation time cost. These two parts varies for different methods.
Full-batch GCN stores all the intermediate embedding matrices for all the L layers, and each matrice has N nodes with dimension K, so its memory complexity for intermediate embedding storage would be O(LN K). Therefore, its total memory complexity is O(LN + LK*). In terms of time complexity, the propagation step which is a sparse-matrix multiplication has time complexity O[l A ).. and the transformation step which is a dense-matie multiplication has time complexity O(N K 2) During the training for L layers and T iterations, the total time complexity would be [ A looK + TLNK3).
For GraphSAGE, it has to store (bsliede) K K-dimension node embeddings, so it has a total memory node + LK?). In terms of time complexity, for each batch, to update one node, have to update Oss activations, each needs (KCfu)) for aggregation and K2 for transfor- mation. Therefore, the total time complexity for GraphSAGE is O(T K shode + K2 (
complexity O(bK 1--1
For VR-GCN, it stores all historical activations, which takes memory of O(LN) and will lead to the total memory complexity OLLN + LK²). The time complexity can be analyzed similarly as GraphSAGE, which would be OLTD & + + Rode).
For FastGCN, it only has store node embeddings in the last layer, and (T. 1))tayer node embeddings in the previous L - 1 layers, so the total memory complexity would be Olbk + (L 1) stayer) + LK2). In terms of time complexity, we need O K Stayer + (L- - 1)T sayer) for for transformation. Note that, we have b < Stagers
aggregation and ((TTK2+ (L (L- (tayer so we ignore the relatively small terms and get the total memory complexity OLLK + LK2), and total time complexity O(T stayer TLK stayer))
For LADIES, its is also layer-wiss wise sampling method. Though it has different sampling strategy as FastGiNN, their memory cost and time cost are the same. There fore, LADIES also has memory complexity (I.K T.22) and time complexity and OTT K + TLK2 TtKKettger))
A.2 SUPPLEMENTARY DISCUSSION OF COORDINATE AND BLOCK COORDINATE DESCENT
Coordinate descent (CD) a classic iterative optimization algorithm that solves an optimization problem by approximately minimizing the objective along each coordinate directions successively.
10

---

Under review as a conference paper at ICLR 2021
Table 4: Statistics of Benchmark Dataset
Dataset
Cora
Citeseer
Pubmed
OGBN-arxiv
Nodes
2708
3327
19717
169343
Edges
5429
4732
44338
1166243
Classes
40
Feature 1433
3703
500
100
In each iteration, it would choose one variable, fix the other components, then optimizing the objec- tive with respect only the single variable. By doing so, only need to solve lower dimensional minimization problem at each iteration, which would he easier. CD algorithm has been discussed in various literatures and has been used in applications for long time (Wright 2015 Wu al. 2008 Shi et al. 2016).
Block coordinate descent (BCD) an extension of CD method. The difference of BCD and the conventional CD is that BCD will do the searching along a coordinate hyperplane instead of a single coordinate direction (Beck Tetruashvili, 2013). i.e. groups variables into blocks, and approximately minimizing the objective with respect to only one block of variables at each iteration while fixing the others.
For BCD algorithm, there exists its parallel implementations. As introduced in the work of (Wright, 2015), we can categorize them into two types: synchronous and asynchronous. For synchronous parallel BCD, we partition the computation into pieces and put different pieces on different pro- cessors, each processor will update part the variables in parallel, then a synchronization step should be conducted to guarantee the consistency of the information shared among all processors before further computation. For asynchronous setting, the difference is we do not have to do the synchronization. As discussed in Section 5 our proposed method can be regarded analogous to BCD and its aforementioned variants.
A.3 SUPPLEMENTARY FOR EXPERIMENTS
Dataset Statistics We use four benchmark dataset: Cora. Citeseer, Pubmed, Reddit, and OGBN- arxiv for the node classification task. The detailed statictics of theses datasets are shown in Table
Hardware We run the experiments on Tesla V100 GPU (16GB).
As introduced before, as NS methods are in general less competitive than the IS methods, we only compare against the IS strategies.
Baselines Detailed introduction of our baselines can be found Section 2
Model Architecture For Cora, Citeseer, Pubmed, we set the number of layers as 2 for all the meth- ods. For OGBN-arxiv, we it to 7.
Parameter Settings For all the methods and datasets, we conduct training for 10 times and record their mean variance. For small datasets (Cora, Citeseer, Pubmed), we set the epoch number as 200, for OGBN-arxiv, we set the epoch number as 500. We choose the model with the best validation performance as convergence point. For the sampling- lrapliigg-ssedd methods: FastGCN and LADIES, we set the sample number as 64, increasing this number would improve the accuracy little bit, but would increase the memory and time cost.
11


Reviews:
[{"review_id": "afneHE8l3Dl", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper gives up end-to-end training and instead trains graph convolution nets layer by layer. The authors call such layer-by-layer training process decoupled greedy learning (DGL-GNN).  In the decoupled greedy learning process, a lazy-update scheme is adopted, that is to exchange information between layers after a certain number of epochs instead of every epoch. The aggregation and transformation process are also separated and performed respectively.  Authors show their proposed engineering methods can save a constant fraction of computation time for graph convolution nets, but for the cost of sacrificing some predictive performance. Authors conduct some experiments on semi-supervised node classification data, reporting less memory and runtime but worse performance.\n\nWhile the paper attempts to study an important problem of scaling graph networks to large graphs, and authors show the proposed methods is capable of saving in terms of memory and runtime by sacrificing predictive performance (which is not surprising at all), it is not suitable for publication at ICLR. \n\nComments: \n\nNovelty is clearly below the bar of ICLR. The proposed method \"layer-by-layer training\" is rather trivial. \n\nNo theoretical or empirical justification is given for giving up end-to-end training. In fact, there are many easy counter-examples to show this approach would fail. Moreover, another problem with this method is the lack of auxiliary labels for each layer. It is easy to find problems where we do not have such layer wise auxiliary labels, and the proposed method would not work. \n\nMoreover, even by doing layer by layer training, the proposed method cannot solve the scaling issue for billion scale networks, so some sampling is still needed.\n\nThis paper is purely experimental; however, the experiments are unsatisfactory:\n- The reported performance is not competitive. In Table 2 and Table 3, test performance of proposed method is worse than prior works, e.g., LGCN, despite consuming even more memory cost. In this sense, practitioners would simply prefer LGCN to the layer-by-layer training. \n- The datasets in experiments are very small, e.g., cora only have a few thousand nodes. What is the value of acceleration (while sacrificing performance) on such small graphs? \n-  More graph neural network architectures need to be tested to show the effectiveness of the method: 1. graph attention network, 2. graph isomorphism network, 3. graphsage, 4. chebyshev GCN. \n-  Missing link prediction benchmarks. \n- Missing baselines: SIGN, GraphSAINT.\n\n\nGrammar mistakes: \nthen after it get fully converged -> gets\n\noptimize next layer -> the next\n\nwe do the same things -> thing\n\nwhich scales exponentially with the number of layers -> factual error", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZiaE-nnbwtm", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "GNN is known to be computationally complicated and it requires significant amount of time and memory. Unlike CNN, inference of the node embedding in GNN requires information be propagated throughout entire graph for each epoch. In this paper, the authors proposed to decouple the inference and aggregation in each layer and learn the layer-wise latent representation separately. The result is to decompose the deep GNN into a series of shadow networks that are connected sequentially. Since the optimization is done independently, it requires significant less time and memory.\n\nThis paper is clearly written and the contribution, literature review is sufficient for reader to understand. The analogy to block coordinate descent is a good part for reader to connect this work with the well-established work in optimization theory. The idea of layerwise decomposition is published in [You et al., 2020] and this paper improve over that by introducing more parallelization. \n\nPros:\n1. The layer-wise decomposition is not a new idea but to optimize GNN, it is not as popular as in normal DL framework. The study on this topic will definitely brings some attention to the community and is enouraged.\n2. The parallelization strategy and lazy update scheme is useful and easy to implement\n3. In the experiment, it shows that the time reduction of LU-DGL-GCN is significant and it does not depends the depth of model. Therefore it is applicable to larger graph and deep structure.\n\nCons:\n1.  Unlike the traditional convex optimization where the block coordinate decent is know to converge to optimal point. In the settings of GCN, there is no proof on how close the block coordinate descent would be towards to traditional GCN. \n2.  Following the above question, it is known that the message-passing and aggregation in one layer only accumulate the information in 1-hop neighborhood and as a result, the embedding it learned only encodes the local structure within the 1-hop neighborhood. In many applications, the structural information of the graph can only be obtained through a wider p-hop neighborhood where p > 1. Traditional GNN take inference on p-hop neighborhood through  optimization of p-layers all together in backpropagation.  While the layer decomposition is attractive in time, LU-DGL-GCN loss the accuracy to infer the structural information in larger neighborhood. And because each layer is optimized using independent classifier on the same target, it is not hard to expect the node representation learned in the bottom layer to be less representative in LU-DGL-GCN compared to the traditional GCN as it does not encode the information beyond the 1-hop neighborhood.\n3.  In early DL, the layer-wise optimization only used as pre-training to obtain a good weight initialization, and after that, a further full training is needed to obtain a representative embedding and model. It is suggested to conduct a full training with weight learned by LU-DGL-GCN and to compare with traditional GCN and to see if there are better solutions. \n4.  It is better to discuss the limitation of the decoupled greedy learning, i.e. the model is myopic in that the resulting layer-wise node embedding only collect information in nearest neighbors. This is esp. an issue for lower-level representation since it lacks of prior learned from upper layer to guide its inference. \n5.  The LU-DGL-GCN is a cascade  of  L independent shadow network models as opposed to one L-layer deep model. It is known that the representative and expressive power of the deep models is higher than that of a sequential connected independent shadow models. Thus compared to LU-DGL-GCN, it sacrifices the representation accuracy and model complexity to obtain a faster training and less memory. This is the tradeoff. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "-tcydzrV9IA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThis paper introduces a method for decoupled layer wise training of graph neural networks. This method has the potential to be parallelized and offers computational and memory savings. The authors provide experimental results on 4 datasets and compare against 4 baselines. Results suggest that the approach achieves comparable results to other methods while being faster by parallelization.\n\n**Strengths**\na) Well written paper with clear notation and illustrations\n\nb) Good analysis of the different algorithms. Lays out complexity and memory costs of related work.\n\nc) Analogy to block coordinate descent intuitively makes sense.\n\nd) Nice contribution towards achieving asynchronous/lazy updates for GNNs. \n\n**Weaknesses**\n\n*Novelty and impact need to be strengthened*\na) As mentioned by the authors, greedy layerwise pre-training is an old idea. When it first came out for Deep NNs, everyone was excited about it but nowadays hardly anyone does it. So if this method has to make a comeback for GNNs then the benefits have to be very compelling.\n\nb) Wrt benefits, the presentation around why this idea is compelling needs to be concretely laid out. It seems like there are some computational and memory benefits via parallelization. But is the additional complexity of parallel training worth it? As the authors mention, there can be non-trivial communication costs between GPUs in addition to the additional code complexity.\n\nc) Absent theoretical convergence guarantees, the proposed approach is a heuristic as it relies on an auxiliary function that’s somewhat arbitrary. Further discussion is needed on why the chosen auxiliary function is a good idea. Some ablation studies with different auxiliary functions are needed to shed light on this particular choice of auxiliary function.\n\n*Experiments can be strengthened*\na) The results seem to trade off accuracy with speed and saved memory and results are reported on 3 small and one large dataset. To truly claim that this method generalizes, the authors would need to strengthen their results across some different tasks (not just semi-supervised classification)  and models (not just GCN).\n\nb) More details are needed regarding the experimental setup. Was there a multi-gpu setup? E.g. OGBN-arxiv has 7 layers. The authors report a total running time of 23.6s. Was this on a 7 GPU setup? \n\nc) More ablation and convergence studies. The authors stop at 200 epochs for the smaller datasets. Does the proposed method reach full-batch GCN accuracy at higher epochs?\n\n**Typos and other fixes**\na) Typo abstract: Should be “ Graph Neural Networks (GNNs) *have* become”\n\nb) Table 6 is missing.\n\nc) Results: “... but higher than all the other efficient GCN training strategies…” ; LGCN is better for pubmed and citeseer\n\nd) “....when comparing the accuracy curves of different Tlayer shown in Figure 3, we find that large Tlazy makes the training more stable…”\ni) Hard to see this from Figure 3. Consider plotting running variance as a shaded overlay. \nii) Plots are too busy. Hard to draw conclusions.\n\n\n**What can the authors do to make the paper better**\na) More thorough experimentation as described in the *Experiments can be strengthened* section.\n\nb) I did not find the code release with the paper. For a paper whose primary claim is computational and memory savings, I think it would be a good idea to include it.\n\nc) Emphasize the novelty of the method otherwise, greedy pre training seems like a lukewarm idea.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a new layer-by-layer training approach for GNN in particular for a large graph. The proposed approach can be easily parallelizable and scale well to a large graph. Reviewers are concerned about the novelty of the approach and the lack of theoretical analysis, and it is not well addressed by the rebuttal. Therefore, this paper is below the acceptance threshold of ICLR. I encourage the authors to revise the paper based on the reviewer's comments and resubmit it to a future venue.
Decision: Reject