Title: Chordal-GCN: Exploiting sparsity in training large-scale graph convolutional networks
Authors: Xin Jiang*
Kewei Cheng*
Song Jiang*
Yizhou Sun
Venue: ICLR 2020
Keywords: graph convolutional network
semi-supervised learning

Abstract: Despite the impressive success of graph convolutional networks (GCNs) on numerous applications, training on large-scale sparse networks remains challenging. Current algorithms require large memory space for storing GCN outputs as well as all the intermediate embeddings. Besides, most of these algorithms involves either random sampling or an approximation of the adjacency matrix, which might unfortunately lose important structure information. In this paper, we propose Chordal-GCN for semi-supervised node classification. The proposed model utilizes the exact graph structure (i.e., without sampling or approximation), while requires limited memory resources compared with the original GCN. Moreover, it leverages the sparsity pattern as well as the clustering structure of the graph. The proposed model first decomposes a large-scale sparse network into several small dense subgraphs (called cliques), and constructs a clique tree. By traversing the tree, GCN training is performed clique by clique, and connections between cliques are exploited via the tree hierarchy. Furthermore, we implement Chordal-GCN on large-scale datasets and demonstrate superior performance.


TL;DR: Chordal-GCN is a scalable graph neural network which exploits the exact graph structure (i.e., without approximation or sampling) and requires limited memory usage.

Review discussion:
In this paper, the authors propose a new method referred to Chordal-GCN to optimize memory usage in large graphs. The authors borrow the ideas from Chordal Sparsity theory to first build a client tree.
Then mini-batch updates are carried out individually on each clique from the leaves following the GCN loss. The authors add an additional consistency loss between shared node with children cliques. Experiments are carried out in four networks with comparison to several baselines.
Strength:
1. The authors study an interesting and important problem to reduce memory usage for GCN in large-scale graphs. The usage of chordal sparsity is interesting and innovative.
2. The authors carry out ablation study on the consistency loss components in the algorithm.
Weakness:
1. One major concern is that the reduction in memory usage is not large enough to justify the huge increment in running-time.
For example, on Cora dataset, the memory is reduced by 4x while the running time is 16x compared to vanilla GCN.
2. It is not clear why Chordal-GCN can achieve better accuracy compared to vanilla GCN. Since Chordal-GCN serves as approximation for GCN as the authors claim that using entire graph will provide better accuracy.
As a result, the vanilla GCN is expected to achieve better accuracy. It would be better if the authors could provide more intuition and explanations.
3. The evaluation does not take the graph preprocessing into consideration. The authors should report the time and memory taken to carry out the preprocessing steps as well.
4. For most real-world large-scale industry networks, it is hard to fit the graph into memory.
Though the GCN training part could run in distributed way, it is not clear how to efficiently build the clique tree in similar method.
5. Given the main purpose of the algorithm is to reduce memory usage for large-scale networks, it is expected to see experiments on larger graphs where the large memory footprint becomes a real issue.
Detailed comments:
1. The description in Section 2.2 is not very clear. It would be better if the authors could provide a more detailed introduction to clique tree and the algorithms used.
2. For the Chordal-GCN in algorithm 1, for epoch 2 onwards, do we also add consistent-loss when training leaves as well?
We thank the reviewer for the constructive reviews. We addressed the questions and concerns of the reviewer accordingly in the following.
Weakness:
1. One major concern is that the reduction in memory usage is not large enough to justify the huge increment in running-time.
For example, on Cora dataset, the memory is reduced by 4x while the running time is 16x compared to vanilla GCN.
Among all the baselines, vanilla GCN always takes the minimum time. Chordal-GCN is slower because Chordal-GCN executes more than one GCN model in every epoch, i.e., one GCN for one clique in Chordal-GCN.
However, the running time of Chordal-GCN is comparable to other baselines.
Actually, in Chordal-GCN there is a trade-off between the memory and the training time. If we divide the graph into more cliques, then the required memory, which is determined by the largest clique size, will be reduced.
But at the same time, the training time will increase because every clique corresponds to one GCN model. We will explore this balance issue in the future.
2. It is not clear why Chordal-GCN can achieve better accuracy compared to vanilla GCN. Since Chordal-GCN serves as approximation for GCN as the authors claim that using entire graph will provide better accuracy.
As a result, the vanilla GCN is expected to achieve better accuracy. It would be better if the authors could provide more intuition and explanations.
Both Chordal-GCN and vanilla GCN use the entire graph---without sampling or any other approximation, so we don’t treat our model as an ‘‘approximation’’ of GCN.
Instead, Chordal-GCN modifies the GCN training procedure such that the training can be performed in a distributed manner.
An intuition is provided in paragraph 3 of the introduction section: in most citation networks, highly-cited papers should have impacts on multiple communities.
Thus, Chordal-GCN treats the ‘‘inter-cluster’’ links and the ‘‘intra-cluster’’ links differently (while GCN doesn’t make a distinction). In training a particular node, Chordal-GCN considers the influence of its neighbors in the same clique first, and the impacts from other cliques are handled via the consistency loss. This different treatment might explain an increase in the accuracy.
3. The evaluation does not take the graph preprocessing into consideration. The authors should report the time and memory taken to carry out the preprocessing steps as well.
The most expensive step in preprocessing is building the clique tree, of which the time and memory are both linear in the number of nonzeros in the adjacency matrix A.
4. For most real-world large-scale industry networks, it is hard to fit the graph into memory.
Though the GCN training part could run in distributed way, it is not clear how to efficiently build the clique tree in similar method.
The SOTA methods for building clique trees [1,2] can be easily extended to a distributed version. Basically, one can start with the node with the largest ID, and then find the clique this node is in. Other neighbors of this node (with largest ID) are certainly in the children clique. Thereby, every time we only need to know the neighbors of the current node, instead of the entire graph.
5. Given the main purpose of the algorithm is to reduce memory usage for large-scale networks, it is expected to see experiments on larger graphs where the large memory footprint becomes a real issue.
Thank you for your suggestion. we plan to include this result in our final draft.
Detailed comments:
1. The description in Section 2.2 is not very clear. It would be better if the authors could provide a more detailed introduction to clique tree and the algorithms used.
Thank you for your advice and sorry for the confusion. The algorithm for building the clique tree is too sophisticated to explain in two pages, and is not the focus of our paper.
So we choose not to include in the preliminaries. We hope the unclearness in Section 2.2 does not affect the understanding of our main model.
2. For the Chordal-GCN in algorithm 1, for epoch 2 onwards, do we also add consistent-loss when training leaves as well?
No. The consistency loss involves the current clique and its children in the clique tree. Since leaves of a tree never have a child (ch(i) is a empty set in Eq. (2)), there is no consistency loss when training leaves.
[1] P. Buneman. A characterization of rigid circuit graphs. Discrete Mathematics. 9:205-212, 1974.
[2] F. Gavril. The intersection graphs of subtrees in trees are exactly chordal graphs. Journal of Combinatorial Theory Series B, 16:47-56, 1974.
This paper leverages the clique tree decomposition of the graph and design a new variant of GCN which does graph convolution on each clique and penalize the inconsistent prediction made on separators of each node and its children. Experiments on citation networks and the reddit network show that the proposed method is efficient.
Overall, this paper could be a significant contribution on improving GCN, with the caveat for some clarifications on the model and experiments. Given these clarifications in an author response, I would be willing to increase the score.
Pros:
1, I like the idea of exploiting graph decomposition. In my opinion, it may not only improve the scalability but also help the model learn representations which better capture the structure or speed up the learning process. It would be great if authors could show some evidence along this line.
2, The examples in Figure 2 and 3 are very helpful in understanding the concepts related to the clique tree decomposition.
3, The summarization of time and memory complexity is very helpful in comparing different models.
4, I read the detailed questions and responses in the open review. It helps me understand more details about the experiments.
Besides the typo of Table 2, I tend to believe that the experimental setup is reasonable and results are convincing although I did not run the code by myself.
Cons & Questions:
1, The main motivation of exploiting the graph decomposition is to save memory such that GCN could be applied to large scale graphs without sacrificing the structural information.
However, the scale of the largest experiments is still less impressive. To strengthen the paper, it would be great to try larger graph datasets which have been used in the literature.
2, I am confused by the writing on the final prediction made by the model.
In particular, do you only keep the prediction of residual or do you average the predictions on the separators? It may be interesting to explore different ways of making predictions based on this decomposition based inference. In general, it would be great to separate the writing of loss (learning) and prediction (inference).
3, Why does Chordal-GCN take significant more epochs than GCN on Reddit and less epochs on all other datasets?
Suggestion:
1, I think the clique tree is very similar if not the same with the junction tree given the node ordering (see section 2.5.2 of [1]). It would be great to discuss the relationship between your chordal graph representation and the tree decomposition used by the probabilistic inference algorithms of graphical models. From the perspective of complexity, the junction tree method and yours both highly depend on the tree-width.
Also, linking to probabilistic inference could help better motivate the method since tree-based inference algorithm is shown to converge faster in the literature.
2, It would be great to discuss and or compare with [2] as it uses graph partition algorithms to get clusters and apply GNN with a propagation schedule which alternates between within-cluster and between-cluster. It is closely related to the chordal-GCN as it uses the decomposition of graph clustering directly rather than the clique tree. Decomposition like multiple overlapping spanning trees are also studied in [2].
[1] Wainwright, M.J. and Jordan, M.I., 2008. Graphical models, exponential families, and variational inference. Foundations and Trends® in Machine Learning, 1(1–2), pp.1-305.
[2] Liao, R., Brockschmidt, M., Tarlow, D., Gaunt, A.L., Urtasun, R. and Zemel, R., 2018. Graph partition neural networks for semi-supervised classification. arXiv preprint arXiv:1803.06272.
Thank authors for the thorough reply! After I read authors' rebuttal and other reviewers' comments, I would like to keep my original rating. Again, I like this idea and believe better exploiting structure in the propagation could improve the inference in many ways. I hope authors could keep improving it, e.g., better motivating the proposed method (memory saving is just one angle which sometimes may need more engineering work to fully verify) and change the experiments accordingly.
We thank the reviewer for the constructive reviews. We addressed the questions and concerns of the reviewer accordingly in the following.
Weakness
1, Thank you for your suggestion. We plan to include this result in our final draft.
2, In our current submission, the testing procedure is identical to the vanilla GCN because storing the entire graph is not the main problem (storing GCN parameters and embeddings needs much more space than storing the graph).
That is why we only include the training phase in Algorithm 1.
Nevertheless, your suggestion might motivate further study in another direction: Comparison between the labels predicted by the entire graph and those predicted by every clique might enlighten better understanding on the graph structure.
For example, when the predictions are different, does it mean that the current clique has a ‘‘negative’’ impact on predicting this node; and is this negative impact due to noise in the network, or in the feature matrix? We believe all these questions are interesting but challenging, and thus beyond the scope of this paper.
3, The outlier in the Reddit dataset is attributed to a bad choice of hyperparameters. With our current hyperparameters, Reddit is decomposed into a giant clique and some small cliques. This unbalanced decomposition cause the abnormal results. We will improve the results in the Reddit dataset by finding better hyperparameters in our final draft.
Suggestion:
1, The clique tree in our paper is indeed the same as the junction tree in [1] because any chordal graph must have a clique tree with the running intersection property (in Definition 2.1 of [1]). The only minor difference is that, in our definition, every node in the clique tree is a ‘‘maximal’’ clique while in some literature, the junction tree does not require maximality. In our humble opinion, the term ‘‘clique tree’’ is more used by the chordal sparsity community while the term ‘‘junction tree’’ is often used by researchers in graphical models and probabilistic networks.
We would have motivated our idea from the perspective of message passing and probabilistic inference.
However, we try to avoid the illusion that Chordal-GCN is a new, sophisticated variant of GCN model;
instead, we would like to persuade readers that Chordal-GCN is a modification in the GCN training procedure.
After all, the idea of linking GCN to probabilistic inference itself is interesting and worth further exploration.
Moreover, we appreciate it a lot if you can notify us with any efficient C implementation of building junction trees, and the tree-based inference algorithms. We are happy to compare the performance of chordal decomposition and the junction tree decomposition in [1].
Although our model still remains the same, the actual running time (especially the preprocessing time) might be improved dramatically.
2, Thank you for bringing the paper [2] into our attention. In our biased opinion, it is more similar to Cluster-GCN [3]. GPNN [2] finds a partition of the nodes and treats inter-cluster and intra-cluster links differently while Chordal-GCN somehow finds a ‘‘partition’’ of the edges and treats clique separators and residuals in a different manner. Building minimum spanning trees in [2] is also used for node partition purpose.
After all, it is a very interesting and highly related paper. We have added the citation and will consider it as an important baseline in our final draft.
Last but not least, we would like to thank you again for your affirmation in our paper. We also appreciate your helpful and insightful comments and suggestions.
[1] Wainwright, M.J. and Jordan, M.I., 2008. Graphical models, exponential families, and variational inference. Foundations and Trends® in Machine Learning, 1(1–2), pp.1-305.
[2] Liao, R., Brockschmidt, M., Tarlow, D., Gaunt, A.L., Urtasun, R. and Zemel, R., 2018. Graph partition neural networks for semi-supervised classification. arXiv preprint arXiv:1803.06272.
[3] W.-L. Chiang, X. Liu, S. Si, Y. Li, S. Bengio, and C.-J. Hsieh. Cluster-GCN: an efficient algorithm for training deep and large graph convolutional networks. KDD 2019.
The authors propose Chordal-GCN which is based on the chordal decomposition method post-ordered clique tree and propagates the features based on the order within each subgraph in order to reduce memory usage. The authors show that Chordal-GCN outperforms GCN [1] on all four datasets and argue that Chordal-GCN reduces memory usage.
The idea of using Chordal graphs to GCN is novel and interesting.
However, my main concern lies in the experiment results.
1) To my best knowledge, the proposed Chordal- match SOTA results on Cora, Citeceer, and Pubmed.
However, since these datasets are small and easy to run, I would like to see the mean and standard deviation of the accuracy of all models you ran. Can you also provide the results of the commonly used "random split setting"[1]?
2) What is the epoch time of the Chordal-GCN? Can you also report it in Table 2? Without including the pre-processing time, we don't know the overall training time of the method.
3) Given that the main concern is the memory usage, the authors should compare to a strong baseline, SGC [2], which is a linear classifier trained on top of propagated features with memory/space complexity O(d) when using mini-batch training. This is much smaller than the proposed method O(Lc_2d + Ld^2).
Also, SGC is at least two magnitudes faster to train (2.7s vs 0.987*410=367.8s + unknown pre-processing time) and more accurate (94.9 vs 94.2) than the proposed Chordal-GCN on the largest Reddit dataset. The authors emphasize that the proposed method is scalable. Please compare it to SGC in Table 2.
Nevertheless, there is some chance that the authors can apply the same method to SGC and speed it up further as long as the preprocessing time is relatively small.
4) Based on Table 2, Cluster-GCN uses less memory and is more accurate and faster to train than Chordal-GCN. Can you justify why people should use the proposed method instead?
5) There are some missing citations. These papers [3,4,5,6] achieved previous SOTA results and should be included in the Tables.
References:
[1] Kipf and Welling: Semi-Supervised Classification with Graph Convolutional Networks (ICLR 2017)
[2] Wu et al.: Simplifying Graph Convolutional Networks (ICML 2019)
[3] Klicpera et al.: Predict then Propagate: Graph Neural Networks meet Personalized PageRank (ICLR 2019)
[4] Gao and Ji: Graph U-Nets (ICML 2019)
[5] Zhang et al.: GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs (UAI 2018)
[6] Fey: Just Jump: Dynamic Neighborhood Aggregation in Graph Neural Networks (ICLR-W 2019)
We thank the reviewer for the constructive reviews. We addressed the questions and concerns of the reviewer accordingly in the following.
1) To my best knowledge, the proposed Chordal- match SOTA results on Cora, Citeceer, and Pubmed.
However, since these datasets are small and easy to run, I would like to see the mean and standard deviation of the accuracy of all models you ran. Can you also provide the results of the commonly used "random split setting"[1]?
Ans: The split in [1] is chosen by the authors, and thus fixed. Existing work [7] has proven that this split of dataset has a significant influence on the classification result. We follow the random held-out strategy and randomly split the dataset for training and test multiple times. This random split strategy is used in all baseline models, and we think this is a fair and consistent setting.
2) What is the epoch time of the Chordal-GCN? Can you also report it in Table 2? Without including the pre-processing time, we don't know the overall training time of the method.
Ans: We have already reported the epoch time in Table 2.
For the preprocessing, the main bottleneck is to build the clique tree, of which the time complexity is linear in the number of nonzeros in the adjacency matrix. In our current implementation, the preprocessing mainly depends on the python package Chompack, and thus it takes more time than Cluster-GCN---as the preprocess in Cluster-GCN depends on a C package.
3) Given that the main concern is the memory usage, the authors should compare to a strong baseline, SGC [2], which is a linear classifier trained on top of propagated features with memory/space complexity O(d) when using mini-batch training. This is much smaller than the proposed method O(Lc_2d + Ld^2).
Also, SGC is at least two magnitudes faster to train (2.7s vs 0.987*410=367.8s + unknown pre-processing time) and more accurate (94.9 vs 94.2) than the proposed Chordal-GCN on the largest Reddit dataset. The authors emphasize that the proposed method is scalable. Please compare it to SGC in Table 2.
Nevertheless, there is some chance that the authors can apply the same method to SGC and speed it up further as long as the preprocessing time is relatively small.
Ans: Thank you for your suggestion. In our biased opinion, Chordal-GCN, as well as all the baselines we used, is a modification in the training phase of the vanilla GCN. Comparatively, SGC should be treated as a totally different graph neural network model. In light of the model structure, SGC only has one layer, and the graph structure $A$ is aggregated with the feature information $X$ before training. This preprocessing step facilitates training but is ‘‘expensive’’ in nature: In preprocessing, one should compute $A^K X$ (in our notation). Computing matrix power $A^K$ involves eigenvalue decomposition of $A$, which is $O(n^3)$. The short preprocessing time in the SGC paper is attributed to highly optimized numerical linear algebra package BLAS and LAPACK, which are written in Fortran language. If clique tree building can be carried out in Fortran or C, the preprocessing time will be limited ($O(||A||_0)$) in Chorda-GCN.
4) Based on Table 2, Cluster-GCN uses less memory and is more accurate and faster to train than Chordal-GCN. Can you justify why people should use the proposed method instead?
Ans: In the first three datasets (Cora, Citeseer, Pubmed), Chordal-GCN uses less memory and achieves better results than Cluster-GCN. We also provide an intuitive explanation in Section 1.
The outlier in Reddit it due to a wrong selection of hyperparameters. With our current hyperparameters, Reddit is decomposed into a giant clique and some small cliques. This unbalanced decomposition causes the abnormal memory cost. We will improve the results in the Reddit dataset by finding better hyperparameters in our final draft.
5) There are some missing citations. These papers [3,4,5,6] achieved previous SOTA results and should be included in the Tables.
Ans: Thank you for bringing these papers into our attention.
However, there are so many papers on GCN models and we can only compare the most related ones. We argue that Chordal-GCN, as well as all the baselines we use, focuses on the training phase of the vanilla GCN. Thereby, variants of GCN models are not considered as baselines in our submission, despite their interestingness.
References:
[1-6] same as above
[7] Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, Stephan Günneman: Pitfalls of Graph Neural Network Evaluation (NeurIPS workshop 2018)

Meta-review:
role: PC
data: comment: The submission is proposed a rejection based on majority review.

Decision: Reject