Chordal-GCN: Exploiting sparsity in training large-scale graph convolutional networks

1 INTRODUCTION . Graph convolutional network ( GCN ) ( Kipf & Welling , 2017 ) is a generalization of convolutional neural networks ( CNNs ) ( LeCun & Bengio , 1998 ) to the graph structure . For a given node , the graph convolution operation aggregates the embeddings ( features ) of its neighbors , followed by a non-linear transformation . By stacking multiple graph convolutional layers , one can learn node representations by utilizing features of its distant neighborhood . The original GCN model , as well as its numerous variations , has shown great success in a variety of applications , including semisupervised node classification ( Kipf & Welling , 2017 ) , inductive node embedding ( Hamilton et al. , 2017 ) , link prediction ( van den Berg et al. , 2017 ) , and knowledge graphs ( Schlichtkrull et al. , 2018 ) . Despite the success of GCNs , training GCNs on large-scale graphs remains challenging due to the memory issue : we need to store all the parameters and outputs of GCN . Thus , the memory space scales linearly in the size of graph while quadratically in the feature dimension ( Chiang et al. , 2019 ; Zou et al. , 2019 ) . This prevents applications of GCN on many real-world networks , where the graphs usually contain millions or even billions of nodes . Methods aimed at large-scale training have been proposed and can be roughly divided into two categories : ( 1 ) sampling-based methods and ( 2 ) clustering-based methods . For sampling-based methods , only a few neighbors for every node will be sampled in every GCN layer , and thus the size of intermediate embeddings for every layer will be reduced for each mini-batch . Works in this track include Hamilton et al . ( 2017 ) ; Chen et al . ( 2018a ; b ) ; Zou et al . ( 2019 ) . However , ignorance of some neighbors might lose important structure information , which is the main drawback of all the sampling methods . Another direction of research notices the sparsity of real-world networks and exploits the clustering structure of the graph . For example , Cluster-GCN ( Chiang et al. , 2019 ) separates the graph into several clusters , and in every iteration of training , only one or a few clusters are picked to calculate the stochastic gradient for the mini-batch . However , Cluster-GCN ignores all the inter-cluster links , which are not negligible in many real-world networks . For example , Figure 1 shows the sparsity pattern of three citation networks . We first rearrange the vertices via an approximate minimum degree ( AMD ) ordering algorithm ( Amestoy et al. , 1996 ) , and then observe a nice arrow pattern in the adjacency matrices . This indicates the existence of some highly-cited papers ( in the right bottom corner ) , which have impacts on multiple communities . In Cluster-GCN , these highly-cited papers are randomly put into one community and the adjacency matrix is approximated by a block-diagonal matrix ; i.e. , Cluster-GCN removes the fletching part of the arrow patterns in Figure 1 . This approximation ruins the beautiful arrow pattern , and thus ignores the multi-community influence of some seminal papers . The above difficulties can be easily tackled by the Chordal-GCN , a novel clustering-based method for the semi-supervised node classification task . Recall that Cluster-GCN ignores all the intercluster links and trains each cluster separately ; comparatively , in Chordal-GCN , we keep all the links , train every cluster separately , and at the same time capture the connections between clusters by an additional loss term . This partially separable training process can be achieved with the help of chordal sparsity theory ( Vandenberghe & Andersen , 2015 ) : we first decompose a large-scale sparse graph into several small dense subgraphs , and we construct a tree of which the nodes are the subgraphs . Note that two subgraphs are adjacent in the tree if they share some vertices . In the training process , we traverse the tree from leaf to root ; and when training on a certain subgraph , we minimize the usual GCN loss , plus an additional term called consistency loss . With the consistency loss , messages in the children subgraphs can be passed to their parent , and thus the relationship between subgraphs is leveraged via the hierarchy of the tree . Therefore , Chordal-GCN exploits the sparsity pattern as well as the clustering structure of the graph without any approximation or random sampling , while requires similar memory space to Cluster-GCN . Our contribution is summarized as follows : • We propose Chordal-GCN for semi-supervised node classification on large-scale sparse networks . The proposed model fully exploits the exact graph structure , while requires limited memory usage on large-scale graphs ( much smaller than the original GCN ) . • Chordal-GCN is able to train a large-scale graph in a partially separable manner ; i.e. , in every iteration , the training is performed on a subgraph , and the connections between subgraphs are handled by a consistency loss . • We analyze the memory and time complexity of Chordal-GCN and compare them with other state-of-the-art GCN models . Also , we evaluate the performance of Chordal-GCN on benchmark datasets and demonstrate superior performance in large-scale datasets . 2 BACKGROUND AND RELATED WORK . 2.1 SEMI-SUPERVISED NODE CLASSIFICATION WITH GCNS . Suppose G = ( V , E ) is an undirected graph with |V | = n and the n × n symmetric matrix A is the adjacency matrix . Every node k is associated with a feature vector xk ∈ Rd , and all the feature vectors are stored in the rows of X ∈ Rn×d . An L-layer graph convolutional network ( GCN ) ( Kipf & Welling , 2017 ) has the layer-wise propagation rule : H ( l+1 ) = σl ( ĀH ( l ) W ( l ) ) , where W ( l ) ∈ Rdl×dl+1 is a weight matrix . The matrix Ā = D̃−1/2ÃD̃−1/2 is the normalized adjacency matrix , where Ã = I + A and D̃ ∈ Rn×d is diagonal with D̃ii = ∑ j Ãij . The matrix H ( l ) ∈ Rn×dl are the activations in the l-th layer ( H ( 0 ) = X and d0 = d ) . The function σl is the activation function for layer l. The output of an L-layer GCN is an n × p matrix H ( L ) , of which the k-th row is the predicted label for node k. Later in the paper , we will train GCN using only a subgraph G ( γ ) , so the notation y〈pred〉k ( W ; Xγ , : , Āγγ ) is used to indicate the predicted label of node k , using the coefficient matrices W = ( W ( 0 ) , . . . , W ( L−1 ) ) , the submatrix Āγγ , and the corresponding feature vectors Xγ , : . Semi-supervised node classification is a popular application of GCN . When applying GCN for this task , we minimize the loss function L〈gcn〉 ( W ; X , Ā ) = ∑ k∈V l ` 1 ( y 〈true〉 k , y 〈pred〉 k ( W ; X , Ā ) ) , ( 1 ) to learn the weight matrices W = ( W ( 0 ) , . . . , W ( L−1 ) ) . In the formula , y〈true〉k is the given true label for node k , and V l ⊆ V is the subset of vertices with given true labels . The function ` 1 ( · , · ) is a loss function , and usually it is the cross entropy loss . Much work has been done to apply GCN on large-scale datasets . For the sampling-based methods , different rules for random selection have been proposed : uniform sampling as in Hamilton et al . ( 2017 ) , or importance sampling in Chen et al . ( 2018a ) . On the other hand , clustering-based methods become more interesting in the recent literature . Besides Cluster-GCN ( Chiang et al. , 2019 ) , Graph Partition Neural Network ( GPNN ) Liao et al . ( 2018 ) also partitions the entire graph into small clusters , and the GCN model alternatively propogates within-cluster and between-cluster . In addition , clustering structure can also be utilized implicitly as a low-rank approximation of the graph Laplacian . This idea is exploited in LanczosNet ( Liao et al. , 2019 ) as well as the Lovász convolutional network ( Yadav et al. , 2019 ) . 2.2 PRELIMINARIES ON CHORDAL GRAPHS . Chordal sparsity has been a classical topic in graph theory and found useful applications in various fields , including database theory ( Beeri et al. , 1983 ) , probabilistic networks ( Pearl , 1988 ; Cowell et al. , 1999 ; Darwiche , 2009 ) , linear algebra ( Rose , 1970 ) , combinatorial optimization ( Golumbic , 2004 ; Gavril , 1972 ) , semidefinite optimization ( Andersen et al. , 2010 ; 2013 ) . In this paper , we will use chordal sparsity theory to exploit the clustering structure . So now we show that a chordal graph can be separated into cliques , and that these cliques form a clique tree with desirable properties . Ordered undirected graph . Given an undirected graph G = ( V , E ) with |V | = n , we can order the vertices from 1 to n , and then we refer to a vertex in V by its order i ∈ [ n ] = { 1 , 2 , . . . , n } . In addition , an index set γ ⊂ [ n ] represents a subset of vertices in V . Cliques . A complete graph is a graph in which every pair of distinct vertices is adjacent . A clique γ ⊆ V of a graph G is a subset of vertices that induces a maximal complete subgraph . The graph induced by γ is denoted G ( γ ) = ( γ , E ( γ ) ) and has edge set E ( γ ) = { { v , w } ∈ E | v , w ∈ γ } . Chordal graph . A path between v0 and vk is a sequence of distinct vertices ( v0 , v1 , v2 , . . . , vk ) with { vi , vi+1 } ∈ E for i = 0 , 1 , . . . , k − 1 . For example , the graph in Figure 2a has a path ( a , e , c , d ) . A chord in a path of an undirected graph is an edge between non-consecutive vertices on the path . For example , in Figure 2a , the edge { d , e } is a chord in the path ( a , e , c , d ) , but in Figure 2b , the path ( b , a , c , d ) is chordless . An undirected graph is chordal if every cycle of length greater than three has a chord . Figure 2b and 2c show two examples of chordal graphs , while the graph in Figure 2a is non-choral . Postordered clique tree . Given a ( connected ) chordal graphG = ( V , E ) , we can always construct a clique tree T c : the nodes of T c are the cliques γ ⊆ V and two cliques are adjacent in T c if they share some vertices in V . In addition , we can arbitrarily pick a clique as the root of the tree , and then every non-rooted clique can be partitioned into clique separators and clique residuals . The clique separator is the intersection of the clique with its parent clique in T c. The clique residual is also called the ( maximal ) supernode , and by definition , all the supernodes form a partition of V . More interestingly , we can always find an ordering of vertices V so that the clique tree T c satisfies the following properties . 1 . The higher neighborhood of every vertex is complete : j , k ∈ adj+ ( i ) implies { j , k } ∈ E , where the higher neighborhood of vertex i is defined as adj+ ( i ) = { j | { i , j } ∈ E , j > i } . This means that every clique and supernode in T c can be written as γi and νi , respectively , where i ∈ V has the lowest order in the clique ( and supernode ) . We denote by i the representative of the supernode νi , by αi = γi\νi the corresponding clique separator , and by V c the set of all representatives . The index set αi is empty if i is the root of T c. 2 . The elements of each supernode are numbered consecutively : νi = { i , i + 1 , . . . , i + ni } with ni = |νi − 1| . 3 . All the representatives i ∈ V c are topologically ordered : i < p ( i ) if i ∈ V c and p ( i ) is the representative of the parent of γi in T c. An ordering satisfying the above three properties is called topological postordering , and a clique tree T c with certain topological postordering is referred to as a postordered clique tree . Figure 3 shows a supernode partition of the chordal graph in Figure 2c with a topological postordering . ( This example is adapted from the survey paper Vandenberghe & Andersen ( 2015 ) . ) In this example , γ5 = { 5 , 6 , 7 , 8 , 10 } , ν5 = { 5 , 6 , 7 } , α5 = { 8 , 10 } . Given a chordal sparsity graph G = ( V , E ) , there exist efficient algorithms ( in O ( |E| ) ) to find a topological postordering , and to generate a postordered clique tree T c together with the parent function p ( i ) and all the index sets ( i.e. , γi , νi , and αi for all i ∈ V c ) ( Lewis et al. , 1989 ; Pothen & Sun , 1990 ) . The whole process is called chordal decomposition ( Vandenberghe & Andersen , 2015 ) .

Summary: In this paper, the authors propose a new method referred to Chordal-GCN to optimize memory usage in large graphs. The authors borrow the ideas from Chordal Sparsity theory to first build a client tree. Then mini-batch updates are carried out individually on each clique from the leaves following the GCN loss. The authors add an additional consistency loss between shared node with children cliques. Experiments are carried out in four networks with comparison to several baselines.