MILE: A Multi-Level Framework for Scalable Graph Embedding

1 Introduction . In recent years , graph embedding has attracted much interest due to its broad applicability for various tasks ( Perozzi et al. , 2014 ; Wang et al. , 2016 ; Henderson et al. , 2012 ) . However , such methods rarely scale to large datasets ( e.g. , graphs with over 1 million nodes ) since they are computationally expensive and often memory intensive . For example , random-walkbased embedding techniques require a large amount of CPU time to generate a sufficient number of walks and train the embedding model . As another example , embedding methods based on matrix factorization , including GraRep ( Cao et al. , 2015 ) and NetMF ( Qiu et al. , 2018 ) , requires constructing an enormous objective matrix ( usually much denser than adjacency matrix ) , on which matrix factorization is performed . Even a medium-size graph with 100K nodes can easily require hundreds of GB of memory using those methods . On the other hand , many graph datasets in the real world tend to be large-scale with millions or even billions of nodes . To the best of our knowledge , none of the existing efforts examines how to scale up graph embedding in a generic way . We make the first attempt to close this gap . We are also interested in the related question of whether the quality of such embeddings can be improved along the way . Specifically , we ask : 1 ) Can we scale up the existing embedding techniques in an agnostic manner so that they can be directly applied to larger datasets ? 2 ) Can the quality of such embedding methods be strengthened by incorporating the holistic view of the graph ? To tackle these problems , we propose a MultI-Level Embedding ( MILE ) framework for graph embedding . Our approach relies on a three-step process : first , we repeatedly coarsen the original graph into smaller ones by employing a hybrid matching strategy ; second , we compute the embeddings on the coarsest graph using an existing embedding techniques - and third , we propose a novel refinement model based on learning a graph convolution network to refine the embeddings from the coarsest graph to the original graph – learning a graph convolution network allows us to compute a refinement procedure that levers the dependencies inherent to the graph structure and the embedding method of choice . To summarize , we find that : • MILE is generalizable : Our MILE framework is agnostic to the underlying graph embedding techniques and treats them as black boxes . • MILE is scalable : MILE can significantly improve the scalability of the embedding methods ( up to 30-fold ) , by reducing the running time and memory consumption . • MILE generates high-quality embeddings : In many cases , we find that the quality of embeddings improves by levering MILE ( in some cases is in excess of 10 % ) . 2 Related Work . Many techniques for graph or network embedding have been proposed in recent years . DeepWalk and Node2Vec generate truncated random walks on graphs and apply the Skip Gram by treating the walks as sentences ( Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ) . LINE learns the node embeddings by preserving the first-order and second-order proximities ( Tang et al. , 2015 ) . Following LINE , SDNE leverages deep neural networks to capture the highly non-linear structure ( Wang et al. , 2016 ) . Other methods construct a particular objective matrix and use matrix factorization techniques to generate embeddings , e.g. , GraRep ( Cao et al. , 2015 ) and NetMF ( Qiu et al. , 2018 ) . This also led to the proliferation of network embedding methods for information-rich graphs , including heterogeneous information networks ( Chang et al. , 2015 ; Dong et al. , 2017 ) and attributed graphs ( Pan et al. , 2016 ; Liang et al. , 2018 ; Yang et al. , 2015 ; Kipf & Welling , 2017 ) . On the other hand , there are very few efforts , focusing on the scalability of network embedding ( Yang et al. , 2017 ; Huang et al. , 2017 ) . First , such efforts are specific to a particular embedding strategy and do not generalize . Second , the scalability of such efforts is limited to moderately sized datasets . Finally , and notably , these efforts at scalability are actually orthogonal to our strategy and can potentially be employed along with our efforts to afford even greater speedup . The closest work to this paper is the very recently proposed HARP ( Chen et al. , 2018 ) , which proposes a hierarchical paradigm for graph embedding based on iterative learning methods ( e.g. , DeepWalk and Node2Vec ) . However , HARP focuses on improving the quality of embeddings by using the learned embeddings from the previous level as the initialized embeddings for the next level , which introduces a huge computational overhead . Moreover , it is not immediately obvious how a HARP like methodology would be extended to other graph embedding techniques ( e.g. , GraRep and NetMF ) in an agnostic manner since such an approach would necessarily require one to modify the embedding methods to preset their initialized embeddings . In this paper , we focus on designing a general-purpose framework to scale up embedding methods treating them as black boxes . 3 Problem Formulation . Let G = ( V , E ) be the input graph where V and E are respectively the node set and edge set . Let A be the adjacency matrix of the graph and we assume G is undirected , though our problem can be easily extended ( Chung , 2005 ; Gleich , 2006 ; Satuluri & Parthasarathy , 2011 ) to directed graph . We first define graph embedding : Definition 3.1 Graph Embedding Given a graph G = ( V , E ) and a dimensionality d ( d |V | ) , the problem of graph embedding is to learn a d-dimension vector representation for each node in G so that graph properties are best preserved . Following this , a graph embedding method is essentially a mapping function f : R|V |×|V | 7→ R|V |×d , whose input is the adjacency matrix A ( or G ) and output is a lower dimension matrix . Motivated by the fact that the majority of graph embedding methods can not scale to large datasets , we seek to speed up existing graph embedding methods without sacrificing quality . We formulate the problem as : Given a graph G = ( V , E ) and a graph embedding method f ( · ) , we aim to realize a strengthened graph embedding method f̂ ( · ) so that it is more scalable than f ( · ) while generating embeddings of comparable or even better quality . 4 Methodology . MILE framework consists of three key phases : graph coarsening , base embedding , and embeddings refining . Figure 1a shows the overview . 4.1 Graph Coarsening . In this phase , the input graph G ( or G0 ) is repeatedly coarsened into a series of smaller graphs G1 , G2 , ... , Gm such that |V0| > |V1| > ... > |Vm| . In order to coarsen a graph from Gi to Gi+1 , multiple nodes in Gi are collapsed to form super-nodes in Gi+1 , and the edges incident on a super-node are the union of the edges on the original nodes in Gi . Here the set of nodes forming a super-node is called a matching . We propose a hybrid matching technique containing two matching strategies that can efficiently coarsen the graph while retaining the global structure . An example is shared in Figure 2 . Structural Equivalence Matching ( SEM ) : Given two vertices u and v in an unweighted graph G , we call they are structurally equivalent if they are incident on the same set of neighborhoods . In figure 2a , node D and E are structurally equivalent . The intuition of matching structually equivalent nodes is that if two vertices are structurally equivalent , then their node embeddings will be similar . Normalized Heavy Edge Matching ( NHEM ) : Heavy edge matching is a popular matching method for graph coarsening ( Karypis & Kumar , 1998 ) . For an unmatched node u in Gi , its heavy edge matching is a pair of vertices ( u , v ) such that the weight of the edge between u and v is the largest . In this paper , we propose to normalize the edge weights when applying heavy edge matching using the formula as follows Wi ( u , v ) = Ai ( u , v ) √ Di ( u , u ) ·Di ( v , v ) . ( 1 ) Here , the weight of an edge is normalized by the degree of the two vertices on which the edge is incident . Intuitively , it penalizes the weights of edges connected with high-degree nodes . As we will show in Sec . 4.3 , this normalization is tightly connected with the graph convolution kernel . Hybrid Matching Method : We use a hybrid of two matching methods above for graph coarsening . To construct Gi+1 from Gi , we first find out all the structural equivalence matching ( SEM ) M1 , where Gi is treated as an unweighted graph . This is followed by the searching of the normalized heavy edge matching ( NHEM ) M2 on Gi . Nodes in each matching are then collapsed into a super-node in Gi+1 . Note that some nodes might not be matched at all and they will be directly copied to Gi+1 . Formally , we build the adjacency matrix Ai+1 of Gi+1 through matrix operations . To this end , we define the matching matrix storing the matching information from graph Gi to Gi+1 as a binary matrix Mi , i+1 ∈ { 0 , 1 } |Vi|×|Vi+1| . The r-th row and c-th column of Mi , i+1 is set to 1 if node r in Gi will be collapsed to super-node c in Gi+1 , and is set to 0 if otherwise . Each column of Mi , i+1 represents a matching with the 1s representing the nodes in it . Each unmatched vertex appears as an individual column in Mi , i+1 with merely one entry set to 1 . Following this formulation , we construct the adjacency matrix of Gi+1 by using Ai+1 = MTi , i+1AiMi , i+1 . ( 2 ) 4.2 Base Embedding on Coarsened Graph . The size of the graph reduces drastically after each iteration of coarsening , halving the size of the graph in the best case . We coarsen the graph for m iterations and apply the graph embedding method f ( · ) on the coarsest graph Gm . Denoting the embeddings on Gm as Em , we have Em = f ( Gm ) . Since our framework is agnostic to the adopted graph embedding method , we can use any graph embedding algorithm for base embedding . 4.3 Refinement of Embeddings . The final phase of MILE is the embeddings refinement phase . Given a series of coarsened graph G0 , G1 , G2 , ... , Gm , their corresponding matching matrix M0,1 , M1,2 , ... , Mm−1 , m , and the node embeddings Em on Gm , we seek to develop an approach to derive the node embeddings of G0 from Gm . To this end , we first study an easier subtask : given a graph Gi , its coarsened graph Gi+1 , the matching matrix Mi , i+1 and the node embeddings Ei+1 on Gi+1 , how to infer the embeddings Ei on graph Gi . Once we solved this subtask , we can then iteratively apply the technique on each pair of consecutive graphs from Gm to G0 and eventually derive the node embeddings on G0 . In this work , we propose to use a graph-based neural network model to perform embeddings refinement . Graph Convolution Network for Refinement Learning : Since we know the matching information between the two consecutive graphs Gi and Gi+1 , we can easily project the node embeddings from the coarse-grained graph Gi+1 to the fine-grained graph Gi using Epi = Mi , i+1Ei+1 ( 3 ) In this case , embedding of a super-node is directly copied to its original node ( s ) . We call Epi the projected embeddings from Gi+1 to Gi , or simply projected embeddings without ambiguity . While this way of simple projection maintains some information of node embeddings , it has obvious limitations that nodes will share the same embeddings if they are matched and collapsed into a super-node during the coarsening phase . This problem will be more serious when the embedding refinement is performed iteratively from Gm , ... , G0 . To address this issue , we propose to use a graph convolution network for embedding refinement . Specifically , we design a graph-based neural network model Ei = R ( Epi , Ai ) , which derives the embeddings Ei on graph Gi based on the projected embeddings Epi and the graph adjacency matrix Ai . Given graph G with adjacency matrix A , we consider the fast approximation of graph convolution from ( Kipf & Welling , 2017 ) . The k-th layer of this neural network model is H ( k ) ( X , A ) = σ ( D̃− 1 2 ÃD̃− 1 2H ( k−1 ) ( X , A ) Θ ( k ) ) ( 4 ) where σ ( · ) is an activation function , Θ ( k ) is a layer-specific trainable weight matrix , and H ( 0 ) ( X , A ) = X . In this paper , we define our embedding refinement model as a l-layer graph convolution model Ei = R ( Epi , Ai ) ≡ H ( l ) ( Epi , Ai ) . ( 5 ) The architecture of the refinement model is shown in Figure 1b . The intuition behind this refinement model is to integrate the structural information of the current graph Gi into the projected embedding Epi by repeatedly performing the spectral graph convolution . Each layer of graph convolution network in Eq . 4 can be regarded as one iteration of embedding propagation in the graph following the re-normalized adjacency matrix D̃− 12 ÃD̃− 12 . Note that this re-normalized matrix is well aligned with the way we conduct normalized heavy edge matching in Eq . 1 . We next discuss how the weight matrix Θ ( k ) is learned . Intricacies of Refinement Learning : The learning of the refinement model is essentially learning Θ ( k ) for each k ∈ [ 1 , l ] according to Eq . 4 . Here we study how to design the learning task and construct the loss function . Since the graph convolution model H ( l ) ( · ) aims to predict the embeddings Ei on graph Gi , we can directly run a base embedding on Gi to generate the “ ground-truth ” embeddings and use the difference between these embeddings and the predicted ones as the loss function for training . We propose to learn Θ ( k ) on the coarsest graph and reuse them across all the levels for refinement . Specifically , we can define the loss function as the mean square error as follows L = 1 |Vm| ∥∥∥Em −H ( l ) ( Mm , m+1Em+1 , Am ) ∥∥∥2 . ( 6 ) We refer to the learning task associated with the above loss function as double-base embedding learning . We point out , however , there are two key drawbacks to this method . First of all , the above loss function requires one more level of coarsening to construct Gm+1 and an extra base embedding on Gm+1 . These two steps , especially the latter , introduce nonnegligible overheads to the MILE framework , which contradicts our motivation of scaling up graph embedding . More importantly , Em might not be a desirable “ ground truth ” for the refined embeddings . This is because most of the embedding methods are invariant to an orthogonal transformation of the embeddings , i.e. , the embeddings can be rotated by an arbitrary orthogonal matrix ( Hamilton et al. , 2017 ) . In other words , the embedding spaces of graph Gm and Gm+1 can be totally different since the two base embeddings are learned independently . Even if we follow the paradigm in ( Chen et al. , 2018 ) and conduct base embedding on Gm using the simple projected embeddings from Gm+1 ( Epm ) as initialization , the embedding space does not naturally generalize and can drift during re-training . One possible solution is to use an alignment procedure to force the embeddings to be aligned between the two graphs ( Hamilton et al. , 2016 ) . But it could be very expensive . In this paper , we propose a very simple method to address the above issues . Instead of conducting an additional level of coarsening , we construct a dummy coarsened graph by simply copying Gm , i.e. , Mm , m+1 = I and Gm+1 = Gm . By doing this , we not only reduce one iteration of graph coarsening , but also avoid performing base embedding on Gm+1 simply because Em+1 = Em . Moreover , the embeddings of Gm and Gm+1 are guaranteed to be in the same space in this case without any drift . With this strategy , we change the loss function for model learning as follows L = 1 |Vm| ∥∥∥Em −H ( l ) ( Em , Am ) ∥∥∥2 . ( 7 ) With the above loss function , we adopt gradient descent with back-propagation to learn the parameters Θ ( k ) , k ∈ [ 1 , l ] . In the subsequent refinement steps , we apply the same set of parameters Θ ( k ) to infer the refined embeddings . We point out that the training of the refinement model is rather efficient as it is done on the coarsest graph . The embeddings refinement process involves merely sparse matrix multiplications using Eq . 5 and is relatively affordable compared to conducting embedding on the original graph . With these different components , we summarize the whole algorithm of our MILE framework in Algorithm 1 . The appendix contains the time complexity of the algorithm in Section A.2 Algorithm 1 Multi-Level Algorithm for Graph Embedding Input : A input graph G0 = ( V0 , E0 ) , # coarsening levels m , and a base embedding method f ( · ) . Output : Graph embeddings E0 on G0 . 1 : Coarsen G0 into G1 , G2 , ... , Gm using proposed hybrid matching method . 2 : Perform base embedding on the coarsest graph Gm ( See Section . 4.2 ) . 3 : Learn the weights Θ ( k ) using the loss function in Eq . 7 . 4 : for i = ( m− 1 ) ... 0 do 5 : Compute the projected embeddings Epi on Gi . 6 : Use Eq . 4 and Eq . 5 to compute refined embeddings Ei . 7 : Return graph embeddings E0 on G0 .

Summary: This paper proposed a multi-Level framework for learning node embeddings for large-scale graphs. The author first coarsens the graphs into different levels of subgraphs. The low-level subgraphs are obtained with the node embeddings of the higher-level graphs with a graph convolutional neural network. By iteratively applying this procedure, the node embeddings of the original graphs can be obtained. Experimental results on several networks (including one network with ~10M node) prove the effective and efficiency of the proposed method over existing state-of-the-art approaches.