MILE: A Multi-Level Framework for Scalable Graph Embedding

1 Introduction . In recent years , graph embedding has attracted much interest due to its broad applicability for various tasks ( Perozzi et al. , 2014 ; Wang et al. , 2016 ; Henderson et al. , 2012 ) . However , such methods rarely scale to large datasets ( e.g. , graphs with over 1 million nodes ) since they are computationally expensive and often memory intensive . For example , random-walkbased embedding techniques require a large amount of CPU time to generate a sufficient number of walks and train the embedding model . As another example , embedding methods based on matrix factorization , including GraRep ( Cao et al. , 2015 ) and NetMF ( Qiu et al. , 2018 ) , requires constructing an enormous objective matrix ( usually much denser than adjacency matrix ) , on which matrix factorization is performed . Even a medium-size graph with 100K nodes can easily require hundreds of GB of memory using those methods . On the other hand , many graph datasets in the real world tend to be large-scale with millions or even billions of nodes . To the best of our knowledge , none of the existing efforts examines how to scale up graph embedding in a generic way . We make the first attempt to close this gap . We are also interested in the related question of whether the quality of such embeddings can be improved along the way . Specifically , we ask : 1 ) Can we scale up the existing embedding techniques in an agnostic manner so that they can be directly applied to larger datasets ? 2 ) Can the quality of such embedding methods be strengthened by incorporating the holistic view of the graph ? To tackle these problems , we propose a MultI-Level Embedding ( MILE ) framework for graph embedding . Our approach relies on a three-step process : first , we repeatedly coarsen the original graph into smaller ones by employing a hybrid matching strategy ; second , we compute the embeddings on the coarsest graph using an existing embedding techniques - and third , we propose a novel refinement model based on learning a graph convolution network to refine the embeddings from the coarsest graph to the original graph – learning a graph convolution network allows us to compute a refinement procedure that levers the dependencies inherent to the graph structure and the embedding method of choice . To summarize , we find that : • MILE is generalizable : Our MILE framework is agnostic to the underlying graph embedding techniques and treats them as black boxes . • MILE is scalable : MILE can significantly improve the scalability of the embedding methods ( up to 30-fold ) , by reducing the running time and memory consumption . • MILE generates high-quality embeddings : In many cases , we find that the quality of embeddings improves by levering MILE ( in some cases is in excess of 10 % ) . 2 Related Work . Many techniques for graph or network embedding have been proposed in recent years . DeepWalk and Node2Vec generate truncated random walks on graphs and apply the Skip Gram by treating the walks as sentences ( Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ) . LINE learns the node embeddings by preserving the first-order and second-order proximities ( Tang et al. , 2015 ) . Following LINE , SDNE leverages deep neural networks to capture the highly non-linear structure ( Wang et al. , 2016 ) . Other methods construct a particular objective matrix and use matrix factorization techniques to generate embeddings , e.g. , GraRep ( Cao et al. , 2015 ) and NetMF ( Qiu et al. , 2018 ) . This also led to the proliferation of network embedding methods for information-rich graphs , including heterogeneous information networks ( Chang et al. , 2015 ; Dong et al. , 2017 ) and attributed graphs ( Pan et al. , 2016 ; Liang et al. , 2018 ; Yang et al. , 2015 ; Kipf & Welling , 2017 ) . On the other hand , there are very few efforts , focusing on the scalability of network embedding ( Yang et al. , 2017 ; Huang et al. , 2017 ) . First , such efforts are specific to a particular embedding strategy and do not generalize . Second , the scalability of such efforts is limited to moderately sized datasets . Finally , and notably , these efforts at scalability are actually orthogonal to our strategy and can potentially be employed along with our efforts to afford even greater speedup . The closest work to this paper is the very recently proposed HARP ( Chen et al. , 2018 ) , which proposes a hierarchical paradigm for graph embedding based on iterative learning methods ( e.g. , DeepWalk and Node2Vec ) . However , HARP focuses on improving the quality of embeddings by using the learned embeddings from the previous level as the initialized embeddings for the next level , which introduces a huge computational overhead . Moreover , it is not immediately obvious how a HARP like methodology would be extended to other graph embedding techniques ( e.g. , GraRep and NetMF ) in an agnostic manner since such an approach would necessarily require one to modify the embedding methods to preset their initialized embeddings . In this paper , we focus on designing a general-purpose framework to scale up embedding methods treating them as black boxes . 3 Problem Formulation . Let G = ( V , E ) be the input graph where V and E are respectively the node set and edge set . Let A be the adjacency matrix of the graph and we assume G is undirected , though our problem can be easily extended ( Chung , 2005 ; Gleich , 2006 ; Satuluri & Parthasarathy , 2011 ) to directed graph . We first define graph embedding : Definition 3.1 Graph Embedding Given a graph G = ( V , E ) and a dimensionality d ( d |V | ) , the problem of graph embedding is to learn a d-dimension vector representation for each node in G so that graph properties are best preserved . Following this , a graph embedding method is essentially a mapping function f : R|V |×|V | 7→ R|V |×d , whose input is the adjacency matrix A ( or G ) and output is a lower dimension matrix . Motivated by the fact that the majority of graph embedding methods can not scale to large datasets , we seek to speed up existing graph embedding methods without sacrificing quality . We formulate the problem as : Given a graph G = ( V , E ) and a graph embedding method f ( · ) , we aim to realize a strengthened graph embedding method f̂ ( · ) so that it is more scalable than f ( · ) while generating embeddings of comparable or even better quality . 4 Methodology . MILE framework consists of three key phases : graph coarsening , base embedding , and embeddings refining . Figure 1a shows the overview . 4.1 Graph Coarsening . In this phase , the input graph G ( or G0 ) is repeatedly coarsened into a series of smaller graphs G1 , G2 , ... , Gm such that |V0| > |V1| > ... > |Vm| . In order to coarsen a graph from Gi to Gi+1 , multiple nodes in Gi are collapsed to form super-nodes in Gi+1 , and the edges incident on a super-node are the union of the edges on the original nodes in Gi . Here the set of nodes forming a super-node is called a matching . We propose a hybrid matching technique containing two matching strategies that can efficiently coarsen the graph while retaining the global structure . An example is shared in Figure 2 . Structural Equivalence Matching ( SEM ) : Given two vertices u and v in an unweighted graph G , we call they are structurally equivalent if they are incident on the same set of neighborhoods . In figure 2a , node D and E are structurally equivalent . The intuition of matching structually equivalent nodes is that if two vertices are structurally equivalent , then their node embeddings will be similar . Normalized Heavy Edge Matching ( NHEM ) : Heavy edge matching is a popular matching method for graph coarsening ( Karypis & Kumar , 1998 ) . For an unmatched node u in Gi , its heavy edge matching is a pair of vertices ( u , v ) such that the weight of the edge between u and v is the largest . In this paper , we propose to normalize the edge weights when applying heavy edge matching using the formula as follows Wi ( u , v ) = Ai ( u , v ) √ Di ( u , u ) ·Di ( v , v ) . ( 1 ) Here , the weight of an edge is normalized by the degree of the two vertices on which the edge is incident . Intuitively , it penalizes the weights of edges connected with high-degree nodes . As we will show in Sec . 4.3 , this normalization is tightly connected with the graph convolution kernel . Hybrid Matching Method : We use a hybrid of two matching methods above for graph coarsening . To construct Gi+1 from Gi , we first find out all the structural equivalence matching ( SEM ) M1 , where Gi is treated as an unweighted graph . This is followed by the searching of the normalized heavy edge matching ( NHEM ) M2 on Gi . Nodes in each matching are then collapsed into a super-node in Gi+1 . Note that some nodes might not be matched at all and they will be directly copied to Gi+1 . Formally , we build the adjacency matrix Ai+1 of Gi+1 through matrix operations . To this end , we define the matching matrix storing the matching information from graph Gi to Gi+1 as a binary matrix Mi , i+1 ∈ { 0 , 1 } |Vi|×|Vi+1| . The r-th row and c-th column of Mi , i+1 is set to 1 if node r in Gi will be collapsed to super-node c in Gi+1 , and is set to 0 if otherwise . Each column of Mi , i+1 represents a matching with the 1s representing the nodes in it . Each unmatched vertex appears as an individual column in Mi , i+1 with merely one entry set to 1 . Following this formulation , we construct the adjacency matrix of Gi+1 by using Ai+1 = MTi , i+1AiMi , i+1 . ( 2 ) 4.2 Base Embedding on Coarsened Graph . The size of the graph reduces drastically after each iteration of coarsening , halving the size of the graph in the best case . We coarsen the graph for m iterations and apply the graph embedding method f ( · ) on the coarsest graph Gm . Denoting the embeddings on Gm as Em , we have Em = f ( Gm ) . Since our framework is agnostic to the adopted graph embedding method , we can use any graph embedding algorithm for base embedding . 4.3 Refinement of Embeddings . The final phase of MILE is the embeddings refinement phase . Given a series of coarsened graph G0 , G1 , G2 , ... , Gm , their corresponding matching matrix M0,1 , M1,2 , ... , Mm−1 , m , and the node embeddings Em on Gm , we seek to develop an approach to derive the node embeddings of G0 from Gm . To this end , we first study an easier subtask : given a graph Gi , its coarsened graph Gi+1 , the matching matrix Mi , i+1 and the node embeddings Ei+1 on Gi+1 , how to infer the embeddings Ei on graph Gi . Once we solved this subtask , we can then iteratively apply the technique on each pair of consecutive graphs from Gm to G0 and eventually derive the node embeddings on G0 . In this work , we propose to use a graph-based neural network model to perform embeddings refinement . Graph Convolution Network for Refinement Learning : Since we know the matching information between the two consecutive graphs Gi and Gi+1 , we can easily project the node embeddings from the coarse-grained graph Gi+1 to the fine-grained graph Gi using Epi = Mi , i+1Ei+1 ( 3 ) In this case , embedding of a super-node is directly copied to its original node ( s ) . We call Epi the projected embeddings from Gi+1 to Gi , or simply projected embeddings without ambiguity . While this way of simple projection maintains some information of node embeddings , it has obvious limitations that nodes will share the same embeddings if they are matched and collapsed into a super-node during the coarsening phase . This problem will be more serious when the embedding refinement is performed iteratively from Gm , ... , G0 . To address this issue , we propose to use a graph convolution network for embedding refinement . Specifically , we design a graph-based neural network model Ei = R ( Epi , Ai ) , which derives the embeddings Ei on graph Gi based on the projected embeddings Epi and the graph adjacency matrix Ai . Given graph G with adjacency matrix A , we consider the fast approximation of graph convolution from ( Kipf & Welling , 2017 ) . The k-th layer of this neural network model is H ( k ) ( X , A ) = σ ( D̃− 1 2 ÃD̃− 1 2H ( k−1 ) ( X , A ) Θ ( k ) ) ( 4 ) where σ ( · ) is an activation function , Θ ( k ) is a layer-specific trainable weight matrix , and H ( 0 ) ( X , A ) = X . In this paper , we define our embedding refinement model as a l-layer graph convolution model Ei = R ( Epi , Ai ) ≡ H ( l ) ( Epi , Ai ) . ( 5 ) The architecture of the refinement model is shown in Figure 1b . The intuition behind this refinement model is to integrate the structural information of the current graph Gi into the projected embedding Epi by repeatedly performing the spectral graph convolution . Each layer of graph convolution network in Eq . 4 can be regarded as one iteration of embedding propagation in the graph following the re-normalized adjacency matrix D̃− 12 ÃD̃− 12 . Note that this re-normalized matrix is well aligned with the way we conduct normalized heavy edge matching in Eq . 1 . We next discuss how the weight matrix Θ ( k ) is learned . Intricacies of Refinement Learning : The learning of the refinement model is essentially learning Θ ( k ) for each k ∈ [ 1 , l ] according to Eq . 4 . Here we study how to design the learning task and construct the loss function . Since the graph convolution model H ( l ) ( · ) aims to predict the embeddings Ei on graph Gi , we can directly run a base embedding on Gi to generate the “ ground-truth ” embeddings and use the difference between these embeddings and the predicted ones as the loss function for training . We propose to learn Θ ( k ) on the coarsest graph and reuse them across all the levels for refinement . Specifically , we can define the loss function as the mean square error as follows L = 1 |Vm| ∥∥∥Em −H ( l ) ( Mm , m+1Em+1 , Am ) ∥∥∥2 . ( 6 ) We refer to the learning task associated with the above loss function as double-base embedding learning . We point out , however , there are two key drawbacks to this method . First of all , the above loss function requires one more level of coarsening to construct Gm+1 and an extra base embedding on Gm+1 . These two steps , especially the latter , introduce nonnegligible overheads to the MILE framework , which contradicts our motivation of scaling up graph embedding . More importantly , Em might not be a desirable “ ground truth ” for the refined embeddings . This is because most of the embedding methods are invariant to an orthogonal transformation of the embeddings , i.e. , the embeddings can be rotated by an arbitrary orthogonal matrix ( Hamilton et al. , 2017 ) . In other words , the embedding spaces of graph Gm and Gm+1 can be totally different since the two base embeddings are learned independently . Even if we follow the paradigm in ( Chen et al. , 2018 ) and conduct base embedding on Gm using the simple projected embeddings from Gm+1 ( Epm ) as initialization , the embedding space does not naturally generalize and can drift during re-training . One possible solution is to use an alignment procedure to force the embeddings to be aligned between the two graphs ( Hamilton et al. , 2016 ) . But it could be very expensive . In this paper , we propose a very simple method to address the above issues . Instead of conducting an additional level of coarsening , we construct a dummy coarsened graph by simply copying Gm , i.e. , Mm , m+1 = I and Gm+1 = Gm . By doing this , we not only reduce one iteration of graph coarsening , but also avoid performing base embedding on Gm+1 simply because Em+1 = Em . Moreover , the embeddings of Gm and Gm+1 are guaranteed to be in the same space in this case without any drift . With this strategy , we change the loss function for model learning as follows L = 1 |Vm| ∥∥∥Em −H ( l ) ( Em , Am ) ∥∥∥2 . ( 7 ) With the above loss function , we adopt gradient descent with back-propagation to learn the parameters Θ ( k ) , k ∈ [ 1 , l ] . In the subsequent refinement steps , we apply the same set of parameters Θ ( k ) to infer the refined embeddings . We point out that the training of the refinement model is rather efficient as it is done on the coarsest graph . The embeddings refinement process involves merely sparse matrix multiplications using Eq . 5 and is relatively affordable compared to conducting embedding on the original graph . With these different components , we summarize the whole algorithm of our MILE framework in Algorithm 1 . The appendix contains the time complexity of the algorithm in Section A.2 Algorithm 1 Multi-Level Algorithm for Graph Embedding Input : A input graph G0 = ( V0 , E0 ) , # coarsening levels m , and a base embedding method f ( · ) . Output : Graph embeddings E0 on G0 . 1 : Coarsen G0 into G1 , G2 , ... , Gm using proposed hybrid matching method . 2 : Perform base embedding on the coarsest graph Gm ( See Section . 4.2 ) . 3 : Learn the weights Θ ( k ) using the loss function in Eq . 7 . 4 : for i = ( m− 1 ) ... 0 do 5 : Compute the projected embeddings Epi on Gi . 6 : Use Eq . 4 and Eq . 5 to compute refined embeddings Ei . 7 : Return graph embeddings E0 on G0 .

Summary: In this submission, the authors propose a three-stage framework for large-scale graph embedding. The proposed method first constructs a small graph by graph coarsening, then applies any existing graph embedding method, and last refines the learned embeddings. It is useful, however, the experimental results are not convincing and cannot support the authors' claims about the proposed method.