GraphZoom: A Multi-level Spectral Approach for Accurate and Scalable Graph Embedding

1 INTRODUCTION . Recent years have seen a surge of interest in graph embedding , which aims to encode nodes , edges , or ( sub ) graphs into low dimensional vectors that maximally preserve graph structural information . Graph embedding techniques have shown promising results for various applications such as vertex classification , link prediction , and community detection ( Zhou et al. , 2018 ) ; ( Cai et al. , 2018 ) ; ( Goyal & Ferrara , 2018 ) . However , current graph embedding methods have several drawbacks in terms of either accuracy or scalability . On the one hand , random-walk-based embedding algorithms , such as DeepWalk ( Perozzi et al. , 2014 ) and node2vec ( Grover & Leskovec , 2016 ) , attempt to embed a graph based on its topology without incorporating node attribute information , which limits their embedding power . Later , graph convolutional networks ( GCN ) are developed with the basic notion that node embeddings should be smoothed over the entire graph ( Kipf & Welling , 2016 ) . While GCN leverages both topology and node attribute information for simplified graph convolution in each layer , it may suffer from high-frequency noise in the initial node features , which compromises ∗Equal contributions 1Source code of GraphZoom is freely available at : github.com/cornell-zhang/GraphZoom . the embedding quality ( Maehara , 2019 ) . On the other hand , few embedding algorithms can scale well to large graphs with millions of nodes due to their high computation and storage cost ( Zhang et al. , 2018a ) . For example , graph neural networks ( GNNs ) such as GraphSAGE ( Hamilton et al. , 2017 ) collectively aggregate feature information from the neighborhood . When stacking multiple GNN layers , the final embedding vector of a node involves the computation of a large number of intermediate embeddings from its neighbors . This will not only cause drastic increase in the amount of computation among nodes , but also lead to high memory usage for storing the intermediate results . In literature , increasing the accuracy and improving the scalability of graph embedding methods are largely viewed as two orthogonal problems . Hence most research efforts are devoted to addressing only one of the problems . For instance , Chen et al . ( 2018 ) and Fu et al . ( 2019 ) proposed multilevel methods to obtain high-quality embeddings by training unsupervised models at every level ; but their techniques do not improve scalability due to the additional training overhead . Liang et al . ( 2018 ) developed a heuristic algorithm to coarsen the graph by merging nodes with similar local structures . They use GCN to refine the embedding results on the coarsened graphs , which is not only time consuming to train but also potentially degrading accuracy when the number of GCN layers increases . More recently , Akbas & Aktas ( 2019 ) proposed a similar strategy to coarsen the graph , where certain useful structural properties of the graph are preserved ( e.g. , local neighborhood proximity ) . However , this work lacks proper refinement methods to improve the embedding quality . In this paper we propose GraphZoom , a multi-level spectral approach to enhancing both the quality and scalability of unsupervised graph embedding methods . More concretely , GraphZoom consists of four major kernels : ( 1 ) graph fusion , ( 2 ) spectral graph coarsening , ( 3 ) graph embedding , and ( 4 ) embedding refinement . The graph fusion kernel first converts the node feature matrix into a feature graph and then fuses it with the original topology graph . The fused graph provides richer information to the ensuing graph embedding step to achieve a higher accuracy . Spectral graph coarsening produces a series of successively coarsened graphs by merging nodes based on their spectral similarities . We show that our coarsening algorithm can effectively and efficiently retain the first few eigenvectors of the graph Laplacian matrix , which is critical for preserving the key graph structures . During the graph embedding step , any of the existing unsupervised graph embedding techniques can be applied to obtain node embeddings for the graph at the coarsest level . 2 Embedding refinement is then employed to refine the embeddings back to the original graph by applying a proper graph filter to ensure embeddings are smoothed over the graph . We evaluate the proposed GraphZoom framework on three transductive benchmarks : Cora , Citeseer and Pubmed citation networks as well as two inductive dataset : PPI and Reddit for vertex classification task . We further test the scalability of our approach on friendster dataset , which contains 8 million nodes and 400 million edges . Our experiments show that GraphZoom can improve the classification accuracy over all baseline embedding methods for both transductive and inductive tasks . Our main technical contributions are summarized as follows : •GraphZoom generates high-quality embeddings . We propose novel algorithms to encode graph structures and node attribute information in a fused graph and exploit graph filtering during refinement to remove high-frequency noise . This results in a relative increase of the embedding accuracy over the prior arts by up to 19.4 % while reducing the execution time by at least 2× . • GraphZoom improves scalability . Our approach can significantly reduce the embedding run time by effectively coarsening the graph without losing the key spectral properties . Experiments show that GraphZoom can accelerate the entire embedding process by up to 40.8× while producing a similar or better accuracy than state-of-the-art techniques . • GraphZoom is highly composable . Our framework is agnostic to underlying graph embedding techniques . Any of the existing unsupervised embedding methods , either transductive or inductive , can be incorporated by GraphZoom in a plug-and-play manner . 2 RELATED WORK . There is a large and active body of research on multi-level graph embedding and graph filtering , from which GraphZoom draws inspiration to boost the performance and speed of unsupervised 2In this work , we do not attempt to preserve node label information in the coarsened graph . Nonetheless , we believe that our approach can be extended to support supervised embedding models such as GAT ( Gulcehre et al. , 2019 ) and PPNP ( Klicpera et al. , 2019 ) , as briefly discussed in Section 5. embedding methods . Due to the space limitation , we only summarize some of the recent efforts in these two areas . Multi-level graph embedding attempts to coarsen the original graph into a series of smaller graphs with decreasing size where existing or new graph embedding techniques can be applied at different coarsening levels . For example , Chen et al . ( 2018 ) ; Lin et al . ( 2019 ) generate a hierarchy of coarsened graphs and perform embedding from the coarsest level to the original one . Fu et al . ( 2019 ) construct and embed multi-level graphs through hierarchical , and the resulting embedding vectors are concatenated to obtain the final node embeddings for the original graph . These methods , however , only focus on improving embedding quality but not the scalability . Later , Zhang et al . ( 2018b ) ; Akbas & Aktas ( 2019 ) attempt to make graph embedding more scalable by only embedding on the coarsest graph . However , their methods lack proper refinement methods to generate high-quality embeddings for the original graph . Liang et al . ( 2018 ) propose MILE , which only trains the coarsest graph to obtain coarse embeddings , and leverages GCN as refinement method to improve embedding quality . However , MILE requires training a GCN model which is very time consuming for large graphs and leading to poor performance when multiple GCN layers are stacked together ( Li et al. , 2018 ) . In contrast to the prior arts , GraphZoom is motivated by theoretical results in spectral graph embedding ( Tang & Liu , 2011 ) and practically efficient so that it can improve both accuracy and scalability of the unsupervised graph embedding tasks . Graph filtering are direct analogs of classical filters in signal processing field , but intended for signals defined on graphs . Shuman et al . ( 2013 ) define graph filters in both vertex and spectral domains , and apply graph filter on image denoising and reconstruction tasks . Wu et al . ( 2019 ) leverage graph filter to simplify GCN model by removing redundant computation . More recently , Maehara ( 2019 ) show the fundamental link between graph embedding and filtering by proving that GCN model implicitly exploits graph filter to remove high-frequency noise from the node feature matrix ; a filter neural network ( gfNN ) is then proposed to derive a stronger graph filter to improve the embedding results . Li et al . ( 2019 ) further derive two generalized graph filters and apply them on graph embedding models to improve their embedding quality for various classification tasks . In GraphZoom we adopt graph filter to properly smooth the intermediate embedding results during the iterative refinement step , which is crucial for improving the quality of the final embedding . 3 GRAPHZOOM FRAMEWORKGraphZoom Framework Figure 1 shows the proposed GraphZoom framework , which consists of four key phases : Phase ( 1 ) is graph fusion , which combines the node attributes and topology information of the original graph to construct a fused weighted graph ; In Phase ( 2 ) , a spectral graph coarsening process is applied to form a hierarchy of coarsened fused graphs with decreasing size ; In Phase ( 3 ) , any of the existing graph embedding methods can be applied to the fused graph at the coarsest level ; In Phase ( 4 ) , the embedding vectors obtained at the coarsest level are mapped onto a finer graph using the mapping operators determined during the coarsening phase . This is followed by a refinement ( or smoothing ) procedure , where the procedure in Phase ( 4 ) is applied in an iterative manner to increasingly finer graphs ; Eventually the embedding vectors for the original graph are obtained . In the rest of this section , we describe each of these four phases in more detail . 3.1 PHASE 1 : GRAPH FUSION . Graph fusion aims to construct a weighted graph that has the same number of nodes as the original graph but potentially different set of edges ( weights ) that encapsulate the original graph topology as well as node attribute information . Specifically , given an undirected graph G = ( V , E ) with N = |V| nodes , its adjacency matrix Atopo ∈ RN×N and its node attribute ( feature ) matrix X ∈ RN×K , where K corresponds to the dimension of node attribute vector , graph fusion can be interpreted as a function f ( · ) that outputs a weighted graph Gfusion = ( V , Efusion ) represented by its adjacency matrix Afusion ∈ RN×N , namely , Afusion = f ( Atopo , X ) . We first convert the initial attribute matrixX into a weighted node attribute graph Gfeat = ( V , Efeat ) by generating a k-nearest-neighbor ( kNN ) graph based on the l2-norm distance between the attribute vectors of each node pair . A straightforward implementation requires comparing all possible node pairs and then selecting top-k nearest neighbors . However , such a naı̈ve approach has a worstcase time complexity of O ( N2 ) , which certainly does not scale to large graphs . Our goal is to construct the attribute graph in nearly linear time by leveraging an efficient spectral graph coarsening scheme that is described in more detail in Section 3.2 . More specifically , our approach starts with coarsening the original graph G to obtain a substantially reduced graph that has much fewer nodes with an O ( |E| ) time complexity . Note that this procedure bears similarity to spectral graph clustering , which aims to group nodes into clusters of high conductance ( Peng et al. , 2015 ) . Once the node clusters are formed through spectral coarsening , we can select the top-k nearest neighbors within each cluster with O ( M2 ) comparisons , where M is the average node count per cluster . Since we have roughly N/M clusters , the time complexity for constructing the approximate kNN graph is O ( MN ) . When a proper coarsening ratio is chosen where M N ( e.g. , M = 50 ) , the overall time complexity will become almost linear . After the attribute graph is formed , we assign a weight to each edge based on the cosine similarity between the attribute vectors of the two incident nodes , namely wi , j = ( Xi , : ·Xj , : ) / ( ‖Xi , :‖‖Xj , :‖ ) , where Xi , : and Xj , : are the attribute vectors of nodes i and j . Finally , we can construct the fused graph by combining the topology graph and the attribute graph using a weighted sum : Afusion = Atopo +βAfeat , where β allows us to balance the topological and node attribute information during the fusion process . This fused graph can be fed into any downstream graph embedding procedures .

Summary: The authors propose a way to fuse information on nodes of a graph with the topology of the graph in the large scale setting. The proposed approach is done in four phases where (i) the covariates in the nodes of the graph is first mapped in the graph space for fusion and fused using linear combination of the topological graph and feature graph, (ii) the resulting "adjacency" matrix will almost surely not be sparse even if the original graph space, so they use eigenvalues of the graph laplacian to coarsen the graph -- remove edges; (iii) they then propose to embed the coarsened graph using "any" unsupervised learning technique; (iv) then the embedded representation is refined using iterative procedures. Cheap procedures are introduced to do Phases (i) and (iv). Experimentally the authors see improvements in the performance using their approach compared to the baselines considered.