GraphZoom: A Multi-level Spectral Approach for Accurate and Scalable Graph Embedding

1 INTRODUCTION . Recent years have seen a surge of interest in graph embedding , which aims to encode nodes , edges , or ( sub ) graphs into low dimensional vectors that maximally preserve graph structural information . Graph embedding techniques have shown promising results for various applications such as vertex classification , link prediction , and community detection ( Zhou et al. , 2018 ) ; ( Cai et al. , 2018 ) ; ( Goyal & Ferrara , 2018 ) . However , current graph embedding methods have several drawbacks in terms of either accuracy or scalability . On the one hand , random-walk-based embedding algorithms , such as DeepWalk ( Perozzi et al. , 2014 ) and node2vec ( Grover & Leskovec , 2016 ) , attempt to embed a graph based on its topology without incorporating node attribute information , which limits their embedding power . Later , graph convolutional networks ( GCN ) are developed with the basic notion that node embeddings should be smoothed over the entire graph ( Kipf & Welling , 2016 ) . While GCN leverages both topology and node attribute information for simplified graph convolution in each layer , it may suffer from high-frequency noise in the initial node features , which compromises ∗Equal contributions 1Source code of GraphZoom is freely available at : github.com/cornell-zhang/GraphZoom . the embedding quality ( Maehara , 2019 ) . On the other hand , few embedding algorithms can scale well to large graphs with millions of nodes due to their high computation and storage cost ( Zhang et al. , 2018a ) . For example , graph neural networks ( GNNs ) such as GraphSAGE ( Hamilton et al. , 2017 ) collectively aggregate feature information from the neighborhood . When stacking multiple GNN layers , the final embedding vector of a node involves the computation of a large number of intermediate embeddings from its neighbors . This will not only cause drastic increase in the amount of computation among nodes , but also lead to high memory usage for storing the intermediate results . In literature , increasing the accuracy and improving the scalability of graph embedding methods are largely viewed as two orthogonal problems . Hence most research efforts are devoted to addressing only one of the problems . For instance , Chen et al . ( 2018 ) and Fu et al . ( 2019 ) proposed multilevel methods to obtain high-quality embeddings by training unsupervised models at every level ; but their techniques do not improve scalability due to the additional training overhead . Liang et al . ( 2018 ) developed a heuristic algorithm to coarsen the graph by merging nodes with similar local structures . They use GCN to refine the embedding results on the coarsened graphs , which is not only time consuming to train but also potentially degrading accuracy when the number of GCN layers increases . More recently , Akbas & Aktas ( 2019 ) proposed a similar strategy to coarsen the graph , where certain useful structural properties of the graph are preserved ( e.g. , local neighborhood proximity ) . However , this work lacks proper refinement methods to improve the embedding quality . In this paper we propose GraphZoom , a multi-level spectral approach to enhancing both the quality and scalability of unsupervised graph embedding methods . More concretely , GraphZoom consists of four major kernels : ( 1 ) graph fusion , ( 2 ) spectral graph coarsening , ( 3 ) graph embedding , and ( 4 ) embedding refinement . The graph fusion kernel first converts the node feature matrix into a feature graph and then fuses it with the original topology graph . The fused graph provides richer information to the ensuing graph embedding step to achieve a higher accuracy . Spectral graph coarsening produces a series of successively coarsened graphs by merging nodes based on their spectral similarities . We show that our coarsening algorithm can effectively and efficiently retain the first few eigenvectors of the graph Laplacian matrix , which is critical for preserving the key graph structures . During the graph embedding step , any of the existing unsupervised graph embedding techniques can be applied to obtain node embeddings for the graph at the coarsest level . 2 Embedding refinement is then employed to refine the embeddings back to the original graph by applying a proper graph filter to ensure embeddings are smoothed over the graph . We evaluate the proposed GraphZoom framework on three transductive benchmarks : Cora , Citeseer and Pubmed citation networks as well as two inductive dataset : PPI and Reddit for vertex classification task . We further test the scalability of our approach on friendster dataset , which contains 8 million nodes and 400 million edges . Our experiments show that GraphZoom can improve the classification accuracy over all baseline embedding methods for both transductive and inductive tasks . Our main technical contributions are summarized as follows : •GraphZoom generates high-quality embeddings . We propose novel algorithms to encode graph structures and node attribute information in a fused graph and exploit graph filtering during refinement to remove high-frequency noise . This results in a relative increase of the embedding accuracy over the prior arts by up to 19.4 % while reducing the execution time by at least 2× . • GraphZoom improves scalability . Our approach can significantly reduce the embedding run time by effectively coarsening the graph without losing the key spectral properties . Experiments show that GraphZoom can accelerate the entire embedding process by up to 40.8× while producing a similar or better accuracy than state-of-the-art techniques . • GraphZoom is highly composable . Our framework is agnostic to underlying graph embedding techniques . Any of the existing unsupervised embedding methods , either transductive or inductive , can be incorporated by GraphZoom in a plug-and-play manner . 2 RELATED WORK . There is a large and active body of research on multi-level graph embedding and graph filtering , from which GraphZoom draws inspiration to boost the performance and speed of unsupervised 2In this work , we do not attempt to preserve node label information in the coarsened graph . Nonetheless , we believe that our approach can be extended to support supervised embedding models such as GAT ( Gulcehre et al. , 2019 ) and PPNP ( Klicpera et al. , 2019 ) , as briefly discussed in Section 5. embedding methods . Due to the space limitation , we only summarize some of the recent efforts in these two areas . Multi-level graph embedding attempts to coarsen the original graph into a series of smaller graphs with decreasing size where existing or new graph embedding techniques can be applied at different coarsening levels . For example , Chen et al . ( 2018 ) ; Lin et al . ( 2019 ) generate a hierarchy of coarsened graphs and perform embedding from the coarsest level to the original one . Fu et al . ( 2019 ) construct and embed multi-level graphs through hierarchical , and the resulting embedding vectors are concatenated to obtain the final node embeddings for the original graph . These methods , however , only focus on improving embedding quality but not the scalability . Later , Zhang et al . ( 2018b ) ; Akbas & Aktas ( 2019 ) attempt to make graph embedding more scalable by only embedding on the coarsest graph . However , their methods lack proper refinement methods to generate high-quality embeddings for the original graph . Liang et al . ( 2018 ) propose MILE , which only trains the coarsest graph to obtain coarse embeddings , and leverages GCN as refinement method to improve embedding quality . However , MILE requires training a GCN model which is very time consuming for large graphs and leading to poor performance when multiple GCN layers are stacked together ( Li et al. , 2018 ) . In contrast to the prior arts , GraphZoom is motivated by theoretical results in spectral graph embedding ( Tang & Liu , 2011 ) and practically efficient so that it can improve both accuracy and scalability of the unsupervised graph embedding tasks . Graph filtering are direct analogs of classical filters in signal processing field , but intended for signals defined on graphs . Shuman et al . ( 2013 ) define graph filters in both vertex and spectral domains , and apply graph filter on image denoising and reconstruction tasks . Wu et al . ( 2019 ) leverage graph filter to simplify GCN model by removing redundant computation . More recently , Maehara ( 2019 ) show the fundamental link between graph embedding and filtering by proving that GCN model implicitly exploits graph filter to remove high-frequency noise from the node feature matrix ; a filter neural network ( gfNN ) is then proposed to derive a stronger graph filter to improve the embedding results . Li et al . ( 2019 ) further derive two generalized graph filters and apply them on graph embedding models to improve their embedding quality for various classification tasks . In GraphZoom we adopt graph filter to properly smooth the intermediate embedding results during the iterative refinement step , which is crucial for improving the quality of the final embedding . 3 GRAPHZOOM FRAMEWORKGraphZoom Framework Figure 1 shows the proposed GraphZoom framework , which consists of four key phases : Phase ( 1 ) is graph fusion , which combines the node attributes and topology information of the original graph to construct a fused weighted graph ; In Phase ( 2 ) , a spectral graph coarsening process is applied to form a hierarchy of coarsened fused graphs with decreasing size ; In Phase ( 3 ) , any of the existing graph embedding methods can be applied to the fused graph at the coarsest level ; In Phase ( 4 ) , the embedding vectors obtained at the coarsest level are mapped onto a finer graph using the mapping operators determined during the coarsening phase . This is followed by a refinement ( or smoothing ) procedure , where the procedure in Phase ( 4 ) is applied in an iterative manner to increasingly finer graphs ; Eventually the embedding vectors for the original graph are obtained . In the rest of this section , we describe each of these four phases in more detail . 3.1 PHASE 1 : GRAPH FUSION . Graph fusion aims to construct a weighted graph that has the same number of nodes as the original graph but potentially different set of edges ( weights ) that encapsulate the original graph topology as well as node attribute information . Specifically , given an undirected graph G = ( V , E ) with N = |V| nodes , its adjacency matrix Atopo ∈ RN×N and its node attribute ( feature ) matrix X ∈ RN×K , where K corresponds to the dimension of node attribute vector , graph fusion can be interpreted as a function f ( · ) that outputs a weighted graph Gfusion = ( V , Efusion ) represented by its adjacency matrix Afusion ∈ RN×N , namely , Afusion = f ( Atopo , X ) . We first convert the initial attribute matrixX into a weighted node attribute graph Gfeat = ( V , Efeat ) by generating a k-nearest-neighbor ( kNN ) graph based on the l2-norm distance between the attribute vectors of each node pair . A straightforward implementation requires comparing all possible node pairs and then selecting top-k nearest neighbors . However , such a naı̈ve approach has a worstcase time complexity of O ( N2 ) , which certainly does not scale to large graphs . Our goal is to construct the attribute graph in nearly linear time by leveraging an efficient spectral graph coarsening scheme that is described in more detail in Section 3.2 . More specifically , our approach starts with coarsening the original graph G to obtain a substantially reduced graph that has much fewer nodes with an O ( |E| ) time complexity . Note that this procedure bears similarity to spectral graph clustering , which aims to group nodes into clusters of high conductance ( Peng et al. , 2015 ) . Once the node clusters are formed through spectral coarsening , we can select the top-k nearest neighbors within each cluster with O ( M2 ) comparisons , where M is the average node count per cluster . Since we have roughly N/M clusters , the time complexity for constructing the approximate kNN graph is O ( MN ) . When a proper coarsening ratio is chosen where M N ( e.g. , M = 50 ) , the overall time complexity will become almost linear . After the attribute graph is formed , we assign a weight to each edge based on the cosine similarity between the attribute vectors of the two incident nodes , namely wi , j = ( Xi , : ·Xj , : ) / ( ‖Xi , :‖‖Xj , :‖ ) , where Xi , : and Xj , : are the attribute vectors of nodes i and j . Finally , we can construct the fused graph by combining the topology graph and the attribute graph using a weighted sum : Afusion = Atopo +βAfeat , where β allows us to balance the topological and node attribute information during the fusion process . This fused graph can be fed into any downstream graph embedding procedures .

Summary: This paper proposes GraphZoom, a framework for augmenting unsupervised graph embedding methods by (a) fusing feature information into the graph topology, (b) learning embeddings on a coarsened graph, and (c) refining the coarsened embeddings to obtain embeddings for the original graph nodes. In particular, a nearest neighbor graph over node features is computed and this adjacency matrix is linearly combined with the original adjacency matrix to obtain a graph with feature information "fused in". The graph is then coarsened using a spectral approach, embeddings are learned on the coarsened graph (via any strategy), and the embeddings are then refined back to the original nodes (again using a spectral approach). The authors take care to heed the advice of Maehara et al. and remove high-frequency information from the features.