GRASPEL: GRAPH SPECTRAL LEARNING AT SCALE

1 INTRODUCTION . Graph construction is playing increasingly important roles in many machine learning and data mining applications . For example , a key step of many existing machine learning methods requires converting potentially high-dimensional data sets into graph representations : it is a common practice to represent each ( high-dimensional ) data point as a node , and assign each edge a weight to encode the similarity between the two nodes ( data points ) . The constructed graphs can be efficiently leveraged to represent the underlying structure of a data set or the relationship between data points ( Jebara et al. , 2009 ; Maier et al. , 2009 ; Liu et al. , 2018 ) . However , how to learn meaningful graphs from large data set at scale still remains a challenging problem . In the past decades , considerable effort has been devoted to the development of graph construction methods . For example , constructing k-nearest-neighbor ( kNN ) graphs requires each node to be connected with its top-k nearest neighbors , while in construction of the -neighborhood graphs all the neighbors within the range of distance will be connected ; to improve the capability of kNN graph in handling multi-scale data , ( Zelnik-Manor & Perona , 2005 ) introduced a self-tuning technique to adjust the local scaling parameter for similarity measurement ; to find meaningful similarity measures between nodes , ( Bach & Jordan , 2006 ) propose to learn the similarities from feature vectors in a supervised setting ; ( Zhu et al. , 2014 ) adopted an information-theoretic definition of data similarity to capture subtle similarity information ; ( Jebara & Shchogolev , 2006 ) proposed to remove spurious edges from kNN graph via b-matching ; ( Pavan & Pelillo , 2007 ) introduced a method for removing noisy edges by selecting the maximum cliques ; ( Premachandran & Kakarala , 2013 ) proposed to leverage collected consensus information form various neighborhoods to improve the robustness of the kNN graph ; ( Nie et al. , 2014 ) proposed to learn the adjacency graph by adaptively assigning neighbors . However , the aforementioned nearest-neighbor ( NN ) based graph construction methods can only capture local manifold information and may not be able to truthfully reveal the global structure of a given data set ( Nie et al. , 2016 ; Liu et al. , 2018 ; Guo , 2015 ) , which can result in over complicated ( with too many edges ) or sometimes misleading graph representations . For example , choosing different numbers of nearest neighbors for constructing kNN graphs may lead to drastically different classification performance in spectral clustering tasks ( Chen et al. , 2018 ) . Several recent graph learning methods leverage emerging graph signal processing ( GSP ) techniques for estimating sparse graph Laplacians , which show very promising results ( Dong et al. , 2016 ; Egilmez et al. , 2017 ; Dong et al. , 2019 ; Kalofolias & Perraudin , 2019 ) . For example , ( Egilmez et al. , 2017 ) addresses the graph learning problem by restricting the precision matrix to be a graph Laplacian and maximizing a posterior estimation of Gaussian Markov Random Field ( GMRF ) , while an L1regularization term is used to promote graph sparsity ; ( Rabbat , 2017 ) provides an error analysis for inferring sparse graphs from smooth signals ; ( Kalofolias & Perraudin , 2019 ) leverages approximate nearest-neighbor ( ANN ) graphs to reduce the number of variables for optimization . However , even the state-of-the-art Laplacian estimation methods for graph learning do not scale well for large data set due to their extremely high algorithm complexity . For example , solving the optimization problem for Laplacian estimation in ( Dong et al. , 2016 ; Kalofolias , 2016 ; Egilmez et al. , 2017 ; Dong et al. , 2019 ) requires O ( N2 ) time complexity per iteration for N data entities and nontrivial parameters tuning for controlling graph sparsity which limits their applications to only very small data sets ( e. g. with up to a few thousands of data points ) . The latest graph learning approach ( Kalofolias & Perraudin , 2019 ) takes advantages of ANN graphs but can still run rather slowly for large data sets . This work for the first time introduces a spectral method ( GRASPEL ) for learning ultra-sparse graphs from data by leveraging the latest results in spectral graph theory ( Feng , 2016 ; 2018 ; Zhao et al. , 2018 ) . There is a clear connection between our approach and the GSP-based Laplacian estimation methods ( Dong et al. , 2016 ; Kalofolias , 2016 ; Egilmez et al. , 2017 ; Kalofolias & Perraudin , 2019 ; Dong et al. , 2019 ) , as well as the classical graphical Lasso framework ( Friedman et al. , 2008 ) . Specifically , by treating p-dimensional data points as p graph signals , GRASPEL learns a graph Laplacian by maximizing its first few eigenvalues as well as the smoothness of graph signals across edges , subject to a graph sparsity constraint . By iteratively interleaving recent nearly-linear time spectral graph sparsification , coarsening and embedding methods ( Feng , 2016 ; 2018 ; Zhao et al. , 2018 ) , GRASPEL enjoys a nearly-linear runtime and space complexity . GRASPEL is similar to the original graphical Lasso ( Friedman et al. , 2008 ) with the precision matrix replaced by a graph Laplacian . GRASPEL iteratively identifies and includes the most spectrallycritical edges into the latest graph , so that the first few Laplacian eigenvalues and eigenvectors can be most significantly perturbed by adding the minimum amount of edges . The iterative graph learning procedure will be terminated when the graph spectra become sufficiently stable ( or graph signals become sufficiently smooth across the graph and lead to rather small Laplacian quadratic forms ) . Comparing with state-of-the-art methods , GRASPEL allows more scalable estimation of attractive Gaussian Markov Random Fields ( GMRFs ) for even very large data set . We show through extensive experiments that GRASPEL can learn high-quality ultra-sparse ( tree-like ) graphs that can be immediately leveraged to significantly improve the efficiency and accuracy of spectral clustering ( SC ) tasks ; the proposed approach also leads to the development of a multilevel t-Distributed Stochastic Neighbor Embedding ( t-SNE ) algorithm that shows significantly improved runtime over existing methods ( Maaten & Hinton , 2008 ; Van Der Maaten , 2014 ) . 2 BACKGROUND OF GRAPH LEARNING VIA LAPLACIAN ESTIMATION . Given M observations on N data entities stored in a data matrix X ∈ RN×M , each column of X can be considered as a signal on a graph . The recent graph learning method ( Dong et al. , 2016 ) aims to estimate a graph Laplacian from X while achieving the following desired characteristics : 1 ) Smoothness of signals on the graph . The graph signals corresponding to the real-world data should be sufficiently smooth on the learned graph structure : the signal values will only change gradually across connected neighboring nodes . The smoothness of a signal x over a undirected graph G = ( V , E , w ) can be measured with Laplacian quadratic form xTLx = ∑ ( p , q ) ∈E wp , q ( x ( p ) − x ( q ) ) 2 , where L = D −W denotes the Laplacian matrix of graph G with D and W denoting the degree and the weighted adjacency matrices of G , and wp , q = W ( p , q ) denotes the weight for edge ( p , q ) . The smaller value of quadratic form indicates the smoother signals across the graph . It is also possible to quantify the smoothness ( Q ) of a set of signals X over graph G using the following matrix trace ( Kalofolias , 2016 ) : Q ( X , L ) = Tr ( XTLX ) , where Tr denotes the matrix trace . 2 ) Sparsity of the estimated graph ( Laplacian ) . Graph sparsity is another critical consideration in graph learning . One of the most important motivations of learning a graph is to use it for downstream data mining or machine learning tasks . Therefore , desired graph learning algorithms should allow better capturing and understanding the global structure ( manifold ) of the data set , while producing sufficiently sparse graphs that can be easily stored and efficiently manipulated in the downstream algorithms , such as graph clustering , partitioning , dimension reduction , data visualization , etc . To this end , the graphical Lasso algorithm ( Friedman et al. , 2008 ) has been proposed to learn the structure in an undirected Gaussian graphical model using L1 regularization to control the sparsity of the precision matrix . Given a sample covariance matrix S and a regularization parameter β , graphical Lasso targets the following objective function : max Θ log det ( Θ ) − Tr ( ΘS ) − β‖Θ‖1 , ( 1 ) over all non-negative definite precision matrices Θ . The first two terms together can be interpreted as the log-likelihood under a Gaussian Markov Random Field ( GMRF ) . β‖Θ‖1 is the sparsity promoting regularization term . This model tries to learn the graph structure by maximizing the penalized log-likelihood . However , the log-determinant problems are very computationally expensive . The emerging GSP-based methods infer the graph by adopting the criterion of signal smoothness ( Kalofolias , 2016 ; Dong et al. , 2016 ; Egilmez et al. , 2017 ; Kalofolias & Perraudin , 2019 ) . However , their extremely high complexities do not allow for learning large-scale graphs involving millions or even hundred thousands of nodes . Furthermore , these methods usually require nontrivial parameters tuning for controlling graph sparsity . 3 GRASPEL : A SPECTRAL APPROACH TO GRAPH LEARNING FROM DATA . At high level , GRASPEL gains insight from recent GSP-based Laplacian estimation methods ( Dong et al. , 2019 ) , aiming to solve the following optimization problem that is similar to the graphical Lasso problem ( Friedman et al. , 2008 ) : maximize : L∈L log det ( L ) − αTr ( XTLX ) − β‖L‖1 , ( 2 ) where L denotes the set of valid Laplacian matrices . It can be shown that the three terms in ( 2 ) are corresponding to log det ( Θ ) , Tr ( ΘS ) and β‖Θ‖1 in ( 1 ) , respectively . When the precision matrix Θ is restricted to be a graph Laplacian , and each column vector in the original data matrix X is treated as a graph signal vector , there is a close connection between our formulation and the graphical Lasso problem . Since graph Laplacians are symmetric and positive definite ( PSD ) matrices ( or M matrices ) with non-positive off-diagonal entries , this formulation will lead to the estimation of attractive GMRFs ( Dong et al. , 2019 ) . 3.1 OVERVIEW OF GRASPEL To achieve good efficiency in graph learning that may involve millions of nodes , instead of solving ( 2 ) directly , GRASPEL leverages a spectral approach for solving ( 2 ) implicitly . Define spectrally-critical edges to be the ones that can most effectively perturb the graph spectral properties , such as the first few Laplacian eigenvalues and eigenvectors . GRASPEL aims to iteratively identify and add the most spectrally-critical edges into the latest graph until no such edges can be found , which consists of the following key steps as illustrated in Figure 1 : Step ( 1 ) : Initial graph construction . Similar to ( Kalofolias & Perraudin , 2019 ) , we start with constructing an ANN graph that can be achieved in O ( N logN ) time , where each edge weight encodes the similarity ( e.g . Gaussian kernel or cosine similarity ) of two data entities ; next , the ANN graph is converted into an ultra-sparse nearest-neighbor ( uNN ) graph with O ( N logN ) edges by leveraging a nearly-linear-time spectral sparsification algorithm ( Feng , 2018 ) . Step ( 2 ) : Spectral graph embedding . We apply a nearly-linear-time spectral embedding procedure ( Zhao et al. , 2018 ) to the current graph so that each node will be associated with a low-dimensional embedding vector ( e.g . vs for node S in Figure 1 ) , where the embedding dimension ( number of eigenvectors ) can be determined based on the largest gaps of the first few ( e.g . 100 ) Laplacian eigenvalues ( Peng et al. , 2015 ) . Step ( 3 ) : Spectrally-critical edge identification . We quickly check the embedding distortion for each candidate edge ( node pair ) defined as η = z emb zdata , where zemb ( zdata ) denotes the distance in the embedding ( data ) vector space . The edges with the largest η are considered as the most spectrally-critical edges and will be added into the latest graph . Step ( 4 ) : Spectral stability checking . After repeating the Steps ( 2 ) - ( 3 ) multiple times for adding new edges , GRASPEL will return the final graph once the overall embedding distortion becomes sufficiently small or stable : if the first few Laplacian eigenvalues do not change much over iterations of adding extra edges , the graph spectra is considered stable ( robust ) since adding more edges does not significantly perturb the key ( smallest ) eigenvalues . Connection between GRASPEL and the formulation ( 2 ) . The original optimization objective function includes the following three components : ( a ) log det ( L ) corresponding to the sum of the logarithmic Laplacian eigenvalues , ( b ) −αTr ( XTLX ) corresponding to the smoothness of signals across the graph 1 , and ( c ) −β ∗ |L|1 corresponding to graph sparsity . Including spectrally-critical edges into the graph will significantly impact the first few Laplacian eigenvalues and eigenvectors key to graph spectral properties , thereby dramatically improving embedding distortion and the overall smoothness of signals across the graph . It can be shown that including any additional edge into the graph will monotonically increase ( a ) while monotonically decreasing both ( b ) and ( c ) . Therefore , when the spectra of the learned graph is not stable adding any spectrally-critical edges into the graph will dramatically increase ( a ) while slightly decreasing ( b ) and ( c ) , since the improved graph signal smoothness will only result in a slight change ( increase ) to Tr ( XTLX ) . Convergence analysis for the GRASPEL framework . The objective function in ( 2 ) will be effectively maximized by including only a small amount of spectrally-critical edges until the first few eigenvalues become sufficiently stable ; when adding extra edges can no longer perturb the first few eigenvalues , ( b ) and ( c ) will start to dominate the objective function value , indicating that the iterations should be terminated . The stopping condition can be controlled by properly setting an embedding distortion threshold for η or parameters α and β . Complexity analysis for the GRASPEL framework . To achieve scalable spectral graph embedding key to identification of spectrally-critical edges in Steps ( 2 ) - ( 3 ) , we will leverage the latest highperformance spectral graph algorithms . Since all the kernel functions involved in GRASPEL , such as ANN graph construction ( Muja & Lowe , 2009 ; 2014 ; Malkov & Yashunin , 2018 ) , spectral graph sparsification ( Feng , 2016 ; 2018 ) , spectral coarsening and embedding ( Zhao et al. , 2018 ) , are all nearly-linear O ( N logN ) time algorithms , the entire spectral graph learning approach GRASPEL also has a nearly-linear time complexity .

Summary: This paper proposes a scalable approach for graph learning from data. At a high-level, it begins with a k-NN graph construction, then node features are embedded to spectral space (embedding to space spanned by eigenvectors of Laplacian). Next, edges that have a large distortion are additionally added to the latest graph. And these steps are repeated until the output graph is stable (i.e., the embedding distortion becomes small). Experimental result for spectral clustering shows that the proposed method can achieve the best accuracy compared to kNN-based methods. For graph recovery, the algorithm also performs better than other Laplacian-based graph learning methods. In addition, the proposed approach runs up to 5 times faster for t-SNE.