Graph Learning via Spectral Densification

1 INTRODUCTION . Graph learning is playing increasingly important roles in many machine learning and data mining applications . For example , a key step of many existing machine learning methods requires converting potentially high-dimensional data sets into graph representations : it is a common practice to represent each ( high-dimensional ) data point as a node , and assign each edge a weight to encode the similarity between the two nodes ( data points ) . The constructed graphs can be efficiently leveraged to represent the underlying structure of a data set or the relationship between data points ( Jebara et al. , 2009 ; Maier et al. , 2009 ; Liu et al. , 2018 ) . However , how to learn meaningful graphs from large data set at scale still remains a challenging problem . Several recent graph learning methods leverage emerging graph signal processing ( GSP ) techniques for estimating sparse graph Laplacians , which show very promising results ( Dong et al. , 2016 ; Egilmez et al. , 2017 ; Dong et al. , 2019 ; Kalofolias & Perraudin , 2019 ) . For example , ( Egilmez et al. , 2017 ) addresses the graph learning problem by restricting the precision matrix to be a graph Laplacian and maximizing a posterior estimation of attractive Gaussian Markov Random Field ( GMRF ) 1 , while an l1-regularization term is used to promote graph sparsity ; ( Rabbat , 2017 ) provides an error analysis for inferring sparse graphs from smooth signals ; ( Kalofolias & Perraudin , 2019 ) leverages approximate nearest-neighbor ( ANN ) graphs to reduce the number of variables for optimization ; ( Kumar et al. , 2019 ) introduces a graph Laplacian learning method by imposing Laplacian spectral constraints . However , even the state-of-the-art Laplacian estimation methods for graph learning do not scale well for large data set due to their extremely high algorithm complexity . For example , solving the optimization problem for Laplacian estimation in ( Dong et al. , 2016 ; Kalofolias , 2016 ; Egilmez et al. , 2017 ; Dong et al. , 2019 ) requires O ( N2 ) time complexity per iteration for N data entities and nontrivial parameters tuning for controlling graph sparsity which limits their applications to only very small data sets ( e. g. with up to a few thousands of data points ) ; the method introduced in ( Carey , 2017 ) leverages Isomap manifold embedding ( Tenenbaum et al. , 2000 ) for graph construction , which requires O ( N3 ) time for manifold construction and thus does not scale to large data set ; the latest graph learning approach ( Kalofolias & Perraudin , 2019 ) takes advantages of ANN graphs , but it still runs very slowly for large data sets ; the Laplacian estimation method with 1If the precision matrix of a GMRF is an M-matrix with all non-negative off-diagonal elements , we call it an attractive GMRF ( Slawski & Hein , 2015 ; Dong et al. , 2019 ) . spectral constraints requires a good graph structure to be provided in advance ( Kumar et al. , 2019 ) , which otherwise can be very costly when going through exhaustive graph structure searches . This work for the first time introduces a spectral graph densification approach ( GRASPEL ) for learning ultra-sparse graphs from data by leveraging the latest results in spectral graph theory ( Feng , 2016 ; 2018 ; Zhao et al. , 2018 ) . GRASPEL has a close connection with prior GSP-based Laplacian estimation methods ( Dong et al. , 2016 ; Kalofolias , 2016 ; Egilmez et al. , 2017 ; Kalofolias & Perraudin , 2019 ; Dong et al. , 2019 ) , and the graphical Lasso method ( Friedman et al. , 2008 ) . By treatingM -dimensional data points asM graph signals , GRASPEL allows efficiently solving a convex problem by iterative identifying and including the most spectrally-critical edges into the latest graph leveraging recent nearly-linear time spectral methods ( Feng , 2016 ; 2018 ; Zhao et al. , 2018 ) . Compared with prior spectral graph sparsification algorithms ( Spielman & Srivastava , 2011 ; Feng , 2016 ) that aim to remove edges from a given graph while preserving key graph spectral properties , GRASPEL aims to add edges into the graph such that the learned graphs will have spectral embedding ( or effective-resistance ) distances encoding the distances between the original input data points . Comparing with state-of-the-art graph learning methods , GRASPEL is more scalable for estimation of attractive Gaussian Markov Random Fields ( GMRFs ) for even very large data set . We summarize the contribution of this work as follows : • We propose a spectral graph densification approach ( GRASPEL ) that allows efficient estimation of attractive Gaussian Markov Random Fields ( GMRFs ) leveraging the latest spectral graph theory . •We show that the graphical Lasso problem with a Laplacian-like precision matrix can be efficiently solved by including spectrally-critical edges to dramatically reduce spectral embedding distortions . • The key to achieving high efficiency is a spectral embedding scheme for finding spectrally-critical edges , allowing each GRASPEL iteration to be completed in O ( N logN ) instead of O ( N2 ) time . • For the first time , we introduce a novel convergence criterion for graph learning tasks based on graph spectral stability : when the maximum embedding distortion becomes small enough , or equivalently when graph spectra become sufficiently stable , GRASPEL iterations can be terminated . • Our experiment results show that the graphs learned from high-dimensional data using GRASPEL can lead to more efficient and accurate spectral clustering ( SC ) as well as dimensionality reduction . 2 BACKGROUND OF GRAPH LEARNING VIA LAPLACIAN ESTIMATION . Given M observations on N data entities in a data matrix X = [ x1 , ... , xM ] ∈ RN×M , each column vector of X can be considered as a signal on a graph . For example , the USPS data set including 9 , 298 images of handwritten digits with each image having 256 pixels will result in a feature matrix X ∈ RN×M with N = 9 , 298 and M = 256 . The recent GSP-based graph learning methods ( Dong et al. , 2016 ) estimate graph Laplacians from X for achieving the following desired characteristics : Smoothness of Graph Signals . The graph signals corresponding to the real-world data should be sufficiently smooth on the learned graph structure : the signal values will only change gradually across connected neighboring nodes . The smoothness of a signal x over an undirected graph G = ( V , E , w ) can be measured with the following Laplacian quadratic form : x > Lx = ∑ ( p , q ) ∈E wp , q ( x ( p ) − x ( q ) ) 2 , ( 1 ) where L = D −W denotes the Laplacian matrix of graph G with D and W denoting the degree and the weighted adjacency matrices of G , and wp , q denotes the weight for edge ( p , q ) . The smaller value of ( 1 ) indicates the smoother signals across the graph . To quantify the smoothness ( Q ) of a set of signals X over graph G , the following matrix trace can be computed ( Kalofolias , 2016 ) : Q ( X , L ) = Tr ( X > LX ) , ( 2 ) where Tr ( • ) denotes the matrix trace . Sparsity of the Estimated Graph ( Laplacian ) . Graph sparsity is another critical consideration in graph learning . One of the most important motivations of learning a graph is to use it for downstream data mining or machine learning tasks . Therefore , desired graph learning algorithms should allow better capturing and understanding the global structure ( manifold ) of the data set , while producing sufficiently sparse graphs that can be easily stored and efficiently manipulated in the downstream algorithms , such as graph clustering , partitioning , dimension reduction , data visualization , etc . To this end , the graphical Lasso algorithm ( Friedman et al. , 2008 ) has been proposed to learn the structure in an undirected Gaussian graphical model using l1 regularization to control the sparsity of the precision matrix . Given a sample covariance matrix S and a regularization parameter β , graphical Lasso targets the following convex optimization task : max Θ : log det ( Θ ) − Tr ( ΘS ) − β‖Θ‖1 , ( 3 ) over all non-negative definite precision matrices Θ . The first two terms together can be interpreted as the log-likelihood under a Gaussian Markov Random Field ( GMRF ) . ‖ • ‖ denotes the entry-wise l1 norm , so β‖Θ‖1 becomes the sparsity promoting regularization term . This model tries to learn the graph structure by maximizing the penalized log-likelihood . When the sample covariance matrix S is obtained from M i.i.d ( independent and identically distributed ) samples X = [ x1 , ... , xM ] where X ∼ N ( 0 , S ) has an N -dimensional Gaussian distribution with zero mean , each element in the precision matrix Θi , j encodes the conditional dependence between variables Xi and Xj . For example , Θi , j = 0 implies that the corresponding variables Xi and Xj are conditionally independent , given the rest . However , the log-determinant problems are very computationally expensive . The emerging GSP-based methods infer the graph by adopting the criterion of signal smoothness ( Kalofolias , 2016 ; Dong et al. , 2016 ; Egilmez et al. , 2017 ; Kalofolias & Perraudin , 2019 ) . However , their extremely high complexities do not even allow for learning graphs with even a few thousands of nodes . Furthermore , these methods require nontrivial parameters tuning for achieving good performance . 3 GRASPEL : GRAPH SPECTRAL LEARNING AT SCALE . Similar to recent GSP-based Laplacian estimation methods , GRASPEL aims to more efficiently solve the following convex problem for estimation of attractive GMRFs ( Dong et al. , 2019 ; Lake & Tenenbaum , 2010 ) that is also similar to the graphical Lasso problem ( Friedman et al. , 2008 ) : max Θ : log det ( Θ ) − 1 M Tr ( X > ΘX ) − β‖Θ‖1 , ( 4 ) where Θ = L + 1σ2 I , L denotes the set of valid graph Laplacian matrices , I denotes the identity matrix , and σ2 > 0 denotes prior feature variance . It can be shown that the three terms in ( 4 ) are corresponding to log det ( Θ ) , Tr ( ΘS ) and β‖Θ‖1 in ( 3 ) , respectively . When each column vector in the data matrix X 2 is treated as a graph signal vector , there is a close connection between our formulation and the graphical Lasso problem . Since Θ = L + 1σ2 I correspond to symmetric and positive definite ( PSD ) matrices ( or M matrices ) with non-positive off-diagonal entries , this formulation will lead to the estimation of attractive GMRFs ( Dong et al. , 2019 ; Slawski & Hein , 2015 ) . In case thatX is non-Gaussian , ( 4 ) can be understood as Laplacian estimation based on minimizing the Bregman divergence between positive definite matrices induced by the function Θ 7→ − log det ( Θ ) ( Slawski & Hein , 2015 ) . 3.1 THEORETICAL BACKGROUND . Express the Laplacian matrix as L = ∑ ( p , q ) ∈E wp , qep , qe > p , q ( 5 ) where ep ∈ RN denotes the standard basis vector with all zero entries except for the p-th entry being 1 , and ep , q = ep − eq . Consider the objective function F in ( 4 ) : F = log det ( Θ ) − 1 M Tr ( X > ΘX ) − β‖Θ‖1 , ( 6 ) which can be further simplified as follows by substituting ( 5 ) into ( 6 ) : F = N∑ i=1 log ( λi + 1/σ 2 ) − 1 M Tr ( X > X ) σ2 + ∑ ( p , q ) ∈E wp , q‖X > ep , q‖22 − 4β ∑ ( p , q ) ∈E wp , q , ( 7 ) 2For each of the N row vectors X ( i , : ) ∈ R1×M where i = 1 , ... , N , the following two-step data preprocessing will be performed : ( 1 ) X ( i , : ) = X ( i , : ) − µi , where µi denotes the sample mean of X ( i , : ) ; then S = XX > M , which results in Tr ( ΘS ) = 1 M Tr ( X > ΘX ) . ( 2 ) Feature normalization by X = X/‖X‖2 . where the Laplacian eigenvectors corresponding to the ascending eigenvalues λi are denoted by ui for i = 1 , ... , N , satisfying : Lui = λiui . ( 8 ) Taking the partial derivative with respect to the weight wp , q of edge ( p , q ) leads to : ∂F ∂wp , q = N∑ i=2 1 λi + 1/σ2 ∂λi ∂wp , q − 1 M ‖X > ep , q‖22 − 4β , ( 9 ) Since the last two terms in ( 9 ) are all fixed ( constant ) values for a given data matrix X where β can be considered as an additional offset added to all data pairs ( candidate edges ) , we can drop the third term by simply setting β = 0 , which will not impact the ranking of candidate edges in graph learning . The above simplification implies the second term alone will effectively penalize graph density for estimating Laplacian-like precision matrix : including more edges will result in a greater trace Tr ( X > ΘX ) . The spectral perturbation analysis in Theorem 1 ( Appendix ) will lead to : ∂λi ∂wp , q = ( u > i ep , q ) 2 . ( 10 ) If we construct a subspace matricesUr for spectral graph embedding by using the first r−1 weighted nontrivial Laplacian eigenvectors as follows : Ur = [ u2√ λ2 + 1/σ2 , ... , ur√ λr + 1/σ2 ] , ( 11 ) then by setting β = 0 , ( 9 ) can be rewritten as follows : ∂F ∂wp , q ≈ ‖U > r ep , q‖22 − 1 M ‖X > ep , q‖22 = zembp , q − 1 M zdatap , q , ( 12 ) where zembp , q = ‖U > r ep , q‖22 and zdatap , q = ‖X > ep , q‖22 denote the ` 2 distances in the spectral embedding space and the original data space , respectively . We note that using a larger r for constructing Ur will allow more accurate estimation of ∂F∂wp , q , but at a higher computational cost .

Summary: This paper studies ways of adding edges to graphs to improve the result of spectral embedding / clustering. It refines existing embeddings using by measuring edges' effect on Laplacian eigenvalues, and adjusting such edges to reduce the distortions. The performance of the algorithm is justified using developments of worst-case efficient algorithms for Laplacian matrices, and experimentally, the algorithm converges quickly when starting with nearest neighbor graphs, and leads to significant increases in accuracy.