Graph Learning via Spectral Densification

1 INTRODUCTION . Graph learning is playing increasingly important roles in many machine learning and data mining applications . For example , a key step of many existing machine learning methods requires converting potentially high-dimensional data sets into graph representations : it is a common practice to represent each ( high-dimensional ) data point as a node , and assign each edge a weight to encode the similarity between the two nodes ( data points ) . The constructed graphs can be efficiently leveraged to represent the underlying structure of a data set or the relationship between data points ( Jebara et al. , 2009 ; Maier et al. , 2009 ; Liu et al. , 2018 ) . However , how to learn meaningful graphs from large data set at scale still remains a challenging problem . Several recent graph learning methods leverage emerging graph signal processing ( GSP ) techniques for estimating sparse graph Laplacians , which show very promising results ( Dong et al. , 2016 ; Egilmez et al. , 2017 ; Dong et al. , 2019 ; Kalofolias & Perraudin , 2019 ) . For example , ( Egilmez et al. , 2017 ) addresses the graph learning problem by restricting the precision matrix to be a graph Laplacian and maximizing a posterior estimation of attractive Gaussian Markov Random Field ( GMRF ) 1 , while an l1-regularization term is used to promote graph sparsity ; ( Rabbat , 2017 ) provides an error analysis for inferring sparse graphs from smooth signals ; ( Kalofolias & Perraudin , 2019 ) leverages approximate nearest-neighbor ( ANN ) graphs to reduce the number of variables for optimization ; ( Kumar et al. , 2019 ) introduces a graph Laplacian learning method by imposing Laplacian spectral constraints . However , even the state-of-the-art Laplacian estimation methods for graph learning do not scale well for large data set due to their extremely high algorithm complexity . For example , solving the optimization problem for Laplacian estimation in ( Dong et al. , 2016 ; Kalofolias , 2016 ; Egilmez et al. , 2017 ; Dong et al. , 2019 ) requires O ( N2 ) time complexity per iteration for N data entities and nontrivial parameters tuning for controlling graph sparsity which limits their applications to only very small data sets ( e. g. with up to a few thousands of data points ) ; the method introduced in ( Carey , 2017 ) leverages Isomap manifold embedding ( Tenenbaum et al. , 2000 ) for graph construction , which requires O ( N3 ) time for manifold construction and thus does not scale to large data set ; the latest graph learning approach ( Kalofolias & Perraudin , 2019 ) takes advantages of ANN graphs , but it still runs very slowly for large data sets ; the Laplacian estimation method with 1If the precision matrix of a GMRF is an M-matrix with all non-negative off-diagonal elements , we call it an attractive GMRF ( Slawski & Hein , 2015 ; Dong et al. , 2019 ) . spectral constraints requires a good graph structure to be provided in advance ( Kumar et al. , 2019 ) , which otherwise can be very costly when going through exhaustive graph structure searches . This work for the first time introduces a spectral graph densification approach ( GRASPEL ) for learning ultra-sparse graphs from data by leveraging the latest results in spectral graph theory ( Feng , 2016 ; 2018 ; Zhao et al. , 2018 ) . GRASPEL has a close connection with prior GSP-based Laplacian estimation methods ( Dong et al. , 2016 ; Kalofolias , 2016 ; Egilmez et al. , 2017 ; Kalofolias & Perraudin , 2019 ; Dong et al. , 2019 ) , and the graphical Lasso method ( Friedman et al. , 2008 ) . By treatingM -dimensional data points asM graph signals , GRASPEL allows efficiently solving a convex problem by iterative identifying and including the most spectrally-critical edges into the latest graph leveraging recent nearly-linear time spectral methods ( Feng , 2016 ; 2018 ; Zhao et al. , 2018 ) . Compared with prior spectral graph sparsification algorithms ( Spielman & Srivastava , 2011 ; Feng , 2016 ) that aim to remove edges from a given graph while preserving key graph spectral properties , GRASPEL aims to add edges into the graph such that the learned graphs will have spectral embedding ( or effective-resistance ) distances encoding the distances between the original input data points . Comparing with state-of-the-art graph learning methods , GRASPEL is more scalable for estimation of attractive Gaussian Markov Random Fields ( GMRFs ) for even very large data set . We summarize the contribution of this work as follows : • We propose a spectral graph densification approach ( GRASPEL ) that allows efficient estimation of attractive Gaussian Markov Random Fields ( GMRFs ) leveraging the latest spectral graph theory . •We show that the graphical Lasso problem with a Laplacian-like precision matrix can be efficiently solved by including spectrally-critical edges to dramatically reduce spectral embedding distortions . • The key to achieving high efficiency is a spectral embedding scheme for finding spectrally-critical edges , allowing each GRASPEL iteration to be completed in O ( N logN ) instead of O ( N2 ) time . • For the first time , we introduce a novel convergence criterion for graph learning tasks based on graph spectral stability : when the maximum embedding distortion becomes small enough , or equivalently when graph spectra become sufficiently stable , GRASPEL iterations can be terminated . • Our experiment results show that the graphs learned from high-dimensional data using GRASPEL can lead to more efficient and accurate spectral clustering ( SC ) as well as dimensionality reduction . 2 BACKGROUND OF GRAPH LEARNING VIA LAPLACIAN ESTIMATION . Given M observations on N data entities in a data matrix X = [ x1 , ... , xM ] ∈ RN×M , each column vector of X can be considered as a signal on a graph . For example , the USPS data set including 9 , 298 images of handwritten digits with each image having 256 pixels will result in a feature matrix X ∈ RN×M with N = 9 , 298 and M = 256 . The recent GSP-based graph learning methods ( Dong et al. , 2016 ) estimate graph Laplacians from X for achieving the following desired characteristics : Smoothness of Graph Signals . The graph signals corresponding to the real-world data should be sufficiently smooth on the learned graph structure : the signal values will only change gradually across connected neighboring nodes . The smoothness of a signal x over an undirected graph G = ( V , E , w ) can be measured with the following Laplacian quadratic form : x > Lx = ∑ ( p , q ) ∈E wp , q ( x ( p ) − x ( q ) ) 2 , ( 1 ) where L = D −W denotes the Laplacian matrix of graph G with D and W denoting the degree and the weighted adjacency matrices of G , and wp , q denotes the weight for edge ( p , q ) . The smaller value of ( 1 ) indicates the smoother signals across the graph . To quantify the smoothness ( Q ) of a set of signals X over graph G , the following matrix trace can be computed ( Kalofolias , 2016 ) : Q ( X , L ) = Tr ( X > LX ) , ( 2 ) where Tr ( • ) denotes the matrix trace . Sparsity of the Estimated Graph ( Laplacian ) . Graph sparsity is another critical consideration in graph learning . One of the most important motivations of learning a graph is to use it for downstream data mining or machine learning tasks . Therefore , desired graph learning algorithms should allow better capturing and understanding the global structure ( manifold ) of the data set , while producing sufficiently sparse graphs that can be easily stored and efficiently manipulated in the downstream algorithms , such as graph clustering , partitioning , dimension reduction , data visualization , etc . To this end , the graphical Lasso algorithm ( Friedman et al. , 2008 ) has been proposed to learn the structure in an undirected Gaussian graphical model using l1 regularization to control the sparsity of the precision matrix . Given a sample covariance matrix S and a regularization parameter β , graphical Lasso targets the following convex optimization task : max Θ : log det ( Θ ) − Tr ( ΘS ) − β‖Θ‖1 , ( 3 ) over all non-negative definite precision matrices Θ . The first two terms together can be interpreted as the log-likelihood under a Gaussian Markov Random Field ( GMRF ) . ‖ • ‖ denotes the entry-wise l1 norm , so β‖Θ‖1 becomes the sparsity promoting regularization term . This model tries to learn the graph structure by maximizing the penalized log-likelihood . When the sample covariance matrix S is obtained from M i.i.d ( independent and identically distributed ) samples X = [ x1 , ... , xM ] where X ∼ N ( 0 , S ) has an N -dimensional Gaussian distribution with zero mean , each element in the precision matrix Θi , j encodes the conditional dependence between variables Xi and Xj . For example , Θi , j = 0 implies that the corresponding variables Xi and Xj are conditionally independent , given the rest . However , the log-determinant problems are very computationally expensive . The emerging GSP-based methods infer the graph by adopting the criterion of signal smoothness ( Kalofolias , 2016 ; Dong et al. , 2016 ; Egilmez et al. , 2017 ; Kalofolias & Perraudin , 2019 ) . However , their extremely high complexities do not even allow for learning graphs with even a few thousands of nodes . Furthermore , these methods require nontrivial parameters tuning for achieving good performance . 3 GRASPEL : GRAPH SPECTRAL LEARNING AT SCALE . Similar to recent GSP-based Laplacian estimation methods , GRASPEL aims to more efficiently solve the following convex problem for estimation of attractive GMRFs ( Dong et al. , 2019 ; Lake & Tenenbaum , 2010 ) that is also similar to the graphical Lasso problem ( Friedman et al. , 2008 ) : max Θ : log det ( Θ ) − 1 M Tr ( X > ΘX ) − β‖Θ‖1 , ( 4 ) where Θ = L + 1σ2 I , L denotes the set of valid graph Laplacian matrices , I denotes the identity matrix , and σ2 > 0 denotes prior feature variance . It can be shown that the three terms in ( 4 ) are corresponding to log det ( Θ ) , Tr ( ΘS ) and β‖Θ‖1 in ( 3 ) , respectively . When each column vector in the data matrix X 2 is treated as a graph signal vector , there is a close connection between our formulation and the graphical Lasso problem . Since Θ = L + 1σ2 I correspond to symmetric and positive definite ( PSD ) matrices ( or M matrices ) with non-positive off-diagonal entries , this formulation will lead to the estimation of attractive GMRFs ( Dong et al. , 2019 ; Slawski & Hein , 2015 ) . In case thatX is non-Gaussian , ( 4 ) can be understood as Laplacian estimation based on minimizing the Bregman divergence between positive definite matrices induced by the function Θ 7→ − log det ( Θ ) ( Slawski & Hein , 2015 ) . 3.1 THEORETICAL BACKGROUND . Express the Laplacian matrix as L = ∑ ( p , q ) ∈E wp , qep , qe > p , q ( 5 ) where ep ∈ RN denotes the standard basis vector with all zero entries except for the p-th entry being 1 , and ep , q = ep − eq . Consider the objective function F in ( 4 ) : F = log det ( Θ ) − 1 M Tr ( X > ΘX ) − β‖Θ‖1 , ( 6 ) which can be further simplified as follows by substituting ( 5 ) into ( 6 ) : F = N∑ i=1 log ( λi + 1/σ 2 ) − 1 M Tr ( X > X ) σ2 + ∑ ( p , q ) ∈E wp , q‖X > ep , q‖22 − 4β ∑ ( p , q ) ∈E wp , q , ( 7 ) 2For each of the N row vectors X ( i , : ) ∈ R1×M where i = 1 , ... , N , the following two-step data preprocessing will be performed : ( 1 ) X ( i , : ) = X ( i , : ) − µi , where µi denotes the sample mean of X ( i , : ) ; then S = XX > M , which results in Tr ( ΘS ) = 1 M Tr ( X > ΘX ) . ( 2 ) Feature normalization by X = X/‖X‖2 . where the Laplacian eigenvectors corresponding to the ascending eigenvalues λi are denoted by ui for i = 1 , ... , N , satisfying : Lui = λiui . ( 8 ) Taking the partial derivative with respect to the weight wp , q of edge ( p , q ) leads to : ∂F ∂wp , q = N∑ i=2 1 λi + 1/σ2 ∂λi ∂wp , q − 1 M ‖X > ep , q‖22 − 4β , ( 9 ) Since the last two terms in ( 9 ) are all fixed ( constant ) values for a given data matrix X where β can be considered as an additional offset added to all data pairs ( candidate edges ) , we can drop the third term by simply setting β = 0 , which will not impact the ranking of candidate edges in graph learning . The above simplification implies the second term alone will effectively penalize graph density for estimating Laplacian-like precision matrix : including more edges will result in a greater trace Tr ( X > ΘX ) . The spectral perturbation analysis in Theorem 1 ( Appendix ) will lead to : ∂λi ∂wp , q = ( u > i ep , q ) 2 . ( 10 ) If we construct a subspace matricesUr for spectral graph embedding by using the first r−1 weighted nontrivial Laplacian eigenvectors as follows : Ur = [ u2√ λ2 + 1/σ2 , ... , ur√ λr + 1/σ2 ] , ( 11 ) then by setting β = 0 , ( 9 ) can be rewritten as follows : ∂F ∂wp , q ≈ ‖U > r ep , q‖22 − 1 M ‖X > ep , q‖22 = zembp , q − 1 M zdatap , q , ( 12 ) where zembp , q = ‖U > r ep , q‖22 and zdatap , q = ‖X > ep , q‖22 denote the ` 2 distances in the spectral embedding space and the original data space , respectively . We note that using a larger r for constructing Ur will allow more accurate estimation of ∂F∂wp , q , but at a higher computational cost .

Summary: The paper proposes a graph learning method for spectral embedding and associated problems such as clustering and dimension reduction. What differentiates the method from much the existing literature is that it focuses approximating an optimal densification of a very sparse initial graph rather than on sparsification of an initial graph, as is more common. The method is based on iteratively identifying edges to add to the graph so as to best improve the corresponding spectral embedding, so called "spectrally critical" edges. The authors motivate spectral criticality in relation to the partial derivatives of an objective function inspired by the log-likelihood of a Gaussian graphical model. In particular, those with the highest partial derivatives will tend to be those which, through their addition to the graph, lead to the greatest increase in this objective. The authors go on to discuss a close connection between spectral criticality and distance distortion when comparing the spectral embedding and the original input space. Since the initial graph is very sparse it can be efficiently determined, and the relatively small number of additional edges which need to be added by the proposed method to obtain a high quality embedding means that the entire procedure can be implemented efficiently.