Implicit SVD for Graph Representation Learning

1 Introduction . Truncated Singular Value Decomposition ( SVD ) provides solutions to a variety of mathematical problems , including computing a matrix rank , its pseudo-inverse , or mapping its rows and columns onto the orthonormal singular bases for low-rank approximations . Machine Learning ( ML ) software frameworks ( such as TensorFlow ) offer efficient SVD implementations , as SVD can estimate solutions for a variaty of tasks , e.g. , in computer vision [ Turk and Pentland , 1991 ] , weather prediction [ Molteni et al. , 1996 ] , recommendation [ Koren et al. , 2009 ] , language [ Deerwester et al. , 1990 , Levy and Goldberg , 2014 ] , and more-relevantly , graph representation learning ( GRL ) [ Qiu et al. , 2018 ] . SVD ’ s benefits include training models , without calculating gradients , to arrive at globally-unique solutions , optimizing Frobenius-norm objectives ( §2.3 ) , without requiring hyperparameters for the learning process , such as the choice of the learning algorithm , step-size , regularization coefficient , etc . Typically , one constructs a design matrix M , such that , its decomposition provides a solution to a task of interest . Unfortunately , existing popular ML frameworks [ Abadi et al. , 2016 , Paszke et al. , 2019 ] can not calculate the SVD of an arbitrary linear matrix given its computation graph : they ∗Part of this work was done during internship at Intel Labs . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . compute the matrix ( entry-wise ) then2 its decomposition . This limits the scalability of these libraries in several cases of interest , such as in GRL , when explicit calculation of the matrix is prohibitive due to memory constraints . These limitations render SVD as impractical for achieving state-of-the-art ( SOTA ) for tasks at hand . This has been circumvented by Qiu et al . [ 2018 ] by sampling M entrywise , but this produces sub-optimal estimation error and experimentally degrades the empirical test performance ( §7 : Experiments ) . We design a software library that allows symbolic definition of M , via composition of matrix operations , and we implement an SVD algorithm that can decompose M from said symbolic representation , without need to compute M. This is valuable for many GRL tasks , where the design matrix M is too large , e.g. , quadratic in the input size . With our implementation , we show that SVD can perform learning , orders of magnitudes faster than current alternatives . Currently , SOTA GRL models are generally graph neural networks trained to optimize cross-entropy objectives . Their inter-layer non-linearities place their ( many ) parameters onto a non-convex objective surface where convergence is rarely verified3 . Nonetheless , these models can be convexified ( §3 ) and trained via SVD , if we remove nonlinearities between layers and swap the cross-entropy objective with Frobenius norm minimization . Undoubtedly , such linearization incurs a drop of accuracy on empirical test performance . Nonetheless , we show that the ( convexified ) model ’ s parameters learned by SVD can provide initialization to deeper ( non-linear ) models , which then can be fine-tuned on cross-entropy objectives . The non-linear models are endowed with our novel Split-ReLu layer , which has twice as many parameters as a ReLu fully-connected layer , and behaves as a linear layer when its parameters reside on some hyperplane ( §5.2 ) . Training on modest hardware ( e.g. , laptop ) is sufficient for this learning pipeline ( convexify → SVD → fine-tune ) yet it trains much faster than current approaches , that are commonly trained on expensive hardware . We summarize our contributions as : 1 . We open-source a flexible python software library that allows symbolic definition of matrices and computes their SVD without explicitly calculating them . 2 . We linearize popular GRL models , and train them via SVD of design matrices . 3 . We show that fine-tuning a few parameters on-top of the SVD initialization sets state-of-the-art on many GRL tasks while , overall , training orders-of-magnitudes faster . 2 Preliminaries & notation . We denote a graph with n nodes and m edges with an adjacency matrix A ∈ Rn×n and additionally , if nodes have ( d-dimensional ) features , with a feature matrix X ∈ Rn×d . If nodes i , j ∈ [ n ] are connected then Aij is set to their edge weight and otherwise Aij = 0 . Further , denote the ( row-wise normalized ) transition matrix as T = D−1A and denote the symmetrically normalized adjacency with self-connections as Â = ( D + I ) − 1 2 ( A + I ) ( D + I ) − 1 2 where I is identity matrix . We review model classes : ( 1 ) network embedding and ( 2 ) message passing that we define as follows . The first inputs a graph ( A , X ) and outputs node embedding matrix Z ∈ Rn×z with z-dimensions per node . Z is then used for an upstream task , e.g. , link prediction . The second class utilizes a function H : Rn×n×Rn×d → Rn×z where the function H ( A , X ) is usually directly trained on the upstream task , e.g. , node classification . In general , the first class is transductive while the second is inductive . 2.1 Network embedding models based on DeepWalk & node2vec . The seminal work of DeepWalk [ Perozzi et al. , 2014 ] embeds nodes of a network using a two-step process : ( i ) simulate random walks on the graph – each walk generating a sequence of node IDs then ( ii ) pass the walks ( node IDs ) to a language word embedding algorithm , e.g . word2vec [ Mikolov et al. , 2013 ] , as-if each walk is a sentence . This work was extended by node2vec [ Grover and Leskovec , 2016 ] among others . It has been shown by Abu-El-Haija et al . [ 2018 ] that the learning outcome of 2TensorFlow can caluclate matrix-free SVD if one implements a LinearOperator , as such , our code could be re-implemented as a routine that can convert TensorGraph to LinearOperator . 3Practitioners rarely verify that∇θJ = 0 , where J is mean train objective and θ are model parameters . the two-step process of DeepWalk is equivalent , in expectation , to optimizing a single objective4 : min Z= { L , R } ∑ ( i , j ) ∈ [ n ] × [ n ] [ − E q∼Q [ T q ] ◦ log σ ( LR > ) − λ ( 1−A ) ◦ log ( 1− σ ( LR > ) ) ] ij , ( 1 ) where L , R ∈ Rn× z2 are named by word2vec as the input and output embedding matrices , ◦ is Hadamard product , and the log ( . ) and the standard logistic σ ( . ) = ( 1 + exp ( . ) ) −1 are applied element-wise . The objective above is weighted cross-entropy where the ( left ) positive term weighs the dot-product L > i Rj by the ( expected ) number of random walks simulated from i and passing through j , and the ( right ) negative term weighs non-edges ( 1−A ) by scalar λ ∈ R+ . The context distribution Q stems from step ( ii ) of the process . In particular , word2vec accepts hyperparameter context window size C for its stochasatic sampling : when it samples a center token ( node ID ) , it then samples its context tokens that are up-to distance c from the center . The integer c is sampled from a coin flip uniform on the integers [ 1 , 2 , . . . , C ] – as detailed by Sec.3.1 of [ Levy et al. , 2015 ] . Therefore , PQ ( q | C ) ∝ C−q+1C . Since q has support on [ C ] , then PQ ( q | C ) = ( 2 ( C+1 ) C ) C−q+1 C . 2.2 Message passing graph networks for ( semi- ) supervised node classification . We are also interested in a class of ( message passing ) graph network models taking the general form : for l = 0 , 1 , . . . L : H ( l+1 ) = σl ( g ( A ) H ( l ) W ( l ) ) ; H ( 0 ) = X ; H = H ( L ) ; ( 2 ) where L is the number of layers , W ( l ) ’ s are trainable parameters , σl ’ s denote element-wise activations ( e.g . logistic or ReLu ) , and g is some ( possibly trainable ) transformation of adjacency matrix . GCN [ Kipf and Welling , 2017 ] set g ( A ) = Â , GAT [ Veličković et al. , 2018 ] set g ( A ) = A ◦ MultiHeadedAttention and GIN [ Xu et al. , 2019 ] as g ( A ) = A + ( 1 + ) I with > 0 . For node classification , it is common to set σL = softmax ( applied row-wise ) , specify the size of WL s.t . H ∈ Rn×y where y is number of classes , and optimize cross-entropy objective : . min { Wj } Lj=1 [ −Y ◦ logH− ( 1−Y ) ◦ log ( 1−H ) ] , where Y is a binary matrix with onehot rows indicating node labels . In semi-supervised settings where not all nodes are labeled , before measuring the objective , subset of rows can be kept in Y and H that correspond to labeled nodes . 2.3 Truncated Singular Value Decomposition ( SVD ) . SVD is an algorithm that approximates any matrix M ∈ Rr×c as a product of three matrices : SVDk ( M ) , arg min U , S , V ||M−USV > ||F subject to U > U = V > V = Ik ; S = diag ( s1 , . . . , sk ) . The orthonormal matrices U ∈ Rr×k and V ∈ Rc×k , respectively , are known as the left- and right-singular bases . The values along diagonal matrix S ∈ Rk×k are known as the singular values . Due to theorem of Eckart and Young [ 1936 ] , SVD recovers the best rank-k approximation of input M , as measured by the Frobenius norm ||.||F . Further , if k ≥ rank ( M ) ⇒ ||.||F = 0 . Popular SVD implementations follow Random Matrix Theory algorithm of Halko et al . [ 2009 ] . The prototype algorithm starts with a random matrix and repeatedly multiplies it by M and by M > , interleaving these multiplications with orthonormalization . Our SVD implementation ( in Appendix ) also follows the prototype of [ Halko et al. , 2009 ] , but with two modifications : ( i ) we replace the recommended orthonormalization step from QR decomposition to Cholesky decomposition , giving us significant computational speedups and ( ii ) our implementation accepts symbolic representation of M ( §4 ) , in lieu of its explicit value ( constrast to TensorFlow and PyTorch , requiring explicit M ) . In §3 , we derive linear first-order approximations of models reviewed in §2.1 & §2.2 and explain how SVD can train them . In §5 , we show how they can be used as initializations of non-linear models . 4Derivation is in [ Abu-El-Haija et al. , 2018 ] . Unfortunately , matrix in Eq . 1 is dense with O ( n2 ) nonzeros .

Summary: In this work, authors design a software framework which allows symbolically representing matrices and computing their SVD implicitly. This paper also derives linear first-order approximations of popular Graph Representation Learning (GRL) models, and they can be trained via SVD. Further, SVD initializations can be used to fine-tune deeper neural networks via cross-entropy on many GRL tasks. Experimental results demonstrate that the proposed model trains orders-of-magnitudes faster than state-of-art methods, while having comparable performance.