Implicit SVD for Graph Representation Learning

1 Introduction . Truncated Singular Value Decomposition ( SVD ) provides solutions to a variety of mathematical problems , including computing a matrix rank , its pseudo-inverse , or mapping its rows and columns onto the orthonormal singular bases for low-rank approximations . Machine Learning ( ML ) software frameworks ( such as TensorFlow ) offer efficient SVD implementations , as SVD can estimate solutions for a variaty of tasks , e.g. , in computer vision [ Turk and Pentland , 1991 ] , weather prediction [ Molteni et al. , 1996 ] , recommendation [ Koren et al. , 2009 ] , language [ Deerwester et al. , 1990 , Levy and Goldberg , 2014 ] , and more-relevantly , graph representation learning ( GRL ) [ Qiu et al. , 2018 ] . SVD ’ s benefits include training models , without calculating gradients , to arrive at globally-unique solutions , optimizing Frobenius-norm objectives ( §2.3 ) , without requiring hyperparameters for the learning process , such as the choice of the learning algorithm , step-size , regularization coefficient , etc . Typically , one constructs a design matrix M , such that , its decomposition provides a solution to a task of interest . Unfortunately , existing popular ML frameworks [ Abadi et al. , 2016 , Paszke et al. , 2019 ] can not calculate the SVD of an arbitrary linear matrix given its computation graph : they ∗Part of this work was done during internship at Intel Labs . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . compute the matrix ( entry-wise ) then2 its decomposition . This limits the scalability of these libraries in several cases of interest , such as in GRL , when explicit calculation of the matrix is prohibitive due to memory constraints . These limitations render SVD as impractical for achieving state-of-the-art ( SOTA ) for tasks at hand . This has been circumvented by Qiu et al . [ 2018 ] by sampling M entrywise , but this produces sub-optimal estimation error and experimentally degrades the empirical test performance ( §7 : Experiments ) . We design a software library that allows symbolic definition of M , via composition of matrix operations , and we implement an SVD algorithm that can decompose M from said symbolic representation , without need to compute M. This is valuable for many GRL tasks , where the design matrix M is too large , e.g. , quadratic in the input size . With our implementation , we show that SVD can perform learning , orders of magnitudes faster than current alternatives . Currently , SOTA GRL models are generally graph neural networks trained to optimize cross-entropy objectives . Their inter-layer non-linearities place their ( many ) parameters onto a non-convex objective surface where convergence is rarely verified3 . Nonetheless , these models can be convexified ( §3 ) and trained via SVD , if we remove nonlinearities between layers and swap the cross-entropy objective with Frobenius norm minimization . Undoubtedly , such linearization incurs a drop of accuracy on empirical test performance . Nonetheless , we show that the ( convexified ) model ’ s parameters learned by SVD can provide initialization to deeper ( non-linear ) models , which then can be fine-tuned on cross-entropy objectives . The non-linear models are endowed with our novel Split-ReLu layer , which has twice as many parameters as a ReLu fully-connected layer , and behaves as a linear layer when its parameters reside on some hyperplane ( §5.2 ) . Training on modest hardware ( e.g. , laptop ) is sufficient for this learning pipeline ( convexify → SVD → fine-tune ) yet it trains much faster than current approaches , that are commonly trained on expensive hardware . We summarize our contributions as : 1 . We open-source a flexible python software library that allows symbolic definition of matrices and computes their SVD without explicitly calculating them . 2 . We linearize popular GRL models , and train them via SVD of design matrices . 3 . We show that fine-tuning a few parameters on-top of the SVD initialization sets state-of-the-art on many GRL tasks while , overall , training orders-of-magnitudes faster . 2 Preliminaries & notation . We denote a graph with n nodes and m edges with an adjacency matrix A ∈ Rn×n and additionally , if nodes have ( d-dimensional ) features , with a feature matrix X ∈ Rn×d . If nodes i , j ∈ [ n ] are connected then Aij is set to their edge weight and otherwise Aij = 0 . Further , denote the ( row-wise normalized ) transition matrix as T = D−1A and denote the symmetrically normalized adjacency with self-connections as Â = ( D + I ) − 1 2 ( A + I ) ( D + I ) − 1 2 where I is identity matrix . We review model classes : ( 1 ) network embedding and ( 2 ) message passing that we define as follows . The first inputs a graph ( A , X ) and outputs node embedding matrix Z ∈ Rn×z with z-dimensions per node . Z is then used for an upstream task , e.g. , link prediction . The second class utilizes a function H : Rn×n×Rn×d → Rn×z where the function H ( A , X ) is usually directly trained on the upstream task , e.g. , node classification . In general , the first class is transductive while the second is inductive . 2.1 Network embedding models based on DeepWalk & node2vec . The seminal work of DeepWalk [ Perozzi et al. , 2014 ] embeds nodes of a network using a two-step process : ( i ) simulate random walks on the graph – each walk generating a sequence of node IDs then ( ii ) pass the walks ( node IDs ) to a language word embedding algorithm , e.g . word2vec [ Mikolov et al. , 2013 ] , as-if each walk is a sentence . This work was extended by node2vec [ Grover and Leskovec , 2016 ] among others . It has been shown by Abu-El-Haija et al . [ 2018 ] that the learning outcome of 2TensorFlow can caluclate matrix-free SVD if one implements a LinearOperator , as such , our code could be re-implemented as a routine that can convert TensorGraph to LinearOperator . 3Practitioners rarely verify that∇θJ = 0 , where J is mean train objective and θ are model parameters . the two-step process of DeepWalk is equivalent , in expectation , to optimizing a single objective4 : min Z= { L , R } ∑ ( i , j ) ∈ [ n ] × [ n ] [ − E q∼Q [ T q ] ◦ log σ ( LR > ) − λ ( 1−A ) ◦ log ( 1− σ ( LR > ) ) ] ij , ( 1 ) where L , R ∈ Rn× z2 are named by word2vec as the input and output embedding matrices , ◦ is Hadamard product , and the log ( . ) and the standard logistic σ ( . ) = ( 1 + exp ( . ) ) −1 are applied element-wise . The objective above is weighted cross-entropy where the ( left ) positive term weighs the dot-product L > i Rj by the ( expected ) number of random walks simulated from i and passing through j , and the ( right ) negative term weighs non-edges ( 1−A ) by scalar λ ∈ R+ . The context distribution Q stems from step ( ii ) of the process . In particular , word2vec accepts hyperparameter context window size C for its stochasatic sampling : when it samples a center token ( node ID ) , it then samples its context tokens that are up-to distance c from the center . The integer c is sampled from a coin flip uniform on the integers [ 1 , 2 , . . . , C ] – as detailed by Sec.3.1 of [ Levy et al. , 2015 ] . Therefore , PQ ( q | C ) ∝ C−q+1C . Since q has support on [ C ] , then PQ ( q | C ) = ( 2 ( C+1 ) C ) C−q+1 C . 2.2 Message passing graph networks for ( semi- ) supervised node classification . We are also interested in a class of ( message passing ) graph network models taking the general form : for l = 0 , 1 , . . . L : H ( l+1 ) = σl ( g ( A ) H ( l ) W ( l ) ) ; H ( 0 ) = X ; H = H ( L ) ; ( 2 ) where L is the number of layers , W ( l ) ’ s are trainable parameters , σl ’ s denote element-wise activations ( e.g . logistic or ReLu ) , and g is some ( possibly trainable ) transformation of adjacency matrix . GCN [ Kipf and Welling , 2017 ] set g ( A ) = Â , GAT [ Veličković et al. , 2018 ] set g ( A ) = A ◦ MultiHeadedAttention and GIN [ Xu et al. , 2019 ] as g ( A ) = A + ( 1 + ) I with > 0 . For node classification , it is common to set σL = softmax ( applied row-wise ) , specify the size of WL s.t . H ∈ Rn×y where y is number of classes , and optimize cross-entropy objective : . min { Wj } Lj=1 [ −Y ◦ logH− ( 1−Y ) ◦ log ( 1−H ) ] , where Y is a binary matrix with onehot rows indicating node labels . In semi-supervised settings where not all nodes are labeled , before measuring the objective , subset of rows can be kept in Y and H that correspond to labeled nodes . 2.3 Truncated Singular Value Decomposition ( SVD ) . SVD is an algorithm that approximates any matrix M ∈ Rr×c as a product of three matrices : SVDk ( M ) , arg min U , S , V ||M−USV > ||F subject to U > U = V > V = Ik ; S = diag ( s1 , . . . , sk ) . The orthonormal matrices U ∈ Rr×k and V ∈ Rc×k , respectively , are known as the left- and right-singular bases . The values along diagonal matrix S ∈ Rk×k are known as the singular values . Due to theorem of Eckart and Young [ 1936 ] , SVD recovers the best rank-k approximation of input M , as measured by the Frobenius norm ||.||F . Further , if k ≥ rank ( M ) ⇒ ||.||F = 0 . Popular SVD implementations follow Random Matrix Theory algorithm of Halko et al . [ 2009 ] . The prototype algorithm starts with a random matrix and repeatedly multiplies it by M and by M > , interleaving these multiplications with orthonormalization . Our SVD implementation ( in Appendix ) also follows the prototype of [ Halko et al. , 2009 ] , but with two modifications : ( i ) we replace the recommended orthonormalization step from QR decomposition to Cholesky decomposition , giving us significant computational speedups and ( ii ) our implementation accepts symbolic representation of M ( §4 ) , in lieu of its explicit value ( constrast to TensorFlow and PyTorch , requiring explicit M ) . In §3 , we derive linear first-order approximations of models reviewed in §2.1 & §2.2 and explain how SVD can train them . In §5 , we show how they can be used as initializations of non-linear models . 4Derivation is in [ Abu-El-Haija et al. , 2018 ] . Unfortunately , matrix in Eq . 1 is dense with O ( n2 ) nonzeros .

Summary: The authors suggest to express large matrices as operations over given manageable leaf matrices. They keep track of the resulting abstract syntax tree (AST) and offer a specialised SVD decomposition implementation that can perform the computation on the final expression by traversing the AST and operating on the individual leaf matrices. They then show how using the SVD operation they can perform learning in various inductive and transductive tasks. The advantage of their proposal is that the process is computationally efficient and they can in practice match SOTA results with speedups reaching several orders of magnitude.