Matrix factorisation and the interpretation of geodesic distance

1 Introduction . Assume we observe , or are given as the result of a computational procedure , data in the form of a symmetric matrix A 2 Rn⇥n which we relate to unobserved vectors Z1 , . . . , Zn 2 Z ⇢ Rd , where d ⌧ n , by A ij = f ( Zi , Zj ) +E ij , 1 i j n , ( 1 ) for some real-valued symmetric function f which will be called a kernel , and a matrix of unobserved perturbations , E 2 Rn⇥n . As illustrative examples , A could be the observed adjacency matrix of a graph with n vertices , a similarity matrix associated with n items , or some matrix-valued estimator of covariance or correlation between n variates . Broadly stated , our goal is to recover Z1 , . . . , Zn given A , up to identifiability constraints . We seek a method to perform this recovery which can be implemented in practice without any knowledge of f , nor of any explicit statistical model for A , and to which we can plug in various matrix-types , be it adjacency , similarity , covariance , or other . At face value , this may seem rather a lot to ask . Without assumptions , neither f , the Zi ’ s , nor even d are identifiable [ 24 ] . However , under quite general assumptions , we will show that a practical solution is provided by the key combination of two procedures : i ) matrix factorisation of A , such as spectral embedding , followed by ii ) nonlinear dimension reduction , such as Isomap [ 57 ] . This combination is effective because of the following facts , only the first of which is already known . The matrix factorisation step approximates high-dimensional images of the Zi ’ s , living near a ddimensional manifold [ 45 ] . Under regularity and non-degeneracy assumptions on the kernel , which for our analysis is taken to be positive definite , geodesic distance in this manifold equals Euclidean 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . geodesic distance on Z , up to simple transformations of the coordinates , for example scaling . Thus a dimension reduction technique which approximates in-manifold geodesic distances can be expected to recover the Zi ’ s , up to a simple transformation . If those assumptions fail , as they must with real data , we may still find that those approximate geodesic distances are useful because they reflect a geometry implicit in the kernel . There are many works which address recovery of the Zi ’ s when one does consider a particular matrix-type , an explicit statistical model , and/or specific form for f . When A is an adjacency matrix and f ( Zi , Zj ) is the probability of an edge between nodes i and j , the construct ( 1 ) is a latent position model [ 22 ] and various scenarios have been studied : Z is a sphere and f ( x , y ) is a function of hx , yi [ 10 ] ; estimation by graph distance when f ( x , y ) is a function of kx yk [ 11 ] ; spectral estimation when f ( x , y ) is a possibly indefinite inner-product [ 53 , 55 , 12 , 46 ] ; inference under a logistic model via MCMC [ 22 ] or variational approximation [ 48 ] . The case where Z is a discrete set of points corresponds to the very widely studied stochastic block model [ 23 ] , see [ 5 , 33 , 44 , 53 , 8 , 14 ] and references therein ; by contrast the methods we propose are directed at the case where Z is continuous . The opposite problem of estimating f , with Zi unknown but assumed uniform on [ 0 , 1 ] , is known as graphon estimation [ 19 ] . Concerning the case when A is a covariance matrix , Latent Variable Gaussian Process models [ 31 , 30 , 59 ] use f ( Zi , Zj ) to define the population covariance between the ith and jth variates under a hierarchical model . When f ( x , y ) = hx , yi , this reduces to a latent variable optimisation counterpart of Probabilistic PCA [ 58 ] and the maximum likelihood estimate of the Zi ’ s is obtained from the eigendecomposition of the empirical covariance matrix , which in our setting could be A . When f is nonlinear , but fixed e.g . to a Radial Basis Function kernel , gradient and variational techniques are available [ 31 , 30 , 59 ] . See the same references for onward connections to kernel PCA [ 50 ] . Our contribution and precursors . The general practice of using dimensionality reduction and geodesic distance to extract latent features from data is common in data science and machine-learning . Examples arise in genomics [ 38 ] , neuroscience [ 63 ] , speech analysis [ 28 , 21 ] and cyber-security [ 9 ] . What sets our work apart from these contributions is that we establish a new rigorous basis for this practice . It has been understood for several years that the high-dimensional embedding obtained by matrix factorisation can be related via a feature map to the Zi of model ( 1 ) [ 26 , 15 , 34 , 56 , 62 , 32 ] . However , it was only recently observed that the embedding must therefore concentrate about a low dimensional set , in the Hausdorff sense [ 45 ] . Our key mathematical contribution is to describe the topology and geometry of this set , proving it is a topological manifold and establishing how in-manifold geodesic distance is related to geodesic distance in Z . Riemannian geometry underlying kernels was sketched in [ 7 ] but without consideration of geodesic distances or rigorous proofs , and not in the context of latent position estimation . The work [ 13 , 61 ] was an inspiration to us , suggesting Isomap as a tool for analysing spectral embeddings , under a latent structure model in which Z is a one-dimensional curve in Rd and f is the inner product . A key feature of our problem setup is that f is unknown , in which case the manifold is not available in closed form and typically lives in an infinite-dimensional space . To our knowledge , we are the first to show why spectral embedding followed by Isomap might recover the true Zi ’ s , or a useful transformation thereof , in general . We complement our mathematical results with experimental evidence , obtained from both simulated and real data , suggesting that alternative combinations of matrix-factorisation and dimensionreduction techniques work too . For the former , we consider the popular node2vec [ 20 ] algorithm , a likelihood-based approach for graphs said to perform matrix factorisation implicitly [ 43 , 64 ] . For the latter , we consider the popular t-SNE [ 37 ] and UMAP [ 39 ] algorithms . As predicted by the theory , a direct low-dimensional matrix factorisation , whether using spectral embedding or node2vec , is less successful . 2 Proposed methods and their rationale . 2.1 Spectral embedding , as estimating ( Zi ) In our mathematical setup ( precise details come later ) the kernel f will be nonnegative definite and will be the associated Mercer feature map . It is well known that each point ( Zi ) then lives in an infinite-dimensional Hilbert space , which will be denoted ` 2 , and the inner-product h ( Zi ) , ( Zj ) i2 in this space equals f ( Zi , Zj ) . The spectral embedding procedure . For p n , we define the p-dimensional spectral embedding of A to be X̂ = [ X̂1 , . . . , X̂n ] > = Û|Ŝ|1/2 2 Rn⇥p , where |Ŝ| 2 Rp⇥p is a diagonal matrix containing the absolute values of the p largest eigenvalues of A , by magnitude , and Û 2 Rn⇥p is a matrix containing corresponding orthonormal eigenvectors , in the same order . The R packages irlba and RSpectra provide fast solutions which can exploit sparse inputs . One should think of X̂i as approximating the vector of first p components of ( Zi ) , denoted p ( Zi ) , up to orthogonal transformation , and this can be formalised to a greater or lesser extent depending on what assumptions are made . There are several situations , e.g . f any polynomial [ 45 ] , the cosine kernel used in Section 4.1 , the degree-corrected [ 29 ] or mixed-membership [ 6 ] stochastic block model , in which only the first p0 ( say ) components of ( · ) are nonzero , where typically p0 d. If , after n reaches p0 , we embed into p = p0 dimensions , then with k · k denoting the Euclidean norm , we have [ 18 ] : max i2 { 1 , ... , n } kQX̂i p ( Zi ) k = OP ⇣ ( logn ) c n1/2 ⌘ , ( 2 ) for a universal constant c 1 , orthogonal matrix Q , under regularity assumptions on the Zi ’ s , f and E ( that Zi are i.i.d. , f ( Zi , Zj ) has finite expectation , and the perturbations Eij are independent and centered with exponential tails ) . This encompasses the case where A is binary , for example a graph adjacency matrix [ 36 , 46 ] . Similar results are available in the cases where A is a Laplacian [ 46 , 40 ] or covariance matrix [ 17 ] . The methods of this paper are based , in practice , on the distances kX̂i X̂jk , which are invariant to orthogonal transformation and so for the purposes of validating kX̂i X̂jk ⇡ k p ( Zi ) p ( Zj ) k the presence of Q in ( 2 ) is immaterial . For X̂i to converge to ( Zi ) more generally , we must let its dimension p grow with n and , at least given the present state of literature , accept weaker consistency results , for example , convergence in Wasserstein distance between QX̂1 , . . .QX̂n and p ( Z1 ) , . . . , p ( Zn ) [ 32 ] . Uniform consistency results , in the style of ( 2 ) , are also available for indefinite [ 46 ] , bipartite , and directed graphs [ 25 ] . These are left for future work because of the complications of Q no longer being orthogonal . Rank selection . In real data , where n is typically fixed , there is no ‘ best ’ way of selecting p , as discussed for example in [ 42 ] . The method of [ 65 ] , based on profile-likelihood , provides a popular , practical choice , taking as input the spectrum of A , and is implemented in the R package igraph . 2.2 Isomap , as estimating Zi . We propose to recover Z1 , . . . , Zn through the following procedure . Algorithm 1 Isomap procedure input p-dimensional points X̂1 , . . . , X̂n 1 : Compute the neighbourhood graph of radius ✏ : a weighted graph connecting i and j , with weight kX̂i X̂jk , if kX̂i X̂jk ✏ 2 : Compute the matrix of shortest paths on the neighbourhood graph , D̂M 2 Rn⇥n 3 : Apply classical multidimensional scaling ( CMDS ) to D̂M into Rd return d-dimensional points Ẑ1 , . . . , Ẑn Theorem 1 below establishes , under general assumptions , that M : = ( Z ) is a topological manifold of Hausdorff dimension exactly that of Z ( as opposed to an upper bound [ 45 ] ) . It also proposes a ‘ change of metric ’ , often trivial , under which a path on Z and its image on M have the same length . This result explains how Ẑ1 , . . . , Ẑn can estimate Z1 , . . . , Zn . First , think of a path from X̂i to X̂j on the neighbourhood graph as a noisy , discrete version of some corresponding continuous path from ( Zi ) to ( Zj ) on M. The length of the first ( the sum of the weights of the edges ) is approximately equal to the length of the second if measured in the standard metric : an infinitesmal step from x to x + dx has length hdx , dxi1/22 . By inversion of we can trace a third path , taking us from Zi to Zj on Z . But to make its length agree with the first two , we must pick a non-standard metric : an infinitesmal step from z to z + dz must be regarded to have length hdz , Hzdzi1/2 = p dz > Hzdz , where H ij z : = @ 2 f @ x ( i ) @ y ( j ) ( z , z ) , i , j 2 { 1 , . . . , d } . The exciting news for practical purposes is that , through elementary calculus , we might establish that Hz is constant in z ( e.g . if f is translation-invariant ) or even proportional to the identity ( e.g . if f is just a function of Euclidean distance ) . In the latter case , if Z is convex , it is not too difficult to see that the length of the shortest path , from ( Zi ) to ( Zj ) on M , must be proportional to the Euclidean distance between Zi and Zj . Thus , the matrix of shortest paths obtained by Isomap ( Step 2 ) approximates the matrix of Euclidean distances between the Zi ( up to scaling ) , from which the Zi themselves can be recovered ( Step 3 ) , up to scaling , rotation , and translation . Of course , we have taken a few liberties in this argument , such as assuming to be invertible and dz > Hzdz to be positive . We will address these rigorously in the next section . Dimension and radius selection . For visualisation in our examples we will pick d = 1 or d = 2 . For other applications , d can be estimated as the approximate rank of D̂M after double-centering [ 60 ] , e.g . via [ 65 ] , again . In practice , we suggest picking ✏ just large enough for the neighbourhood graph to be connected or , if the data have outliers , a fixed quantile ( such as 5 % ) of D̂M , removing nodes outside the largest connected component . The same recommendations apply if the k-nearest neighbour graph is used instead of the ✏-neighbourhood graph .

Summary: This paper considers the problem of sensing a notion of the true distance between nodes of a  network. This allows to estimate the true position of these nodes up to pairwise scaling. The  cradle of this approach is essentially organized into two steps: a) compute a matrix factorization  of the matrix associated with the graph, i.e., spectral embedding, and b) perform a nonlinear  dimensionality reduction step. The main observation, subject to necessary conditions, is that  Euclidean distances of the spectral embeddings agree with geodesic distances between the  graph nodes. Numerical experiments provide evidence that the proposed approach works.