Matrix factorisation and the interpretation of geodesic distance

1 Introduction . Assume we observe , or are given as the result of a computational procedure , data in the form of a symmetric matrix A 2 Rn⇥n which we relate to unobserved vectors Z1 , . . . , Zn 2 Z ⇢ Rd , where d ⌧ n , by A ij = f ( Zi , Zj ) +E ij , 1 i j n , ( 1 ) for some real-valued symmetric function f which will be called a kernel , and a matrix of unobserved perturbations , E 2 Rn⇥n . As illustrative examples , A could be the observed adjacency matrix of a graph with n vertices , a similarity matrix associated with n items , or some matrix-valued estimator of covariance or correlation between n variates . Broadly stated , our goal is to recover Z1 , . . . , Zn given A , up to identifiability constraints . We seek a method to perform this recovery which can be implemented in practice without any knowledge of f , nor of any explicit statistical model for A , and to which we can plug in various matrix-types , be it adjacency , similarity , covariance , or other . At face value , this may seem rather a lot to ask . Without assumptions , neither f , the Zi ’ s , nor even d are identifiable [ 24 ] . However , under quite general assumptions , we will show that a practical solution is provided by the key combination of two procedures : i ) matrix factorisation of A , such as spectral embedding , followed by ii ) nonlinear dimension reduction , such as Isomap [ 57 ] . This combination is effective because of the following facts , only the first of which is already known . The matrix factorisation step approximates high-dimensional images of the Zi ’ s , living near a ddimensional manifold [ 45 ] . Under regularity and non-degeneracy assumptions on the kernel , which for our analysis is taken to be positive definite , geodesic distance in this manifold equals Euclidean 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . geodesic distance on Z , up to simple transformations of the coordinates , for example scaling . Thus a dimension reduction technique which approximates in-manifold geodesic distances can be expected to recover the Zi ’ s , up to a simple transformation . If those assumptions fail , as they must with real data , we may still find that those approximate geodesic distances are useful because they reflect a geometry implicit in the kernel . There are many works which address recovery of the Zi ’ s when one does consider a particular matrix-type , an explicit statistical model , and/or specific form for f . When A is an adjacency matrix and f ( Zi , Zj ) is the probability of an edge between nodes i and j , the construct ( 1 ) is a latent position model [ 22 ] and various scenarios have been studied : Z is a sphere and f ( x , y ) is a function of hx , yi [ 10 ] ; estimation by graph distance when f ( x , y ) is a function of kx yk [ 11 ] ; spectral estimation when f ( x , y ) is a possibly indefinite inner-product [ 53 , 55 , 12 , 46 ] ; inference under a logistic model via MCMC [ 22 ] or variational approximation [ 48 ] . The case where Z is a discrete set of points corresponds to the very widely studied stochastic block model [ 23 ] , see [ 5 , 33 , 44 , 53 , 8 , 14 ] and references therein ; by contrast the methods we propose are directed at the case where Z is continuous . The opposite problem of estimating f , with Zi unknown but assumed uniform on [ 0 , 1 ] , is known as graphon estimation [ 19 ] . Concerning the case when A is a covariance matrix , Latent Variable Gaussian Process models [ 31 , 30 , 59 ] use f ( Zi , Zj ) to define the population covariance between the ith and jth variates under a hierarchical model . When f ( x , y ) = hx , yi , this reduces to a latent variable optimisation counterpart of Probabilistic PCA [ 58 ] and the maximum likelihood estimate of the Zi ’ s is obtained from the eigendecomposition of the empirical covariance matrix , which in our setting could be A . When f is nonlinear , but fixed e.g . to a Radial Basis Function kernel , gradient and variational techniques are available [ 31 , 30 , 59 ] . See the same references for onward connections to kernel PCA [ 50 ] . Our contribution and precursors . The general practice of using dimensionality reduction and geodesic distance to extract latent features from data is common in data science and machine-learning . Examples arise in genomics [ 38 ] , neuroscience [ 63 ] , speech analysis [ 28 , 21 ] and cyber-security [ 9 ] . What sets our work apart from these contributions is that we establish a new rigorous basis for this practice . It has been understood for several years that the high-dimensional embedding obtained by matrix factorisation can be related via a feature map to the Zi of model ( 1 ) [ 26 , 15 , 34 , 56 , 62 , 32 ] . However , it was only recently observed that the embedding must therefore concentrate about a low dimensional set , in the Hausdorff sense [ 45 ] . Our key mathematical contribution is to describe the topology and geometry of this set , proving it is a topological manifold and establishing how in-manifold geodesic distance is related to geodesic distance in Z . Riemannian geometry underlying kernels was sketched in [ 7 ] but without consideration of geodesic distances or rigorous proofs , and not in the context of latent position estimation . The work [ 13 , 61 ] was an inspiration to us , suggesting Isomap as a tool for analysing spectral embeddings , under a latent structure model in which Z is a one-dimensional curve in Rd and f is the inner product . A key feature of our problem setup is that f is unknown , in which case the manifold is not available in closed form and typically lives in an infinite-dimensional space . To our knowledge , we are the first to show why spectral embedding followed by Isomap might recover the true Zi ’ s , or a useful transformation thereof , in general . We complement our mathematical results with experimental evidence , obtained from both simulated and real data , suggesting that alternative combinations of matrix-factorisation and dimensionreduction techniques work too . For the former , we consider the popular node2vec [ 20 ] algorithm , a likelihood-based approach for graphs said to perform matrix factorisation implicitly [ 43 , 64 ] . For the latter , we consider the popular t-SNE [ 37 ] and UMAP [ 39 ] algorithms . As predicted by the theory , a direct low-dimensional matrix factorisation , whether using spectral embedding or node2vec , is less successful . 2 Proposed methods and their rationale . 2.1 Spectral embedding , as estimating ( Zi ) In our mathematical setup ( precise details come later ) the kernel f will be nonnegative definite and will be the associated Mercer feature map . It is well known that each point ( Zi ) then lives in an infinite-dimensional Hilbert space , which will be denoted ` 2 , and the inner-product h ( Zi ) , ( Zj ) i2 in this space equals f ( Zi , Zj ) . The spectral embedding procedure . For p n , we define the p-dimensional spectral embedding of A to be X̂ = [ X̂1 , . . . , X̂n ] > = Û|Ŝ|1/2 2 Rn⇥p , where |Ŝ| 2 Rp⇥p is a diagonal matrix containing the absolute values of the p largest eigenvalues of A , by magnitude , and Û 2 Rn⇥p is a matrix containing corresponding orthonormal eigenvectors , in the same order . The R packages irlba and RSpectra provide fast solutions which can exploit sparse inputs . One should think of X̂i as approximating the vector of first p components of ( Zi ) , denoted p ( Zi ) , up to orthogonal transformation , and this can be formalised to a greater or lesser extent depending on what assumptions are made . There are several situations , e.g . f any polynomial [ 45 ] , the cosine kernel used in Section 4.1 , the degree-corrected [ 29 ] or mixed-membership [ 6 ] stochastic block model , in which only the first p0 ( say ) components of ( · ) are nonzero , where typically p0 d. If , after n reaches p0 , we embed into p = p0 dimensions , then with k · k denoting the Euclidean norm , we have [ 18 ] : max i2 { 1 , ... , n } kQX̂i p ( Zi ) k = OP ⇣ ( logn ) c n1/2 ⌘ , ( 2 ) for a universal constant c 1 , orthogonal matrix Q , under regularity assumptions on the Zi ’ s , f and E ( that Zi are i.i.d. , f ( Zi , Zj ) has finite expectation , and the perturbations Eij are independent and centered with exponential tails ) . This encompasses the case where A is binary , for example a graph adjacency matrix [ 36 , 46 ] . Similar results are available in the cases where A is a Laplacian [ 46 , 40 ] or covariance matrix [ 17 ] . The methods of this paper are based , in practice , on the distances kX̂i X̂jk , which are invariant to orthogonal transformation and so for the purposes of validating kX̂i X̂jk ⇡ k p ( Zi ) p ( Zj ) k the presence of Q in ( 2 ) is immaterial . For X̂i to converge to ( Zi ) more generally , we must let its dimension p grow with n and , at least given the present state of literature , accept weaker consistency results , for example , convergence in Wasserstein distance between QX̂1 , . . .QX̂n and p ( Z1 ) , . . . , p ( Zn ) [ 32 ] . Uniform consistency results , in the style of ( 2 ) , are also available for indefinite [ 46 ] , bipartite , and directed graphs [ 25 ] . These are left for future work because of the complications of Q no longer being orthogonal . Rank selection . In real data , where n is typically fixed , there is no ‘ best ’ way of selecting p , as discussed for example in [ 42 ] . The method of [ 65 ] , based on profile-likelihood , provides a popular , practical choice , taking as input the spectrum of A , and is implemented in the R package igraph . 2.2 Isomap , as estimating Zi . We propose to recover Z1 , . . . , Zn through the following procedure . Algorithm 1 Isomap procedure input p-dimensional points X̂1 , . . . , X̂n 1 : Compute the neighbourhood graph of radius ✏ : a weighted graph connecting i and j , with weight kX̂i X̂jk , if kX̂i X̂jk ✏ 2 : Compute the matrix of shortest paths on the neighbourhood graph , D̂M 2 Rn⇥n 3 : Apply classical multidimensional scaling ( CMDS ) to D̂M into Rd return d-dimensional points Ẑ1 , . . . , Ẑn Theorem 1 below establishes , under general assumptions , that M : = ( Z ) is a topological manifold of Hausdorff dimension exactly that of Z ( as opposed to an upper bound [ 45 ] ) . It also proposes a ‘ change of metric ’ , often trivial , under which a path on Z and its image on M have the same length . This result explains how Ẑ1 , . . . , Ẑn can estimate Z1 , . . . , Zn . First , think of a path from X̂i to X̂j on the neighbourhood graph as a noisy , discrete version of some corresponding continuous path from ( Zi ) to ( Zj ) on M. The length of the first ( the sum of the weights of the edges ) is approximately equal to the length of the second if measured in the standard metric : an infinitesmal step from x to x + dx has length hdx , dxi1/22 . By inversion of we can trace a third path , taking us from Zi to Zj on Z . But to make its length agree with the first two , we must pick a non-standard metric : an infinitesmal step from z to z + dz must be regarded to have length hdz , Hzdzi1/2 = p dz > Hzdz , where H ij z : = @ 2 f @ x ( i ) @ y ( j ) ( z , z ) , i , j 2 { 1 , . . . , d } . The exciting news for practical purposes is that , through elementary calculus , we might establish that Hz is constant in z ( e.g . if f is translation-invariant ) or even proportional to the identity ( e.g . if f is just a function of Euclidean distance ) . In the latter case , if Z is convex , it is not too difficult to see that the length of the shortest path , from ( Zi ) to ( Zj ) on M , must be proportional to the Euclidean distance between Zi and Zj . Thus , the matrix of shortest paths obtained by Isomap ( Step 2 ) approximates the matrix of Euclidean distances between the Zi ( up to scaling ) , from which the Zi themselves can be recovered ( Step 3 ) , up to scaling , rotation , and translation . Of course , we have taken a few liberties in this argument , such as assuming to be invertible and dz > Hzdz to be positive . We will address these rigorously in the next section . Dimension and radius selection . For visualisation in our examples we will pick d = 1 or d = 2 . For other applications , d can be estimated as the approximate rank of D̂M after double-centering [ 60 ] , e.g . via [ 65 ] , again . In practice , we suggest picking ✏ just large enough for the neighbourhood graph to be connected or , if the data have outliers , a fixed quantile ( such as 5 % ) of D̂M , removing nodes outside the largest connected component . The same recommendations apply if the k-nearest neighbour graph is used instead of the ✏-neighbourhood graph .

Summary: This paper considers the problem of finding the latent variables $Z$ given noisy observations of $f(Z_i, Z_j)$ with unknown $f$, and suggests applying matrix factorization (e.g. spectral embedding) + nonlinear dimension reduction (e.g. isomap). It shows mathematically that under some assumptions on the kernel $f$, finding geodesic distance in the manifold (which the matrix factorization step approximates) is equivalent to finding the geodesic distance in the original latent space. Experiments on simulated and real-world data are provided.