How can classical multidimensional scaling go wrong?

1 Introduction . Multidimensional scaling ( MDS ) refers to a class of techniques for embedding data into Euclidean space given pairwise dissimilarities [ Carroll and Arabie , 1998 , Borg and Groenen , 2005 ] . Apart from the general usefulness of dimensionality reduction , MDS has been used in a wide variety of applications including data visualization , data preprocessing , network analysis , bioinformatics , and data exploration . Due to its long history and being well studied , MDS has many variations such as non-metric MDS [ Shepard , 1962a , b ] , multi-way MDS [ Kroonenberg , 2008 ] , multi-view MDS [ Bai et al. , 2017 ] , confirmatory or constrained MDS [ Heiser and Meulman , 1983 ] , etc . ( See France and Carroll [ 2010 ] , Cox and Cox [ 2008 ] for surveys ) . The basic MDS formulation involves minimizing an objective function over a space of embeddings . There are two main objective functions associated with MDS : STRESS and STRAIN . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The STRAIN objective ( Equation 1 below ) was introduced by Torgerson [ 1952 ] , whose algorithm to solve for this objective is now commonly referred to as the classical MDS algorithm ( cMDS ) . Xcmds : = argmin X2Rr⇥n X T X ✓ V DV 2 ◆ . ( 1 ) Here V is the centering matrix given by V : = I 1nJ , and I is the identity matrix and J is the matrix of all ones . cMDS first centers the squares of the given distance matrix and then uses its spectral decomposition to extract the low dimensional embedding . cMDS is one of the oldest and most popular methods for MDS , and its popularity is in part due to the fact that this decomposition is fast and can scale to large matrices . The point set produced by cMDS , however , is not necessarily the point set whose Euclidean distance matrix is closest under say Frobenius norm to the input dissimilarity matrix . This type of objective is instead captured by STRESS , which comes in a variety of related forms . In particular , in this paper we consider the SSTRESS objective ( see Equation 2 ) . Specifically , given an embedding X , let EDM ( X ) be the corresponding Euclidean distance matrix , that is EDM ( X ) ij = kXi Xjk2F , where Xi , Xj are the ith and jth columns of X . If D is a dissimilarity matrix whose entries are squared , then we are interested in the matrix , Dt : = argmin D0=EDM ( X ) , X2Rr⇥n kD0 Dk2F . ( 2 ) Note that the reason we assume our dissimilarity matrix has squared entries is because the standard EDM characterizations uses squared entries ( see further discussion of EDMs below ) . Equation 2 is a well studied objective [ Takane et al. , 1977 , Hayden and Wells , 1988 , Qi and Yuan , 2014 ] . There are a number of similarly defined objectives . If one considers this objective when the matrix entries are not squared ( i.e . q D0ij p Dij ) , then it is referred to as STRESS . If one further normalizes each entry of the matrix difference by the input distance value ( i.e . ( q D0ij p Dij ) / p Dij ) then it is called Sammon Stress . In this paper , we are less concerned with the differences between different types of Stress , and instead focus on how the cMDS solution behaves generally under a Stress type objective . Thus for simplicity we focus on SSTRESS . It is important to note that there are algorithms to solve the SSTRESS objective , but the main drawback is that they are slow in comparison to cMDS [ Takane et al. , 1977 , Hayden and Wells , 1988 , Qi and Yuan , 2014 ] . Thus , many practitioners default to using cMDS and do not optimize for SSTRESS . In this paper , we shed light on the theoretical and practical differences between optimizing for these two objectives . Let Dcmds : = EDM ( Xcmds ) , where Xcmds is the solution to Equation 1 , and let Dt be the solution to Equation 2 . We are interested in understanding the quantity err : = kD Dcmdsk2F . ( 3 ) Doing so will provide practitioners with multiple advantages and will guide the development of better algorithms . In particular , 1 . Understanding err is the first step in rigorously quantifying the robustness of the cMDS algorithm . 2 . If err is guaranteed to be small , then we can use the cMDS algorithm without having to worry about loss in quality of the solution . 3 . If err is big , we can make an informed decision about the benefits of the speed of the cMDS algorithm versus the quality of the solution . 4 . Understanding when err is big helps us design algorithms to approximate Dt that perform better when cMDS fails . Contributions . Our main theorem , Theorem 1 , decomposes err into three components . This decomposition gives insight into when and why cMDS can fail with respect to the SSTRESS objective . In particular , for Euclidean inputs , err naturally decreases as the embedding dimension increases . For non-Euclidean inputs , however , our decomposition shows that after an initial decrease , counterintuitively err can actually increase as the embedding dimension increases . In practice one may not know a priori what dimension to embed into , though one might assume it suffices to embed into some sufficiently large dimension . Importantly , these results demonstrate that when using cMDS to embed , choosing a dimension too large can actually increase error . This degradation of the cMDS solution is of particular concern in relation to the robustness in the presence of noisy or missing data , as may often be the case for real world data . Several authors [ Cayton and Dasgupta , 2006 , Mandanas and Kotropoulos , 2016 , Forero and Giannakis , 2012 ] have proposed variations to specifically address robustness with cMDS . However , our decomposition of err , suggests a novel approach . Specifically , by attempting to directly correct for the problematic term in our decomposition ( which resulted in err increasing with dimension ) we produce a new lower bound solution . We show empirically that this lower bound corrects for err increasing , both by itself and when used as a seed for cMDS . Crucially the running time of our new approach is comparable to cMDS , rather than the prohibitively expensive optimal SSTRES solution . Finally , and perhaps more importantly , we show that if we add noise or missing entries to real world data sets , then our new solution outperforms cMDS in terms of the downstream task of classification accuracy , under various classifiers . Moreover , our decomposition can be used to quickly predict the dimension where the increase in err might occur . The main contributions of our paper are as follows . 1 . We decompose the error in Equation 3 into three terms that depend on the eigenvalues of a matrix obtained from D. Using this analysis , we show that there is a term that tells us that as we increase the dimension that we embed into , eventually , the error starts increasing . 2 . We verify , using classification as a downstream task , that this increase in the error for cMDS results in the degradation of the quality of the embedding , as demonstrated by the classification accuracy decreasing . 3 . Using this analysis , we provide an efficiently computable algorithm that returns a matrix Dl such that if Dt is the solution to Equation 2 , then kDl DkF kDt DkF , and empirically we see that kDl DtkF kDcmds DtkF . 4 . While Dl is not metric , when given as input to cMDS instead of D , it results in solutions that are empirically better than the cMDS solution . In particular , this modified procedure results in a more natural decreasing of the error as the dimension increases and has better classification accuracy . 2 Preliminaries and Background . In this section , we lay out the preliminary definitions and necessary key structural characterizations of Euclidean distance matrices . cMDS algorithm . For completeness , we include the classical multidimensional scaling algorithm in Algorithm 1 . Algorithm 1 Classical Multidimensional Scaling . 1 : function CMDS ( D , r ) 2 : X = V ⇤D ⇤ V/2 3 : Compute µ1 . . . µr > 0 , U as the eigenvalues and eigenvectors of X 4 : return U ⇤ diag ( pµ1 , . . . , p µr , 0 , . . . , 0 ) EDM Matrices Definition 1 . D 2 Rn⇥n is a Euclidean Distance Matrix ( EDM ) if and only if there exists a d 2 N such that there are points x1 , . . . , xn 2 Rd with Dij = kxi xjk22 . Note that unlike other distance matrices , an EDM consists of the squares of the ( Euclidean ) distances between the points . This form permits several important structural characterizations of the cone of EDM matrices . 1 . Gower [ 1985 ] , Schoenberg [ 1935 ] show that a symmetric matrix D is an EDM if only if F : = ( I 1sT ) D ( I s1T ) is a positive semi-definite matrix for all s such that 1T s = 1 and Ds 6= 0 . 2 . Schoenberg [ 1938 ] showed that D is an EDM if and only if exp ( D ) is a PSD matrix with 1s along the diagonal for all > 0 . Note here exp is element wise exponentiation of the matrix . 3 . Another characterization is given by Hayden and Wells [ 1988 ] in which D is an EDM if and only if D is symmetric , has 0s on the diagonal ( i.e. , the matrix is hollow ) , and D̂ is negative semi-definite , where D̂ is defined as follows QDQ = D̂ f f T ⇠ . ( 4 ) Here f is a vector and Q = I 2 vT v vv T for v = [ 1 , . . . , 1 , 1 + p n ] T . ( 5 ) Note Q is Householder reflector matrix ( in particular , it ’ s unitary ) and it reflects a vector about the hyperplane span ( v ) ? . The characterization from Hayden and Wells [ 1988 ] is the main characterization that we shall use . Hence we establish some important notation . Definition 2 . Given any symmetric matrix A 2 Rn⇥n , let us define Â 2 Rn 1⇥n 1 , f ( A ) 2 Rn 1 , and ⇠ ( A ) 2 R as follows QAQ = Â f ( A ) f ( A ) T ⇠ ( A ) . In addition to characterizations of the EDM cone , we are also interested in the dimension of the EDM . Definition 3 . Given an EDM D , the dimensionality of D is the smallest dimension d , such that there exist points x1 , . . . , xn 2 Rd with Dij = kxi xjk22 . Let E ( r ) be the set of EDM matrices whose dimensionality is at most r. Using Hayden and Wells [ 1988 ] ’ s characterization of EDMs , Qi and Yuan [ 2014 ] show D 2 E ( r ) if and only if D is symmetric , hollow ( i.e. , 0s along the main diagonal ) , and D̂ in Equation 4 is negative semi-definite with rank at most r. Conjugation matrices : Q and V . Conjugating distance matrices either by Q ( in Equation 5 ) or by the centering matrix V is an important component of understanding both EDMs and the cMDS algorithm . We observe that V is also ( essentially ) a Householder matrix like Q . Let w = [ 1 , . . . , 1 ] T and observe that J = wwT so that V = I 1wTwww T . Qi and Yuan [ 2014 ] establishes an important connection between Q and V that we make use of in our analysis in Section 3 . Specifically , for any symmetric matrix A , we have that V AV = Q Â 0 0 0 Q . ( 6 ) Here Â is the matrix given by Definition 2 . This connection gives a new framework in which we can understand the cMDS algorithm . We know that given D , cMDS first computes V DV/2 . Using Equation 6 , this is equal to Q D̂ 0 0 0 Q/2 . Thus , when cMDS computes the spectral decomposition of V DV/2 , this is equivalent of computing the spectral decomposition of D̂ . Then using the characterization of E ( r ) from Qi and Yuan [ 2014 ] , setting all but the largest r eigenvalues to 0 might seem like the optimal solution . However , this procedure ignores condition that the matrix must be hollow . As we shall show , this results in sub-optimal solutions .

Summary: The paper considers the classic embedding method known as multidimensional scaling and focus on the so called STRAIN  and SSTRESS objectives, both designed for embedding a given squared distance matrix into a lower dimensional space, often for visualization, but potentially also for reducing problem size for downstream tasks.   The idea is that the SSTRESS is the better objective to optimize as it captures how well the distances of the embedded data points approximate the input distances, but computationally hard, so maybe it is enough to just use the solution for the simpler STRAIN objective, that boils down to Singular Value Decomposition and that may work fine.  The paper analyzes the solution of STRAIN in terms of the more relevant SSTRESS objective, and shows a decomposition of the error of the STRAIN solution as measured with the STRESS objective and analyzes this objective to understand what happens. The punchline is that if the input distance matrix is not Euclidian (no data set with these pairwise distances exist in any dimension), then the SSTRESS error of the embedding achieved by optimizing STRAIN can  actually increase with increased embedding dimension. A property that seems counterintuitive as a higher dimension should allow improved embeddings. The paper shows  experimentally that this phenomena actually occurs in practice as well and that the performance is nicely explained by the error decomposition proved and discussed in the paper. In particular the experiments show that the classic Isomap embedding has this issue.  Furthermore, if these STRAIN embeddings are used for a simple downstream task like standard classification, the results become increasingly worse as the embedding size is increased.