How can classical multidimensional scaling go wrong?

1 Introduction . Multidimensional scaling ( MDS ) refers to a class of techniques for embedding data into Euclidean space given pairwise dissimilarities [ Carroll and Arabie , 1998 , Borg and Groenen , 2005 ] . Apart from the general usefulness of dimensionality reduction , MDS has been used in a wide variety of applications including data visualization , data preprocessing , network analysis , bioinformatics , and data exploration . Due to its long history and being well studied , MDS has many variations such as non-metric MDS [ Shepard , 1962a , b ] , multi-way MDS [ Kroonenberg , 2008 ] , multi-view MDS [ Bai et al. , 2017 ] , confirmatory or constrained MDS [ Heiser and Meulman , 1983 ] , etc . ( See France and Carroll [ 2010 ] , Cox and Cox [ 2008 ] for surveys ) . The basic MDS formulation involves minimizing an objective function over a space of embeddings . There are two main objective functions associated with MDS : STRESS and STRAIN . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The STRAIN objective ( Equation 1 below ) was introduced by Torgerson [ 1952 ] , whose algorithm to solve for this objective is now commonly referred to as the classical MDS algorithm ( cMDS ) . Xcmds : = argmin X2Rr⇥n X T X ✓ V DV 2 ◆ . ( 1 ) Here V is the centering matrix given by V : = I 1nJ , and I is the identity matrix and J is the matrix of all ones . cMDS first centers the squares of the given distance matrix and then uses its spectral decomposition to extract the low dimensional embedding . cMDS is one of the oldest and most popular methods for MDS , and its popularity is in part due to the fact that this decomposition is fast and can scale to large matrices . The point set produced by cMDS , however , is not necessarily the point set whose Euclidean distance matrix is closest under say Frobenius norm to the input dissimilarity matrix . This type of objective is instead captured by STRESS , which comes in a variety of related forms . In particular , in this paper we consider the SSTRESS objective ( see Equation 2 ) . Specifically , given an embedding X , let EDM ( X ) be the corresponding Euclidean distance matrix , that is EDM ( X ) ij = kXi Xjk2F , where Xi , Xj are the ith and jth columns of X . If D is a dissimilarity matrix whose entries are squared , then we are interested in the matrix , Dt : = argmin D0=EDM ( X ) , X2Rr⇥n kD0 Dk2F . ( 2 ) Note that the reason we assume our dissimilarity matrix has squared entries is because the standard EDM characterizations uses squared entries ( see further discussion of EDMs below ) . Equation 2 is a well studied objective [ Takane et al. , 1977 , Hayden and Wells , 1988 , Qi and Yuan , 2014 ] . There are a number of similarly defined objectives . If one considers this objective when the matrix entries are not squared ( i.e . q D0ij p Dij ) , then it is referred to as STRESS . If one further normalizes each entry of the matrix difference by the input distance value ( i.e . ( q D0ij p Dij ) / p Dij ) then it is called Sammon Stress . In this paper , we are less concerned with the differences between different types of Stress , and instead focus on how the cMDS solution behaves generally under a Stress type objective . Thus for simplicity we focus on SSTRESS . It is important to note that there are algorithms to solve the SSTRESS objective , but the main drawback is that they are slow in comparison to cMDS [ Takane et al. , 1977 , Hayden and Wells , 1988 , Qi and Yuan , 2014 ] . Thus , many practitioners default to using cMDS and do not optimize for SSTRESS . In this paper , we shed light on the theoretical and practical differences between optimizing for these two objectives . Let Dcmds : = EDM ( Xcmds ) , where Xcmds is the solution to Equation 1 , and let Dt be the solution to Equation 2 . We are interested in understanding the quantity err : = kD Dcmdsk2F . ( 3 ) Doing so will provide practitioners with multiple advantages and will guide the development of better algorithms . In particular , 1 . Understanding err is the first step in rigorously quantifying the robustness of the cMDS algorithm . 2 . If err is guaranteed to be small , then we can use the cMDS algorithm without having to worry about loss in quality of the solution . 3 . If err is big , we can make an informed decision about the benefits of the speed of the cMDS algorithm versus the quality of the solution . 4 . Understanding when err is big helps us design algorithms to approximate Dt that perform better when cMDS fails . Contributions . Our main theorem , Theorem 1 , decomposes err into three components . This decomposition gives insight into when and why cMDS can fail with respect to the SSTRESS objective . In particular , for Euclidean inputs , err naturally decreases as the embedding dimension increases . For non-Euclidean inputs , however , our decomposition shows that after an initial decrease , counterintuitively err can actually increase as the embedding dimension increases . In practice one may not know a priori what dimension to embed into , though one might assume it suffices to embed into some sufficiently large dimension . Importantly , these results demonstrate that when using cMDS to embed , choosing a dimension too large can actually increase error . This degradation of the cMDS solution is of particular concern in relation to the robustness in the presence of noisy or missing data , as may often be the case for real world data . Several authors [ Cayton and Dasgupta , 2006 , Mandanas and Kotropoulos , 2016 , Forero and Giannakis , 2012 ] have proposed variations to specifically address robustness with cMDS . However , our decomposition of err , suggests a novel approach . Specifically , by attempting to directly correct for the problematic term in our decomposition ( which resulted in err increasing with dimension ) we produce a new lower bound solution . We show empirically that this lower bound corrects for err increasing , both by itself and when used as a seed for cMDS . Crucially the running time of our new approach is comparable to cMDS , rather than the prohibitively expensive optimal SSTRES solution . Finally , and perhaps more importantly , we show that if we add noise or missing entries to real world data sets , then our new solution outperforms cMDS in terms of the downstream task of classification accuracy , under various classifiers . Moreover , our decomposition can be used to quickly predict the dimension where the increase in err might occur . The main contributions of our paper are as follows . 1 . We decompose the error in Equation 3 into three terms that depend on the eigenvalues of a matrix obtained from D. Using this analysis , we show that there is a term that tells us that as we increase the dimension that we embed into , eventually , the error starts increasing . 2 . We verify , using classification as a downstream task , that this increase in the error for cMDS results in the degradation of the quality of the embedding , as demonstrated by the classification accuracy decreasing . 3 . Using this analysis , we provide an efficiently computable algorithm that returns a matrix Dl such that if Dt is the solution to Equation 2 , then kDl DkF kDt DkF , and empirically we see that kDl DtkF kDcmds DtkF . 4 . While Dl is not metric , when given as input to cMDS instead of D , it results in solutions that are empirically better than the cMDS solution . In particular , this modified procedure results in a more natural decreasing of the error as the dimension increases and has better classification accuracy . 2 Preliminaries and Background . In this section , we lay out the preliminary definitions and necessary key structural characterizations of Euclidean distance matrices . cMDS algorithm . For completeness , we include the classical multidimensional scaling algorithm in Algorithm 1 . Algorithm 1 Classical Multidimensional Scaling . 1 : function CMDS ( D , r ) 2 : X = V ⇤D ⇤ V/2 3 : Compute µ1 . . . µr > 0 , U as the eigenvalues and eigenvectors of X 4 : return U ⇤ diag ( pµ1 , . . . , p µr , 0 , . . . , 0 ) EDM Matrices Definition 1 . D 2 Rn⇥n is a Euclidean Distance Matrix ( EDM ) if and only if there exists a d 2 N such that there are points x1 , . . . , xn 2 Rd with Dij = kxi xjk22 . Note that unlike other distance matrices , an EDM consists of the squares of the ( Euclidean ) distances between the points . This form permits several important structural characterizations of the cone of EDM matrices . 1 . Gower [ 1985 ] , Schoenberg [ 1935 ] show that a symmetric matrix D is an EDM if only if F : = ( I 1sT ) D ( I s1T ) is a positive semi-definite matrix for all s such that 1T s = 1 and Ds 6= 0 . 2 . Schoenberg [ 1938 ] showed that D is an EDM if and only if exp ( D ) is a PSD matrix with 1s along the diagonal for all > 0 . Note here exp is element wise exponentiation of the matrix . 3 . Another characterization is given by Hayden and Wells [ 1988 ] in which D is an EDM if and only if D is symmetric , has 0s on the diagonal ( i.e. , the matrix is hollow ) , and D̂ is negative semi-definite , where D̂ is defined as follows QDQ = D̂ f f T ⇠ . ( 4 ) Here f is a vector and Q = I 2 vT v vv T for v = [ 1 , . . . , 1 , 1 + p n ] T . ( 5 ) Note Q is Householder reflector matrix ( in particular , it ’ s unitary ) and it reflects a vector about the hyperplane span ( v ) ? . The characterization from Hayden and Wells [ 1988 ] is the main characterization that we shall use . Hence we establish some important notation . Definition 2 . Given any symmetric matrix A 2 Rn⇥n , let us define Â 2 Rn 1⇥n 1 , f ( A ) 2 Rn 1 , and ⇠ ( A ) 2 R as follows QAQ = Â f ( A ) f ( A ) T ⇠ ( A ) . In addition to characterizations of the EDM cone , we are also interested in the dimension of the EDM . Definition 3 . Given an EDM D , the dimensionality of D is the smallest dimension d , such that there exist points x1 , . . . , xn 2 Rd with Dij = kxi xjk22 . Let E ( r ) be the set of EDM matrices whose dimensionality is at most r. Using Hayden and Wells [ 1988 ] ’ s characterization of EDMs , Qi and Yuan [ 2014 ] show D 2 E ( r ) if and only if D is symmetric , hollow ( i.e. , 0s along the main diagonal ) , and D̂ in Equation 4 is negative semi-definite with rank at most r. Conjugation matrices : Q and V . Conjugating distance matrices either by Q ( in Equation 5 ) or by the centering matrix V is an important component of understanding both EDMs and the cMDS algorithm . We observe that V is also ( essentially ) a Householder matrix like Q . Let w = [ 1 , . . . , 1 ] T and observe that J = wwT so that V = I 1wTwww T . Qi and Yuan [ 2014 ] establishes an important connection between Q and V that we make use of in our analysis in Section 3 . Specifically , for any symmetric matrix A , we have that V AV = Q Â 0 0 0 Q . ( 6 ) Here Â is the matrix given by Definition 2 . This connection gives a new framework in which we can understand the cMDS algorithm . We know that given D , cMDS first computes V DV/2 . Using Equation 6 , this is equal to Q D̂ 0 0 0 Q/2 . Thus , when cMDS computes the spectral decomposition of V DV/2 , this is equivalent of computing the spectral decomposition of D̂ . Then using the characterization of E ( r ) from Qi and Yuan [ 2014 ] , setting all but the largest r eigenvalues to 0 might seem like the optimal solution . However , this procedure ignores condition that the matrix must be hollow . As we shall show , this results in sub-optimal solutions .

Summary: A new decomposition of the classical multidimensional scaling (cMDS) error term is generated and used to get additional insights of the method.  As I first came across the draft, I naively thought; “Well, cMDS is extremely well known, how can one innovate on such topic?”. Moreover, the departing justification stating cMDS does not excel in non-linear mappings wasn’t that great; I mean cMDS ASSUMES Euclidean space and linearity so it was no wonder that used out of its natural scope it may be suboptimal. But I am very glad to say that the authors proved my naivety wrong very quickly! There is a very clear contribution, the paper is strongly supported, and reading enjoyable.  STRENGTHS + The unusual “late comer”. Innovating in a method which is so well known is difficult.  WEAKNESSES + There are some unclear decisions and justifications but it is somewhat contradictory that I’m criticizing this when, I in the place of the authors, I would very likely have proceed equally… Notwithstanding, I’m indicating this below, in case a second thought can be given + There is an unconventional mixture of theoretical claims with experimental testing whereby some “theoretical proofs” are complemented with heuristics verified experimentally, but actually not theoretically proven.