Title: Neuc-MDS: Non-Euclidean Multidimensional Scaling Through Bilinear Forms
Venue: NeurIPS 2024
Accepted: yes
Keywords: multidimensional scaling

dimension reduction

non-Euclidean geometry
Abstract: We introduce \textbf{N}on-\textbf{Euc}lidean-\textbf{MDS} (Neuc-MDS), which extends Multidimensional Scaling (MDS) to generate outputs that can be non-Euclidean and non-metric. The main idea is to generalize the inner product to other symmetric bilinear forms to utilize the negative eigenvalues of dissimiliarity Gram matrices. Neuc-MDS efficiently optimizes the choice of (both positive and negative) eigenvalues of the dissimilarity Gram matrix to reduce STRESS, the sum of squared pairwise error. We provide an in-depth error analysis and proofs of the optimality in minimizing lower bounds of STRESS. We demonstrate Neuc-MDS's ability to address limitations of classical MDS raised by prior research, and test it on various synthetic and real-world datasets in comparison with both linear and non-linear dimension reduction methods.

Neuc-MDS: Non-Euclidean Multidimensional Scaling Through Bilinear Forms

Abstract

We introduce N onEuc lideanMDS (Neuc-MDS), an extension of classical Multidimensional Scaling (MDS) that accommodates non-Euclidean and non-metric inputs. The main idea is to generalize the standard inner product to symmetric bilinear forms to utilize the negative eigenvalues of dissimilarity Gram matrices. Neuc-MDS efficiently optimizes the choice of (both positive and negative) eigenvalues of the dissimilarity Gram matrix to reduce STRESS, the sum of squared pairwise error. We provide an in-depth error analysis and proofs of the optimality in minimizing lower bounds of STRESS. We demonstrate Neuc-MDS's ability to address limitations of classical MDS raised by prior research, and test it on various synthetic and real-world datasets in comparison with both linear and non-linear dimension reduction methods.

1 Introduction

Many datasets in applications adopt dissimilarities that are non-Euclidean and/or nonmetric. Examples of such popular dissimilarity measures [19, 7] include Minkowski distance ( L p ), cosine similarity, Hamming, Jaccard, Mahalanobis, Chebyshev, and KL-divergence. Studies in psychology have long recognized that human perception of similarity is not a metric [42]. Further, dissimilarity matrices with negative entries (e.g.: cosine similarity, correlation, signed distance) have also been widely used in various problems. Negative inner product norms also have deep connections to hyperbolic spaces as well as the study of spacetime in special relativity theory.

In many machine learning practices, embedding in low dimensional vector space is often explicitly or implicitly done within the data processing pipeline. Such embedding may already need to consider more general dissimilarities. For example, one branch of graph learning adopts embedding in non-Euclidean spaces (e.g., hyperbolic spaces [9]), and machine learning for physics model and data (AI4Sicence) needs to consider more general inner product norms [22]. In transformer models [45, 13], the attention mechanism can also be viewed as learning a general bilinear form on tokens. Despite the wide adoption of general dissimilarity measures in practice, theoretical study of embedding and dimension reduction for non-Euclidean non-metric data appears to be still very limited.

In this paper we consider one of the most classical algorithms for data embedding and dimension reduction - multidimensional scaling - and develop a non-Euclidean, non-metric version with theoretical performance guarantee.

Background on MDS. Classical multidimensional scaling (cMDS) takes as input a Euclidean distance matrix (EDM) , i.e., a symmetric matrix D ∈ R n × n where each entry is the squared Euclidean distance between two points in Euclidean space, and recovers the Euclidean coordinates. Using a standard double centering trick, D can be turned into a

Gram matrix B = X T X where X encodes the Euclidean coordinates. For the purpose of producing a low-dimensional vector, classical MDS takes k eigenvectors corresponding to top k largest eigenvalues of the Gram matrix B . This minimizes the strain , the difference (Frobenius norm) in terms of the Gram matrix.

When the input distance matrix is not a Euclidean distance matrix, this problem is called metric MDS [1]. Metric MDS considers the minimization of STRESS [28], defined as the sum of squared difference of pairwise embedding distances to the input dissimilarities. Minimizing STRESS makes the problem to be non-linear and there is no closed-form solution, though one can use either gradient descent or Newton's method [35, 26]. Nevertheless, in practice, cMDS is often applied for non-Euclidean distance matrix. In this case, the centered matrix B is no longer positive semi-definite. The common practice is keep the top positive eigenvalues and throw away the negative eigenvalues.

Two recent papers [41, 38] pointed out that classical MDS produces suboptimal solutions on non-Euclidean distance matrix when considering STRESS. This is not a surprise, since cMDS, minimizing strain, does not minimize STRESS. However, a more problematic issue is that when using more dimensions in cMDS (i.e., increasing k ), the STRESS error first drops and then increases. We call this phenomena Dimensionality Paradox . It is theoretically unsatisfactory and counter-intuitive that embeddings by classical MDS using more dimensions could yield worse results.

The error analysis in [38] sheds light on this issue. When the input matrix is Non-Euclidean, the negative eigenvalues carry crucial information. cMDS, keeping only positive eigenvalues, is intrinsically biased - the more positive eigenvalues used the more it deviates from the input data. A real eradication of this issue must address the root cause, i.e., applying an algorithm meant for Euclidean geometry on non-Euclidean data.

Our Contributions. We extend multidimensional scaling to non-Euclidean geometry, by generalizing the dissimilarity function from the standard inner product (which defines Euclidean geometry) to the broader family of symmetric bilinear forms Φ( u, v ) = u T Av , where the symmetric matrix A does not have to be positive semi-definite. For dimension reduction, we look for both a low dimensional vector representation and an associated bilinear form, that together approximate the input dissimilarity matrix with minimum STRESS error. Specifically, the key contributions are as follows:

❼ We conduct an in-depth analysis on STRESS error for any chosen subset of k < n eigenvalues of the input centered dissimilarity matrix. We propose Neuc-MDS, an efficient algorithm that finds the best subset of eigenvalues to minimize a lower bound of STRESS.

❼ Beyond the constraints of eigenvalue subsets, we extend our findings to the general linear combinations of eigenvalues. Our advanced algorithm, Neuc-MDS + , finds the best linear combination of eigenvalues to minimize the lower bound objective.

❼ We provide theoretical analysis for the asymptotic behavior of cMDS and Neuc-MDS on random symmetric matrices. First, both necessarily produce large STRESS if the target dimension k = o ( n ) - on completely unstructured data aggressive dimension reduction shall not be expected. Further, when k = Θ( n ), the STRESS of Neuc-MDS monotonically decreases to 0 while the STRESS of cMDS increases and eventually reaches a plateau.

❼ Empirically we evaluate Neuc-MDS and Neuc-MDS + on ten diverse datasets encompassing different domains. The experiment results show that both methods substantially outperform previous baselines on STRESS and average distortion, and fully resolve the issue of dimensionality paradox in cMDS. Our codes are available on Github 2 .

2 Related Work

Our work is in the general family of similarity learning [25, 3] with dimension reduction, going beyond metric learning and embedding. Due to the huge amount of literature on this topic we only mention those that are most relevant.

MDS Family and Embedding in Euclidean Spaces. As one of the most useful embedding and dimension reduction techniques in practice, the MDS family has many variants. Non-metric MDS [36, 37] considers a monotonically increasing function f on input dissimilarity and minimizes STRESS between { f ( D ij ) } and embedded squared Euclidean distances. Generalized multidimensional scaling (GMD) considers the target space as an arbitrary smooth surface [6]. In addition, non-linear dimension reduction methods such as Isomap [39], Laplacian Eigenmaps [2], LLE [34], t-SNE [23, 44] consider data points from a non-linear high-dimensional manifold and extract distances defined on the manifold. In all these methods, the points are still embedded in Euclidean spaces. Some of these methods such as Isomap directly apply cMDS as the final step. If the dissimilarity is highly non-Euclidean, we can replace cMDS by Neuc-MDS to get performance improvement.

Dimension Reduction in Non-Euclidean Spaces. There is also prior work that finds embedding of manifold data in non-Euclidean spaces, e.g., on a piece-wise connected manifold [50, 5], on a sphere [11, 15, 48], and in hyperbolic spaces [15, 48]. Very recently, there is study of Johnson-Lindenstrauss style dimension reduction for weighted Euclidean space [30], hyperbolic space [4], as well as PCA [8], dimension-reduction [16], and t-SNE in hyperbolic space [20]. Our dissimilarity function generalizes beyond hyperbolic distances.

3 Dimension Reduction with Bilinear Forms

̸

Let P denote a dataset of size n . Let D ∈ R n × n be the dissimilarity matrix of dataset P , D ij = D ji is a real-valued symmetric dissimilarity measure between pair p i , p j ∈ P , i = j ∈ [ n ], and all diagonal entries D ii = 0 (i.e., D is a hollow matrix). D is the analog of squared Euclidean distance matrix in classical MDS. But here D is not necessarily Euclidean, may not be a metric (e.g. violating triangle inequality) and may have negative entries. Our goal is to obtain (1) a low-dimensional vector representation for each element in P , and (2) a function f that computes a dissimilarity measure using the calculated low dimensional vectors. Since the input dissimilarities are not necessarily Euclidean nor a metric, we look for the function f beyond Euclidean distances, but stay within a broader family of inner products or bilinear forms.

̸

Formally, we have the following problem definition.

Definition 1 (Non-Euclidean Dimension Reduction) . Given a symmetric dissimilarity matrix D of a dataset P of size n and a natural number k ≤ n , find a collection of n k -dimensional vectors ˆ P = (ˆ p 1 , · · · , ˆ p n : ˆ p i ∈ R k ) with a bilinear form f : R k × R k → R , f ( u, v ) = u T Av , such that the STRESS error || ˆ D -D || 2 F for the dissimilarity matrix ˆ D of ˆ P given by ˆ D ij = f ( ˆ p i , ˆ p j ) is minimized.

3.1 MDS in the Lens of Bilinear Forms

A special case of a symmetric bilinear form is the standard inner product ⟨ , ⟩ on R n . Indeed the inner product ⟨ u, v ⟩ = u T v is a symmetric positive definite bilinear form. The inner product of u -v and u -v is precisely the squared Euclidean distance || u -v || 2 . The Euclidean space is R n equipped with the standard inner product. Thus metric geometry of Euclidean space is governed by the inner product. On the other hand, a basic theorem of linear algebra states that a finite dimensional vector space equipped with a symmetric positive definite bilinear form is isometric to the Euclidean space of the same dimension.

Thus geometry of a positive definite symmetric bilinear form, or symmetric positive definite matrices, is nothing but Euclidean geometry.

When D represents the inner products of pairwise differences (i.e., squared Euclidean distances) of n points P in R d , D is called a Euclidean distance matrix (EDM) . It can be shown that by taking B = -1 2 CDC , where C = I -1 n 1 n 1 T n is the centering matrix and 1 n is a vector of ones, one obtains the Gram matrix B = X T X where X is d × n dimensional matrix of the n coordinates of dimension d . The matrix B is a symmetric positive semi-definite matrix. Thus using eigendecomposition of B one can recover the coordinates X . This procedure is classical multidimensional scaling (cMDS) [40]. Furthermore, if one would like to use k -dimension coordinates with k < d , classical MDS suggests to take the eigenvectors corresponding to the k largest positive eigenvalues of the Gram matrix B . This minimizes the strain , the difference (Frobenius norm ∥ · ∥ F ) in terms of the Gram matrix.

Now consider a general symmetric dissimilarity matrix D of size n × n , it naturally associates R n with a symmetric bilinear form Φ. In many real-world situations, the dissimilarity matrix is not positive, nor negative definite, i.e., the bilinear form is indefinite. This means the intrinsic geometry ( R n , Φ) is non-Euclidean. The relationship between the Gram matrix and square distance matrix still holds for indefinite bilinear forms. See Appendix A

In practice, classical MDS is often the default choice even when the input dissimilarity matrix D is not an EDM, i.e., the centered matrix B = -1 2 CDC is not positive semidefinite. Classical MDS, which simply drops negative eigenvalues of B to produce positive semi-definiteness does not respect the geometry well. Indeed, multiple researchers have observed suboptimal and counter-intuitive performance [41, 38]. For example, increasing k may lead to increased STRESS error - keeping more dimensions makes the approximation worse! On a second thought, such results are not surprising. If the input data does not carry Euclidean geometry, forcing it through a procedure for Euclidean geometry is fundamentally problematic.

Our main observation is that a vector space with an indefinite symmetric bilinear form has its own intrinsic geometry. This geometry, even though may not be metrical, carries the most accurate information about the dissimilarity matrix and the datasets. Suppose the centered matrix B = -1 2 CDC has p > 0 positive eigenvalues and q > 0 negative eigenvalues. The associated indefinite bilinear form Φ has signature ( p, q ). One such example is

The geometry of a finite dimensional vector space with a symmetric bilinear form of signature ( p, q ) where p, q > 0 is much less developed compared to Euclidean geometry. When q = 1, this is the Minkowski geometry and is closely related to relativity theory in physics and hyperbolic geometry [33] in mathematics. For general ( p, q ), one should probably abandon the notion of distance for indefinite spaces. Namely, the expression Φ( u -v, u -v ), even if it is positive, should not be considered as the square of the distance between two points u, v . According to [33], one calls √ Φ( u -v, u -v ) the Lorenzian distance between u, v . Despite the term distance in the name, the Lorenzian distance does not satisfy triangle inequality in general.

3.2 Non-Euclidean MDS

We propose Non-Euclidean MDS , a novel linear dimension reduction technique using bilinear forms. For a vector v , we use Diag( v ) as the diagonal matrix with v along the main diagonal and zero everywhere else. For any given symmetric dissimilarity matrix D ∈ R n × n , let the eigen decomposition of the centralization of D be given as follows:

where U ∈ R n × n is the orthogonal matrix of eigenspace and Λ ∈ R n × n is the diagonal matrix Λ =: Diag( λ ) with eigenvalues λ ≜ ( λ 1 , · · · , λ n ) T , λ 1 ≥ λ 2 ≥ · · · ≥ λ n . By using an algorithm that will be discussed in Section 4 we will choose k of the eigenvalues, represented by a binary indicator vector w = ( w 1 , · · · , w n ) ∈ { 0 , 1 } n with a value of 1 (or 0) indicating the corresponding eigenvalue is chosen (or not chosen). Let

Note that √ Λ contains complex numbers for non-PSD matrix D . Since w has only k nonzero values, we can drop the n -k zero rows in X (corresponding to the eigenvalues not selected) and have a k -dimensional vector representation of the data. Now we can derive dissimilarities by defining

See 1 for details.

Algorithm 1 : Non-Euclidean Multidimensional Scaling

Input : n × n dissimilarity matrix D , integer k ≤ n .

Output : k × n matrix X of k -dim vectors

B = - 1 2 CDC , where C = I - 1 n 1 n 1 T n ;

Compute eigenvalue vector λ and eigenvectors U of B : λ = ( λ 1 , λ 2 , · · · λ n ) T with λ 1 ≥ λ 2 ≥ · · · ≥ λ n ;

Compute the indicator vector of k selected eigenvalues w = EV-Selection( λ , k ); √

Compute X by Λ · Diag( w ) · U T with n - k zero rows dropped;

In the description above, we allow the coordinates in X to take complex numbers, such that we can take ˆ D i,j to be the standard dot product of X i -X j with X i -X j . Alternatively, we can keep X to take real coordinates, i.e., X = √ Λ ′ · Diag( w ) · U T with Λ ′ = Diag( | λ 1 | , · · · , | λ n | ). Again we drop the n -k zero rows in X and take a bilinear form f ( u, v ) = u T Av , where A is a k by k diagonal matrix, with the element at ( i, i ) to be 1 (or -1) if the corresponding eigenvalue chosen is positive/negative. Notice that the bilinear form takes precisely the format of ( p, q )-distance as in Equation (2).

We have a few remarks in place: First we are not throwing away the negative eigenvalues. As will be explained later we actually keep eigenvalues of largest magnitude and some could be negative. As a consequence ˆ D may have negative real values, which is expected as we are moving away from Euclidean distances and input entries in D may even start to be negative. Second, when D is an EDM, i.e., all eigenvalues are non-negative, Neuc-MDS reduces to classical MDS. Last, similar to cMDS, our method also starts with computing the eigenvalues of the Gram matrix. For large datasets, fast (approximation) algorithm for partial SVD can also be applied to our methods. For example, on n × n symmetric matrices, using power methods one can iteratively compute partial SVD up to k largest/smallest eigenvalues in O ( kn 2 ). With randomness introduced, it can be reduced to O (log k · n 2 ) [18, 27]. For really large datasets, the dissimilarity matrix with size O ( n 2 ) might already be too large to be acquired or stored, one may extend MDS through local embedding methods like Landmark MDS [12] or Local MDS [10]. This approach can also be applied to neuc-MDS to substantially speed up computation without suffering too much on performance (See Section 6 for empirical results).

4 Theoretical Results for Non-Euclidean MDS

To establish the foundation of theoretical analysis, we first analyze the STRESS error of Neuc-MDS and decompose it into three terms. Next we show an efficient algorithm that minimizes the first two terms that dominate. Last, we examine Neuc-MDS and classical MDS on random Gaussian matrices.

4.1 Error Analysis

Inspired by the analysis of STRESS of classical MDS [38], we adopt a similar approach and decompose the STRESS error into three terms. Let λ ∈ R n be the vector of all eigenvalues λ = ( λ 1 , · · · , λ n ) T , λ (2) ∈ R n be the vector of all squared eigenvalues λ (2) = ( λ 2 1 , · · · , λ 2 n ) T , ¯ w ≜ 1 n -w be the indicator vector of dropped eigenvalues, ¯ w ∈ { 0 , 1 } n , and ⊙ be the Hadamard product. We have the following result with proof in Appendix B.

Note that C 1 / 4 is the sum of squared eigenvalues that are dropped, and C 2 / 4 is the square of the sum of eigenvalues that are dropped. Individually, we can minimize C 1 by keeping eigenvalues of large absolute value; and C 2 by balancing the dropped eigenvalues such that the summation has a small magnitude. For term C 3 , from Equation 21 in Proof B, we know C 3 ≥ 0. In [38] it is argued that if one takes the approximation ∥ ( U ⊙ U )( ¯ w ⊙ λ ) ∥ 2 F ≈ ∥ 1 √ n 1 n 1 T n ( ¯ w ⊙ λ ) ∥ 2 F for a random orthogonal matrix U , then C 3 ≈ 0. Although it is empirically observed in [38] that C 3 is roughly constant and hence negligible for optimization, there are some cases in our experiments in which C 3 is not negligible.

In light of Theorem 2, we would like to approximately optimizing the STRESS by minimizing the lower bound C 1 + C 2 , which can be formulated as a quadratic integer programming problem: Given a set of n values L = { λ i } and a positive integer k > 0, choose a k -subset S ⊆ L such that

When λ i 's are all positive, the best choice is to take the k largest eigenvalues of L , i.e., the cMDS' solution. However, with a mixture of positive and negative eigenvalues, taking the top k largest eigenvalues is no longer optimal - specifically, as k increases, the first error term is monotonically reduced but the second error term could start going up. In the following subsection, we discuss an optimal algorithm to solve Equation (5).

4.2 An Optimal Algorithm for Eigenvalue Selection

The optimization problem described in Equation (5) is a special case of the family of quadratic integer programming problems. Though in general, quadratic integer programming is NP-hard [31], we show that this particular one is actually solvable in polynomial time. Formally, we have:

Theorem 3. For the optimization problem defined in (5), there exits an optimal solution with r largest positive values and s smallest negative values in L , r + s = k . And, there is an O ( n ) -algorithm that outputs an optimal solution.

Our algorithm (2) is greedy and iteratively selects the eigenvalue with the highest absolute value. Specifically, let S be the set of selected eigenvalues and H ( S ) be the sum of eigenvalues not selected. Initially S = ∅ . In each iteration, if H ( S ) < 0, select the negative eigenvalues remained of the greatest magnitude and add it to S ; if H ( S ) > 0, pick the largest positive one. If H ( S ) = 0, pick the eigenvalue with the greatest magnitude. The complete proof of Theorem 3 is delayed to Appendix C.

4.3 Analysis on Random Symmetric Matrices

Here, we analyze the important error term C 1 + C 2 in Theorem 2 for cMDS and Neuc-MDS when the input (centered) dissimilarity matrix is a symmetric random matrix. Our analysis is established upon Wigner's famous Semicircle Law [46, 47], which states the following.

Proposition 4 (Semicircle Law [47]) . Suppose B ∈ R n × n is a symmetric random matrix where every element is independently distributed with equal densities and second moments σ 2 . Let S a,b ( B ) be the number of eigenvalues of B that lie in the interval ( a √ n, b √ n ) . Then the expected value E ( S a,b ( B )) of S ( B ) satisfies

Our results are formally stated as follows with proof in Appendix D.

Theorem 5. Suppose a random symmetric matrix B ∈ R n × n where B ij is independently distributed with equal densities and second moments σ 2 , is taken as the centered matrix 3 to classical MDS and Neuc-MDS, both selecting k eigenvalues with k ≤ n . Let e C denote the C 1 + C 2 error for cMDS and e N for Neuc-MDS, we have:

k = o ( n ) , e C ≈ n 2 σ 2 (1 + 4 k 2 -4 k ) , and e N ≈ n 2 σ 2 (1 -4 k )

when n n n 2. when k = cn , with c → 1 , e N ≈ 0 . When c ≥ 1 / 2 , e C ≈ 0 . 1801 · n 3 σ 2 .

By Theorem 5, Neuc-MDS has a strictly better C 1 + C 2 error than cMDS. Second, if we take | S | = k ≪ n = |L| , the term C 1 = ∑ λ ∈L\ S λ 2 is almost equal to ∑ λ ∈L λ 2 ≈ n 2 σ 2 . Therefore the STRESS error of both classical MDS and Neuc-MDS cannot be very small if the target dimension k is o ( n ). Lastly, when k = cn , with c → 1, e N is monotonically decreasing and eventually reaches 0. On the other hand, for cMDS, when c ≥ 1 / 2, e C reaches a plateau at about 0 . 1801 · n 3 σ 2 . Notice that this error has an extra factor of n compared to the error for small k .

For real world data the Gram matrix is likely far from a random matrix. The analysis above points out that aggressive dimension reduction can be indeed only a luxury for structured data, even if we use inner products that are not limited to Euclidean distances. Second, any real world data carries some random measurement noise. When the scale of such random noise becomes non-negligible, STRESS error introduced by such noise cannot be small with aggressive dimension reduction. We would recommend practitioners to examine the spectrum of eigenvalues to gain insights on the power or limit in reducing dimensions.

5 Beyond Binary Selection of Eigenvalues

Both cMDS and Neuc-MDS choose a subset of k eigenvalues from the input Gram matrix. This can be considered as applying Diag( w ) to the eigenvalue vector Λ (Equation 4) to filter some eigenvalues out. Such operators can be viewed as a special case of more general low-rank linear maps T : R n → R n with rank( T ) = k ≤ n .

Here we consider a new family of embeddings ˜ X = U ˜ Λ 1 / 2 with ˜ Λ being a rankk diagonal matrix whose diagonal entries are given by some ˜ λ ∈ R n with only k nonzero entries. Note that there always exists a rankk linear operator T such that ˜ λ = T ( λ ). Then we ask if we can improve Neuc-MDS further with a rankk linear map of λ ? This question can be answered by the following theorem:

Theorem 6. Given a dissimilarity matrix D ∈ R n × n with eigenvalues λ ∈ R n of its Gram matrix -CDC/ 2 = U Λ U T , for any rankk ( k ≤ n ) diagonal matrix ˜ Λ = Diag( ˜ λ ) with ˜ λ ∈ R n with k nonzero entries only on coordinates given by some index k -set W ⊆ [ n ] , let w := 1 W ∈ { 0 , 1 } n be the indicator vector of W , and let ˜ D be the dissimilarity matrix reconstructed from ˜ X = ˜ Λ 1 / 2 U T . Then the STRESS error of ˜ D can be expressed as:

where ∆ λ := λ -˜ λ . The first two terms, ˜ C 1 + ˜ C 2 , as a lower bound of the STRESS, is minimized as

Remark 7. Recall the decomposition of STRESS in Theorem 2, the lower bound 4 ¯ w T λ (2) + 4( ¯ w T λ ) 2 = C 1 + C 2 / ( k +1) ≤ C 1 + C 2 . Therefore, it has a better lower bound of STRESS compared to Neuc-MDS .

Theorem 6 provides a constructive way to obtain the optimal ˜ λ that (approximately) minimizes the STRESS error for a prefixed w which is determined by an indicator set W served as the constraints of nonzero entries on ˜ λ . The next question is how to find an optimal k -set W on which the optimal ˜ λ ∗ has the lowest lower bound. Essentially, we need to solve the following optimization problem:

Proposition 8. For the optimization problem (8), there exits an optimal solution with r largest positive values and s smallest negative values in L , r + s = k . And, there is an O ( n ) -algorithm that outputs the optimal solution.

The algorithm follows a similar greedy manner as EV-Selection , with only one adjustment: in each step, compare the two marginal gains provided by the largest positive eigenvalue and the lowest negative eigenvalue among the remained ones, and choose the positive/negative eigenvalue of largest magnitude if the corresponding gain is smaller. We name the algorithm Neuc-MDS + , and present the complete algorithm with its correctness proof in Appendix C.

6 Experiments

This section presents experimental results of Neuc-MDS and Neuc-MDS + . First, we evaluate the performance on dissimilarity error of two proposed algorithms comparing with closelyrelated baselines on three metrics: STRESS, distortion and additive error. Then we show that the dimensionality paradox issue observed on cMDS is fully resolved by Neuc-MDS and Neuc-MDS + .

Synthetic Data. We introduce two synthetic datasets: Random-simplex and Euclideanball , both with non-Euclidean dissimilarities. See details in Appendix E.1. On a high level, suppose the dataset has size n , we construct a Random-simplex such that for each vertex, the first n -1 coordinates virtually form a simplex, while the last coordinate almost dominates the distances between the other points, which creates a large negative eigenvalue for the Gram matrix. The Euclidean-ball dataset (similar to Delft's balls [14]) considers n balls of different radii with the distance of two balls defined as the smallest distance of two points on the two respective balls. The dissimilarities by this definition no longer satisfy triangle inequality.

Real-world Data. We consider both genomics and image data. For genomics data, we include 5 datasets from the Curated Microarray Database (CuMiDa) [17], each indicating a certain type of cancer. Following the practice mentioned in [43], pairwise dissimilarities are generated with entropic affinity with the diagonal as zero. We also test three celebrated image datasets: MNIST, Fashion-MNIST and CIFAR-10. The dissimilarity matrix for each dataset captures 1000 images randomly sampled for each class. We use three measures in [38] to calculate the dissimilarities.

An essential guidance of our choice of datasets is the existence of substantial negative eigenvalues. Otherwise, Neuc-MDS and cMDS are equivalent. Figure 1 illustrates the eigenvalue distribution of the Renal dataset. Table 1 shows the basic statistics of each dataset and the number of negative eigenvalues of the Gram matrix. All datasets are non-Euclidean and non-metric.

EigenvaluesDistributionofRenal

30

Count

20

10

0

-0.003

-0.002

-0.001

0.000

0.001

0.002

0.003

Baselines. We include cMDS, Lower-MDS [38] and SMACOF (Scaling by MAjorizing a COmplicated Function) [29] as baselines. Lower-MDS [38] looks for a symmetric, lowrank, trace-zero, and positive semi-definite matrix. SMACOF minimizes STRESS using majorization and is one of the best nonlinear optimization algorithms for MDS. All methods are deterministic therefore variance is not concerned.

Performance on Dissimilarity Error. In addition to STRESS as the primary metric, we also test the average distortion (i.e. multiplicative error) and scaled additive error on all datasets. For all metrics, a smaller value indicates a more favorable performance. Limited by space, we leave scaled additive error together with other observations in Appendix E. With k as the target dimension, for synthetic datasets and images, we set k = 100, for genomics data, k = 20. The details are presented in Table 2.

The results 4 show that in terms of STRESS, Neuc-MDS and Neuc-MDS + outperform cMDS and Lower-MDS consistently by a large margin. SMACOF has comparable performance with Neuc-MDS in a couple data sets but can go out of bound in others. For average distortion, the genomics datasets differentiate different methods drastically while the rest datasets produce comparable results. Neuc-MDS + occasionally gives slightly higher STRESS than Neuc-MDS. Recall that both methods focus on optimizing for a lower bound of STRESS (in Theorem 2 and Theorem 6). This shows that C 3 may play a role in practice.

Neuc-MDS Addresses 'Dimensionality Paradox'. Dimensionality paradox of classical MDS refers to the observation that STRESS increases as the dimension goes up. When raising this concern, [38] proposes a Lower-MDS algorithm as mitigation. We show that Neuc-MDS and Neuc-MDS + address this issue even better. For Lower-MDS the target dimension cannot be larger than the number of positive eigenvalues. Our methods do not have this limitation. Figure 2 shows STRESS on Random-simplex and Renal. In Randomsimplex, cMDS has an increasing STRESS with k = 75 ∼ 100 then stops, and in Renal the STRESS keeps increasing. In contrast, Lower-MDS converges promptly while Neuc-MDS and Neuc-MDS + achieve even lower STRESS. Results on other datasets are in Appendix E.4.

Comparing Stress on Random-simplexDataset

ComparingStressonRenalDataset

40

Neuc-MDS Stress

0.125

Neuc-MDS Stress

125

Neuc-MDS+ Stress

Neuc-MDS+Stress

100

20

MDS Stress

0.100

MDS Stress

Stress

75

Lower-MDS Stress

Stress

0.075

Lower-MDS Stress

0.075

50

0

50

100

150

0.050

0.050

0.025

25

0.025

0

10

20

30

0

0.000

0

200

400

600

800

1000

0

20

40

60

80

100

120

140

TargetDimensionk

TargetDimensionk

Landmark MDS Landmark MDS [12] is a heuristic to speed up classical MDS. One chooses a small number of landmarks and apply MDS on the landmarks first. The coordinates of the remaining points are obtained through a triangulation step with respect to the landmarks. We can use the same heuristic to speed up Neuc-MDS. On the random-simplex dataset, with only 25% points randomly chosen as landmarks the STRESS is only a factor of 1 . 0644 of the STRESS obtained by Neuc-MDS. If we use only 10% points as landmarks, the final STRESS is only 1 . 0898 of the STRESS of Neuc-MDS. This shows that Neuc-MDS can also be significantly accelerated using the landmark idea, achieving nearly the same STRESS.

7 Discussion and Conclusion

This paper presents an extension of classical MDS to non-Euclidean non-metric settings. We would like to mention a few future directions. Since we step out of the domain of Euclidean embedding, both the input dissimilarity matrix and the one obtained after dimension reduction can have negative values. Therefore if one would like to feed the output dissimilarity matrix to another data processing module that by default requires non-negative values, special care must be taken to address the negative values. In experiments, we discovered that Neuc-MDS + achieves similar stress as Neuc-MDS and produces much fewer negatives values in the output dissimilarity matrix (Appendix E.3). How to effectively use such dissimilarities in downstream learning and inference tasks would be a major future work. Note that the geometry of general bilinear forms is a largely unexplored territory.

Acknowledgments and Disclosure of Funding

The authors acknowledge funding support through NSF IIS-2229876, DMS-2220271, DMS2311064, CCF-2208663, CCF-2118953, and CRCNS-2207440.

References

R. Beals, D. H. Krantz, and A. Tversky. Foundations of multidimensional scaling. Psychol. Rev. , 75(2):127-142, Mar. 1968.

M. Belkin and P. Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. , 15(6):1373-1396, June 2003.

A. Bellet, A. Habrard, and M. Sebban. A survey on metric learning for feature vectors and structured data. arXiv 1306.6709, June 2013.

I. Benjamini and Y. Makarychev. Dimension reduction for hyperbolic space. Proceedings of the American Mathematical Society , 137(2):695-698, 2009.

M. Brand. Charting a manifold. In Proceedings of the 15th International Conference on Neural Information Processing Systems , volume 15, pages 985-992, 2002.

A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Generalized multidimensional scaling: a framework for isometry-invariant partial surface matching. Proc. Natl. Acad. Sci. U. S. A. , 103(5):1168-1172, Jan. 2006.

S.-H. Cha. Comprehensive survey on distance/similarity measures between probability density functions. Int. J. Math. Model. Meth. Appl. Sci. , 1(4):300-307, 2007.

I. Chami, A. Gu, D. P. Nguyen, and C. Re. HoroPCA: Hyperbolic dimensionality reduction via horospherical projections. In Proceedings of the 38th International Conference on Machine Learning , pages 1419-1429, 2021.

I. Chami, R. Ying, C. R´ e, and J. Leskovec. Hyperbolic graph convolutional neural networks. Adv. Neural Inf. Process. Syst. , 32:4869-4880, Dec. 2019.

L. Chen and A. Buja. Local multidimensional scaling for nonlinear dimension reduction, graph drawing, and proximity analysis. J. Am. Stat. Assoc. , 104(485):209-219, Mar. 2009.

T. F. Cox and M. A. A. Cox. Multidimensional scaling on a sphere. Communications in Statistics - Theory and Methods , 20(9):2943-2953, Jan. 1991.

V. de Silva and J. B. Tenenbaum. Sparse multidimensional scaling using landmark points. Technical Report, Stanford University, 2004.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16 × 16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations , 2021.

R. P. W. Duin and E. P¸ ekalska. Non-Euclidean dissimilarities: Causes and informativeness. In Structural, Syntactic, and Statistical Pattern Recognition , pages 324-333. Springer Berlin Heidelberg, 2010.

A. Elad, Y. Keller, and R. Kimmel. Texture mapping via spherical multi-dimensional scaling. In Scale Space and PDE Methods in Computer Vision , pages 443-455. Springer Berlin Heidelberg, 2005.

X. Fan, C.-H. Yang, and B. C. Vemuri. Nested hyperbolic spaces for dimensionality reduction and hyperbolic NN design. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 356-365, 2022.

B. C. Feltes, E. B. Chandelier, B. I. Grisci, and M. Dorn. CuMiDa: an extensively curated microarray database for benchmarking and testing of machine learning approaches in cancer research. Journal of Computational Biology , 26(4):376-386, 2019.

G. H. Golub and C. F. Van Loan. Matrix Computations . Johns Hopkins University Press, Feb. 2013.

A. A. Goshtasby. Similarity and dissimilarity measures. In A. A. Goshtasby, editor, Image Registration: Principles, Tools and Methods , pages 7-66. Springer London, London, 2012.

Y. Guo, H. Guo, and S. Yu. CO-SNE: Dimensionality reduction and visualization for hyperbolic data. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11-20, 2022.

T. Hayden and J. Wells. Approximation by matrices positive semidefinite on a subspace. Linear Algebra and its Applications , 109:115-130, 1988.

Y.-H. He and J. M. P´ erez Ipi˜ na. Machine-learning the classification of spacetimes. Phys. Lett. B , 832:137213, Sept. 2022.

G. E. Hinton and S. Roweis. Stochastic neighbor embedding. In Proceedings of the 15th International Conference on Neural Information Processing Systems , volume 15, page 857-864, 2002.

R. A. Horn and C. R. Johnson. Topics in matrix analysis . Cambridge University Press, 1994.

B. Kulis. Metric learning: A survey. Foundations and Trends ➤ in Machine Learning , 5(4):287-364, 2013.

J. A. Lee and M. Verleysen. Nonlinear Dimensionality Reduction . Springer New York, 2007.

M. W. Mahoney. Randomized Algorithms for Matrices and Data . Now Publishers, 2011.

K. V. Mardia. Some properties of clasical multi-dimesional scaling. Communications in Statistics - Theory and Methods , 7(13):1233-1241, Jan. 1978.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

P. Pellizzoni, S. Moretti, and F. Silvestri. Dimensionality reduction on complex vector spaces for Euclidean distance with dynamic weights. arXiv 2212.06605, Dec. 2022.

D. Pisinger. The quadratic knapsack problem-a survey. Discrete Appl. Math. , 155(5):623-648, Mar. 2007.

H.-D. Qi and X. Yuan. Computing the nearest Euclidean distance matrix with low embedding dimensions. Math. Program. , 147(1-2):351-389, Oct 2014.

J. G. Ratcliffe. Foundations of Hyperbolic Manifolds . Springer International Publishing, 2006.

S. T. Roweis and L. K. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science , 290(5500):2323-2326, 2000.

J. W. Sammon. A nonlinear mapping for data structure analysis. IEEE Trans. Comput. , C-18(5):401-409, May 1969.

R. N. Shepard. The analysis of proximities: Multidimensional scaling with an unknown distance function. I. Psychometrika , 27(2):125-140, June 1962.

R. N. Shepard. The analysis of proximities: Multidimensional scaling with an unknown distance function. II. Psychometrika , 27(3):219-246, Sept. 1962.

R. Sonthalia, G. Van Buskirk, B. Raichel, and A. Gilbert. How can classical multidimensional scaling go wrong? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 12304-12315, 2021.

J. B. Tenenbaum, V. de Silva, and J. C. Langford. A global geometric framework for nonlinear dimensionality reduction. Science , 290(5500):2319-2323, 2000.

W. S. Torgerson. Multidimensional scaling: I. Theory and method. Psychometrika , 17(4):401-419, Dec. 1952.

J. Tsang and R. Pereira. Taking all positive eigenvectors is suboptimal in classical multidimensional scaling. SIAM J. Optim. , 26(4):2080-2090, Jan. 2016.

A. Tversky and I. Gati. Similarity, separability, and the triangle inequality. Psychol. Rev. , 89(2):123-154, Mar. 1982.

H. Van Assel, T. Vayer, R. Flamary, and N. Courty. SNEkhorn: Dimension reduction with symmetric entropic affinities. arXiv preprint arXiv:2305.13797 , 2023.

L. van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research , 9:2579-2605, 2008.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems , volume 30, page 6000-6010, 2017.

E. P. Wigner. Characteristic vectors of bordered matrices with infinite dimensions. Annals of Mathematics , 62(3):548-564, 1955.

E. P. Wigner. On the distribution of the roots of certain symmetric matrices. Annals of Mathematics , 67(2):325-327, 1958.

R. C. Wilson, E. R. Hancock, E. Pekalska, and R. P. W. Duin. Spherical and hyperbolic embeddings of data. IEEE Trans. Pattern Anal. Mach. Intell. , 36(11):2255-2269, Nov. 2014.

W. Xu, R. C. Wilson, and E. R. Hancock. Determining the cause of negative dissimilarity eigenvalues. In Computer Analysis of Images and Patterns , pages 589-597. Springer Berlin Heidelberg, 2011.

Z. Zhang and H. Zha. Principal manifolds and nonlinear dimensionality reduction via tangent space alignment. SIAM J. Sci. Comput. , 26(1):313-338, Jan. 2004.

A Hollow Matrices and Relationship Between Gram and Squared Distance Matrices

We will prove the following proposition relating hollow symmetric matrices and squared 'distance' matrices.

Proposition 9. (1) Given any symmetric matrix G = [ g ij ] n × n , there exists a symmetric bilinear form ⟨ , ⟩ on R n and n vectors v 1 , ..., v n such that g ij = ⟨ v i , v j ⟩ .

(2) Given any hollow symmetric matrix M = [ m ij ] n × n , there exists a symmetric bilinear form ⟨ , ⟩ on R n and n vectors v 1 , ..., v n such that m ij = ⟨ v i -v j , v i -v j ⟩ for all i, j .

Note that the hollow condition m ii = 0 is necessary.

Proof. Part (1) is a well known fact from linear algebra. We take v 1 , ..., v n to be the standard basis of R n . From the symmetric matrix G , one defines the symmetric bilinear form by the formula ⟨ u, v ⟩ = u T Gv and vice versa.

To see part (2), consider the centered symmetric matrix B = CMC where C = I -J n is the double centering matrix and J is the matrix whose entries are 1. One can see that its ( i, j )-th entry b ij of matrix B is given by

One can easily verify from the formula that the centering condition holds:

By part (1) of the proposition, one finds n vectors v 1 , ..., v n in R n and a symmetric bilinear form ⟨ , ⟩ on R n such that -2 b ij = ⟨ v i , v j ⟩ . Now we claim that m ij = ⟨ v i -v j , v i -v j ⟩ , i.e., -2 m ij = b ii + b jj -2 b ij .

Indeed, by dropping the constant term 1 n 2 ∑ n k =1 ∑ n l =1 m kl in (9), we have

The last step uses the fact that M is a hollow matrix, i.e., m ii = m jj = 0.

In summary, we call G = [ ⟨ v i , v j ⟩ ] the Gram matrix and M = [ ⟨ v i -v j , v i -v j ⟩ ] and the squared distance matrix of a symmetric bilinear form ⟨ , ⟩ . Obviously, we can determine M from G by the formula ⟨ v i -v j , v i -v j ⟩ = ⟨ v i , v i ⟩ + ⟨ v j , v j ⟩-2 ⟨ v i , v j ⟩ , i.e., m ij = g ii + g jj -2 g ij . We can also recover the Gram matrix G from M from the double centering construction CMC if the center of the vectors v 1 , ..., v n is 0, i.e., ∑ n i =1 v i = 0. To see this, given a squared distance matrix M associated to n vectors v 1 , .., v n , we can replace v i 's by v i -w for a fixed vector w without changing M . Now by taking w = 1 n ∑ n i =1 v i to be the center of v 1 , ..., v n , we can normalize ∑ n i =1 v i = 0. In this case, formula (9) states that

where m ij = ⟨ v i -v j , v i -v j ⟩ .

B Proofs of Error Analysis of Non-Euclidean MDS

In this section, we want to prove our main Theorem 6 for the error analysis of STRESS error. Note that Theorem 2 is a special case of Theorem 6 for ∆ λ ⊙ w = 0, which is exactly the case in MDS and Neuc-MDS with ˜ λ given by λ ⊙ w .

Theorem 6. Given a dissimilarity matrix D ∈ R n × n with eigenvalues λ ∈ R n of its Gram matrix -CDC/ 2 = U Λ U T , for any rankk ( k ≤ n ) diagonal matrix ˜ Λ = Diag( ˜ λ ) with ˜ λ ∈ R n with k nonzero entries only on coordinates given by some index k -set W ⊆ [ n ] , let w := 1 W ∈ { 0 , 1 } n be the indicator vector of W , and let ˜ D be the dissimilarity matrix reconstructed from ˜ X = ˜ Λ 1 / 2 U T . Then the STRESS error of ˜ D can be expressed as: ∥ ˜ D -D ∥ 2 F = ˜ C 1 + ˜ C 2 + ˜ C 3 with

where ∆ λ := λ -˜ λ . The first two terms, ˜ C 1 + ˜ C 2 , as a lower bound of the STRESS, is minimized as

The proof will essentially follow the pipeline in [38]. Our results and proofs can be viewed as an extension to a more general family of problems. We first show that the STRESS error can be decomposed into three terms ˜ C 1 + ˜ C 2 + ˜ C 3 . Then we analyze the approximate lower bound ˜ C 1 + ˜ C 2 and figure out the minimum.

We first introduce some notations and lemmas we need to use later. Recall that C = I -1 n 1 n 1 T n is the centering matrix. First note the following fact:

Lemma 10. The n -dimensional vector 1 n is an eigenvector with eigenvalue 0 of the matrices C, U and ˜ X T ˜ X .

Proof. By definition of C = I -1 n 1 n 1 T n , it is easy to check C 1 n = 0. Also by definitions of U and ˜ X T ˜ X we can get similar results.

As a corollary, we have the following lemma:

Lemma 11. -C ˜ DC/ 2 = C ˜ X T ˜ XC = ˜ X T ˜ X .

Proof.

Let Q be a Householder reflector matrix defined as follows:

For any symmetric matrix A ∈ R n × n , let Φ( A ) ∈ R ( n -1) × ( n -1) , f ( A ) ∈ R n -1 , ξ ( A ) ∈ R be given by QAQ through the following equation:

We need the following properties of Q :

Lemma 12 ([32]) . For a symmetric matrix A , we have

Lemma 13 ([21, 38]) . For any matrix M , we use diag( M ) = ( M i,i ) to denote the column vector of M 's diagonal entries. For a symmetric hollow matrix A

Now we are ready to prove our main theorem.

For the first term:

For the second term:

From 0 = Tr( D -˜ D ) = Tr(Φ( D ) -Φ( ˜ D )) + ( ξ ( D ) -ξ ( ˜ D

For the third term:

Taking the difference and then using a theorem on diagonalization and the Hadamard product from [24], we have:

Therefore,

Finally, we have that

The above arguments prove the first part of the Theorem.

Now we want to analyze the lower bound of

Denote C := ¯ w T λ , Z := w T ∆ λ . By the Cauchy-Schwartz inequality, we know that

The equality is obtained when ∆ λ has the same value on all coordinates of W . Now check that

The last inequality achieves the lower bound with Z = -k 1+ k C through the standard analysis of univariate quadratic equations. Based on the above arguments, we have that

Since the lower bounds from the Cauchy-Schwartz inequality and the univariate quadratic equation are all tight, we can conclude that the final lower bound is tight and it can be achieved with

This finishes the proof.

C Proofs of Eigenvalue Selection Algorithm

Proof of Theorem 3. For any subset S ⊆ L , define F ( S ) = ∑ λ ∈L\ S λ 2 + (∑ λ ∈L\ S λ ) 2 . Note that this is the value we want to minimize (i.e., C 1 + C 2 ). Without loss of generality, we may assume that none of the chosen eigenvalues in S is 0, since it does nothing to change F ( S ).

We first prove the first claim of Theorem 3.

Proof. First, we show that an optimal solution S of size k exists. Assume we have some optimal solution S of size less than k . Let b ∈ L \ S . Then we have:

Now it is clear that if H ( S ) is 0, we can add any eigenvalue to S and F ( S ) would not change. If H ( S ) is not 0, then adding an eigenvalue to S that matches the sign of H ( S ), would reduce F ( S ) and the opposite would increase F ( S ). We will continue to make use of this fact. Adding an eigenvalue that matches the sign of H ( S ) is also always possible since

H ( S ) is the sum of the eigenvalues we are choosing from. Thus, we can just continue to add eigenvalues to S until it is of size k .

Next, assuming S is of size k , we show that it must be the one as described in Theorem 3. We can prove this by contradiction. Without loss of generality, we may assume that S contains a positive eigenvalue b such that there exists some a ∈ L\ S satisfying a > b . First, we observe that since S is optimal:

which reduces to H ( S ) ≥ -b . Then, we have:

We know that b -a < 0 and b > 0, H ( S ) ≥ -b , so we get that the expression is negative. Thus, S ∪ { a } \ { b } has a lower value under F and contradicts our assumption that S is optimal.

Let S be the optimal solution from Theorem 3. Let T be the solution obtained by EVSelection. We will show that F ( S ) = F ( T ). We use contradiction again. Without loss of generality, we assume that S has more positive eigenvalues than T . Now consider H ( S ∩ T ). From a similar argument used in the proof of Theorem 3 (1), due to S \ T containing only positive eigenvalues, H ( S ∩ T ) ≥ 0. Now if H ( S ∩ T ) = 0, we know that S \ T contains one eigenvalue, - otherwise, removing a positive eigenvalue from S would decrease F ( S ). Then, we know that F ( S ) = F ( T ) and that is a contradiction. If H ( S ∩ T ) > 0, then we have reached a contradiction in the definition of T because EV-Selection will only choose eigenvalues in T \ S when all negative eigenvalues in T ∩ S , Q , has been chosen. However, for all subsets P of positive eigenvalues of T , H ( P ∪ Q ) > 0, so no eigenvalue in T \ S will ever be chosen by EV-Selection.

Algorithm 3 : Eigenvalue Selection for Generalized Neuc-MDS

Proof of Proposition 8. For any subset S ⊆ L , define F ( S ) = ∑ λ ∈L\ S λ 2 + 1 | S | +1 (∑ λ ∈L\ S λ ) 2 . Note that this is the value we want to minimize (i.e., C 1 + 1 | S | +1 C 2 ). Without loss of generality, we may assume that none of the chosen eigenvalues in S is 0, since it does nothing to change F ( S ).

We first prove the first claim in Proposition 8.

Proof. First, we show that an optimal solution S of size k exists. Assume we have some optimal solution S of size less than k . Let b ∈ L \ S . Then we have:

Now it is clear that if H ( S ) is 0, adding any eigenvalue to S would not increase F ( S ). If H ( S ) is not 0, then adding an eigenvalue to S that matches the sign of H ( S ), would reduce F ( S ). We will continue to make use of this fact. Adding an eigenvalue that matches the sign of H ( S ) is also always possible since H ( S ) is the sum of the eigenvalues we are choosing from. Thus, we can just continue to add eigenvalues to S until it is of size k .

Next, we will show there must be an S as described in Proposition 8. We can prove this by contradiction. Without loss of generality, we may assume some optimal S contains a positive eigenvalue b such that there exists some a ∈ L\ S satisfying a > b . We can directly observe since S is optimal

Since we know b -a < 0, we get that ( | S | + 2) b + | S | a + 2 H ( S ) ≤ 0. First, for the case where it is equal to 0, we can just take out b and put in a . If S still does not have the greatest positive eigenvalues, we can repeat our analysis and examine another candidate positive eigenvalue to be replaced. In the case that it is strictly negative, then we have H ( S ) < 0 since | S | , b, a > 0. Then there must exist some negative eigenvalue c that has not been chosen and if we replace a with c in our analysis, we get that b -c > 0 and ( | S | +2) b + | S | c +2 H ( S ) < 0 which gives F ( S ∪ { c } \ { b } ) < F ( S ) which contradicts the optimality of S .

Let S be the optimal solution from Proposition 8. Let T be the solution obtained by ?? 3. We will show that F ( S ) = F ( T ). Note that now that we know | S | = k , we now define F = ∑ λ ∈L\ S λ 2 + 1 k +1 (∑ λ ∈L\ S λ ) 2 . We use contradiction again. Without loss of generality, we assume that S has more positive eigenvalues than T . Let a be the largest eigenvalue in magnitude from T \ S . Let T ′ be the set chosen by ?? 3 right before choosing a . Then let b be the largest eigenvalue in S \ T ′ . By definition of T , we know that:

Now let's compare that with:

Now clearly H ( S ) + b ≤ H ( T ′ ). In the case that H ( S ) + b = H ( T ′ ), we clearly have S ∪ { a } \ { b } = T , so we get F ( S ) = F ( T ). In the case that H ( S ) + b < H ( T ′ ), we also know that a -b < 0, so we get F ( S ∪ { a } \ { b } ) -F ( S ) < F ( T ′ ∪ { a } ) -F ( T ′ ∪ { b } ) ≤ 0. That violates the fact that S is an optimal set, so we have a contradiction.

D Computations on Random Matrices

We compare the classical multidimensional scaling and our non-Euclidean multidimensional scaling under the random matrix context, and give the proof of Theorem 5.

We start with a symmetric random matrix B ∈ R n × n as in Proposition 4, and want to select k eigenvalues by using the classical and non-Euclidean multidimensional scaling.

We start with classical multidimensional scaling. To select the largest k eigenvalues, we need to select all eigenvalues greater than (2 -r ) √ nσ such that

There is not an explicit way to write r as a function of k . For a fixed r , since we want to drop all eigenvalues smaller than (2 -r ) √ nσ , we have

For non-Euclidean multidimensional scaling, to select the k eigenvalues with largest magnitude, we need to select all eigenvalues with magnitude greater than (2 -r ) √ nσ such that

For a fixed r , since we want to drop all eigenvalues with magnitude smaller than (2 -r ) √ nσ , we have

We first consider the case that k = o ( n ), i.e. we want to reduce the dimension from n to a much smaller k . Under this assumption, r is a very small positive number. In the classical case, by applying the Taylor series to equation (38), we get r ≈ ( 3 πk 2 n ) 2 3 . Then we plug it into equation (39) to get

Similarly, in the non-Euclidean case, we apply the Taylor series to equation (40), we get r ≈ ( 3 πk 4 n ) 2 3 . Then we plug it into equation (41) to get

This finishes the proof of Theorem 5 (1).

Now we turn to the case that k = cn for a constant c ∈ [0 , 1]. We first solve c = k n in equations (38) and (40) to get the corresponding r , then plug the r -values in equations (39) and (41) to get the corresponding e C and e N , respectively. Since equations (38) and (40) can not be solved explicitly, we can only get numerical values of e C and e N . For c ∈ (0 , 0 . 5], numerical values of e C and e N as shown in Table 4. For c ∈ [0 . 5 , 1], the classical multidimensional scaling stabilizes, with e C ≈ (0 . 5 + 0 . 1801 · n ) n 2 σ 2 . Meanwhile, the error e N for non-Euclidean multidimensional scaling decreases to zero as c increases to 1. More precisely, if c = 1 -ϵ with very small ϵ > 0, equations (40) and (41) give

Some numerical values of e N with c ∈ [0 . 5 , 1] are shown in Table 5. This finishes the proof of Theorem 5 (2).

E More Details and Results on Experiments

In this appendix we fill the missing details in the experiment section. Our experiments are implemented with Intel Core i9 CPU of 32GB memory, no GPU is required and the execution time is no longer than 30 seconds. In the rest of this section, we first introduce the generation process of synthetic datasets in Appendix E.1; Next we define the evaluation metrics and perturbation approaches in Appendix E.2; More results on dissimilarities regarding the scaled additive error, negative distances and negative eigenvalues selected, are demonstrated in Appendix E.3. With respect to the dimensionality paradox , we provide illustrations similar to Figure 2 on other datasets in Appendix E.4.

E.1 Synthetic Datasets Generation

The random-simplex dataset is generated with the idea of creating a simplex with one added dimension that generates all of the variance in the the distances such that the added dimension corresponds with a negative eigenvalue. This way, we should have a dimension with a large negative eigenvalue that is impactful on the stress of the embedding.

Specifically, this dataset was generated with points x 1 , ...x 1000 , each point having 1000 dimensions. For each point x i , the first 100 coordinates were chosen uniformly randomly between 0 and .01. The next 899 coordinates were chosen uniformly randomly between 0 and √ 0 . 5 / 899. The last coordinate is i × 0 . 3 / 1000. Then, the distance between x i and x j was found in the following way:

where x i ( k ) is the k th coordinate of x i . The first 999 coordinates essentially form a simplex, because selecting a large number of coordinates randomly from the same distribution causes all of the distances to converge to a normal distribution with low variance. Thus, all of the distances between points using just the first 999 coordinates are very similar. Then, the last coordinate has 'large' discrepancies between all of the points, causing most of the differences in the distances.

The Euclidean-ball metric was inspired by [49]. The set of distances between objects in space does not follow triangle inequality, so it becomes a non-euclidean dissimilarity measure. The particular dataset we use generates the distances by first randomly selecting 1000 points

uniformly in a 10 dimensional hypercube of length 100. Each of these points is now the center of a ball. Then, for each ball, with probability 0.9, we select a radius uniformly randomly between 0 and 5. With probability 0.1, we let the radius be 0.8 times the distance between the center of this ball to the closest other ball. This closest other ball is based on the radius of the other ball if it has already been decided. The final distance matrix is just filled by the distances between the 1000 balls. In this way, we attempt to create some balls with large radius, so that the triangle inequality is heavily violated, therefore causing the dataset to deviate more from the Euclidean setting.

E.2 Evaluation and Perturbation Metrics

In addition to STRESS, our evaluation metric include scaled additive error and average geometric distortion. Scaled additive error by definition is allowing scaling of the distance matrix of our embedding before calculating stress. To do this, we first ran the embedding method (e.g., cMDS, Neuc-MDS, or other methods) on the given dissimilarity squared matrix D to get the output dissimilarity squared matrix ˜ D . Then, we would flatten both D and ˜ D to vectors, and project D onto the line through ˜ D to get E . Then we calculate ∥ D -E ∥ to get scaled additive error. Clearly, this is equivalent to allowing scaling of ˜ D to find the minimum stress. For average geometric distortion, we again input D to get ˜ D , but this time, we evaluate the distortion of each dissimilarity by dividing entries of √ D by entries of √ ˜ D . Since distortion is not well defined when dissimilarities become complex, we skip those. Then, with the remaining valid distortions, we scale the distortions so that there's an equal number of them greater than 1 and less than 1, then for those that remain less than 1, we take their reciprocal. Then, we finally take the geometric average of all of these distortions. In this way, we basically again allowed scaling of the dissimilarities to find the minimum possible average distortion.

Next, we had to decide on metrics to use for the image datasets. We took the following 3 metrics from [38]. For the first metric, the distance between two images is first finding the Euclidean distance between their coordinate representations, and then adding Gaussian noise to the distances. Because the noise is added directly onto the distances, the end result is highly likely to be non-Euclidean. For the second metric, we build a k nearest neighbor graph of the images based on their Euclidean distances. Since the end result is a graph structure, the shortest path distances are also highly likely to be non-Euclidean. For the third metric, we randomly removed entries from the coordinate representation of the images. Then, the distance between two images is decided by taking the Euclidean distance between coordinates that both images still included. Here, the Euclidean property breaks down again because triangle inequality would no longer have to hold. In Table 2, we choose the k -NN metric with k = 2. In the next Appendix section, we show results for higher k and other metrics applied to the image datasets. As for genomics datasets, we use the entropic affinities commonly used in t-SNE methods. The main idea is to apply an adaptive kernel, whose bandwidth depends on a parameter termed as perplexity, to pairwise data entries.

E.3 More Results on Dissimilarity Error

In addition to STRESS and average distortion already reported in Table 2, we further show the third metric: scaled additive error as defined in Appendix E.2. Note that our algorithms may produce negative distances, we also report number of negative distances and number of negative eigenvalues selected. All original results (without scaling) on 10 datasets are demonstrated in Table 6. Recall we mentioned the results in Table 2 contains some scaling on the synthetic data and images, we also provide the original values in Table 7.

On scaled additive error, our proposed methods still consistently outperform cMDS and Lower-MDS considerably. Thus, we have shown that on all metrics our methods yield more favorable results. It is interesting to observe that Neuc-MDS + produces much fewer negative distances than Neuc-MDS, though the number of negative eigenvalues selected are quite close. Further, two methods have similar performance on STRESS and other metrics. Therefore in practice, if there is a considerable concern on negative distances, Neuc-MDS + should become a better choice.

0.6

1.0

0.8

Lower-MDS Stress

Neuc-MDS Stress

Neuc-MDS+ Stress

MDS Stress

Comparing Stress on Brain Dataset

0.20

0.15

Comparing Stress on Breast Dataset

Neuc-MDS Stress

Neuc-MDS+ Stress

0.15

Comparing Stress on Colorectal Dataset

Neuc-MDS+ Stress

Neuc-MDS Stress

0.25

0.20

Comparing Stress on Leukemia Dataset

Neuc-MDS+ Stress

Neuc-MDS Stress

0.2

0.10

0.05

Lower-MDS Stress

MDS Stress

0.05

e

Lower-MDS Stress

MDS Stress

0.10

0.05

Lower-MDS Stress

MDS Stress

0.0

0.00

0.00

0.00

0

20

40

60

80

100

120

0

20

40

60

80

100

120

140

0

25

50

75

100

125

150

175

200

0

50

100

150

200

250

Target Dimension k

Target Dimension k

Target Dimension k

Target Dimension k

1e7

Comparing Stress on Ball Dataset

Comparing Stress on MNIST Dataset

Comparing Stress on Fashion MNIST Dataset

Comparing Stress on CIFAR10 Dataset

1.25

1.00

Neuc-MDS Stress

Neuc-MDS+ Stress

MDS Stress

Lower-MDS Stress

40000

60000

MDS Stress

Lower-MDS Stress

Neuc-MDS Stress

Neuc-MDS+ Stress

30000

40000

Neuc-MDS Stress

Neuc-MDS+ Stress

MDS Stress

Lower-MDS Stress

ssans

40000

30000

MDS Stress

Lower-MDS StreSs

Neuc-MDS Stress

Neuc-MDS+ Stress

20000

0.50

0.25

20000

10000

10000

0.00

0

0

0

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

009

800

1000

0

200

400

600

800

1000

Target Dimension k

Target Dimension k

Target Dimension k

Target Dimension k

Until now all results reported on images are using k NN metric with k = 2 to produce non-Euclidean dissimilarities. There could be other perturbations for the same purpose, as mentioned in Appendix E.2. We further test with k = 10 and the other two metrics: adding noise and random removal on the STRESS. We add noise sampled from Gaussian distribution with variance as the maximum instance-wise difference scaled by 500; For random removal, we skip 50 images for each. The results are shown in Table 8.

E.4 More Results on Dimensionality Paradox

The purpose of this section is to show the consistent performance of Neuc-MDS and NeucMDS + on all datasets, with respect to mitigating the dimensionality paradox issue. Therefore, in addition to Figure 2, we provide the same plots for other datasets, and the parameters all follow the main setup. Figure 3 gives a clear illustration that Neuc-MDS and Neuc-MDS + always have lower STRESS than cMDS and Lower-MDS as dimension grows.

F NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We propose a generalization of classical multidimensional scaling to handle non-Euclidean non-metric data. We presented both a new algorithm, theoretical proofs on its performance, and experimental results.

Guidelines:

❼ The answer NA means that the abstract and introduction do not include the claims made in the paper.

❼ The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

❼ The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

❼ It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed the limitation (and future work) of the proposed approach. These include: 1) developing downstream data analysis for using nonEuclidean non-metrics data. 2) speeding up the algorithm for extremely large data (e.g., by developing landmark based non-Euclidean MDS algorithms).

Guidelines:

❼ The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

❼ The authors are encouraged to create a separate 'Limitations' section in their paper.

❼ The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

❼ The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

❼ The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speechto-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

❼ The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

❼ If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

❼ While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in

favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please refer to the theorem statements. We included all proofs in the Appendix.

Guidelines:

❼ The answer NA means that the paper does not include theoretical results.

❼ All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

❼ All assumptions should be clearly stated or referenced in the statement of any theorems.

❼ The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

❼ Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

❼ Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provided all details of our experiments in the Appendix. We also shared our code using an anonymous Github repository. See the footnote on Page 2.

Guidelines:

❼ The answer NA means that the paper does not include experiments.

❼ If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

❼ If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

❼ Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

❼ While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provided all details of our experiments in the Appendix. We also shared our code using an anonymous Github repository. See the footnote on Page 2.

Guidelines:

❼ The answer NA means that paper does not include experiments requiring code.

❼ Please see the NeurIPS code and data submission guidelines ( https://nips. cc/public/guides/CodeSubmissionPolicy ) for more details.

❼ While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

❼ The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

❼ The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

❼ The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

❼ At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

❼ Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provided all details and parameters used in our experiments in the Appendix.

Guidelines:

❼ The answer NA means that the paper does not include experiments.

❼ The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

❼ The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Since our algorithm and benchmark algorithms are deterministic, all experimental results are deterministic once the input data is fixed. We mentioned this in Table 2.

Guidelines:

❼ The answer NA means that the paper does not include experiments.

❼ The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

❼ The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

❼ The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

❼ The assumptions made should be given (e.g., Normally distributed errors).

❼ It should be clear whether the error bar is the standard deviation or the standard error of the mean.

❼ It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

❼ For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

❼ If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We included in the appendix computer resources

Guidelines:

❼ The answer NA means that the paper does not include experiments.

❼ The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

❼ The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

❼ The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: N/A

Guidelines:

❼ The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

❼ If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

❼ The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We included a short paragraph of broader societal impacts of the work at the end of the appendix.

Guidelines:

❼ The answer NA means that there is no societal impact of the work performed.

❼ If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

❼ Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

❼ The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

❼ The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

❼ If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: N/A

Guidelines:

❼ The answer NA means that the paper poses no such risks.

❼ Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

❼ Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

❼ We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cited software packages used in the paper. No licensed data was used in the experiments.

Guidelines:

❼ The answer NA means that the paper does not use existing assets.

❼ The authors should cite the original paper that produced the code package or dataset.

❼ The authors should state which version of the asset is used and, if possible, include a URL.

❼ The name of the license (e.g., CC-BY 4.0) should be included for each asset.

❼ For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

❼ If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

❼ For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

❼ If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Documentation is supplied together with the code.

Guidelines:

❼ The answer NA means that the paper does not release new assets.

❼ Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

❼ The paper should discuss whether and how consent was obtained from people whose asset is used.

❼ At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: N/A

Guidelines:

❼ The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

❼ Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

❼ According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: N/A

Guidelines:

❼ The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

❼ Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

❼ We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

❼ For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a variant of MDS that is able to deal with data with a non-Euclidean structure. The main idea is to generalize the inner product to a bilinear form, hence admitting some relationships corresponding to \"negative eigenvalues,\" interpreted as in an inner product form $u^TAv$. Although a closed-form solution is not possible, the authors optimize over a lower bound. The paper ends with empirical evaluations both on synthetic and real data, with promising results.\n\nSTRENGTHS:\nThe paper is well written in general. The introduction and motivation are very clear. Although most of the bounds and some ideas are already in [38], the idea of including negative eigenvalues and its implementation is interesting.\n\nAnother strength is the availability of the code.\n\nWEAKNESSES:\nThe introduction and problem definition are quite detailed (which is good), but I think that some of these facts may be considered well-known, like basic linear algebra facts. This could be used to include more specific material with the page restriction.\n\nI also think that the experimental section needs a subsection to interpret the embeddings. Throughout the paper (and supplementary material) there are numerical results reported, but the embeddings are never plotted. At least for some toy example.\n\nQUESTIONS:\nI have a couple of questions related to real/complex values.\n\nIn line 99. why real/imaginary-valued dissimilarity when in the next line it reads $D\\in \\mathbb{R}^{n \\times n}$ ?\n\nIn equation (3). Then $X$ is complex (since $\\sqrt{\\Lambda}$ is). Because in the problem formulation you are looking for real embeddings.\n\nAlso related to that. If $X$ is complex, in the equation at the end of that page (page 4), shouldn't it be the complex conjugate instead of only the transpose?\n\nSomehow related to these last two points. Does it make sense to include the sign of the eigenvalue in Diag(w) (and therefore $w$ can have values in $\\{0,1,-1\\}$, and the (squared root) absolute value of the eigenvalues in $\\sqrt{\\Lambda}$?\nSomething like the Generalized RDPG in \"A statistical interpretation of spectral embedding: the generalised random dot product graph\" by Rubin-Delanchy et al.\n\nIn line 303:  \"all dataset are non-metric\", but then in Table 1 there's a column indicating wether the dataset is metric.\n\nMinor comments:\n- In equation (1), use \\left( \\right)\n- Line 259:  \"The we ask\"\n\nLIMITATIONS:\nAlthough there is no specific section for this, some limitations are commented on throughout the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Neuc-MDS, a novel extension of classical Multidimensional Scaling (MDS) designed to handle non-Euclidean and non-metric dissimilarities in datasets. The goal is to create accurate low-dimensional embeddings while minimizing the STRESS (sum of squared pairwise error)\n\nSTRENGTHS:\n1.\tNeuc-MDS extends the concept of the inner product to a broader class of symmetric bilinear forms, allowing the incorporation of both positive and negative eigenvalues from the dissimilarity Gram matrix. This generalization helps capture the underlying structure of non-Euclidean data more effectively than classical MDS, which typically discards negative eigenvalues.\n2.\tNeuc-MDS is specifically designed to handle non-Euclidean and non-metric dissimilarities, making it versatile for a wide range of applications where traditional MDS falls short. The method is backed by a thorough theoretical analysis, providing guarantees for minimizing STRESS. This includes a detailed decomposition of the STRESS error and the demonstration of optimality in eigenvalue selection.\n3.\tNeuc-MDS is capable of working with a variety of dissimilarity measures that are commonly used in practice but are not Euclidean, such as cosine similarity, Hamming distance, and Jaccard index. This broadens its applicability to different fields and types of data.\n\nWEAKNESSES:\n1.\tThe theoretical guarantees provided by Neuc-MDS are based on certain assumptions about the data and dissimilarity matrices. If these conditions are not met in practice, the performance and guarantees may not hold.\n2.\tSome theoretical results rely on properties of random matrices (e.g., Wigner's Semicircle Law). The applicability of these results to structured or real-world datasets, which may not exhibit such random properties, is unclear. The use of Lorenzian distance or other non-standard measures may not satisfy traditional distance properties (e.g., triangle inequality), potentially leading to confusion or misinterpretation in applications that rely on classical distance metrics.\n3.\tThe algorithm involves eigenvalue decomposition and optimization over eigenvalue selections, which can be computationally intensive for large datasets. The scalability of Neuc-MDS, especially for very large datasets, may pose a practical challenge.\n4.\tThe implementation of Neuc-MDS involves several complex steps, including eigenvalue selection and optimization. This complexity may hinder its adoption by practitioners who require straightforward and easily implementable solutions.\n\nQUESTIONS:\n1.\tHow does Neuc-MDS perform on extremely large datasets compared to cMDS and other dimensionality reduction methods in terms of runtime and memory usage? Are there specific strategies or heuristics recommended for choosing the subset of eigenvalues in very large-scale applications to balance between computational efficiency and accuracy?\n2.\tHow do the negative eigenvalues impact the interpretability of the resulting low-dimensional embeddings in practical applications, such as in visualization or clustering? What are the recommended practices for handling and interpreting dissimilarity matrices with negative entries generated by Neuc-MDS in downstream tasks?\n3.\tTo what extent do the theoretical guarantees of Neuc-MDS hold for real-world datasets that do not conform to the random matrix assumptions used in the analysis?\n4.\tHow does Neuc-MDS compare with other non-linear dimensionality reduction methods (e.g., t-SNE, UMAP) in preserving the global and local structure of non-Euclidean datasets?\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The authors introduce Non-Euclidean-MDS (Neuc-MDS), an extension of Multidimensional Scaling (MDS) that accommodates non-Euclidean and non-metric outputs, efficiently optimizes the choice of (both positive and negative) eigenvalues of the dissimilarity Gram matrix to reduce STRESS. The results seem to be promising and the error analysis looks solid.

STRENGTHS:
1. the error analysis seems solid.
2. the topic is important which can extend to non-Euclidean MDS
3. the experiments look promising

WEAKNESSES:
1. The overall writing is not easy to follow, for example from line 119-131, your statement is long, but I am still confused how to construct A. Apparently if A is Identity matrix, it degenerates into traditional one. Your contribution is to say A doesn't need to be Identity or even PSD, but how to construct A remains unknown to me.
2. I don't agree that A can be non-PSD, in which case the inner product of v with itself can be negative, I feel wired.
3. Line 137 you claim X can be recovered from B, however this is not precise. 
4. the formatting can be improved, for example the fontsize Table 3 can be reduced and figure 3 can be better.

QUESTIONS:
What is the advantage of Neuc-MDS over Neuc-MDS+ or the opposite direction? From the experiments, I didn't find difference.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Non-Euclidean Multidimensional Scaling (Neuc-MDS), an extension of classical Multidimensional Scaling (MDS) that can handle non-Euclidean and non-metric data. The key ideas and contributions are:\n\n1. It generalizes the inner product to more general symmetric bilinear forms, allowing the use of both positive and negative eigenvalues of dissimilarity matrices.\n\n2. Neuc-MDS efficiently optimizes the choice of eigenvalues to minimize STRESS (sum of squared pairwise errors). The authors set up the problem as a quadratic integer program but show that it has an optimal solution and that it can be found in a greedy fashion.\n\n3. The authors provide theoretical analysis of the error and prove optimality in minimizing lower bounds of STRESS.\n\n4. They introduce two algorithms: Neuc-MDS and an advanced version Neuc-MDS+.\n\n5. Theoretical analysis is provided for the asymptotic behavior of classical MDS and Neuc-MDS on random symmetric matrices.\n\n6. Experimental results on 10 diverse datasets show that Neuc-MDS and Neuc-MDS+ outperform previous methods on STRESS and average distortion metrics.\n\n7. The proposed methods resolve the \"dimensionality paradox\" issue of classical MDS, where increasing dimensions can lead to worse performance.\n\n8. The approach is applicable to both image and text data, and can handle various non-Euclidean dissimilarity measures.\n\n9. The paper provides detailed proofs, algorithm descriptions, and experimental results in the appendices.\n\nOverall, this work extends MDS to non-Euclidean spaces, providing both theoretical guarantees and practical improvements over existing methods for dimension reduction and data embedding tasks.\n\nSTRENGTHS:\nThis paper is well-written and it clearly outlines the problem and the solution. The analysis is non-trivial and the experiments are thorough.\n\nWEAKNESSES:\nWhile the paper presents a novel approach with several strengths, there are a few potential weaknesses or limitations that can be identified:\n\n1. Computational complexity: For large datasets, the method may still be computationally intensive, as it requires eigendecomposition of the full dissimilarity matrix. While the authors mention the possibility of using approximation algorithms for partial SVD, they don't provide detailed analysis or experiments on very large-scale datasets.\n\n2. Limited comparison with non-linear methods: The paper primarily compares Neuc-MDS with classical MDS and other linear dimension reduction techniques. A comparison with popular non-linear methods like t-SNE or UMAP could provide more context on its performance relative to the state-of-the-art in dimension reduction.\n\n3. Interpretability: The use of general bilinear forms, while mathematically elegant, may make the resulting embeddings less interpretable compared to Euclidean embeddings, especially for domain experts not familiar with non-Euclidean geometries.\n\n4. Downstream tasks: The paper focuses on the quality of the embedding itself (via STRESS and distortion metrics) but doesn't extensively explore how these embeddings perform in downstream machine learning tasks compared to other methods.\n\n5. Negative distances: The method can produce negative distances, which may be problematic for some applications or require special handling in downstream tasks. While this is mentioned, strategies for dealing with negative distances are not fully explored nor is the impact on \"real\" data sets discussed.\n\n6. A discussion of just how non-Euclidean \"standard\" data sets are would be useful. Is this problem of non-Euclidean distances a substantial one or not? \n\n7. This brings me to a more detailed discussion of the experiments: I'd be interested in seeing a quasi-synthetic example of a real data set with non-Euclidean distances used (say, graph distances from a nearest neighbor graph). What impact does that make on the results?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nnot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-25"}