The Multidimensional Scaling (MDS) algorithm for dimensionality reduction
By: Joseph Imperial
Tags: Machine Learning, Dimensionality Reduction, Data Science, Algorithms, Feature Selection

Photo by Aditya Chinchure on Unsplash

Introduction

In this article, we’ll talk about the dimensionality reduction in general, its importance with the use and growth of large sources of data (ex. Big Data), and discuss one of the oldest dimensionality reduction techniques that have been used by researches of various fields (Education, Psychology, Finance, etc) for representing data in a lower dimension space — the Multidimensional Scaling algorithm.

With bigger data comes the need for a lower representation

With the advent of the use of data-driven technologies, massive amounts of very high-dimensional (with a large number of features) or unstructured data are continuously produced and stored with much cheaper cost than they used to be [1]. As people, industries, and research works are becoming more and more data-driven, so does the need for its interpretation and analysis. As such, a significant increase in the number of organizations has invested in developing products using utilizing data to address their monitoring, experimentation, data analysis, simulations, and other knowledge and business needs.

The problem with the rise of real-world data comes from the method of how to represent them. By representing them, we mean the selection of top, descriptive features as representations of the data itself. Since real-world data is usually highly unstructured and sometimes collected in extremely large quantities, the extraction of features or properties of the data for analysis and research poses a big and complex challenge for all researchers and industries using data as primary source fuel. Thus, in terms of statistical accuracy, dimension reduction and variables selection play pivotal roles in analyzing high dimensional data.

Dimensionality reduction in a nutshell

Dimensionality reduction or dimension reduction is the process of reducing the number of random variables under consideration by obtaining a set of principal variables [2].

The aim of dimension reduction procedures is to summarize the original p-dimensional data space in the form of lower k-dimensional components subspace. To achieve this goal, statistical and mathematical theory provided many approaches like Principal component analysis (PCA), Linear discriminant analysis (LDA), Factor Analysis, etc.

The MDS algorithm

Multidimensional scaling, also known as Principal Coordinates Analysis (PCoA), Torgerson Scaling or Torgerson–Gower scaling, is a statistical technique originating in psychometrics. The data used for multidimensional scaling (MDS) are dissimilarities between pairs of objects.

The main objective of MDS is to represent these dissimilarities as distances between points in a low dimensional space such that the distances correspond as closely as possible to the dissimilarities.

The vanilla or classical MDS

The classical, non-revised version of MDS originated in the 1930s when Young and Householder [5] showed how starting with a matrix of distances between points in a Euclidean space, coordinates for the points can be found such that distances are preserved. There exist, however, a number of non-Euclidean distance measures, which are limited to very specific research questions [5]. Eventually, Torgerson brought the subject to popularity using the technique for scaling. Thus, giving birth to multidimensional scaling as dimensionality reduction and visualization technique for high-dimensional data. The algorithm for the classical version of multidimensional scaling is described below:

TABLE I. THE CLASSICAL MULTIDIMENSIONAL SCALING ALGORITHM

As shown in the algorithm, a Euclidean space of, at most, n-1 dimensions could be found so that distances in the space equaled original dissimilarities. Usually, matrix B used in the procedure will be of rank n-1 and so the full n-1 dimensions are needed in the space, and hence little has been gained in dimension reduction of the data.

In addition, Gower [7] was the first to state clearly the formulation and the importance of the classical scaling technique, and from this selection of the first p “principal coordinates” for the configuration, he coined the name “principal coordinates analysis” (PCA). Principal coordinates analysis is now synonymous with classical multidimensional scaling, as also is the term metric scaling. However metric scaling encompasses more than this one technique.

The spectral decomposition of the matrix B, the distances between the points in the n-1 dimensional Euclidean space are given by the equation below.

Decomposition equation for matrix B

Hence, if many of the eigenvalues are “small”, then their contribution to the squared distance d2rs can be neglected. If only p eigenvalues are retained (in relation to the target number of projected dimensions) as being significantly large, then the p dimensional Euclidean space formed for the first p eigenvalues and with xr truncated to the first p elements can be used to represent the objects in the lower dimension for interpretation.

In addition to this, in classical scaling, the negative eigenvalues (and its eigenvectors) are simply ignored as an error. Hopefully, p will be small, preferably 2 or 3, for ease of graphical representation [8].

Data for MDS

The data used for MDS analyses is usually coined as ‘proximities’. Proximities indicate the overall similarity (or dissimilarity) of the elements in the data. MDS will look for a spatial configuration of the elements so that the distances between the elements match their proximities as closely as possible. Often the data are arranged in a square matrix — the proximity matrix. There are two major groups of methods for deriving proximities: direct, and indirect methods.

For direct methods, subjects might either assign a numerical similarity or dissimilarity value to each pair of objects or provide a ranking of the pairs with respect to their similarity or dissimilarity. Both approaches are direct methods of collecting proximity data.

Indirect methods for proximity data do not require that a subject assigns a numerical value to the elements of the proximity matrix directly. Rather, the proximity matrix itself is derived from other measures [9]. An example of this is data from confusion matrices or from correlation matrices.

However, unlike univariate statistical methods, the outcome of an MDS analysis is more dependent on the decisions that are taken beforehand. According to Wickelmaier [9], the data-collection stage one should be aware that asking for similarity rather than for dissimilarity ratings might affect the results. For example, a similarity judgment cannot simply be regarded as the “inverse” of a dissimilarity judgment. Further, a decision between direct and indirect methods, and symmetric versus asymmetric methods of data collection, respectively, have to be taken.

Example using MDS

The purpose of a multidimensional scaling analysis is communicated most easily by an example or two. One example by Ekman [4] in the field of Psychology collected data to study the perception of fourteen different colors. Every pair of colors was judged by a respondent from having ‘no similarity’ to being ‘identical’. The obtained scores can be scaled in such a way that identical colors are denoted by 0, and completely different colors by 1. The averages of these dissimilarity scores over the 31 respondents are presented in the table below.

TABLE II. DISSIMILARITIES OF COLORS WITH WAVELENGTHS

Starting from wavelength 434, the colors range from bluish-purple, blue, green, yellow, to red. In this example, the dissimilarities are symmetric. Likewise, the extent to which colors with wavelengths 490 and 584 are the same is equal to that of colors 584 and 490. Therefore, it suffices to only present the lower triangular part of the data in the table. Also, the diagonal is not of interest in MDS because the distance of an object with itself is necessarily zero.

Figure 1. Transformed two-dimensional matrix from Table I data

As stated in the definition, MDS tries to represent the dissimilarities in Table I in a map to make it more interpretable and easier for analysis of correlations and such. Figure 1 presents such an MDS map in 2 dimensions. The colors, denoted by their wavelengths, are represented in the shape of a circle. The interpretation of this flattened map should be done in terms of the depicted inter-point distances.

Note that, as distances do not change under rotation, a rotation of the plot does not affect the interpretation.

Similarly, a translation of the solution (that is, a shift of all coordinates by a fixed value per dimension) does not change the distances either, nor does a reflection of one or both axes.

Figure 2. Generated map of British cities using MDS

Another example by Cox and Cox[10] used road distances between some British cities. For illustration, the journey times by road between twelve British cities were subjected to classical multidimensional scaling. Figure 2 above shows the configuration of points produced by the technique. There is a striking similarity between the positions of the points representing the cities and the positions of the same cities seen in a geographical map of Great Britain, except of course the cities in Figure 2 appear to be reflected about a line and rotated from the geographical map usually presented in an atlas.

Can MDS be used as a visualization technique?

It is said that it is easy to observe empirically that many of the manifold learning methods (like MDS) are not good for data visualization. The reason is that they have been designed to find a d-dimensional manifold if the inherent dimensionality of data is d. For visualization, the display needs to have d = 2 or d = 3; that is, the dimensionality may need to be reduced beyond the inherent dimensionality of data.

According to Kaski and Jaakko [10], it is well-known that a high-dimensional data set cannot, in general, be faithfully represented in a lower-dimensional space, such as the plane with d = 2. Hence a visualization method needs to choose what kinds of errors to make. The choice naturally should depend on the visualization goal; it turns out that under a specific but general goal the choice can be expressed as an interesting tradeoff as shown below.

Figure 3. Two kinds of errors when projecting to a lower-dimensional space

According to Kaski, when the task is to visualize which data points are similar, the visualization can have two kinds of errors as shown in the figure above, it can miss some similarities (it can place similar points far apart as false negatives) or it can bring dissimilar data points to close together as false positives.

Where should MDS be applied?

According to Young [8], multidimensional scaling may be applied (and misapplied) to a very wide range of types of data. Technically, any matrix of some raw or transformed data is a candidate for analysis by some type of multidimensional scaling method, if the elements of the data matrix indicate that strength or degree of relation between the objects or events represented by the rows and columns of the data matrix. We call such data ‘relational’ data with examples such as correlations, distances, proximities, similarities, multiple rating scales, preference matrices, etc.

Multidimensional scaling methods have been designed for all types of relational data matrices, including symmetric and asymmetric matrices, rectangular and squares matrices, matrices with or without missing elements, equally and unequally replicated data matrices, two-way and multi-way matrices, and other types of matrices.

References:

[1] Jianqing, F., Fang, H., Han L. (2014). Challenges of Big Data analysis, National Science Review, Volume 1, Issue 2, Pages 293–314, https://doi.org/10.1093/nsr/nwt032

[2] Roweis, S. T.; Saul, L. K. (2000). “Nonlinear Dimensionality Reduction by Locally Linear Embedding”. Science. 290 (5500): 2323–2326. Bibcode:2000Sci…290.2323R. CiteSeerX 10.1.1.111.3313. DOI:10.1126/science.290.5500.2323. PMID 11125150.

[3] Borg, I., & Groenen, P. J. (2010). Modern multidimensional scaling: Theory and applications. New York, NY: Springer New York.

[4] Ekman, G. (1954). Dimensions of color vision. Journal of Psychology, 38, 467–474

[5] Young, G. and Householder, A.S. (1938) Discussion of a set of points in terms of their mutual distances. Psychometrika, 3, 19–22.

[6] Borg, I., & Groenen, P. (1997). Modern multidimensional scaling: theory and applications. New York: Springer.

[7] Gower, J.C. (1966) Some distance properties of latent root and vector methods in multivariate analysis. Biometrika, 53, 325–338.

[8] Young, F. W. (2017). Multidimensional Scaling: History, theory, and applications. S.l.: Psychology Press.

[9] Wickelmaier, F. (2003). An Introduction to MDS.

[10] Kaski, S., & Jaakko, P. (2011, February 17). Dimensionality Reduction for Data Visualization. Dimensionality Reduction for Data Visualization, 28(2). doi:10.1109/MSP.2010.940003

[11] Cox, T. F., & Cox, M. A. (1994). Multidimensional scaling. London: Chapman & Hall.