Reducing Confusion about Dimensionality Reduction
By: Kevin Macias-Matsuura
Tags: Principal Component, Projection, Dimensionality Reduction

Consider the following sentence: “While you are out, I need you to go to the store and purchase some apples”. How much can you reduce this sentence while still maintaining its original meaning? One way would be to rewrite it as: “Go to the store and purchase apples”. Our original sentence contains 16 words while our rewritten sentence contains 7 (quite the Hemmingwayesque reduction). Rephrasing our sentence in this way still gets the point across. We do not necessarily need the dependent clause “While you are out” as you can’t purchase apples inside (unless that inside is a store) and we know from the sentence we need to go to the store. We also don’t need “I need you” because it can be inferred that the “you” is the person that is being spoken to. Additionally “some” is indefinite so we don’t really gain much information by leaving it in. If we really wanted to push it we could, arguably, take a note from the shortest line in the King James Bible (Jesus wept) and simply state: “Buy apples”.

The idea of dimensionality reduction is, in some ways, very similar to our sentence reduction example. Dimensionality reduction seeks to reduce the complexity and noise of your data while retaining the relationships we are trying to capture. Below are some simple pros and cons to consider before jumping into more detail:

Pros:

Speeds up training performance

In certain cases, can reduce noise

Able to plot info for data visualizations (2-D and 3-D)

Can reduce overfitting (high-dimensional datasets are sparse causing training to lead to overfitting) Cons:

Reducing dimensionality causes loss of information (like in compressing a FLAC audio file to MP3)

Makes pipelines more complex and harder to maintain

Can make your models/system under-perform due to information loss

When working with datasets containing thousands or even millions of features and observations the number of dimensions can become staggeringly large. Simply increasing the density of observations, given a high number of features, to fill in the sparseness of high-dimensional datasets seems like a fix but is a dangerous game as the number of features can easily surpass the number of observable atoms in the universe and our computer systems are simply not capable of handling that type of load. When dealing in the higher dimensions, the human brain has an increasingly difficult time comprehending and visualizing the shape and relationships between points. Take the four dimensional object of a tesseract in the picture below for example.

Taken from wikipedia (link)

At a point, the tesseract looks like small cube inside a larger cube connected to each other by a trapezoid. In reality, “the hypersurface of the tesseract consists of eight cubical cells” of the same shape (paragraph 1). Our brain and eyes have a difficult time understanding an object like this because we are so used to living and seeing in the three dimensions. What if we consider not just 4-Dimensions but 10, 100, or even thousands? The aforementioned reasons are why we need techniques like dimensionality reduction.

One way we implement dimensionality reduction is using the technique of “projection”. This involves projecting points in a higher-dimensional space onto a lower dimensional plane. Take a minute to observe the image below.

Given the 3D space, the crosses mark the data points projected onto each flat 2D plane. For example, simply imagine the points as marbles falling onto the floor and landing on each black cross (only they stay where they land and don’t bounce or roll around). While not the same graph, by reducing the dimensions our 3D plane might end up looking something like the graph below.

Provided by wikipedia (link)

When reducing dimensions we need to be aware of which dimensions we remove. By removing different dimensions we can get vastly different results depending on how the points are distributed. A commonly used example of this is the “Swiss Roll” dataset.

Dinoj Surendran (link)

As you can see, depending on how we choose to “unroll” the data onto a 2D plane, the result can be very different. In most cases, lowering the dimensions will give you a simpler interpretation and decision boundary. Take the example from Aurélien Géron’s book Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems (2nd Edition). The 3D plot in the top-left corner would have a complex decision boundary but when it is unrolled onto a 2D plane, like in the top-right, the decision boundary becomes much more clear.

To reiterate, this is valid in most cases, but depending on your dataset, it may also be the case that plotting in a higher dimension can give you more insight and a better decision boundary. Take the bottom row of plots from the graphs above. The bottom-left 3D plot has a clear decision boundary at X1 = 5. The same data unrolled in the bottom-right 2D graph becomes more complicated as the green data points no longer have a definite pattern to delineate the decision boundary.

The most popular way we implement dimensionality reduction is called the Principal Component Analysis (PCA). This technique incorporates and expands upon projection in order to plot the correlations among your predictors onto a 2-D graph. This process requires a whole other blog post but a simple summary here will suffice. In PCA principal components along with their corresponding loadings give us a sense of which combination of predictors explain the larger parts of our variance. This can give us some idea about which predictors might be valuable in explaining the data and which might not.

While this post does not give a deep-dive into the math or application of dimensionality reduction techniques, understanding what dimensionality reduction does is the first step in understanding what it can be used for. Working with multiple dimensions can be mind-blowingly complicated and I encourage you to take take some time to build some intuition and basic understanding before jumping head first into the multi-dimensional pool.