The Curse of Dimensionality and its Cure
By: Nikhil Raj
Tags: Feature Engineering, Dimensionality Reduction, Python, Data Science, Machine Learning

Avada Kedavra (aka The Killing Curse) may be the most deadly curse of all but in this Muggle world, we have to deal with something much worse than that.

The Curse of Dimensionality

There is a particular species of Muggles also known as Data Scientists, who have to deal with this curse in their day to day job.

(Okay, so enough with the harry potter references and let's get right into business.)

The curse of dimensionality is a problem that arises when we are working with a lot of data having multiple features or we can say it as high dimensional data. The dimension of the data means the no. of features or columns in our dataset.

The problems that we face while dealing with high-dimensional data is that it becomes very challenging to identify meaningful patterns while analyzing and visualizing the data and it also degrades the Machine Learning model’s accuracy while decreasing the computation speed as well, i.e. training the model will become much slower as the dimensions increase.

With the increase in dimensions, there are more chances for the occurrence of multicollinearity as well.

Suppose we have 4 data points in one dimension(only one feature in the data set). So, it can be easily represented with the help of a line and the dimension space is equal to 4.

Now if we add one more feature, then this will cause an increase in dimension space to 4*4 =16. Again if we add one more feature to it, dimension space will increase to 4*4*4 = 64, and so on (4 dimensions(4*4*4*4=256) etc.). So as the dimensions keep on increasing, dimensions space increases exponentially.

Hughes Phenomenon

This phenomenon states that “with a fixed number of training samples, the average (expected) predictive power of a classifier or regressor first increases as the number of dimensions or features used is increased but beyond a certain dimensionality it starts deteriorating instead of improving steadily”.

So, with the increase in the no. of dimensions, it becomes very cumbersome to calculate distance between observations as well, and hence, all the Machine Learning algorithm that relies on calculating distance between observations find it very troublesome to work with high dimensional data, such as segmentation and clustering algorithms like KNN, K-Means etc.

So as we now know that, high dimensional data is responsible for the curse of dimensionality, but why do we have such a huge no. of dimensions in our data in the first place.

Generally, in the data gathering or extraction phase, we collect data from a lot of all possible places and combine it, which results in a huge no. of features in our dataset. So, to solve real-world business problems we generally require a lot of information and hence more features are present in the data.

We usually have categorical features in our dataset as well, but a lot of Machine Learning models don’t work well with categorical data. So, what we do is use feature encoding technique(conversion of a categorical variable to numerical features) such as one-hot encoding, which creates a lot of dummy variables for each category, i.e it increases the number of dimensions.

As we know that Avada Kedavra is an instantaneous kill curse and the only thing that can save you from the curse is the resurrection stone, and so here in the muggle world, data scientists have come up with a cure for the curse of dimensionality as well.

(Okay, I promise, this was the last harry potter reference in this article)

CURE

Dimensionality Reduction

Dimensionality reduction is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data, ideally close to its natural dimension or in simple words, it means reducing the dimensions of our data set. (Wikipedia reference)

Advantages of Dimensionality Reduction:

It reduces the dimensions of our dataset, and hence it reduces storage space. It reduces computation time as less no. of dimensions mean less computing is required which means that algorithms train faster than compared to before. Multicollinearity is reduced. Models accuracy may improve as well. It helps in visualizing data and identifying a meaningful pattern in our dataset more easily as visualizing in a 1-D, 2-D or 3-D space is much easier than compared to more no. of dimensions.

Dimensionality Reduction can be divided into two types, Feature Selection and Feature Extraction.

Now, let’s briefly explore a few of the Dimensionality Reduction techniques.

First, Feature Selection is selecting only the important features or eliminating the less useful features from our dataset.

Some of the Feature selection methods are –

Remove those feature which has a huge no. of missing values (more than 50–60%) because it will not have sufficient information to provide us. We can set a particular threshold value, and if the missing values are more than this threshold then we will remove all those features.

(more than 50–60%) because it will not have sufficient information to provide us. We can set a particular threshold value, and if the missing values are more than this threshold then we will remove all those features. Remove the features having a high correlation. A high correlation between 2 variables means that both are carrying similar information and dropping either of the two will not have that much impact on our outcome. We can find the correlation of our data frame using the corr() function of pandas. And we can set a threshold, and if the correlation coefficient crosses that threshold, we can drop it.

A high correlation between 2 variables means that both are carrying similar information and dropping either of the two will not have that much impact on our outcome. We can find the correlation of our data frame using the corr() function of pandas. And we can set a threshold, and if the correlation coefficient crosses that threshold, we can drop it. Feature Selection using Random Forrest — Random Forrest has an inbuilt feature of providing feature importance metric but before that, we need to convert our data into numerical form as Random Forrest takes only numeric inputs. (Note- Drop the ID variables and Target variables as well)

We will import Random Forrest Regressor from the scikit-learn library. And, then we will fit our data frame in the model. And, after that with the help of the “feature_importances_” attribute, we can find the importance of the features and plot them for visualization.

Now, we have plotted the top 15 features having the highest importance out of all the features.

And, then we can remove the features whose importance is not that significant.

(Note — While working with Feature Selection, please do keep domain knowledge in mind, as these techniques are prone to loss of information.)

Now we will look into some of the Feature Extraction methods as well –

1. PCA — Principal Component Analysis

PCA is a linear dimensionality reduction algorithm that helps us in extracting a new set of variables from an existing large set of variables which are called Principal Components.

Principal components are extracted in such a way that the first principal component explains maximum variance in the dataset, then the Second principal component tries to explain the remaining variance in the dataset and is uncorrelated to the first and then the Third principal component tries to explain the variance which is not explained by the first two principal components and so on.

So basically, what PCA does, it finds the best linear combinations of the original variables so that the variance or spread along the new variable is maximum.

The above curve quantifies how much of the total variance is contained within the first N components. Above, we can see that with the digits the first 10 components contain approximately 75% of the total variance, while we need around 20–25 components to describe close to 90% of the variance.

To observe the distribution of components, let’s use the iris dataset from the sklearn library. We will then import PCA from the sklearn library, and let’s chose 3 components and then fit and transform the data set. After that, we will plot the components using matplotlib’s scatter plot.

2. (t-SNE) t-Distributed Stochastic Neighbor Embedding

It is a non-linear dimensionality reduction algorithm used for exploring high-dimensional data. It finds patterns in the data based on the similarity of data points with features. The similarity of points is calculated as the conditional probability that a point A would choose point B as its neighbour.

It then tries to minimize the difference between these conditional probabilities in high-dimensional space and low-dimensional space for a perfect representation of data points in lower-dimensional space.

Now, to visualize the components let’s import TSNE from the sklearn library and fit the data. TSNE has its fair share of advantages over other methods, out of which the hyperparameters are one of the most important ones. We can fine-tune it to get the most benefit out of it.

As we can observe that t-SNE outputs provide better results than PCA and other linear dimensionality reduction models as well. This is because a linear method such as classical scaling is not good at modelling curved manifolds.

Some of the disadvantages of Dimensionality Reduction are that it may lead to some amount of data loss and sometimes the results are not always the best for visualization which makes it difficult to interpret.

So we have covered the concept of the curse of dimensionality and some of the methods of dimensionality reduction.

Thanks for Reading, and Keep Learning.

And if you found this article helpful, then please follow me on LinkedIn.

Evanesco