Understanding Dimensionality Reduction for Machine Learning | by Rajat S | Towards Data Science

Recently, I was asked to work on a dataset that was a bit on the heavy side. It was so large that my Excel program would stop responding for a few minutes while loading, and manually exploring the data started to become a real pain.
I am still a newbie in the Machine Learning world. So, I had to go back the Excel multiple times to decide which features should be used in the Machine Learning model.
But after an excruciatingly long and mind wrecking time, I was able to build a model that would give me some decent results! I started to feel really good about myself and started to believe that I could make it as a Machine Learning Engineer!
But then I came across my next obstacle: How do I show my results to ordinary people?
I could make two columns where one would show the real-world results and the other column would show the results given to me by the ML model. But this is not the most visually appealing way to go. While machines are better than us at understanding large sets of numeric data, the human mind finds it easier to understand visual data like graphs and plots.
So a plot is the best way to go forward. But our screen is two dimensional and our data can have more than two features where each feature can be considered as a dimension. One of the two dimensions of the plot is going to be the output. So the question we should ask ourselves is: How do we represent all of our features, which could be anywhere from hundreds to thousands, in a single dimension?
This is where dimensionality reduction comes into play!
Dimensionality Reduction is a technique in Machine Learning that reduces the number of features in your dataset. The great thing about dimensionality reduction is that it does not negatively affect your machine learning model’s performance. In some cases, this technique has even increased the accuracy of the model.
By reducing the number of features in our dataset, we are also reducing the storage space required to store the data, our python compiler will need less time to go through the dataset.
In this post, we will take a look at two of the most popular types of Dimensionality Reduction techniques, Principle Component Analysis (PCA) and Linear Discriminant Analysis (LDA).
Before we move further, let’s make sure that your system is ready to work with some python code.
There two major versions of Python: 2.x and 3.x. I would suggest that you make sure that you have the 3.x version installed on your system. If that is not the case, then you can get it from here.
Next, we need a good code editor that will help make your coding experience easier and enjoyable. My personal choice is the amazing VSCode, but you can also try out Spyder or Jupyter Notebook. There are also a couple of online code editors like Repl and Google Colab that you can try. Using online editor would let you skip the time and storage wasting task of installing Python Libraries.
We will be working on the Wine Dataset. This dataset is completely numeric and it does not contain any missing values, making it perfect for this article as I won’t have to waste time on Data-Preprocessing techniques such as converting categorical data to numeric data and imputing data into empty cells.
Let’s start by importing some Python Libraries that we will need in our script:
import pandas as pdfrom sklearn.preprocessing import StandardScaler
We also need to load the wine dataset into our python script. Let’s do that using the read_csv() function from the pandas library:
dataset_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv'dataset = pd.read_csv(dataset_url, sep=';')
The dataset looks something like this:
The quality column will be the dependent variable of our Machine Learning Model, and the rest of the columns will be the independent variables. So let’s split the dataset into two parts accordingly:
X = dataset.iloc[:, 0:-1].valuesy = dataset.iloc[:, -1].values
X contains 11 columns, and y the 12th column.
We also need to perform some feature scaling on the data. We are using the StandardScalar function to do this for us. This function will transform our data in such a way that the data’s distribution will have a mean of 0 and a standard deviation of 1. We will perform the feature scaling on the independent variables since we want our dependent variable to stay as pristine as possible.
sc = StandardScaler()X = sc.fit_transform(X)
We are now ready to perform dimensionality reduction! Let’s start with PCA first, and then perform LDA. I would suggest to make a new python script file and copy your code into it, and use it later with the LDA part.
Principal Component Analysis (PCA) is one of the most popular dimensionality reduction algorithms out there and can also be used for noise filtering, stock market predictions, data visualization, and much more.
PCA reduces the number of features in the dataset by detecting the correlation between them. When the correlation between the features is strong enough, they will get merged together and form a single feature.
We could take a look at the Math behind the algorithm and try to understand how it works, but I found it easier to get into the code and see what outputs we get.
Picking up from where we left our code in the previous section, let’s start by import the PCA class from sklearn:
from sklearn.decomposition import PCA
Next, we need to create a local instance of the PCA class.
pca = PCA(n_components=None)
We are passing a parameter called n_components which we will intially pass a value of None.
Next, we will call the fit_transform method on pca and pass it the X_train dataset as input. We will also call the transform method and pass the X_test dataset as input
X = pca.fit_transform(X)
Once this is done, we need to get the explained variance of different principal components of the dataset. Our pca object has a property called explained_variance_ratio_ that contains an array of percentages.
variances = pca.explained_variance_ratio_
If you print this variable, you will get an array similar to this:
[0.28173931 0.1750827  0.1409585  0.11029387 0.08720837 0.05996439  0.05307193 0.03845061 0.0313311  0.01648483 0.00541439]
We can see that the elements of this array are in descending order. But what do these values mean? Each of the values explains the amount of variance given by each of the 11 principal components in our dataset.
So if only take the first principal component, we will have 28.17% variance. If we take two principal components, we will have the sum of the first two elements of the array, which is around 45.68% variance, and so on.
Now we can our Python compiler, and replace the None value passed to n_components inside PCA with the desired number of dimensions. Let’s see what we get by selecting 1 as the desired number of dimensions
pca = PCA(n_components=2)X = pca.fit_transform(X)// Output[[-1.61952988]   [-0.79916993]   [-0.74847909]   ...   [-1.45612897]   [-2.27051793]   [-0.42697475]]
Voila! We have successfully reduced the number of dimensions of our input dataset from 11 to just 1!
Similar to PCA, LDA is also a dimensionality reduction algorithm. But unlike PCA, LDA will also find the features that maximize the separation between multiple classes.
You might ask yourself: why should I bother with this new algorithm that basically does the same thing as the previous algorithm?
Both PCA and LDA are dimensionality reduction algorithms. But where PCA is considered to be an unsupervised algorithm, LDA, on the other hand, is considered to be supervised.
Supervised and Unsupervised? These are common machine learning terms that define whether an algorithm uses the dependent variable or not, respectively. So LDA uses the dependent variable, and PCA, as we have seen before, does not.
Let’s get to some code and understand how to implement the LDA algorithm in our python code.
If you have not already done so, then take the code from the getting started section and paste it in a new python script file. Then, import the LinearDiscriminantAnalysis class from sklearn.
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
Next, we create a local object called lda from the LinearDiscriminantAnalysis. We will also pass it a parameter called n_components. Now we could do the whole thing where we find the percentage of variance introduced by each principal component. But I am just going to skip that part and set the n_components parameter to the value of 1.
lda = LinearDiscriminantAnalysis(n_component=1)
One last thing to do! Call the fit method on lda and pass it the X and y datasets as input, and then chain a transform method with X as its input.
X = lda.fit(X,y).transform(X)// Output[[-1.51304437]   [-1.28152255]   [-1.11875163]   ...   [ 0.71637389]   [-0.32030997]   [ 0.8645305 ]]
Tada! 🎉 We have now successfully reduced the dimensions of X to 1!
Thank you for reading this post on dimensionality reduction. I hope that it helped you even a little bit on your journey in the machine learning. Dimensionality Reduction Algorithms are truly amazing and really helpful not just for visualizing your results, but also to help reduce processing time and the strain on your system by preventing it from reading large datasets.