Dimensionality Reduction Approaches | by Prerna Singh | Towards Data Science

The full explosion of big data has persuaded us that there is more to it. While it is true, of course, that a large amount of training data allows the machine learning model to learn more rules and generalize better to new data, it is also true that an indiscriminate introduction of low-quality data and input features may introduce too much noise and at the same time slow down the training considerably. And, in the presence of a dataset with a very large number of columns of data, it is good practice to look at how many of these data features are really useful for the model.
In machine learning, we tend to add as many features as possible at first, to grab useful indicators and get a more accurate result. Nevertheless, the model’s output will decrease after a certain level, with the rising number of elements. This phenomenon is often called the “Curse of Dimensionality.”
The curse of dimensionality exists because the density of the sample decreases exponentially as the dimensionality increases. If we continue to add features without also increasing the number of training samples, the dimensionality of the feature space will expand and become sparser and sparser. Because of this sparsity, it becomes much simpler for the machine learning model to find the right solution which is highly likely to lead to overfitting. Overfitting occurs when the model is too close to a particular set of data and is not generalizing well.
However, can we overcome the curse of dimensionality and avoid overfitting particularly when we have a lot of features and relatively few samples of training? One common approach is that of reducing dimensionality in feature space. Reduction of dimensionality is the method of reducing with consideration the dimensionality of the function space by obtaining a collection of principal features. Reduction of the dimensionality can be further divided into a collection of features and extraction of features.
The selection of features tries to pick a subset of the original features to be used in the machine learning model. In this way, we could delete redundant and obsolete characteristics without incurring much information loss. The extraction of a feature is also called the projection of a feature. Although the selection of features returns a subset of the original features, the extraction of features creates new features by projecting the data to a space of lesser dimensions in the high-dimensional space. Informative and non-redundant functionality can also be extracted from this method.
We can use the selection of features and extraction of features together. In addition to reducing overfitting and redundancy, the reduction in dimensionality also leads to better human interpretations and lower computational costs with model simplification.
It reduces the time and storage space required.The removal of multicollinearity improves the interpretation of the parameters of the machine learning model.It becomes easier to visualize the data when reduced to very low dimensions such as 2D or 3D.Reduce space complexity.More interpretable because it removes noise thus provides a simpler explanation.To mitigate “curse of dimensionality”.
It reduces the time and storage space required.
The removal of multicollinearity improves the interpretation of the parameters of the machine learning model.
It becomes easier to visualize the data when reduced to very low dimensions such as 2D or 3D.
Reduce space complexity.
More interpretable because it removes noise thus provides a simpler explanation.
To mitigate “curse of dimensionality”.
We discussed the benefits of dimension reduction and provided an overview of dimension reduction requirements. We will now discuss in detail the two key techniques for dimension reduction i.e.
PCA (Principal Component Analysis)LDA (Linear Discriminant Analysis)
PCA (Principal Component Analysis)
LDA (Linear Discriminant Analysis)
For example, references, Wine dataset from the UCI Machine Learning Repository would be used. This dataset, which is fairly small, has six target class and eleven-dimensional feature set (i.e. eleven different features like fixed acidity, pH value, alcohol content, and so on to predict the quality of wine).
Datasets can be downloaded following the link below.https://github.com/psi49/Revisiting_MachineLearning/blob/master/winequality_red.csv
Principal Component Analysis (PCA) is an unsupervised learning algorithm as it ignores the class labels (the so-called principal components) that maximize the variance in a dataset, to find the directions. In other words, PCA is basically a summarization of data. For example, to obtain quality/type of wine, we can use different features of wine such as its pH value, alcohol content, the color of the wine, acidity content and so on, however, many of these features will be redundant or dummy feature variables (can be derived from other features), therefore causing to train the model on unnecessary features. In short, we can get the type of wine with fewer feature variables and this is what actually PCA does inside the box.
Note that, PCA does not select a set of features and discard other features, but it infers some new features, which best describe the type of class (in our case — type of wine) from the existing features.
Previously, while deriving a formal definition, we came up with a phrase — maximize the variance in a dataset. Now the question arises — what the word ‘variance’ has to do with PCA? Remember, our main task is to define a feature set that distinguishes one type of wine from another. Imagine that you land up with a set of features that are unable to distinguish the types of wine, therefore, these sets of features are useless. This type of dimensionality reduction will deteriorate your model accuracy and, in cases will lead to under-fitting of data. Therefore, PCA looks for properties that show as much variation across datasets as possible.
PCA works on eigenvectors and eigenvalues of the covariance matrix, which is the equivalent of fitting those straight, principal-component lines to the variance of the data. Why? Because PCA determines the lines of variance in the dataset which are called principal components with the first principal component having the maximum variance, the second principal component having a second maximum variance, and so on.
Now let’s jump to the implementation of PCA using sklearn on the example of the wine dataset and understand how the data is transformed from a higher dimensionality to lower dimensionality.
Let’s perform the PCA on wine dataset and analyze by visual representation:
import numpy as npimport pandas as pddf=pd.read_csv(‘winequality_red.csv’)def isQuality(quality):if quality > 6:     return 1if (quality >= 5) and (quality <= 6):     return 2else:   return 0
The given data has six different quality types. For the sake of understanding the dimensionality reduction concept. The quality of the wine is grouped into three target classes if you like you can work on six target classes as well.
Lets used the above function to convert into three wine quality classes.
df[‘isQuality’] = df[‘quality’].apply(isQuality)print(‘New classes are defined for the quality of wines:\n’,df[‘isQuality’].value_counts())
New classes are defined for the quality of wines:
Class 2:1319
Class 1: 217
Class 0: 63
Drop the quality column from the dataset as we have created a new column with three wine quality classes.
df=df.drop(‘quality’, axis=1)
Divide data into features and target class.
feature=df.iloc[:,0:11]target=df[‘isQuality’]print(feature.shape)print(target.shape)
The shape of the features and target column.
(1599, 11)
(1599,1)
Split the data into train and test using train_test_split function from sklearn.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(feature, target, test_size = 0.2, random_state = 42)
Scaling the Dataset
Scaling the dataset — Can use min-max normalization for scaling the dataset with mean zero and a unit standard deviation.
This estimator scales and translates each feature individually such that it is in the given range on the dataset set, e.g. between zero and one.
from sklearn.preprocessing import MinMaxScalerscaler = MinMaxScaler()X_train=scaler.fit(X_train).transform(X_train)X_test=scaler.fit(X_test).transform(X_test)
Apply PCA with inbuilt PCA function in sklearn:
from sklearn.decomposition import PCApca = PCA(n_components=2)X_train = pca.fit(X_train).transform(X_train)X_test= pca.fit(X_test).transform(X_test)
Here, the feature is the training dataset and n_component is the number of PCA components we want to derive from our existing feature set.
So using the sklearn, PCA is like a black box (explained above), you give a scaled feature set as an input to sklearn PCA and get PCA components as output which can be used as an input to data training algorithms. For the PCA algorithm, it is necessary to perform data scaling before we find principal components.
Training the Algorithm
Here we will train principal components using a simple Logistic Regression tool to solve this classification problem. To do so, execute the following code:
from sklearn.linear_model import LogisticRegressionclf = LogisticRegression(random_state = 0)clf.fit(X_train, y_train)y_pred = clf.predict(X_test)# Predicting the test set result using. Predict function under Random Forest Classifiery_pred = clf.predict(X_test)
Evaluating the Algorithm
For classification problems, the metrics used to evaluate an algorithm are accuracy, confusion matrix, precision, recall, and F1 values. Execute the following script to find these values:
from sklearn.metrics import classification_report, confusion_matrix, accuracy_scoreprint(confusion_matrix(y_test,y_pred))print(classification_report(y_test,y_pred))print(accuracy_score(y_test, y_pred))
The output will look something like this:
84.06% of accuracy we have achieved only using two feature sets out of eleven features in the original dataset. I would not consider this too bad. Further model accuracy can be improved using hyperparameter optimization of the logistic regression model or using some other models from the sklearn library. Try training model for more than 2 and less than 6 principal components. That would be a fun exercise to play around with PCA.
Here we will plot two graphs one by considering only two columns out of eleven columns and the second graph we will plot for principal components to understand how PCA makes visualization easy for high dimensional data.
import seaborn as snsimport matplotlib.pyplot as pltsns.set(rc={‘figure.figsize’:(12,7)})sns.scatterplot(x=”free sulfur dioxide”, y=”total sulfur dioxide”, hue=”isQuality”,data=df,palette=”Set1")plt.show()
sns.scatterplot(pca_df[:,0],pca_df[:,1], hue=”isQuality”,data=df, palette=”Set1")plt.show()
The graph above shows the plotting of data points using two features vs. Plotting of data points using principal components (which is basically a summarization of all the features in our dataset) i.e. PCA_1 and PCA_. This helps us to identify one more use of PCA i.e. proper and separable visualization of our dataset along with their “Quality” class tags. If you look at the graph, you can visualize the whole dataset properly with only two feature variables as compared to having thirteen feature variables. Though they do not differentiate well among (As see between sulfur dioxide and total sulfur dioxide) the wine quality classes but helps in reducing the feature set with minimal loss of information.
So far, we have made a pretty good understanding of PCA with some hands-on examples. We have seen how few components are sufficient to train models and also PCA is a great tool to reduce data dimension and give us the power to visualize data properly.
Let’s move forward and dive into one more tool for dimension reduction “Linear Discriminant Analysis”. I will be using the same wine dataset to provide hands-on experience.
Linear Discriminants is a statistical method of dimensionality reduction that provides the highest possible discrimination among various classes, used in machine learning to find the linear combination of features, which can separate two or more classes of objects with the best performance. The method is based on discriminant functions that are estimated based on a set of data called a training set. These discriminant functions are linear with respect to the characteristic vector.
The aim of LDA is to maximize the between-class variance and minimize the within-class variance, through a linear discriminant function, under the assumption that data in every class are described by a Gaussian probability density function with the same covariance.
It is good to understand the mathematics behind the LDA but I would not like to overwhelm anyone. We will understand by keeping things simple so it will be easy to follow for anyone.
LDA helps you find the boundaries of class clusters. This projects your data points on a line in order to differentiate the clusters as much as possible, with each cluster having a relatively close distance to a centroid.
So, the question arises-how are these clusters identified and how do we get LDA’s reduced feature set?
Basically, LDA considers a centroid of data points for each class. For example, with eleven different features, LDA will use the eleven different feature datasets to find the centroid of each of its classes. Depending on this, it now defines a new dimension which is nothing more than an axis that should fulfill two requirements.
Maximize the distance between each class ‘Centroid.Minimize the variance within each group (which LDA calls scatter, and is represented by s2).
Maximize the distance between each class ‘Centroid.
Minimize the variance within each group (which LDA calls scatter, and is represented by s2).
Hence the essence is (mean_a — mean_b)/(S_a — S_b)
(mean_a — mean_b) = ideally large (S_a — S_b) = ideally small
Here mean is nothing more than the class centroid. Variance is nothing but the spread of data across the plane. So, if the data variance is small then there will be less overlapping between the classes and the overall distinction between the different classes will be preserved.
So whatever coordinate of the new axis meets these two conditions, they form the new dataset dimension.
Now let’s jump to implementing LDA using sklearn on wine dataset and see how the data transitions from a higher dimensionality to a lower dimensionality.
Let’s perform the LDA on wine dataset and analyze graphically:
All the necessary libraries are already imported let’s import LDA with inbuilt LDA function in sklearn.
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDAlda = LDA(n_components=2)lda_df = lda.fit(feature, target).transform(feature)sns.scatterplot(lda_df[:,0],lda_df[:,1], hue=”isQuality”,data=df, palette=”Set1")plt.show()
Note: I wanted to clarify that feature scaling such as (standardization) does not change the overall results of an LDA and thus may be optional.
In the PCA graph, you can visualize the whole dataset properly differentiate well among the classes. Whereas in the LDA graph the classes are not clearly separable due to the uneven distribution of classes. This is the major difference between PCA and LDA.
To summarize, PCA performs better in situations where there are fewer samples per class. In this wine dataset, we have high counts of class 2 (1319) followed by class 1 (217) and lastly class 0 (63).
Although LDA works better with a large multi-class dataset where class separability is an important factor while reducing dimensionality.
The general LDA approach is very similar to a Principal Component Analysis, but in addition to finding the component axes that maximize the variance of our data (PCA), we are additionally interested in the axes that maximize the separation between multiple classes (LDA).
Finally, let’s see how we can choose the number of n_components in dimensionality reduction methods within sklearn. The simple approach is to use explained_variance_ratio_ and plot the graph to see how many components we need to retain variance by the new feature set. Let’s understand by doing this
scaler_df=scaler.fit(df).transform(df)pca = PCA().fit(scaler_df)plt.figure()plt.plot(np.cumsum(pca.explained_variance_ratio_))plt.xlabel(‘Number of n_Components’)plt.ylabel(‘Variance (%)’) #for each component present in the datasetplt.title(‘Wine Dataset Explained Variance’)plt.show()plt.ylabel(‘Variance (%)’) #for each component present in the datasetplt.title(‘Wine Dataset Explained Variance’)plt.show()
This plot tells us that selecting 8 components we can preserve something more than 90 % of the total variance of the wine data. It makes sense, we’ll not use 100% of our variance, because it denotes all components, and we want only the principal ones.
Now by using n_components=8 we can perform the same analysis and train any model. It is worth playing around. Higher-level Python libraries for machine learning actually made learning pretty easy, unlike earlier days when we have to hard code everything from scratch.