Dimensionality Reduction — Can PCA improve the performance of a classification model? | by Satyam Kumar | Towards Data Science

Principal Component Analysis (PCA) is a common feature extraction technique in data science that employs matrix factorization to reduce the dimensionality of data into lower space.
In real-world datasets, there are often too many features in the data. The higher the number of features harder it is to visualize the data and work on it. Sometimes most of the features are correlated, and hence redundant. Hence feature extraction comes into play.
The dataset used in this article is Ionosphere Dataset from the UCI machine learning repository. It is a binary class classification problem. There are 351 observations with 34 features.
Importing necessary libraries and reading the dataset
Preprocessing of dataset
Standardization
The training data has 34 features.
After preprocessing of data, training data is trained using Logistic Regression algorithm for binary class classification
Finetuning Logistic Regression model to find the best parameters
Compute training and test accuracy and f1 score.
Training an LR model using 34 features for c=10**0
Compute training and test accuracy and f1 score
Results obtained by training the entire “X_train” data having 34 features,
the test f1-score is 0.90, as 14 values are misclassified as observed in the confusion matrix.
To extract features from the dataset using the PCA technique, firstly we need to find the percentage of variance explained as dimensionality decreases.
Notations,λ: eigenvalued: number of dimension of original datasetk: number of dimensions of new feature space
From the above plot, it is observed that for 15 dimensions the percentage of variance explained is 90%. This means we are preserving 90% of variance by projecting higher dimensionality (34) into lower space (15).
Now the training data after PCA dimensionality reduction has 15 features.
After preprocessing of data, training data is trained using Logistic Regression algorithm for binary class classification
Finetuning Logistic Regression model to find the best parameters
Compute training and test accuracy and f1 score.
Training an LR model using 15 feature for c=10**0
Compute training and test accuracy and f1 score
Results obtained by training the PCA data with 15 features,
the test f1-score is 0.896, as 12 values are misclassified as observed in the confusion matrix.
After concatenating original data with 34 features and PCA data with 15 features, we form a dataset of 49 features.
After preprocessing of data, training data is trained using Logistic Regression algorithm for binary class classification
Finetuning Logistic Regression model to find the best parameters
Compute training and test accuracy and f1 score.
Training an LR model using 15 feature for c=10**0
Compute training and test accuracy and f1 score
From the above pretty table, we can observe that,
An LR model trained using the raw preprocessed dataset with 34 features, we get an F1-score of 90%.
An LR model trained only with extracted 15 features using PCA, we get an F1-score of 89%.
An LR model trained with a combination of the above two data, we get an F1-score of 92%.
Let’s observe the change in Confusion Matrix results for the above-mentioned 3 models.
Hence we conclude that using only PCA extracted features with only 50% of numbers of features from original data we get 1% less F1-score. But if we combine both data we improve the metric of 2% to get the final F1-score of 91%.
Click below to get code:
colab.research.google.com
Thank You for Reading