Machine Learning: Dimensionality Reduction via Linear Discriminant Analysis
By: Benjamin Obi Tayo Ph.D.
Tags: Machine Learning, Python, Data Transformation, Data Processing, Data Science

A machine learning algorithm (such as classification, clustering or regression) uses a training dataset to determine weight factors that can be applied to unseen data for predictive purposes. Before implementing a machine learning algorithm, it is necessary to select only relevant features in the training dataset. The process of transforming a dataset in order to select only relevant features necessary for training is called dimensionality reduction. Dimensionality reduction is important because of three main reasons:

Prevents Overfitting: A high-dimensional dataset having too many features can sometimes lead to overfitting (model captures both real and random effects). Simplicity: An over-complex model having too many features can be hard to interpret especially when features are correlated with each other. Computational Efficiency: A model trained on a lower-dimensional dataset is computationally efficient (execution of algorithm requires less computational time).

Dimensionality reduction therefore plays a crucial role in data preprocessing.

Implementation of Dimensionality Reduction

There are several models for dimensionality reduction in machine learning such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), Stepwise Regression, and Regularized Regression (such as LASSO). We focus here on PCA and LDA, which are widely used for classification problems.

PCA and LDA are two data preprocessing linear transformation techniques that are often used for dimensionality reduction in order to select relevant features that can be used in the final machine learning algorithm. PCA is an unsupervised algorithm that is used for feature extraction in high-dimensional and correlated data. PCA achieves dimensionality reduction by transforming features into orthogonal component axes of maximum variance in a dataset. An implementation of PCA using iris dataset can be found here: https://github.com/bot13956/principal_component_analysis_iris_dataset

The goal of LDA is to find the feature subspace that optimizes class separability and reduce dimensionality (see figure below). Hence LDA is a supervised algorithm. In this article, we illustrate the implementation of LDA using the iris dataset. An in-depth description of PCA and LDA can be found in this book: Python Machine Learning by Sebastian Raschka, Chapter 5.

Figure 1: LDA algorithm transforms from old to new feature subspace so as to optimize class separability and reduce dimensionality. Picture adapted from: “Python Machine Learning by Sebastian Raschka”.

The code for implementing LDA is found here: https://github.com/bot13956/linear-discriminant-analysis-iris-dataset/blob/master/LDA_irisdataset.ipynb

The output from an LDA calculation using the iris dataset is shown in figure 2 below:

Figure 2: Linear separability of iris classes in the LDA subspace. Notice that the LD1 component captures most of the class discriminability.

By analyzing the cumulative discriminability (see code: https://github.com/bot13956/linear-discriminant-analysis-iris-dataset/blob/master/LDA_irisdataset.ipynb), we can show that the LD1 and LD2 components capture 100% of the total discriminability. Hence when we perform classification (using logistic regression or support vector machine) on the LDA subspace, we can train the model on a lower two-dimensional LDA transformed dataset. Since the original iris dataset was four-dimensional (4 features), we remark that LDA transformation achieves both class separability as well as dimensionality reduction.

In summary, we have shown how the LDA algorithm can be implemented using the iris dataset for illustrative purposes. In a previous article, we showed how the PCA algorithm can be implemented using the iris dataset as well. Here are the two links:

PCA implementation:

https://github.com/bot13956/principal_component_analysis_iris_dataset

LDA implementation:

https://github.com/bot13956/linear-discriminant-analysis-iris-dataset

Thanks for reading.