What is Supervised Linear Discriminant Analysis(LDA) ~ PCA
By: Bob Rupak Roy
Tags: Machine Learning, Analytics, Data Science, Dimensionality Reduction, Supervised Learning

What is PCA first of all?

Principal Component Analysis or PCA is a statistical procedure that allows us to summarize/extract the only important data that explains the whole dataset.

Principal component analysis today is one of the most popular multivariate statistical techniques. PCA is the mother method for multivariate data analysis MVDA

It has been widely used in the areas of pattern recognition and signal processing and in statistical analysis to reduce the dimension, in simple words, to understand and extract only the important factors that explains the whole data. Thus helps in avoiding unnecessary data to be processed.

Now since we got a basic idea of what is pca. Let’s understand what is Linear Discriminant Analysis (LDA) type for dimensionality reduction.

Both PCA and LDA are linear transformation techniques used for dimensionality reduction. However PCA is unsupervised and LDA is a supervised one with the fact that the dependent variable (DV) is considered makes LDA a supervised model.

Let’s understand this with the help of an example. For this example we will use one of the famous available dataset ‘wine_quality’ were we have the key components factors of wine and its customer.

Wine Quality Dataset

What we will do is with LDA we will try to find out the few key component factors that describes the whole dataset and to validate we will cross check it with performance metrics.

Let’s get started!

First we will import the required libraries and import the dataset then define our independent variable X and dependent variable y then split the dataset into train and test.

import numpy as np

import matplotlib.pyplot as plt

import pandas as pd #Importing the dataset

dataset = pd.read_csv('Wine.csv')

X = dataset.iloc[:, 0:13].values

y = dataset.iloc[:, 13].values #Splitting the dataset into the Training set and Test set

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

We will also normalize/scale the data into a common range to reduce the magnitude / spread of data points without losing its original meaning.

#Feature Scaling

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()

X_train = sc.fit_transform(X_train)

X_test = sc.transform(X_test)

Time to apply LDA!

#LDA

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

lda = LDA(n_components = 2)

X_train = lda.fit_transform(X_train, y_train)

X_test = lda.transform(X_test)

Its almost the same like PCA only

‘Explained_variance = pca.explained_variance_ratio’ is not required as we are not looking for independent variables that explains the most variance. Now we are looking for independent variables that separates the most classes of our dependent variable.

PCA vs LDA

Now its time to fit our data in a model and will check how much accuracy we can derive.

#Fitting Logistic Regression to the Training set

from sklearn.linear_model import LogisticRegression

classifier = LogisticRegression(random_state = 0)

classifier.fit(X_train, y_train) #Predicting the Test set results

y_pred = classifier.predict(X_test)

Here as you can see, we are using logistic regression, random_state is just a seed number to avoid randomness for the set of data for each time the algorithm computes. Then we will predict with an unseen data (X_test)

We will use our regular confusion matrix then accuracy score metric.

#Confusion Matrix

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_test, y_pred) #Another evaluation Metrics

print('Accuracy Score:', metrics.accuracy_score(y_test, y_pred))

Well indeed LDA perfectly separated the independent variables across the classes for our test dataset(unseen data) and also our model accuracy is 100%

Now think of this having more than 10000 columns and we are able to extract only few principal components that explains the whole dataset.

Just imagine how helpful PCA Principal Components Analysis it will be.

Let’s put all of the codes together.