Linear Discriminant Analysis — An Introduction | by Pritha Saha | Towards Data Science

LDA is a dimensionality reduction algorithm, similar to PCA. However while PCA is an unsupervised algorithm that focusses on maximising variance in a dataset, LDA is a supervised algorithm that maximises separability between classes.
In a classification problem set up the objective is to ensure maximum separability or discrimination of classes.
Suppose we have a dataset with two columns — one explanatory variable and a binary target variable (with values 1 and 0). The distribution of the binary variable is as per below:
The green dots represent 1 and the red ones represent 0. Since there is only one explanatory variable, it is denoted by one axis (X). However, if we try to place a linear divider to demarcate the data points, we will not be able to do it successfully since the points are scattered across the axis.
Hence it seems that one explanatory variable is not enough to predict the binary outcome. So we will bring in another feature X2 and check the distribution of points in the 2 dimensional space.
It seems that in 2 dimensional space the demarcation of outputs is better than before. However, increasing dimensions might not be a good idea in a dataset which already has several features. In those situations, LDA comes to our rescue by minimising the dimensions. In the below figure the target classes are projected on a new axis:
The classes are now easily demarcated. LDA transforms the original features to a new axis, called Linear Discriminant (LD), thereby reducing dimensions and ensuring maximum separability of the classes.
In order to put this separability in numerical terms, we would need a metric that measures the separability. Calculating the difference between means of the two classes could be one such measure. The higher difference would indicate an increased distance between the points. However, this method does not take the spread of the data into cognisance. Hence even a higher mean cannot ensure that some of the classes don’t overlap with each other.
The second measure is taking both the mean and variance within classes into consideration. To ensure maximum separability we would then maximise the difference between means while minimising the variance.
The score is calculated as (M1-M2)2/(S12+S22).
If there are three explanatory variables- X1, X2, X3, LDA will transform them into three axes — LD1, LD2 and LD3. These three axes would rank first, second and third on the basis of the calculated score.
Hence LDA helps us to both reduce dimensions and classify target values.
We will go through an example to see how LDA achieves both the objectives. The below data shows a fictional dataset by IBM, which records employee data and attrition. The objective is to predict attrition of employees, based on different factors like age, years worked, nature of travel, education etc.
sns.countplot(train['Attrition'])plt.show()
There are around 1470 records, out of which 237 employees have left the organisation and 1233 haven’t. Yes has been coded as 1 and No is coded as 0.
Attrition of employees if not predicted correctly can lead to losing valuable people, resulting in reduced efficiency of the organisation, reduced morale among team members etc. Hence it is necessary to correctly predict which employee is likely to leave. In other words, if we predict an employee will stay, but actually the employee leaves the company, the number of False Negatives increase. Our objective would be to minimise False Negatives and hence increase Recall (TP/(TP+FN)).
We will try classifying the classes using KNN:
knn=KNeighborsClassifier(n_neighbors=10,weights='distance',algorithm='auto', p=3)start_time = time.time()knn.fit(X_train_sc,y_train)end_time = time.time()time_knn = end_time-start_timeprint(time_knn)pred = knn.predict(X_test_sc)print(confusion_matrix(y_test,pred))print(classification_report(y_test,pred))
Time taken to fit KNN : 0.0058078765869140625
Recall is very poor for the employees who left at 0.05.
We will now use LDA as a classification algorithm and check the results.
lda_0 = LDA()lda_0.fit(X_train_sc, y_train)y_test_pred_0 = lda_0.predict(X_test_sc)print(confusion_matrix(y_test, y_test_pred_0))print(classification_report(y_test, y_test_pred_0))
Recall has increased drastically to 0.36
Finally, we will transform the training set with LDA and then use KNN.
#Transformation by LDAlda_1 = LDA(n_components = 1, solver='eigen', shrinkage='auto')X_train_lda = lda_1.fit_transform(X_train_lda, y_train_lda)X_test_lda = lda_1.transform(X_test_lda)
This is followed by Standard Scaling.
sc3 = StandardScaler()X_train_lda_sc = sc3.fit_transform(X_train_lda)X_test_lda_sc = sc3.transform(X_test_lda)
Now we apply KNN on the transformed data.
knn=KNeighborsClassifier(n_neighbors=8,weights='distance',algorithm='auto', p=3)start_time = time.time()knn.fit(X_train_lda_sc,y_train_lda)end_time = time.time()time_lda = end_time-start_timeprint(time_lda)print(confusion_matrix(y_test_lda,pred))print(classification_report(y_test_lda,pred))
Time taken to run KNN on transformed data: 0.0024199485778808594
Recall is now increased by 6%.
Also, the time taken by KNN to fit the LDA transformed data is 50% of the time taken by KNN alone.
The entire code is available here.
Hope I have been able to demonstrate the use of LDA, both for classification and transforming data into different axes!
Stay tuned for more! As always, any feedback is appreciated.