LDA: Linear Discriminant Analysis — How to Improve Your Models with Supervised Dimensionality Reduction | by Saul Dobilas | Towards Data Science

Linear Discriminant Analysis (LDA) is a commonly used dimensionality reduction technique. However, despite the similarities to Principal Component Analysis (PCA), it differs in one crucial aspect.
Instead of finding new axes (dimensions) that maximize the variation in the data, it focuses on maximizing the separability among the known categories (classes) in the target variable.
In this article, I give an intuitive explanation of how LDA works while highlighting the differences to PCA. At the same time, I provide a Python example of performing Linear Discriminant Analysis on real-life data.
The category of Machine Learning techniques LDA belongs to
Intuitive explanation of how LDA works
Python example of performing LDA on real-life data
Conclusions
Unlike Principal Component Analysis (PCA), LDA requires you to provide features and class labels for your target. Hence, despite being a dimensionality reduction technique similar to PCA, it sits within the supervised branch of Machine Learning.
The below graph is interactive, so please click on different categories to enlarge and reveal more👇.
If you enjoy Data Science and Machine Learning, please subscribe to get an email whenever I publish a new story.
The easiest way to grasp the concepts of LDA is by working through an example. Hence, instead of focusing on the maths behind the algorithm, I have created a visual explanation for us to go through.
Assume we have collected a bunch of data on apartment prices in the city. We categorized them into “expensive” and “cheap” based on some threshold, say $1M.
I.e., <$1M = cheap, and ≥$1M = expensive. 
We display the data on a scatterplot where the x and y axes represent the latitude and longitude, giving us the apartment’s location. Meanwhile, color represents the category/class (cheap vs. expensive). Note that we have already standardized* the data in this fictitious example, so it is centered around the origin.
*Standartization is a data transformation technique that rescales the data so each attribute has a mean of 0 and a standard deviation of 1. This transformation can be described with the following formula:
Let’s take a look at the scatterplot of our example data:
Now let’s compare the PCA and LDA process to understand how these algorithms work and how they differ.
Since PCA is an unsupervised Machine learning technique, we do not need to provide a target variable with category labels. This means that PCA does not care about whether the apartment is part of a “cheap” or “expensive” category.
Nevertheless, I have kept the color labels in the below graph to highlight the main difference from LDA.
The goal of PCA is to capture the maximum amount of variance, which is achieved by the algorithm finding a line that minimizes the distances from data points to that line. Interestingly, this is equivalent to maximizing the spread of data point projections on that same line, which is why we can capture the maximum amount of variance.
As you can see from the graph above, we have done a good job finding a new axis that preserves most of the variance, enabling us to go down from two dimensions to one. This is what it looks like after we map the data to our new single dimension (PC1):
You can clearly see that, in this scenario, we were able to preserve most of the variance but still lost some information that would have been useful when trying to separate the two categories (classes).
If you would like to get a more in-depth understanding of PCA, you can refer to my previous article:
towardsdatascience.com
Now let’s go back to our original example data and apply LDA instead of PCA. Quick reminder, the goal of LDA is to maximize the separability of the known categories in our target variable (“cheap”, “expensive”) while at the same time reducing dimensions.
Here is what the new axis looks like when we perform LDA:
As you can see, the choice of the new axis, in this scenario, is very different from that of PCA. In this case, we end up losing a significant amount of variance by mapping data onto LD1. However, we achieve better separation of the two categories:
While the separation is imperfect, it is visibly better, with only a few overlapping observations from the two categories.
Two key criteria are used when finding the “best” line for our new axis, which are considered simultaneously.
Maximizing the distance (d2).2 categories — when you have two classes in your target variable, the distance refers to the difference between the mean (μ) of class 1 and the mean of class 2. More than 2 categories — when you have three or more classes in your target variable, the algorithm first finds a central point to all of the data and then measures the distance from each category mean (μ) to that central point.
Maximizing the distance (d2).2 categories — when you have two classes in your target variable, the distance refers to the difference between the mean (μ) of class 1 and the mean of class 2. More than 2 categories — when you have three or more classes in your target variable, the algorithm first finds a central point to all of the data and then measures the distance from each category mean (μ) to that central point.
2. Minimize the variation, also known as “scatter” in LDA (s2).
Let’s illustrate the two criteria on a graph:
In practice, the calculations are typically performed with the help of either Singular Value Decomposition (SVD) or Eigenvalue decomposition. Both of these options are available in sklearn’s LDA implementation, which we will use in the next section.
Finally, it’s time for the fun stuff where we get to apply LDA using Python. Let’s start by getting the right libraries and data for our analysis.
We will use the following data and libraries:
House price data from Kaggle
Scikit-learn library for1) encoding categorical class labels (OrdinalEncoder);2) feature standardization (StandardScaler);3) performing Principal Component Analysis (PCA);4) performing Linear Discriminant Analysis (LDA);5) creating a decision tree-based prediction model (DecisionTreeClassifier);6) model evaluation (classification_report)
Plotly for data visualizations
Pandas for data manipulation
Let’s import all the libraries:
Next, we download an ingest house price data from Kaggle.
Here’s a snippet of what the data looks like:
For this example, we want to create a multiclass target variable. Hence, we take ‘Y house price of unit area’ and divide it into 3 equal categories. We call them: ‘1.Affordable’ (bottom 33%), ‘2.Mid-range’ (middle 33%), and ‘3.Expensive’ (top 33%).
Since we have created categorical labels while the algorithm requires numerical ones, let’s use the Ordinal encoder to convert them to numeric. [Note, we could have directly assigned numeric labels, but the categorical ones will make it easier for us to read the upcoming visualization].
To keep the ability to visualize the data, we will limit ourselves to the first 3 feature variables: ‘X1 transaction date’, ‘X2 house age’, and ‘X3 distance to the nearest MRT station’.
Let’s see what the data looks like by plotting these three features on the 3D scatterplot:
We can see that the data is scattered significantly. Also, we note that cheaper real estate is located further away from MRT stations, while the expensive ones tend to be much closer to MRT and also newer.
One final step in data preparation is the standardization of the features.
Note that we should also split our data into train and test samples at this stage, but we’ll work with the entire dataset to keep things simple for this example.
For interest, we’ll also perform PCA so we can compare the results of PCA to the results of LDA.
Interestingly, explained variance ratios for PC1 and PC2 only cover ~69% of the total variance, meaning that we have lost about 31% of the total variance when going down from 3 dimensions to 2. This is largely due to little correlation between the 3 features.
Here is a scatter plot of the resulting 2D plot after applying PCA:
While we did lose 31% of the variance, we still managed to keep a lot of the spread across the two new dimensions. However, since PCA only considers features and not the target, the three categories (classes) ended up all mixed together.
Next, let’s see how LDA compares to PCA.
We will use Eigen decomposition as our solver (sklearn implementation enables you to choose between SVD, LSQR, and Eigen) and set the components parameter (number of dimensions) to 2, leaving all the rest as default. Note that this time we also need to supply our target (y) to the algorithm.
As before, we also visualize the results on a 2D scatterplot.
There is a notable difference in LDA results when compared to PCA. Since the goal of PCA is to separate target categories instead of maximizing variance, it managed to find new axes that reduced the spread of blue (‘2.Mid-range’) and red (‘3.Expensive’) points, keeping them largely within their own space. Meanwhile, despite the large variance remaining in the green (‘1.Affordable) class, the separation from red and blue is pretty good.
To validate what we see in these scatter plots, let’s build a couple of decision tree models and see how well we can predict ‘Price Band’ based on features created using PCA and LDA.
Let’s set up a reusable function that we can quickly call to train a model and display the results.
Now let’s use it to build a model with PCA-transformed features.
Here are the model results:
As expected, the results are not great, with accuracy only at 64%. Mixing of the categories (classes) after applying PCA has definitely made it harder to predict the price label.
Let’s repeat the same with LDA-transformed features.
This time the model seems to be much better, with nearly 79% accuracy. These results support our earlier intuition that LDA has better preserved the information relevant to predicting an apartment’s price when compared to PCA.
LDA is a great tool when we want to reduce the dimensionality of our data while keeping as much information relevant to our prediction target.
However, the direct comparison of PCA and LDA might not be completely fair because they are two different techniques meant for different purposes. For example, PCA is an unsupervised learning technique, while LDA falls under the supervised branch of ML.
Hence, please don’t take the results above as proof of LDA superiority over PCA. Instead, please first assess the applicability of these algorithms to your unique case.
I sincerely hope my article has helped you better understand Linear Discriminant Analysis, enabling you to incorporate it into your own Data Science projects.
Cheers! 👏Saul Dobilas
If you have already spent your learning budget for this month, please remember me next time. My personalized link to join Medium is:
solclover.com
If you liked this story, here are couple more articles that you might enjoy: