Principal Component Analysis algorithm in Real-Life: Discovering patterns in a real-estate dataset | by Carolina Bento | Towards Data Science

Principal Component Analysis, PCA for short, is an unsupervised learning technique used to surface the core patterns in the data.
In this article, we’re going through how PCA works with the real-life example of a real estate agent who wants to understand why some of their listings are taking too long to close, and how we can use PCA to encode a smaller dataset.
Your friend Maggie is a real-estate agent and she is puzzled about why some of the properties her company manages have not been sold for more than six months. She knows you’re a Data Scientist and asks if you could help her get some insight into what’s happening.
You start by doing some exploratory analysis. You sanity-check the data to see if there are missing or inconsistent values, and then it’s time to look for patterns in the data.
Maggie wants to understand why these properties stay unsold for so long. She wants to know what are the characteristics and patterns in the data that make these properties take so long to get sold.
But finding patterns in a dataset that is big and has lots of attributes, usually referred to as features, is impossible to do by hand.
On top of this, your dataset might have several redundant features. You might suspect some features are slight variations of other features. And in some cases, some features might be pure noise, not revealing much about the core patterns in the data.
For instance, you notice that in Maggie’s dataset there are features for:
Total number of doors in a house,
Total number of bedrooms,
Total number of bathrooms.
Having a feature for the total number of doors doesn’t seem like it could reveal something critical about the data, since we have the other two features. Putting it in statistical terms, the total number of doors in a house is most likely a redundant feature and doesn’t explain the variance in the data.
Ok, no big deal! If the total number of doors in a house is a redundant feature, we can remove it from the dataset. This makes sense in our case, but in a dataset with hundreds or even thousands of features, we need to be more cautious.
In a dataset with a large number of features, we need to be sure that a feature is redundant or noise before removing it. We might be removing important information about the data without knowing.
We should only remove features we’re sure will not impact the patterns in the data or the prediction results.
To find patterns in this dataset, you immediately think about Principal Component Analysis.
Principal component analysis (PCA) is an unsupervised learning method, meaning it’s used when we don’t have a label or target for each observation in our dataset.
While in supervised learning methods we must have a label for each observation, because the goal is to predict a class or a number, in unsupervised learning the goal is to look for patterns in the data.
So, you can help our friend Maggie and use PCA to figure out what are the core characteristics of the properties that take longer to sell.
The goal of Principal Component Analysis is to apply linear transformations to the data, so you minimize noise and redundancy. The result is a dataset that reveals the most important characteristics of the data.
After applying PCA, you’ll have a set of Principal Components, ranked in descending order of how much they contribute to describing patterns in the data. In statistical parlance, they are ranked according to how much variance they explain.
The first principal component is the most important at describing variance in the data. The remaining principal components are less critical expressing the variability of patterns in the data.
Behind the scenes, Principal Component Analysis uses statistical tools to identify noise and redundancy in the dataset. It uses the covariance matrix to analyze:
Variance of each feature. It will show if a feature is relevant or pure noise, and we can read that in the diagonal cells.
Strength of linear relationship between pairs of features. This helps to spot redundant features, and it’s read in all non-diagonal values.
The covariance matrix highlights the things that get in the way of seeing patterns in the data. So, at the end of the day, what PCA will produce a set of principal components which:
Reduce noise, by maximizing feature variance.
Reduce redundancy, by minimizing the covariance between pairs of features.
The basis of PCA is the covariance matrix and, in practice, there are two approaches to identify the principal components:
Calculate the eigenvectors of the covariance matrix.
Calculate the Singular Value Decomposition of the covariance matrix.
Before diving into Maggie’s data, PCA requires that we prepare our dataset. The original dataset looks something like this:
In the exploratory analysis phase we’d also want to create a pairplot and to spot any clear correlations between features. But, even in this small dummy dataset with only 17 features, the resulting pairplot is enormous and slow to compute. Which makes sense, since we have to plot 17 choose 2 = 136 individual plots.
The first step in PCA is to normalize the data and make sure all features are normally distributed. The data will then follow a standard normal distribution, with mean equal to zero and standard deviation equal to one.
Normalizing the data transforms each feature into a common representation, and makes all values across features comparable. It is especially important to normalize features when they have different scales. For instance, when one feature is in miles and the other in minutes.
Normalization also guarantees that principal components are:
Linearly independent. Each principal component is a linear combination that is not made of other principal components.
Orthogonal. Meaning all principal components make a 90 degree angle with each other.
There are several ways to normalize your features, usually called feature scaling. One of them is the Z-score Normalization, also referred to as Standardization.
To apply Z-score normalization to your dataset you update the value each data point such that:
We took this detour to talk about normalization, but most statistical libraries will do that for you. With Python libraries like ScikitLearn or statsmodels, you just need to set a few parameters.
At the end of the process, PCA will encode your features into principal components. But it’s important to note that principal components don’t necessarily map one-to-one with features. They are a new representation of the dataset and can encode several features at once.
Another step in our exploratory analysis, is to take a look at the covariance matrix. PCA normalizes the data and creates the covariance matrix, but it’s useful to visualize the linear relationships between features.
To read this matrix, we should first focus in the darker areas:
On the non-diagonal, the pairs of redundant features. Confirming our initial hunch bedrooms and number_doors have a covariance of 97%. This means there’s a strong linear relationship between these two features. As number_doors increases by one unit, bedrooms increases by one unit and vice-versa. This pair of features encodes a similar pattern, so we can consider them as redundant.
On the diagonals, we see that all features have high variance. It looks like all features are relevant and none of them are pure noise.
Here’s how to plot the covariance matrix.
import numpy as npimport pandas as pdimport seaborn as snsimport matplotlib.pyplot as plt# Load datasetdataset = pd.read_csv('src/dataset.csv')pca = PCA(dataset, standardize=True, method='eig')normalized_dataset = pca.transformed_data# Covariance Matrix# bias =True, so dataset is normalized# rowvar = False, each column represents a variable, i.e., a feature. This way we compute the covariance of features as whole instead of the covariance of each rowcovariance_df = pd.DataFrame(data=np.cov(normalized_dataset, bias=True, rowvar=False), columns=dataset.columns)# Plot Covariance Matrixplt.subplots(figsize=(20, 20))sns.heatmap(covariance_df, cmap='Blues', linewidths=.7, annot=True, fmt='.2f', yticklabels=dataset.columns)plt.show()
We’re going to use the Python library statsmodels to perform PCA. You can also use ScikitLearn, but statsmodels provides a bit more flexibility. For instance, you can choose if you want to use eigenvectors or Singular Value Decomposition.
So, for this example we’ll set the following flags in the statsmodel PCA method:
standardize is set to True, so we standardize the data with mean zero and variance 1.
method is set to eig, so the eigenvectors of the covariance matrix become our principal components.
We’ll let parameter ncomp unset, so the PCA returns all principal components, i.e., are as many as the number of features.
import pandas as pdfrom sklearn import preprocessing# Load datasetdataset = pd.read_csv('dataset.csv')# Run PCApca = PCA(dataset, standardize=True, method='eig')components_df = pca.factors
So, components_df is an array with all the principal components, i.e., the eigenvectors of the covariance matrix.
But this by itself doesn’t tell us much about the data. We need to find a way to interpret these results and connect it to our original dataset.
We’ll start by computing the correlation between each principal component and each feature in the original dataset. The key here is to do the pairwise correlation of the principal components and features as a whole, instead of between individual values.
It’s easier if we visualize the correlation matrix. On top of a color gradient we’ll tweak the plot a bit to:
Create some space between cells, for better readability and aesthetics with linewidths=.7.
Add the correlation value to each cell with annot=True. So it’s easier to spot the differences between cells values.
Truncate the value in each cell to have 2 significant digits, for better readability, with fmt=’.2f’.
And we have a correlation matrix 😀
Right away we can see the highest correlation values associated with the first principal component, comp_00.
Principal components are sorted by of how much of the variability in the dataset they encode. So, it makes sense the first principal components have higher correlation with the dataset features.
These features associated with the first principal component, are the most relevant to determine the patterns in the data.
We also see there’s not a one-to-one relationship between features and principal components. The first principal component is has a strong positive correlation with several features.
import pandas as pdimport seaborn as snsimport matplotlib.pyplot as plt# Load datasetdataset = pd.read_csv('dataset.csv')# Run PCApca = PCA(dataset, standardize=True, method='eig')components_df = pca.factorscombined_df = pd.concat([dataset, components_df], axis=1)correlation = combined_df.corr()# This matrix will have the correlation between:# 1. feature vs features# 2. feature vs principal component# 3. principal component vs principal component# We're removing part of the output to keep only the correlation between features and principal componentscorrelation_plot_data = correlation[:-len(components_df.columns)].loc[:, 'comp_00':]# plot correlation matrixfig, ax = plt.subplots(figsize=(20, 7))sns.heatmap(correlation, cmap='YlGnBu', linewidths=.7, annot=True, fmt='.2f')plt.show()
The correlation matrix shows which features are correlated with first principal component and, therefore which features will change together. One good example, since there’s a positive correlation between the first principal component and bedrooms and square footage, we can say that as you increase the number of bedrooms or baths, the square footage tends to increase.
But there are also a few negative correlations with the first principal component, for instance with transit accessibility, proximity to city center and supermarket proximity. This shows that bigger houses, with higher square footage and have more bedrooms, will tend to be farther away from the city centre, from supermarket and have lower transit accessibility.
Since Maggie is analyzing the features of houses that haven’t been sold for more than 6 months, she can look into these negative correlations to find hints for why these houses have remained unsold for so long.
The first principal component has strong positive correlation with 11 of the 17 features in the dataset. This means it encodes, or represents, the patterns of these features.
This is great, because we can now represent more information with less data. The trade-off here, is that we lose interpretability.
When we are plotting features against each other, it’s easier to interpret the patterns. But since principal components encode several features, when we plot them it’s not clear:
What features each principal components encode,
What proportion of each feature has more impact on the patterns we’re seeing.
To understand how much each feature contributes to a principal components, we need to look at the loadings of each principal component.
Loadings are the weights applied to each feature in the linear combination that results in the score of the principal component.
Looking only at the first two principal components we can see that:
bedrooms and number_doors have more information captured in the first principal component.
hardwood_floors and city_center have more information captured in the second principal component.
Here’s how you can inspect the loadings.
import pandas as pdfrom statsmodels.multivariate.pca import PCA# Load datasetdataset = pd.read_csv('dataset.csv')pca = PCA(dataset, standardize=True, method='eig')loadings = pca.loadingsprint(loadings)
One of the most popular applications of PCA is dimensionality reduction. It’s a way to create a new representation of your dataset that is:
Smaller than the original dataset,
Keeps only its core patterns and characteristics.
Picking principal components to represent a smaller version of your dataset is a complex topic in of itself. It’s called Factor Analysis, because we can also refer to components as factors.
The traditional methods for picking factors include:
Kaiser criterion
Explained variance
Scree plot
With the Kaiser criterion we pick only the principal components that have eigenvalues greater than 1. Each eigenvalue is associated with one eigenvector which, in fact, is a principal component.
If we were to use this method, we’d exclude the last three principal components.
import pandas as pdfrom statsmodels.multivariate.pca import PCA# Load datasetdataset = pd.read_csv('src/dataset.csv')pca = PCA(dataset, standardize=True, method='eig')eigen_values = pd.DataFrame(data=pca.eigenvals.values, columns=['eigenvalue'])print(eigen_values)
With this method, you pick the total amount of variance in the dataset you want principal components to encode. Usually these cut-off values are 80% or 90%.
If we were to use this method, we have:
The first six principal components explain a bit more than 80% of the variance,
The first nine principal components would roughly explain 90% of the variance in our dataset.
We’d use the first six or first nine principal components, depending on the cut-off we’ve established.
import pandas as pdfrom statsmodels.multivariate.pca import PCA# Load datasetdataset = pd.read_csv('src/dataset.csv')pca = PCA(dataset, standardize=True, method='eig')# Cumulative Variance Explainedcumulative_variance_explained = pd.DataFrame(data=pca.rsquare.values, columns=['cumulative_var'])print(cumulative_variance_explained)
In statistics, R-squared is the metric used to determine how much of the variance of a variable is explained by another variable. Hence the name for this attribute of the PCA model.
With the scree plot we pick components in a more visual way. We plot the eigenvalue for each component and use the elbow method to determine our cut-off point.
The idea is to follow the decreasing eigenvalues and see where the values make an elbow. We only include the principal component that lead to the elbow point.
This is a less precise method, because it depends on where each person sees the elbow.
import pandas as pdimport matplotlib.pyplot as pltfrom statsmodels.multivariate.pca import PCA# Load datasetdataset = pd.read_csv('src/dataset.csv')pca = PCA(dataset, standardize=True, method='eig')plt.subplots(figsize=(10, 10))pca.plot_scree()plt.show()
Hope you enjoyed this example. With this randomly generated dataset, we went through each step of PCA and covered:
How PCA works.
What are principal components and how the relate to the original dataset.
How to decide which principal components to keep, in case we want to shrink our dataset.
Thanks for reading!