Understand Principal Component Analysis And Implement It From Scratch
By: Omar Boufeloussen
Tags: Deep Learning, Machine Learning, Artificial Intelligence, Python, Data Science

Principal component analysis is a technique used for dimensionality reduction. It’s widely used for data visualization by extracting information from a dataset with n features (e.g n dimensions) and representing this dataset in a space with lower dimensions (e.g m dimensions with m<n). So, it’s basically a technique that allow us to reduce dimensions for better visualizations while preserving information as much as possible.

Actually a lot of people find a hard time understanding this technique and interpreting it. For this reason, I’ve decided to break it down with a practical implementation through these steps :

Dataset description :

For this task we choose the classic breast cancer dataset for its simplicity in order to show how PCA is useful for visualization. Breast cancer dataset holds 12 features as characteristics of the cell nuclei present in the image and label classifying type of tumor diagnosed as malignant 0 and benign 1 :

The first 5 line of the dataset

The features labeled 0 represent 37% of the dataset while the features labeled 1 represent more than 62% of the data :

So, the goal of implementing PCA is reducing the 12 dimensions into 2D dimensions so we can visualize if our dataset is visually separated into two classes.

Standardization :

Features standardization is defined as :

Standardization is important in the sense that it makes all the features having the same range of values, so we use the formula above to normalize all features. This how the data would look like after standardization :

Covariance Matrix :

To understand the covariance matrix we suppose that we have only 5 features a, b, c, d, e :

The values of variance on the diagonal represents the measure of the spread of a certain feature:

The elements other than the diagonal elements are the covariance of each feature with the other 4 features which allow us to measure how two features vary together :

Singular value decomposition :

We apply singular value decomposition on the covariance matrix to get a matrix of eigenvectors and diagonal matrix containing the eigenvalues of each eigenvector :

Given A is a covariance matrix means it's a symmetric matrix. Thus, the eigenvectors are orthogonal which allow us to obtain new uncorrelated features. And the eigenvalue represents how much information retained by each new feature, that's why we choose the first m components sorting the eigenvectors according to their eigenvalues in decreasing order.

Now, enough talking lets move to the code :

Implementation of PCA :

Now after having an idea about how to get the most important components sorted in decreasing order, we choose the first 2 components representing 2D plan that allow us to retain as much information as possible :

def compute_pca(X, n_components=2): # mean center the data

X_demeaned = X-X.mean() # calculate the covariance matrix

covariance_matrix = np.cov(X_demeaned.T) # calculate eigenvectors & eigenvalues of the covariance matrix

eigen_vals, eigen_vecs =np.linalg.eigh(covariance_matrix) # sort eigenvalue in increasing order (get the indices from the sort)

idx_sorted = np.argsort(eigen_vals)



# reverse the order so that it's from highest to lowest.

idx_sorted_decreasing = np.flip(idx_sorted) # sort the eigen values by idx_sorted_decreasing

eigen_vals_sorted = eigen_vals[idx_sorted_decreasing] # sort eigenvectors using the idx_sorted_decreasing indices

eigen_vecs_sorted = eigen_vecs[:,idx_sorted_decreasing] # select the first n eigenvectors (n is desired dimension

# of rescaled data array, or dims_rescaled_data)

eigen_vecs_subset = eigen_vecs_sorted[:,:2] # transform the data by multiplying the transpose of the eigenvectors

# with the transpose of the de-meaned data

# Then take the transpose of that product.

X_reduced = np.dot(eigen_vecs_subset.T,X_demeaned.T).T return X_reduced

In this python code we’ve implemented each step explained above in order to compute the first 2 components. Here's the result of projecting the dataset’s features on those 2 components to visualize our dataset :

Scatter plot

As you can see the data is visually separated into two classes which means we'd get a good results training a basic classification model on this dataset. For the full code check the notebook on GitHub.

You can check my last article on Sentiment Analysis From Scratch With Logistic Regression.