A Better Understanding of Principal Component Analysis by implementing it on MNIST Dataset using Python.
By: Rohith Reddy Anugu
Tags: Dimensionality Reduction, Principal Component, Sklearn, Visualization, Python

This is my first article on medium. Here, I’ll be giving a quick overview of what Principal Component Analysis is, why do we need it and how to do it using Python on MNIST Dataset.

Principal Component Analysis

Principal Component Analysis (PCA) is a statistical technique that is used to reduce the dimensionality of the data (reduce the number of features in the dataset) by selecting the most significant features that capture most of the information about the dataset.

PCA operates by finding the Principal Components of data, (directions where there is the most variance in the data) and project it onto a smaller dimensional subspace while retaining most of the information. By projecting our data into a smaller space, we’re reducing the dimensionality of our feature space. The feature that causes highest variance is the first Principal Component. The feature that is responsible for second highest variance is considered the second Principal Component, and so on.

In simple words, PCA is a method of extracting important features (in the form of components) from a large set of features available in a dataset.

MNIST Dataset

MNIST is a computer vision dataset which contains images of the handwritten digits with each image being 28 pixels in height and 28 pixels in width, for a total of 784 pixels.

Now we will be performing PCA for visualization on this dataset using sklearn .

from sklearn import decomposition

pca=decomposition.PCA() # configuring the parameteres

# the number of components = 2



pca.n_components = 2

pca_data = pca.fit_transform(standardized_data) # pca_reduced will contain the 2-d projects of simple data

print("shape of pca_reduced.shape = ", pca_data.shape) shape of pca_reduced.shape = (42000, 2) # attaching the label for each 2-d data point

pca_data = np.vstack((pca_data.T, l)).T



# creating a new data fram which help us in ploting the result data

pca_df = pd.DataFrame(data=pca_data, columns=("1st_principal", "2nd_principal", "label"))

sn.FacetGrid(pca_df, hue="label", size=6).map(plt.scatter, '1st_principal', '2nd_principal').add_legend()

plt.show()

PCA is one of the way to reduce high dimension features (say 784 in MNIST dataset in our example) to lower dimension without losing the variance of the original data.

Steps to arrive PCA:

lets take our dataset as matrix A

Step 1: Preprocess A i.e standardize the dataset.

Step 2 : Find the co-variance of the A , let’s say co-variance matrix as S.

Step 3: Find the eigen values and eigen vectors for the co-variance matrix, S

Step 4 : Find the dot product of the eigen vector and co-variance matrix , S

Eigen Values — Gives us the percentage of variance of the features.

Eigen Vectors — Gives us the direction of the features .

# PCA for diemnsionality reduction (not visualization)

pca.n_components = 784

pca_data = pca.fit_transform(standardized_data)



percentage_var_explained = pca.explained_variance_ / np.sum(pca.explained_variance_);



cum_var_explained = np.cumsum(percentage_var_explained)

# Plot the PCA spectrum

plt.figure(1, figsize=(6, 4))



plt.clf()

plt.plot(cum_var_explained, linewidth=2)

plt.axis('tight')

plt.grid()

plt.xlabel('n_components')

plt.ylabel('Cumulative_explained_variance')

plt.show()





# If we take 200-dimensions, approx. 90% of variance is expalined.

Here we plot the cumulative sum of variance with the component. Here 200 components explain the almost 90% variance. So we can reduce the dimension according to the required variance.

Reference:https://colah.github.io/posts/2014-10-Visualizing-MNIST/