Classifying MNIST Digits Using PCA + Deep Learning | by Muriel Kosaka | Towards Data Science

One of the many important concepts in Data Science includes Principal Component Analysis (PCA) which is an unsupervised learning method. It is often used to as a dimensionality reduction method for large datasets or simplify their complexity — this is done by transforming a large set of variables into a small one while retaining most of the variation in the dataset. PCA reduces data by geometrically projecting it onto lower dimensions which in turn are called as Principal Components(PC). This method’s goal is to find the best summary of our data by using the least amount of principal components, by choosing our principal components we minimize the distance between the original data and its projected values on the principal components.
Deep Learning is a subset of Machine Learning, inspired by the structure of a human brain. Deep learning algorithms attempt to draw similar conclusions as humans would by continually analyzing data with a given logical structure. To achieve this, deep learning uses a multi-layered structure of algorithms called neural networks. Whether it’s Alexa, Siri or Cortana, deep learning helps them understand speech and the language. Deep Learning uses different types of neural network architectures like object recognition, image and sound classification, and object detection for different types of problems. The more data a Deep Learning algorithm is trained on, the more accurate it is.
The MNIST Dataset (Modified National Institute of Standards and Technology database) is one of the more popular datasets among deep learning enthusiasts. This dataset contains 42,000 labeled grayscale images (28 x 28 pixel) of handwritten digits from 0–9 in their training set and 28,000 unlabeled test images.
In this blog, I will be demonstrating how to use PCA in building a CNN model to recognize handwritten digits from the MNIST Dataset to achieve high accuracy.
Import the libraries and load the dataset: Importing the necessary libraries, packages, and MNIST datasetPreprocess the dataPCA ImplementationCreate CNN modelTrain and Evaluate the Model
Import the libraries and load the dataset: Importing the necessary libraries, packages, and MNIST dataset
Preprocess the data
PCA Implementation
Create CNN model
Train and Evaluate the Model
We first import all the necessary libraries we will need to train our data, and then load our MNIST dataset which can be found here — you can also use keras.datasets to import the mnist dataset!
# Import Libraries%matplotlib inlineimport pandas as pdfrom sklearn.decomposition import PCAimport numpy as npimport matplotlib.pyplot as pltfrom sklearn.preprocessing import StandardScalerfrom keras.utils.np_utils import to_categoricalfrom sklearn.preprocessing import LabelEncoderfrom tensorflow.keras.layers import Input,InputLayer, Dense, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2Dfrom tensorflow.keras.layers import AveragePooling2D, MaxPooling2D, Dropoutfrom tensorflow.keras.models import Sequential,Modelfrom tensorflow.keras.optimizers import SGDfrom tensorflow.keras.callbacks import ModelCheckpoint,LearningRateSchedulerimport kerasfrom tensorflow.keras import backend as Kfrom sklearn.preprocessing import StandardScalerfrom scipy.linalg import eigh# Load the Datasettrain = pd.read_csv("data/train.csv")test = pd.read_csv("data/test.csv")train.head()
Sample images from the dataset:
First, we will split the dataset into training and testing sets
y_train = train['label']X_train = train.drop(['label'], axis=1)X_test = testX_train.shape, y_train.shape, X_test.shape#Output((42000, 784), (42000,), (28000, 784))X_train = X_train/255X_test = X_test/255
Note that we do not have y_test because the labels were not given for the test set. Next, we will normalize the data which is important in PCA implementation as it projects your original data onto directions which maximize the variance.
Now, we will implement PCA on our data. First we will standardize our data and calculate our covariance matrix. Covariance can be thought of as the direction of linear relationship between the variables. The covariance matrix is the first step in dimensionality reduction because it gives an idea of the number of features that strongly relate, and it is usually the first step in dimensionality reduction because it gives an idea of the number of strongly related features so that those features can be discarded.
standardized_scalar = StandardScaler()standardized_data = standardized_scalar.fit_transform(X_train)standardized_data.shape# Output(42000, 784)cov_matrix = np.matmul(standardized_data.T, standardized_data)cov_matrix.shape# Output(784,784)
Next, we will calculate eigenvalues and eigenvectors. Eigenvectors and eigenvalues ​​of a covariance matrix (or correlation) describe the source of the PCA. Eigenvectors (main components) determine the direction of the new attribute space, and eigenvalues ​​determine its magnitude.
# Calculate eigenvalues and eigenvectorslambdas, vectors = eigh(cov_matrix, eigvals=(782, 783))vectors.shape# Output(784,2)vectors = vectors.Tvectors.shape# (2, 784)
Then, we will calculate unit vectors and its new coordinates. Unit vectors are vectors with a magnitude of 1, and we are trying to find the unit vector that maximizes the variance.
new_coordinates = np.matmul(vectors, standardized_data.T)print(new_coordinates.shape)new_coordinates = np.vstack((new_coordinates, y_train)).T# Output(2, 42000)df_new = pd.DataFrame(new_coordinates, columns=["f1", "f2", "labels"])df_new.head()
Let’s see a visualization of the cumulative variance retained across the number of components (784):
Before we build our CNN model, we will need to take some preprocessing steps. This includes converting the training set into an array, reshaping the input data to get it in the shape which the model expect to receive later, and encoding the image labels.
X_train = np.array(X_train)y_train = np.array(y_train)X_train = X_train.reshape(X_train.shape[0], 28, 28, 1)print(X_train.shape, y_train.shape)# Output(42000, 28, 28, 1) (42000,)nclasses = y_train.max() - y_train.min() + 1y_train = to_categorical(y_train, num_classes = nclasses)print("Shape of ytrain after encoding: ", y_train.shape)# OutputShape of ytrain after encoding:  (42000, 10)
Now we can build our CNN model, I built a 2D CNN model with 3 layers and 1 fully connected layer.
input_shape = (28,28,1)X_input = Input(input_shape)# layer 1x = Conv2D(64,(3,3),strides=(1,1),name='layer_conv1',padding='same')(X_input)x = BatchNormalization()(x)x = Activation('relu')(x)x = MaxPooling2D((2,2),name='maxPool1')(x)# layer 2x = Conv2D(32,(3,3),strides=(1,1),name='layer_conv2',padding='same')(x)x = BatchNormalization()(x)x = Activation('relu')(x)x = MaxPooling2D((2,2),name='maxPool2')(x)# layer 3x = Conv2D(32,(3,3),strides=(1,1),name='conv3',padding='same')(x)x = BatchNormalization()(x)x = Activation('relu')(x)x = MaxPooling2D((2,2), name='maxPool3')(x)# fcx = Flatten()(x)x = Dense(64,activation ='relu',name='fc0')(x)x = Dropout(0.25)(x)x = Dense(32,activation ='relu',name='fc1')(x)x = Dropout(0.25)(x)x = Dense(10,activation ='softmax',name='fc2')(x)conv_model = Model(inputs=X_input, outputs=x, name='Predict')conv_model.summary()
After 40 epochs, we achieved an accuracy of 99.8% !
From this article, we classify handwritten digits using neural networks and preprocessing the data using PCA. For more information about PCA and it’s uses, please see the following helpful articles. Thank you for reading and all code is available on my Github! :)